# Safe Coder CLI

Standalone CLI tool for documentation crawling with SPA support, error detection, and code validation.

## Overview

`@gulibs/safe-coder-cli` is an independent command-line tool that crawls documentation websites and generates structured output. It supports both static sites and Single Page Applications (SPAs) using browser automation.

This CLI is designed to work standalone or as part of the Safe Coder ecosystem, where it's called by the `@gulibs/safe-coder` MCP Server.

## Features

- **HTTP & Browser Crawling**: Supports both static HTTP crawling and browser-based rendering for SPAs
- **Intelligent Content Extraction**: Cleans and structures documentation content
- **Parallel Processing**: Multi-worker support for faster crawling
- **Progress Reporting**: Real-time progress updates via stderr
- **JSON Output**: Machine-readable JSON output for programmatic use
- **Skill Generation**: Generates AI-ready SKILL files from documentation
- **Checkpoint Support**: Resume interrupted crawls
- **Proxy Support**: Configure HTTP/HTTPS proxies

## Installation

### Global Installation (Recommended)

```bash
npm install -g @gulibs/safe-coder-cli
```

Or using yarn:

```bash
yarn global add @gulibs/safe-coder-cli
```

Or using pnpm:

```bash
pnpm add -g @gulibs/safe-coder-cli
```

### Verify Installation

```bash
safe-coder-cli --version
safe-coder-cli --help
```

## Usage

### Basic Crawl

```bash
safe-coder-cli crawl https://react.dev
```

### Crawl with Options

```bash
# Limit pages and depth
safe-coder-cli crawl https://react.dev --max-pages 50 --max-depth 3

# Use multiple workers for faster crawling
safe-coder-cli crawl https://react.dev --workers 5

# Force browser automation for SPAs
safe-coder-cli crawl https://spa-site.com --spa-strategy auto --browser playwright

# Save output to directory
safe-coder-cli crawl https://react.dev --output-dir ./skills
```

### JSON Output (for MCP Integration)

```bash
# Output machine-readable JSON
safe-coder-cli crawl https://react.dev --output-format json

# Capture output to file
safe-coder-cli crawl https://react.dev --output-format json > output.json
```

## Command Reference

### `crawl <url> [options]`

Crawl documentation website and optionally generate skill file.

#### Options

- `-c, --config <path>` - Path to configuration file
- `-b, --browser <type>` - Browser type: `puppeteer` | `playwright`
- `-d, --max-depth <number>` - Maximum crawl depth (default: 3)
- `-p, --max-pages <number>` - Maximum number of pages to crawl (default: 50)
- `-w, --workers <number>` - Number of parallel workers (default: 1)
- `--spa-strategy <type>` - SPA strategy: `smart` | `auto` | `manual` (default: smart)
- `-o, --output-dir <path>` - Output directory for skill files
- `-f, --filename <name>` - Skill name for directory and file names
- `--checkpoint` - Enable checkpoint/resume functionality
- `--resume` - Resume from last checkpoint if available
- `--rate-limit <ms>` - Delay in milliseconds between requests (default: 500)
- `--output-format <format>` - Output format: `json` | `pretty` (default: pretty)
- `--include-paths <paths>` - Additional path patterns to include (comma-separated)
- `--exclude-paths <paths>` - Path patterns to exclude (comma-separated)

### `detect-errors <file> [options]`

Detect errors and warnings in code files.

```bash
safe-coder-cli detect-errors ./src/app.ts
safe-coder-cli detect-errors ./src/app.ts --format json
```

### `validate-code <file> [options]`

Validate and optionally fix code errors.

```bash
safe-coder-cli validate-code ./src/app.ts
safe-coder-cli validate-code ./src/app.ts --output ./src/app.fixed.ts
```

## Configuration File

Create a `.doc-crawler.json` file in your project root:

```json
{
  "browser": "puppeteer",
  "spaStrategy": "smart",
  "crawl": {
    "maxDepth": 3,
    "maxPages": 200,
    "workers": 5,
    "rateLimit": 300,
    "checkpoint": {
      "enabled": true,
      "interval": 50
    }
  },
  "proxy": "http://127.0.0.1:7890"
}
```

## Output Format

### JSON Output Structure

When using `--output-format json`, the CLI outputs:

```json
{
  "success": true,
  "data": {
    "source": {
      "url": "https://react.dev",
      "crawledAt": "2024-01-15T10:30:00.000Z",
      "pageCount": 50,
      "depth": 3
    },
    "pages": [
      {
        "url": "https://react.dev/learn",
        "title": "Learn React",
        "content": "...",
        "wordCount": 1500,
        "codeBlocks": 5,
        "headings": ["Getting Started", "Components"]
      }
    ],
    "metadata": {
      "technology": "react.dev",
      "categories": ["tutorial", "api", "guide"]
    },
    "statistics": {
      "totalPages": 50,
      "maxDepthReached": 3,
      "errors": 0
    },
    "skill": {
      "skillMd": "...",
      "quality": 85
    }
  }
}
```

### Progress Output (stderr)

Progress information is output to stderr in JSON format:

```json
{"type":"progress","message":"Crawled 10/50 pages","timestamp":"...","current":10,"total":50,"percentage":20}
```

## Browser Setup

For SPA crawling, you need Chrome/Chromium installed:

### macOS

```bash
brew install --cask google-chrome
```

### Windows

```bash
winget install Google.Chrome
```

### Linux

```bash
sudo apt install google-chrome-stable
```

### Custom Browser Path

```bash
export CHROME_PATH=/path/to/chrome
```

## Environment Variables

- `CHROME_PATH` - Path to Chrome executable
- `HTTP_PROXY` - HTTP proxy URL
- `HTTPS_PROXY` - HTTPS proxy URL
- `LOG_LEVEL` - Log level (INFO, DEBUG, ERROR)

## Integration with MCP Server

The CLI is designed to be called by `@gulibs/safe-coder` MCP Server. The MCP Server:

1. Checks if CLI is installed
2. Spawns CLI with appropriate parameters
3. Monitors progress via stderr
4. Parses JSON output from stdout
5. Post-processes results and generates SKILL guidance

## Examples

### Simple Documentation Crawl

```bash
safe-coder-cli crawl https://docs.example.com --max-pages 30
```

### Fast Parallel Crawl

```bash
safe-coder-cli crawl https://docs.example.com --workers 8 --max-pages 200
```

### SPA Site with Browser

```bash
safe-coder-cli crawl https://spa-site.com --spa-strategy auto --browser playwright
```

### Generate SKILL and Save

```bash
safe-coder-cli crawl https://react.dev \
  --output-dir ~/.cursor/skills \
  --filename react-docs \
  --max-pages 100
```

### JSON Output for Scripting

```bash
safe-coder-cli crawl https://docs.example.com \
  --output-format json \
  --max-pages 20 > output.json

# Process with jq
cat output.json | jq '.data.statistics'
```

## Troubleshooting

### CLI Not Found

After installation, if `safe-coder-cli` is not found:

```bash
# Check npm global bin path
npm config get prefix

# Add to PATH if needed (macOS/Linux)
export PATH="$(npm config get prefix)/bin:$PATH"
```

### Browser Not Found

If you see "Chrome/Chromium not found":

1. Install Chrome (see Browser Setup above)
2. Set `CHROME_PATH` environment variable
3. Or install full puppeteer: `npm install -g puppeteer`

### Permission Errors

On Linux/macOS, you may need sudo for global installation:

```bash
sudo npm install -g @gulibs/safe-coder-cli
```

Or use a version manager like nvm to avoid sudo.

## Development

```bash
# Clone repository
git clone <repository-url>
cd safe-coder-cli

# Install dependencies
npm install

# Build
npm run build

# Link for local testing
npm link

# Test
safe-coder-cli --version
```

## License

MIT

## Related Projects

- [`@gulibs/safe-coder`](https://www.npmjs.com/package/@gulibs/safe-coder) - MCP Server that orchestrates this CLI