# LLM Researcher

A lightweight MCP (Model Context Protocol) server for LLM orchestration that provides efficient web content search and extraction capabilities. This CLI tool enables LLMs to search DuckDuckGo and extract clean, LLM-friendly content from web pages.

Built with **TypeScript**, **tsup**, and **vitest** for modern development experience.

## Features

- **MCP Server Support**: Provides Model Context Protocol server for LLM integration
- **Free Operation**: Uses DuckDuckGo HTML endpoint (no API costs)
- **GitHub Code Search**: Search GitHub repositories for code examples and implementation patterns
- **Smart Content Extraction**: Playwright + @mozilla/readability for clean content
- **LLM-Optimized Output**: Sanitized Markdown (h1-h3, bold, italic, links only)
- **Rate Limited**: Respects DuckDuckGo with 1 req/sec limit
- **Cross-Platform**: Works on macOS, Linux, and WSL
- **Multiple Modes**: CLI, MCP server, search, direct URL, and interactive modes
- **Type Safe**: Full TypeScript implementation with strict typing
- **Modern Tooling**: Built with tsup bundler and vitest testing

## Installation

### Prerequisites

- Node.js 20.0.0 or higher
- No local Chrome installation required (uses Playwright's bundled Chromium)

### Setup

```bash
# Clone or download the project
cd light-research-mcp

# Install dependencies (using pnpm)
pnpm install

# Build the project
pnpm build

# Install Playwright browsers
pnpm install-browsers

# Optional: Link globally for system-wide access
pnpm link --global
```

## Usage

### MCP Server Mode

Use as a Model Context Protocol server to provide search and content extraction tools to LLMs:

```bash
# Start MCP server (stdio transport)
llmresearcher --mcp

# The server provides these tools to MCP clients:
# - github_code_search: Search GitHub repositories for code
# - duckduckgo_web_search: Search the web with DuckDuckGo
# - extract_content: Extract detailed content from URLs
```

#### Setting up with Claude Code

```bash
# Add as an MCP server to Claude Code
claude mcp add light-research-mcp /path/to/light-research-mcp/dist/bin/llmresearcher.js --mcp

# Or with project scope for team sharing
claude mcp add light-research-mcp -s project /path/to/light-research-mcp/dist/bin/llmresearcher.js --mcp

# List configured servers
claude mcp list

# Check server status
claude mcp get light-research-mcp
```

#### MCP Tool Usage Examples

Once configured, you can use these tools in Claude:

```
> Search for React hooks examples on GitHub
Tool: github_code_search
Query: "useState useEffect hooks language:javascript"

> Search for TypeScript best practices
Tool: duckduckgo_web_search  
Query: "TypeScript best practices 2024"
Locale: us-en (or wt-wt for no region)

> Extract content from a search result
Tool: extract_content
URL: https://example.com/article-from-search-results
```

### Command Line Interface

```bash
# Search mode - Search DuckDuckGo and interactively browse results
llmresearcher "machine learning transformers"

# GitHub Code Search mode - Search GitHub for code
llmresearcher -g "useState hooks language:typescript"

# Direct URL mode - Extract content from specific URL
llmresearcher -u https://example.com/article

# Interactive mode - Enter interactive search session
llmresearcher

# Verbose logging - See detailed operation logs
llmresearcher -v "search query"

# MCP Server mode - Start as Model Context Protocol server
llmresearcher --mcp
```

## Development

### Scripts

```bash
# Build the project
pnpm build

# Build in watch mode (for development)
pnpm dev

# Run tests
pnpm test

# Run tests in CI mode (single run)
pnpm test:run

# Type checking
pnpm type-check

# Clean build artifacts
pnpm clean

# Install Playwright browsers
pnpm install-browsers
```

### Interactive Commands

When in search results view:
- **1-10**: Select a result by number
- **b** or **back**: Return to search results
- **open \<n>**: Open result #n in external browser
- **q** or **quit**: Exit the program

When viewing content:
- **b** or **back**: Return to search results
- **/\<term>**: Search for term within the extracted content
- **open**: Open current page in external browser
- **q** or **quit**: Exit the program

## Configuration

### Environment Variables

Create a `.env` file in the project root:

```env
USER_AGENT=Mozilla/5.0 (compatible; LLMResearcher/1.0)
TIMEOUT=30000
MAX_RETRIES=3
RATE_LIMIT_DELAY=1000
CACHE_ENABLED=true
MAX_RESULTS=10
```

### Configuration File

Create `~/.llmresearcherrc` in your home directory:

```json
{
  "userAgent": "Mozilla/5.0 (compatible; LLMResearcher/1.0)",
  "timeout": 30000,
  "maxRetries": 3,
  "rateLimitDelay": 1000,
  "cacheEnabled": true,
  "maxResults": 10
}
```

### Configuration Options

| Option | Default | Description |
|--------|---------|-------------|
| `userAgent` | `Mozilla/5.0 (compatible; LLMResearcher/1.0)` | User agent for HTTP requests |
| `timeout` | `30000` | Request timeout in milliseconds |
| `maxRetries` | `3` | Maximum retry attempts for failed requests |
| `rateLimitDelay` | `1000` | Delay between requests in milliseconds |
| `cacheEnabled` | `true` | Enable/disable local caching |
| `maxResults` | `10` | Maximum search results to display |

## Architecture

### Core Components

1. **MCPResearchServer** (`src/mcp-server.ts`)
   - Model Context Protocol server implementation
   - Three main tools: github_code_search, duckduckgo_web_search, extract_content
   - JSON-based responses for LLM consumption

2. **DuckDuckGoSearcher** (`src/search.ts`)
   - HTML scraping of DuckDuckGo search results with locale support
   - URL decoding for `/l/?uddg=` format links
   - Rate limiting and retry logic

3. **GitHubCodeSearcher** (`src/github-code-search.ts`)
   - GitHub Code Search API integration via gh CLI
   - Advanced query support with language, repo, and file filters
   - Authentication and rate limiting

4. **ContentExtractor** (`src/extractor.ts`)
   - Playwright-based page rendering with resource blocking
   - @mozilla/readability for main content extraction
   - DOMPurify sanitization and Markdown conversion

5. **CLIInterface** (`src/cli.ts`)
   - Interactive command-line interface
   - Search result navigation
   - Content viewing and text search

6. **Configuration** (`src/config.ts`)
   - Environment and RC file configuration loading
   - Verbose logging support

### Content Processing Pipeline

#### MCP Server Mode
1. **Search**: 
   - DuckDuckGo: HTML endpoint → Parse results → JSON response with pagination
   - GitHub: Code Search API → Format results → JSON response with code snippets
2. **Extract**: URL from search results → Playwright navigation → Content extraction
3. **Process**: @mozilla/readability → DOMPurify sanitization → Clean JSON output
4. **Output**: Structured JSON for LLM consumption

#### CLI Mode  
1. **Search**: DuckDuckGo HTML endpoint → Parse results → Display numbered list
2. **Extract**: Playwright navigation → Resource blocking → JS rendering
3. **Process**: @mozilla/readability → DOMPurify sanitization → Turndown Markdown
4. **Output**: Clean Markdown with h1-h3, **bold**, *italic*, [links](url) only

### Security Features

- **Resource Blocking**: Prevents loading of images, CSS, fonts for speed and security
- **Content Sanitization**: DOMPurify removes scripts, iframes, and dangerous elements
- **Limited Markdown**: Only allows safe formatting elements (h1-h3, strong, em, a)
- **Rate Limiting**: Respects DuckDuckGo's rate limits with exponential backoff

## Examples

### MCP Server Usage with Claude Code

#### 1. GitHub Code Search

```
You: "Find React hook examples for state management"

Claude uses github_code_search tool:
{
  "query": "useState useReducer state management language:javascript",
  "results": [
    {
      "title": "facebook/react/packages/react/src/ReactHooks.js",
      "url": "https://raw.githubusercontent.com/facebook/react/main/packages/react/src/ReactHooks.js",
      "snippet": "function useState(initialState) {\n  return dispatcher.useState(initialState);\n}"
    }
  ],
  "pagination": {
    "currentPage": 1,
    "hasNextPage": true,
    "nextPageToken": "2"
  }
}
```

#### 2. Web Search with Locale

```
You: "Search for Vue.js tutorials in Japanese"

Claude uses duckduckgo_web_search tool:
{
  "query": "Vue.js チュートリアル 入門",
  "locale": "jp-jp",
  "results": [
    {
      "title": "Vue.js入門ガイド",
      "url": "https://example.com/vue-tutorial",
      "snippet": "Vue.jsの基本的な使い方を学ぶチュートリアル..."
    }
  ]
}
```

#### 3. Content Extraction

```
You: "Extract the full content from that Vue.js tutorial"

Claude uses extract_content tool:
{
  "url": "https://example.com/vue-tutorial",
  "title": "Vue.js入門ガイド",
  "extractedAt": "2024-01-15T10:30:00.000Z",
  "content": "# Vue.js入門ガイド\n\nVue.jsは...\n\n## インストール\n\n..."
}
```

### CLI Examples

#### Basic Search

```bash
$ llmresearcher "python web scraping"

🔍 Search Results:
══════════════════════════════════════════════════

1. Python Web Scraping Tutorial
   URL: https://realpython.com/python-web-scraping-practical-introduction/
   Complete guide to web scraping with Python using requests and Beautiful Soup...

2. Web Scraping with Python - BeautifulSoup and requests
   URL: https://www.dataquest.io/blog/web-scraping-python-tutorial/
   Learn how to scrape websites with Python, Beautiful Soup, and requests...

══════════════════════════════════════════════════
Commands: [1-10] select result | b) back | q) quit | open <n>) open in browser

> 1

📥 Extracting content from: Python Web Scraping Tutorial

📄 Content:
══════════════════════════════════════════════════

**Python Web Scraping Tutorial**
Source: https://realpython.com/python-web-scraping-practical-introduction/
Extracted: 2024-01-15T10:30:00.000Z

──────────────────────────────────────────────────

# Python Web Scraping: A Practical Introduction

Web scraping is the process of collecting and parsing raw data from the web...

## What Is Web Scraping?

Web scraping is a technique to automatically access and extract large amounts...

══════════════════════════════════════════════════
Commands: b) back to results | /<term>) search in text | q) quit | open) open in browser

> /beautiful soup

🔍 Found 3 matches for "beautiful soup":
──────────────────────────────────────────────────
Line 15: Beautiful Soup is a Python library for parsing HTML and XML documents.
Line 42: from bs4 import BeautifulSoup
Line 67: soup = BeautifulSoup(html_content, 'html.parser')
```

### Direct URL Mode

```bash
$ llmresearcher -u https://docs.python.org/3/tutorial/

📄 Content:
══════════════════════════════════════════════════

**The Python Tutorial**
Source: https://docs.python.org/3/tutorial/
Extracted: 2024-01-15T10:35:00.000Z

──────────────────────────────────────────────────

# The Python Tutorial

Python is an easy to learn, powerful programming language...

## An Informal Introduction to Python

In the following examples, input and output are distinguished...
```

### Verbose Mode

```bash
$ llmresearcher -v "nodejs tutorial"

[VERBOSE] Searching: https://duckduckgo.com/html/?q=nodejs%20tutorial&kl=us-en
[VERBOSE] Response: 200 in 847ms
[VERBOSE] Parsed 10 results
[VERBOSE] Launching browser...
[VERBOSE] Blocking resource: https://example.com/style.css
[VERBOSE] Blocking resource: https://example.com/image.png
[VERBOSE] Navigating to page...
[VERBOSE] Page loaded in 1243ms
[VERBOSE] Processing content with Readability...
[VERBOSE] Readability extraction successful
[VERBOSE] Closing browser...
```

## Testing

### Running Tests

```bash
# Run tests in watch mode
pnpm test

# Run tests once (CI mode)
pnpm test:run

# Run tests with coverage
pnpm test -- --coverage
```

### Test Coverage

The test suite includes:

- **Unit Tests**: Individual component testing
  - `search.test.ts`: DuckDuckGo search functionality, URL decoding, rate limiting
  - `extractor.test.ts`: Content extraction, Markdown conversion, resource management
  - `config.test.ts`: Configuration validation and environment handling

- **Integration Tests**: End-to-end workflow testing
  - `integration.test.ts`: Complete search-to-extraction workflows, error handling, cleanup

### Test Features

- **Fast**: Powered by vitest for quick feedback
- **Type-safe**: Full TypeScript support in tests
- **Isolated**: Each test cleans up its resources
- **Comprehensive**: Covers search, extraction, configuration, and integration scenarios

## Troubleshooting

### Common Issues

**"Browser not found" Error**
```bash
pnpm install-browsers
```

**Rate Limiting Issues**
- The tool automatically handles rate limiting with 1-second delays
- If you encounter 429 errors, the tool will automatically retry with exponential backoff

**Content Extraction Failures**
- Some sites may block automated access
- The tool includes fallback extraction methods (main → body content)
- Use verbose mode (`-v`) to see detailed error information

**Permission Denied (Unix/Linux)**
```bash
chmod +x bin/llmresearcher.js
```

### Performance Optimization

The tool is optimized for speed:
- **Resource Blocking**: Automatically blocks images, CSS, fonts
- **Network Idle**: Waits for JavaScript to complete rendering
- **Content Caching**: Supports local caching to avoid repeated requests
- **Minimal Dependencies**: Uses lightweight, focused libraries

## Development

### Project Structure

```
light-research-mcp/
├── dist/                      # Built JavaScript files (generated)
│   ├── bin/
│   │   └── llmresearcher.js   # CLI entry point (executable)
│   └── *.js                   # Compiled TypeScript modules
├── src/                       # TypeScript source files
│   ├── bin.ts                 # CLI entry point
│   ├── index.ts               # Main LLMResearcher class
│   ├── mcp-server.ts          # MCP server implementation
│   ├── search.ts              # DuckDuckGo search implementation
│   ├── github-code-search.ts  # GitHub Code Search implementation
│   ├── extractor.ts           # Content extraction with Playwright
│   ├── cli.ts                 # Interactive CLI interface
│   ├── config.ts              # Configuration management
│   └── types.ts               # TypeScript type definitions
├── test/                      # Test files (vitest)
│   ├── search.test.ts         # Search functionality tests
│   ├── extractor.test.ts      # Content extraction tests
│   ├── config.test.ts         # Configuration tests
│   ├── mcp-locale.test.ts     # MCP locale functionality tests
│   ├── mcp-content-extractor.test.ts # MCP content extractor tests
│   └── integration.test.ts    # End-to-end integration tests
├── tsconfig.json              # TypeScript configuration
├── tsup.config.ts             # Build configuration
├── vitest.config.ts           # Test configuration
├── package.json
└── README.md
```

### Dependencies

#### Runtime Dependencies
- **@modelcontextprotocol/sdk**: Model Context Protocol server implementation
- **@mozilla/readability**: Content extraction from HTML
- **cheerio**: HTML parsing for search results
- **commander**: CLI argument parsing
- **dompurify**: HTML sanitization
- **dotenv**: Environment variable loading
- **jsdom**: DOM manipulation for server-side processing
- **playwright**: Browser automation for JS rendering
- **turndown**: HTML to Markdown conversion

#### Development Dependencies
- **typescript**: TypeScript compiler
- **tsup**: Fast TypeScript bundler
- **vitest**: Fast unit test framework
- **@types/***: TypeScript type definitions

## License

MIT License - see LICENSE file for details.

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

## Roadmap

### Planned Features

- **Enhanced MCP Tools**: Additional specialized search tools for documentation, APIs, etc.
- **Caching Layer**: SQLite-based URL → Markdown caching with 24-hour TTL
- **Search Engine Abstraction**: Support for Brave Search, Bing, and other engines
- **Content Summarization**: Optional AI-powered content summarization
- **Export Formats**: JSON, plain text, and other output formats
- **Batch Processing**: Process multiple URLs from file input
- **SSE Transport**: Support for Server-Sent Events MCP transport

### Performance Improvements

- **Parallel Processing**: Concurrent content extraction for multiple results
- **Smart Caching**: Intelligent cache invalidation based on content freshness
- **Memory Optimization**: Streaming content processing for large documents