# termux-web-tools

[![npm version](https://img.shields.io/npm/v/@erdium/pi-termux-web-tools.svg)](https://www.npmjs.com/package/@erdium/pi-termux-web-tools)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Node.js](https://img.shields.io/badge/node-%3E%3D22-brightgreen.svg)](https://nodejs.org/)
[![Pi Package](https://img.shields.io/badge/Pi-package-blue.svg)](https://pi.dev/)
[![Termux Compatible](https://img.shields.io/badge/Termux-compatible-green.svg)](https://termux.dev/)

Web search and URL reading tools for [Pi Coding Agent](https://github.com/earendil-works/pi). **Works on Termux (Android), Linux, macOS, and Windows.** No native modules required.

## Features

- 🔍 **Web Search** - Search using DuckDuckGo with SearXNG fallback
- 📄 **URL Reading** - Fetch and convert web pages to clean Markdown
- 🔗 **Link Extraction** - Extract all links from a webpage
- 📚 **GitHub README** - Fetch repository READMEs directly

## Screenshot

![termux-web-tools screenshot](https://raw.githubusercontent.com/erdium/pi-termux-web-tools/main/screenshot.png)

```
┌─────────────────────────────────────────────────┐
│  Search results for: "TypeScript best practices"         │
│                                                          │
│  1. TypeScript Best Practices - 2024 Guide               │
│     https://example.com/ts-guide                         │
│     Learn about generics, utility types...               │
│                                                          │
│  2. Advanced TypeScript Patterns                         │
│     https://example.com/patterns                         │
│     Explore conditional types...                         │
└─────────────────────────────────────────────────┘
```

## Installation

### Via Pi (Recommended)

```bash
pi install npm:@erdium/pi-termux-web-tools
```

### Via npm

```bash
npm install -g @erdium/pi-termux-web-tools
```

### From Source

```bash
git clone https://github.com/erdium/pi-termux-web-tools.git
cd pi-termux-web-tools
npm install
npm run build
pi install .
```

### Termux (Android)

```bash
# Install Pi first (if not installed)
npm install -g @earendil-works/pi-coding-agent

# Install this extension
pi install npm:@erdium/pi-termux-web-tools
```

No `node-gyp`, no `playwright`, no `puppeteer` - just pure JavaScript.

## Quick Start

After installation, the tools are automatically available to the agent:

```
You: Search for "TypeScript generics tutorial"
Agent: [Uses web_search tool to find results]

You: Read this article: https://example.com/article
Agent: [Uses read_url tool to fetch and convert content]

You: Show me the README for https://github.com/microsoft/typescript
Agent: [Uses github_readme tool to fetch README]
```

## Available Tools

### `web_search`

Search the web using DuckDuckGo HTML endpoint with automatic SearXNG fallback.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `query` | string | ✅ | - | Search query |
| `maxResults` | number | ❌ | 5 | Max results (1-20) |

**Example:**
```typescript
web_search({ query: "TypeScript generics", maxResults: 3 })
```

### `read_url`

Fetch a URL and convert it to clean Markdown. Uses Jina Reader first, then falls back to direct fetch.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `url` | string | ✅ | - | URL to read |
| `maxLength` | number | ❌ | 50000 | Max output length |

**Features:**
- Strips scripts, styles, navigation, ads
- Converts HTML to clean Markdown
- Handles timeouts gracefully

**Example:**
```typescript
read_url({ url: "https://example.com/article" })
```

### `extract_links`

Extract all links from a webpage.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `url` | string | ✅ | - | URL to extract links from |

**Example:**
```typescript
extract_links({ url: "https://example.com" })
```

### `github_readme`

Fetch the README.md from a GitHub repository.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `url` | string | ✅ | - | GitHub repository URL |

**Example:**
```typescript
github_readme({ url: "https://github.com/microsoft/typescript" })
```

## Commands

### `/clear-web-cache`

Clear the in-memory request cache. Useful if you're getting stale results.

## Configuration

### Environment Variables

No configuration required. Optional environment variables for future providers:

```bash
# Future: Brave Search API
export BRAVE_SEARCH_API_KEY="your-key"

# Future: Tavily Search API
export TAVILY_API_KEY="your-key"
```

### Request Settings

- **Timeout:** 15 seconds (configurable per request)
- **Retries:** Up to 2 with exponential backoff
- **Cache:** 5-minute in-memory cache

## Termux Compatibility

This extension is specifically designed for Termux/Android:

| Feature | Status |
|---------|--------|
| No native modules | ✅ |
| No node-gyp compilation | ✅ |
| No playwright/puppeteer | ✅ |
| Uses standard Node.js APIs | ✅ |
| Works with Node.js 22+ | ✅ |

## FAQ

### Q: Does this work on Termux?

**A:** Yes! This extension is specifically designed for Termux/Android with zero native dependencies.

### Q: Do I need API keys?

**A:** No API keys are required. The extension uses DuckDuckGo (free) and SearXNG (free) for search.

### Q: Is my data private?

**A:** Search queries are sent to DuckDuckGo/SearXNG. URLs you request are fetched. No data is stored permanently.

### Q: Can I use this offline?

**A:** No, this extension requires network access for web search and URL fetching.

### Q: How do I clear cached results?

**A:** Use the `/clear-web-cache` command in Pi.

## Troubleshooting

### Search not working

1. Check your network connection
2. Try `/clear-web-cache` to clear stale cache
3. DuckDuckGo may rate-limit; wait a few minutes

### URL reading fails

1. Check if the URL is accessible
2. Some sites block automated requests
3. Try a different URL to isolate the issue

### Extension not loading

1. Verify installation: `pi list`
2. Check Pi version: `pi --version`
3. Reinstall: `pi remove @erdium/pi-termux-web-tools && pi install npm:@erdium/pi-termux-web-tools`

## Architecture

```
termux-web-tools/
├── src/
│   ├── index.ts              # Main extension (4 tools)
│   ├── cache.ts              # In-memory cache with TTL
│   ├── fetch-utils.ts        # Fetch with retry/timeout
│   ├── html-to-md.ts         # HTML → Markdown converter
│   └── search-providers.ts   # Search provider implementations
├── dist/                     # Compiled JavaScript
├── package.json
├── tsconfig.json
└── README.md
```

## Future Improvements

- [ ] Brave Search API integration
- [ ] Tavily Search API integration
- [ ] Exa Search API integration
- [ ] Google Custom Search integration
- [ ] Bing Search API integration
- [ ] Firecrawl integration
- [ ] PDF reading support
- [ ] Image OCR support
- [ ] Persistent disk cache

## Contributing

Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

[MIT](LICENSE)

## Support

- 📝 [Report Issues](https://github.com/erdium/pi-termux-web-tools/issues)
- 💬 [Discussions](https://github.com/erdium/pi-termux-web-tools/discussions)
- ⭐ [Star on GitHub](https://github.com/erdium/pi-termux-web-tools)

---

Built with ❤️ for the Pi community