---
name: fetch
description: "Fetch and extract content from any source: web pages to markdown (clip), library/framework docs (docs), and YouTube transcripts. Auto-detects source type. Triggers on 'clip', 'convert to markdown', 'grab this page', 'look up the docs for X', 'get the API reference for X', or a URL to read."
---

# Fetch

Auto-detect the source type and route to its handler.

## Input Detection

| Input | Handler |
| --- | --- |
| YouTube URL (`youtube.com/watch`, `youtu.be/`, `youtube.com/shorts/`, `youtube.com/embed/`) | **Transcript** |
| A library/tool name with a documentation question ("docs for X", "how do I use X", "API reference for X") | **Docs** |
| Any other URL or file path, or "clip" keyword | **Clip** |

Tie-breakers: a YouTube URL **always** routes to Transcript, even when the user says "clip". A direct URL to a documentation page is Clip — fetch that page; Docs *discovers* documentation from a library name or question.

---

## 1. Clip — Web Pages & Files to Markdown

Convert URLs, files, or a mix of both to Markdown through the markdown-anything API.

### Prerequisites

- `curl` and `jq`.
- The API endpoint: in this skill's directory, `source .env` if it exists, else `source .config` (the committed template); either sets `$URL`. An optional `MD_CONVERT_KEY` environment variable adds the `X-API-Key` header.

### How to clip

**1. Classify inputs:** an input starting with `http://` or `https://` is a URL; a path that exists on disk is a file. URLs alone use JSON mode; any file uses multipart mode.

**2. Call the API** (`$PAGE`, `$PAGE1`, `$PAGE2` are the pages to clip; `$URL` is the endpoint from Prerequisites):

```bash
# Single URL (JSON mode)
curl -sf -H "Content-Type: application/json" \
  ${MD_CONVERT_KEY:+-H "X-API-Key: $MD_CONVERT_KEY"} \
  -d "{\"url\": \"$PAGE\"}" \
  "$URL/convert"

# Multiple URLs
curl -sf -H "Content-Type: application/json" \
  ${MD_CONVERT_KEY:+-H "X-API-Key: $MD_CONVERT_KEY"} \
  -d "{\"urls\": [\"$PAGE1\", \"$PAGE2\"]}" \
  "$URL/convert"

# Files (multipart mode)
curl -sf \
  ${MD_CONVERT_KEY:+-H "X-API-Key: $MD_CONVERT_KEY"} \
  -F "files=@/path/to/file.pdf" \
  "$URL/convert"

# Mixed files + URLs
curl -sf \
  ${MD_CONVERT_KEY:+-H "X-API-Key: $MD_CONVERT_KEY"} \
  -F "files=@/path/to/file.pdf" \
  -F "url=https://example.com" \
  "$URL/convert"
```

**3. Parse the response:**

```json
{
  "success": true,
  "results": [
    {
      "name": "source-name",
      "source": "url|file",
      "format": "markdown|error",
      "tokens": 1234,
      "data": "# The markdown content..."
    }
  ]
}
```

Take the content from `.results[].data`. A result whose `.format` is `"error"` → report its `.error`.

**4. Post-process the markdown:**

Strip (top-down): skip-to-content links, site logo/branding, navigation menus, search UI, login links, breadcrumbs, cookie banners.

Strip (bottom-up): related-articles sections, site-wide footers, promotional CTAs, app-store badges, tracking pixels.

Clean up: remove trailing empty links, collapse 3+ blank lines to 2, trim whitespace.

**5. Output:**

Pick the destination from the response's `tokens` field:

- The user gave a save path or said "save" → write the file; confirm with path and token count.
- Small result (≲1,500 tokens) → display the cleaned markdown directly.
- Larger result → save to a sensibly named `.md` file in the working directory; show a 3–5 line summary plus the path.

**Always** report the token count.

### Options

| User says                   | Maps to                     |
| --------------------------- | --------------------------- |
| "wait for .content to load" | `waitForSelector` option    |
| "skip pdf metadata"         | `--pdf-no-meta`             |
| "save to X"                 | Write output to file path X |
| "raw" / "no clean"          | Skip post-processing        |

---

## 2. Docs — Library Documentation Discovery

Find and digest documentation for a library, framework, or tool. Work down this ladder: drop to the next rung **only** when the current one fails, and **never** retry a rung that already failed.

1. **context7.com llms.txt** — pre-digested docs for most popular projects
2. **The project's own llms.txt**
3. **Repository analysis** — clone + repomix when no llms.txt exists
4. **Web research** — last resort for scattered or unofficial docs

### Rung 1: context7

- GitHub-hosted project: `https://context7.com/{org}/{repo}/llms.txt`
- Docs website: `https://context7.com/websites/{normalized-domain-path}/llms.txt`
- **Focused question → filter by topic:** append `?topic={query}` to return only the matching sections. Prefer it whenever the user asks about one feature rather than a whole library — "the date picker in shadcn/ui" → `https://context7.com/shadcn-ui/ui/llms.txt?topic=date picker`.

### Rung 2: the project's own llms.txt

WebSearch `"[library name] llms.txt"`, or probe the common locations: `https://docs.[library].com/llms.txt`, `https://[library].dev/llms.txt`. A 404 or empty file → try the GitHub README or the official docs site, then the next rung.

### Rung 3: repository analysis

```bash
DOCS_DIR=$(mktemp -d)
git clone --depth 1 [repo-url] "$DOCS_DIR"
cd "$DOCS_DIR" && npx -y repomix --output repomix-output.xml
```

Read `repomix-output.xml` and extract the documentation: README, `docs/`, `examples/`, CONTRIBUTING. Report repository health (stars, last activity) with the answer — it dates the material.

### Rung 4: web research

Launch parallel researcher agents with distinct beats: official sources (registry page, website, release notes), tutorials, community threads (Stack Overflow, GitHub issues/discussions), API references found in the wild. Cross-check findings and flag conflicting or low-quality sources.

### Reading many URLs

An llms.txt often lists many page URLs. Read a few inline; past that, fan out to Explore agents, grouping related URLs on one agent:

| URLs  | Approach                                                                                  |
| ----- | ----------------------------------------------------------------------------------------- |
| 1–3   | read inline with WebFetch — no agents                                                      |
| 4–10  | 3–5 Explore agents                                                                         |
| 11–20 | 5–7 Explore agents                                                                         |
| 20+   | two phases: critical pages first (≤7 agents), a second batch only if still unanswered      |

Synthesize the agents' findings into one organized answer — **never** concatenate their outputs. When sources disagree, prefer the official or latest one and say why they disagree (usually version drift).

---

## 3. Transcript — YouTube Video Transcripts

### Usage

Run from this skill's directory (the one containing this SKILL.md):

```bash
uv run scripts/get_transcript.py "VIDEO_URL_OR_ID"               # plain text
uv run scripts/get_transcript.py "VIDEO_URL_OR_ID" --timestamps  # with timestamps
```

### Supported URL Formats

- `https://www.youtube.com/watch?v=VIDEO_ID`
- `https://www.youtube.com/shorts/VIDEO_ID`
- `https://youtu.be/VIDEO_ID`
- `https://youtube.com/embed/VIDEO_ID`
- Raw video ID (11 characters)

### Output Rules

- Preserve the transcript verbatim — **never** alter or omit words; reflowing whitespace for readability is allowed.
- Without timestamps, group the text into complete paragraphs — **never** break mid-sentence.
- **Always** save to a markdown file with the frontmatter below, never display the transcript in chat; confirm briefly with filename, title, and summary.

### Frontmatter & Naming

1. Generate a descriptive title from the video content.
2. Filename = title + `.md`, special characters removed.
3. Write the YAML frontmatter:

```yaml
---
title: <descriptive title>
summary: <2-4 sentence summary>
url: <full YouTube URL>
date: <publish date or today>
tags:
  - <tag-1>
  - <tag-2>
comment:
---
```

---

## Error Handling

- **Clip**: neither `.env` nor `.config` sets `URL` → tell the user to create `.env`. A non-200 response or `format: "error"` → report the error; **never** fall back silently to WebFetch.
- **Docs**: a rung fails → take the next one. When every rung fails, report which were tried.
- **Transcript**: captions unavailable (disabled, or missing for the language) → tell the user and suggest alternatives.