---
name: research-extract
description: >-
  Ingest and analyze content from YouTube, podcasts, blogs, PDFs, and audio files.
  Extract structured insights using parallel agent teams. Generate
  Show Notes and Cheat Sheet HTML variants. Use /research-extract when you want
  to analyze any content source and extract key insights, quotes, themes,
  challenges, solutions, frameworks, and external resources.
---

# Research Extract

Ingest content from various sources and extract structured insights using parallel agent team processing.

## First-Time Setup

Run once to install Python dependencies:

```bash
bash .claude/skills/research-extract/scripts/setup.sh
```

This checks for required system dependencies (yt-dlp, pdftotext, whisper).

## Commands

When the user invokes this skill, determine their intent:

### Ingest Commands
- "ingest [url]" or "add [url]" → Ingest a new source
- "ingest [url] as [slug]" → Ingest with custom slug name
- "ingest [filepath]" → Ingest a local file (text, PDF, audio)

### Analysis Commands
- "analyze [slug]" or "extract from [slug]" → Run full extraction with agent team
- "analyze [url] as [slug]" → Ingest AND run extraction in one step

### Variant Commands
- "variants [slug]" → Generate Show Notes + Cheat Sheet HTML from consolidated JSON

### Query Commands
- "list sources" → Show all ingested sources
- "show [slug]" → Show source details and extractions
- "progress" → Check extraction progress

---

## Source Naming

Always use descriptive slugs, not numeric IDs.

When ingesting, derive or ask for a slug:
- YouTube: `mfm-6-ideas`, `lex-altman-interview`
- Blog: `paul-graham-founder-mode`
- File: use filename without extension

The slug is used for all file paths and database lookups.

---

## Parallel Extraction Workflow

This is the core workflow for extracting insights from content.

### Step 1: Ingest the source

```bash
python3 .claude/skills/research-extract/scripts/ingest.py "[URL_OR_PATH]" --slug [SLUG]
```

This will:
- Auto-detect source type (YouTube, blog, PDF, text, audio)
- Download captions or transcribe with Whisper
- Store metadata and transcript in `.research-extract/sources/`

### Step 2: Chunk the transcript

```bash
python3 .claude/skills/research-extract/scripts/extract.py chunk --slug [SLUG]
```

This will:
- Split transcript into ~15k character chunks
- Save chunks to `.research-extract/chunks/[slug]_chunk_N.json`
- Initialize progress tracking
- Report number of chunks created

### Step 3: Create team and spawn teammates

Create an Agent Team to coordinate parallel extraction:

**3a. Create the team:**
```
TeamCreate with team_name: "extract-[SLUG]"
```

**3b. Create tasks** - one task per chunk. This lets teammates self-balance workload:
```
For a transcript with 6 chunks, create 6 tasks:

TaskCreate: "Extract chunk 0 for [SLUG]"
TaskCreate: "Extract chunk 1 for [SLUG]"
...
TaskCreate: "Extract chunk 5 for [SLUG]"
```

Each task description should include the full processing instructions (see teammate prompt below).

**3c. Spawn 3 teammates** in a SINGLE message (parallel tool calls):

```
Task tool (x3) with:
  team_name: "extract-[SLUG]"
  name: "extractor-1" / "extractor-2" / "extractor-3"
  subagent_type: "general-purpose"
```

**Teammate prompt template:**
```
You are an extraction teammate on team "extract-[SLUG]".

Your workflow:
1. Check TaskList for pending tasks with no owner
2. Claim a task using TaskUpdate (set owner to your name, status to in_progress)
3. Process the chunk:
   a. Read chunk: python3 .claude/skills/research-extract/scripts/process_chunk.py show --slug [SLUG] --chunk-id [CHUNK_ID]
   b. Analyze the content for: key insights, notable quotes, themes, challenges, solutions/approaches, action items, frameworks/models, and external resources
   c. Save result: python3 .claude/skills/research-extract/scripts/process_chunk.py save --slug [SLUG] --chunk-id [CHUNK_ID] --result '<json>'
4. Mark task completed via TaskUpdate
5. Check TaskList again - claim the next pending task
6. Repeat until no pending tasks remain

JSON format for save:
{
  "chunk_id": X,
  "key_insights": [
    {"title": "...", "description": "...", "speaker": "Name", "quote": "...", "significance": "..."}
  ],
  "quotes": [
    {"quote": "...", "speaker": "Name", "context": "..."}
  ],
  "themes": ["theme1", "theme2"],
  "challenges": [
    {"title": "...", "description": "...", "speaker": "Name", "quote": "..."}
  ],
  "solutions_approaches": [
    {"title": "...", "description": "...", "speaker": "Name", "implementation": "...", "quote": "..."}
  ],
  "action_items": [
    {"action": "...", "context": "...", "speaker": "Name"}
  ],
  "frameworks_models": [
    {"name": "...", "description": "...", "speaker": "Name", "quote": "..."}
  ],
  "external_resources": [
    {"type": "book|podcast|tool|person|course|website|other", "name": "...", "author": "...", "speaker": "who mentioned", "context": "...", "quote": "..."}
  ]
}

IMPORTANT: Capture ALL external resources - books, people referenced/quoted, tools, platforms, podcasts, courses, frameworks.
Work autonomously. Claim tasks, process them, move to next. Stop when no tasks remain.
```

**IMPORTANT:**
- Create 1 task per chunk (teammates self-balance by claiming work)
- Spawn ALL teammates in a SINGLE message (parallel tool calls)
- Use `subagent_type="general-purpose"` and `team_name="extract-[SLUG]"`
- 3 teammates is the default; adjust only for very small (1-2 chunks = 1 teammate) or very large (15+ chunks = 5 teammates) jobs

### Step 4: Wait for completion

Teammates send automatic idle notifications as they finish work. Check progress via:
```
TaskList - shows task completion status across all teammates
```

Fallback:
```bash
python3 .claude/skills/research-extract/scripts/extract.py progress
```

### Step 5: Merge results and clean up team

Once all tasks show completed in TaskList:

**5a. Merge results:**
```bash
python3 .claude/skills/research-extract/scripts/extract.py merge --slug [SLUG]
```
This combines all chunk extractions into `.research-extract/exports/[slug]_merged.json`

**5b. Shut down teammates:**
```
SendMessage type: "shutdown_request" to extractor-1
SendMessage type: "shutdown_request" to extractor-2
SendMessage type: "shutdown_request" to extractor-3
```

**5c. Clean up team:**
```
TeamDelete
```

### Step 6: Consolidate and rank

After merging, create a consolidated analysis that:
- Deduplicates similar insights, challenges, and solutions
- Ranks by frequency/importance
- Identifies top quotes
- Groups external resources by type
- Counts theme frequency

Save to `.research-extract/exports/[slug]_consolidated.json` with this structure:

```json
{
  "source": "Source title",
  "speakers": ["Speaker 1", "Speaker 2"],
  "url": "https://...",

  "key_insights": [
    {
      "rank": 1,
      "title": "Insight title",
      "description": "Synthesized description",
      "evidence": ["quote 1", "quote 2"],
      "speakers": ["Speaker 1"]
    }
  ],

  "themes": [
    {"theme": "Theme name", "frequency": 5}
  ],

  "challenges": [
    {
      "rank": 1,
      "title": "Challenge title",
      "description": "What the challenge is",
      "evidence": ["quote"],
      "speakers": ["Speaker 1"]
    }
  ],

  "solutions_approaches": [
    {
      "rank": 1,
      "title": "Solution title",
      "description": "What it is",
      "implementation": "How to do it",
      "evidence": ["quote"],
      "speakers": ["Speaker 1"]
    }
  ],

  "action_items": [
    {"action": "What to do", "context": "Why", "speaker": "Who"}
  ],

  "frameworks_models": [
    {"name": "Framework name", "description": "How it works", "speaker": "Who"}
  ],

  "top_quotes": [
    {
      "quote": "...",
      "speaker": "...",
      "context": "Why this quote matters"
    }
  ],

  "external_resources": {
    "books": [
      {"title": "Book Title", "author": "Author", "mentioned_by": "Speaker", "context": "Why referenced"}
    ],
    "people": [
      {"name": "Person Name", "mentioned_by": "Speaker", "context": "Why referenced"}
    ],
    "tools": [
      {"name": "Tool Name", "mentioned_by": "Speaker", "context": "How used"}
    ],
    "other": [
      {"type": "podcast|course|website|framework", "name": "...", "mentioned_by": "Speaker", "context": "..."}
    ]
  },

  "metadata": {
    "extraction_date": "2025-01-01T00:00:00",
    "total_chunks": 6,
    "source_type": "youtube"
  }
}
```

### Step 7: Generate writeup files

After consolidation, generate two markdown files:

**File 1: `.research-extract/writeups/[slug]-notes.md`** (Quick Reference)

Structure:
- Header with source, speakers, date
- Sections matching main topics from the content
- Bullet points with key facts
- Timestamps as clickable links where available: `[(4:57)](url#t=4m57s)`
- Key quotes in blockquotes with speaker attribution
- Table of best quotes at bottom

**File 2: `.research-extract/writeups/[slug]-writeup.md`** (Essay)

Structure:
- Introduction framing the topic's significance
- Body sections exploring each major theme with developed paragraphs
- Quotes woven into prose with timestamp citations
- Conclusion synthesizing implications

**Writing style:** See `STYLE_GUIDE.md` for detailed guidance on paragraph style, quote integration, voice, and punctuation rules.

### Step 8: Generate variant pages (on demand)

When the user runs `variants [slug]`:

1. Read consolidated JSON from `.research-extract/exports/[slug]_consolidated.json`
2. Read raw transcript from `.research-extract/sources/[slug].txt` (for timestamps)
3. Read template specs from `VARIANTS.md`
4. Create output directory: `.research-extract/variants/[slug]/`
5. Generate two self-contained HTML files:
   - `show-notes.html` — podcast companion page with timestamped sections, quote callouts, resource grids
   - `cheat-sheet.html` — print-optimized landscape reference card with color-coded sections
6. Open both files in the browser: `open .research-extract/variants/[slug]/show-notes.html .research-extract/variants/[slug]/cheat-sheet.html`

Follow all layout, CSS, and generation rules in VARIANTS.md exactly. Every insight, quote, tool, framework, and action item from the consolidated JSON should appear in at least one template.

### Step 9: Present summary to user

Show:
- Source slug and title
- Key insights found (ranked)
- Challenges and solutions found (ranked)
- Key themes with frequency
- Top 3-5 quotes
- External resources mentioned (books, people, tools, etc.)
- Links to generated files

---

## Storage Locations

All data stored in `{project_root}/.research-extract/`:

- **Source metadata**: `sources/[slug].json`
- **Transcripts**: `sources/[slug].txt`
- **Chunks**: `chunks/[slug]_chunk_N.json`
- **Chunk results**: `exports/[slug]_chunk_N_result.json`
- **Merged results**: `exports/[slug]_merged.json`
- **Consolidated**: `exports/[slug]_consolidated.json`
- **Progress**: `extraction_progress_[slug].json`
- **Notes writeup**: `writeups/[slug]-notes.md`
- **Essay writeup**: `writeups/[slug]-writeup.md`
- **Variant pages**: `variants/[slug]/show-notes.html`, `variants/[slug]/cheat-sheet.html`

---

## For Queries

List sources (reads `sources/*.json` flat files):
```bash
python3 -c "
import sys; sys.path.insert(0, '.claude/skills/research-extract/scripts')
from db import list_sources
for s in list_sources():
    print(f'{s[\"slug\"]}: [{s[\"source_type\"]}] {s[\"title\"]}')"
```

Get source details (reads `sources/{slug}.json` + `sources/{slug}.txt`):
```bash
python3 -c "
import sys; sys.path.insert(0, '.claude/skills/research-extract/scripts')
from db import get_source_by_slug
source = get_source_by_slug('[SLUG]')
print(f'Title: {source[\"title\"]}')
print(f'Type: {source[\"source_type\"]}')"
```

---

## Tips

- Always use Agent Teams for large transcripts (>20k chars)
- 3 teammates is the default; scale to 1 for tiny jobs (1-2 chunks) or 5 for large (15+ chunks)
- Create 1 task per chunk, teammates self-balance by claiming work from TaskList
- Merge deduplicates overlapping insights automatically
- Consolidation step is where ranking and synthesis happens
- Writeups and variants require the consolidation step to complete first
- Run `variants [slug]` to generate Show Notes + Cheat Sheet HTML pages
- Always shut down teammates and call TeamDelete after merging