# @vinxi/scraper

Agent-native scraping engine. An AI agent (or you) writes a small per-site scraper with `defineScraper()`; the runtime provides stealth browsing, HTTP transport, checkpointing, task queues, rate-limit-aware pacing, and streaming JSONL output. A companion set of reconnaissance tools helps the agent *understand* a site first — hydration payload walking, XHR capture, visual audits, and validation scorecards.

Designed to be driven by the [`site-reconnaissance` agent skill](https://github.com/nksaraf/skills):

```bash
npx skills add nksaraf/skills --skill site-reconnaissance
```

## Install

Runs on [Bun](https://bun.sh).

```bash
bun add @vinxi/scraper
bun add -d playwright   # only needed for browser-tier scrapers
bunx playwright install chromium
```

## Write a scraper

Scrapers live in `scrapers/<id>.ts` in your project:

```typescript
// scrapers/example-quotes.ts
import { defineScraper } from "@vinxi/scraper"

export default defineScraper({
  id: "example-quotes",
  source: "quotes.toscrape.com",
  async run(ctx) {
    const res = await ctx.fetch("https://quotes.toscrape.com/")
    const html = await res.text()
    // ...extract with cheerio, or use ctx.browser for JS-rendered sites
    ctx.pushData({ quote: "...", author: "..." })
  },
})
```

Run it from your project root:

```bash
bunx vinxi-scraper run example-quotes --city=delhi      # --key=value args reach ctx.args
bunx vinxi-scraper run example-quotes --rpm=60 --min-delay=500 --concurrency=8
bunx vinxi-scraper list
bunx vinxi-scraper export example-quotes --format json   # or csv | jsonl
```

Each run streams to `data/results/<id>/<timestamp>.jsonl`, and `data/results/<id>/latest.jsonl` points at the most recent successful run. Checkpoints and task queues persist under `data/` so interrupted runs resume. Set `SCRAPER_HOME` to point the CLI at a different project root.

Pacing defaults to a polite 30 requests/min with a 2s floor per domain — tune it per run with `--rpm`, `--min-delay` (ms), and `--concurrency`. CSV export JSON-encodes nested columns (e.g. a `variants` array), so commerce rows round-trip cleanly.

## The scraper context

`run(ctx)` receives:

| | |
|---|---|
| `ctx.fetch` | HTTP client with realistic headers, proxy-rotation-ready |
| `ctx.browser` | Lazy stealth Playwright facade — `newPage()` survives Akamai / Cloudflare / DataDome checks |
| `ctx.pushData` | Stream a result row out (persisted immediately) |
| `ctx.checkpoint` | Key-value store persisted to disk |
| `ctx.tasks` | Deduplicating, persistent task queue |
| `ctx.human` | Human-like pauses, mouse movement, warmup navigation, smooth scrolling |
| `ctx.log` / `ctx.sleep` / `ctx.args` | Structured logging, pacing, CLI args (`--key=value`) |

## Reconnaissance tools

```typescript
import {
  extractAllHydrationPayloads, walkSchema, filterLeaves, // hydration payloads (__NEXT_DATA__, JSON-LD, window.*)
  startXhrCapture, capturedToPayloads,                   // intercept JSON/XHR traffic before navigation
  auditPage, findMissingFields,                          // visual ground truth: screenshots + label/value pairs
  scoreCard, formatScoreCard,                            // completeness / quality / authenticity / freshness scoring
} from "@vinxi/scraper/recon"
```

These are deliberately noisy scaffolding: they help an agent see a page fast, then the agent writes a small per-site extractor. The methodology — six recon phases with quality gates, the recon–extract loop, validation scorecards — is documented in the [site-reconnaissance skill](https://github.com/nksaraf/skills).

## Canonical retail schema

`@vinxi/scraper/retail` exports `RetailStore` — a cross-brand store/dealer/branch/ATM location schema — plus helpers: `makeRetailStore()` (build a record from a partial, filling safe defaults), `extraOf()`, `normaliseHours()`, `normaliseCountry()`.

## Validation scorecard

`scoreCard()` scores a cohort of **store-locator / geo** records (0–100 across completeness, data quality, source authenticity, freshness). It expects `RetailStore` shapes — build them with `makeRetailStore()`. It defaults to India geography; pass `geoBounds` / `expectedMetros` for other regions.

```typescript
import { scoreCard, formatScoreCard } from "@vinxi/scraper/recon"
import { makeRetailStore } from "@vinxi/scraper/retail"

const stores = rows.map((r) => makeRetailStore({ brand: "Acme", name: r.name, lat: r.lat, lng: r.lng, address: r.address }))
const card = scoreCard(stores, {
  brand: "Acme",
  sourceDomain: "acme.com",
  officialDomain: "acme.com",
  authoritativeCount: 366,
  geoBounds: { minLat: 24, maxLat: 50, minLng: -125, maxLng: -66 }, // US (omit for India default)
  expectedMetros: ["New York", "Los Angeles", "Chicago"],
})
console.log(formatScoreCard(card)) // → band: high/medium/low + evidence
```

> The scorecard is for physical-location data. A **product catalog** (SKUs, prices, variants) has no geographic scorecard — validate those by sampling fields and reconciling counts instead.

## License

MIT