# @jobo/scraper-sdk

TypeScript SDK for building Jobo scraper providers — scrape job listings and details from any ATS.

## Overview

The Scraper SDK provides the types, base classes, and utilities needed to build a scraper provider that integrates with the Jobo platform. Each provider targets a specific ATS (Applicant Tracking System) and implements a standardized interface for discovering and extracting job postings.

### Key Concepts

- **Provider** — A scraper implementation targeting one ATS (e.g. Greenhouse, Lever, Workday).
- **Manifest** — A `manifest.json` declaring the provider's identity, URL patterns, and configuration.
- **Context** — Platform-injected services (HTTP client, browser client, logger) that handle proxying, fingerprinting, and rate limiting. Scrapers **must not** make HTTP requests outside the context.
- **Category** — Scrapers are either `"full"` (all details on the listing page) or `"split"` (minimal listings + secondary detail fetch).
- **Transport** — How data is retrieved: `"api"` (JSON/REST), `"html"` (static HTML), or `"browser"` (headless JS rendering).

### Architecture

```
┌─────────────────────────────────────────────────────────┐
│                    Jobo Platform                         │
│  ┌──────────┐  ┌──────────┐  ┌────────────────────┐    │
│  │  Proxy   │  │ Finger-  │  │   Rate Limiter     │    │
│  │  Pool    │  │ printing │  │   & Retry Engine    │    │
│  └────┬─────┘  └────┬─────┘  └─────────┬──────────┘    │
│       └──────────────┴──────────────────┘               │
│                      │                                   │
│              ┌───────▼───────┐                           │
│              │ ScraperContext │  ← Injected into provider │
│              │  .http        │                           │
│              │  .browser     │                           │
│              │  .log         │                           │
│              └───────┬───────┘                           │
└──────────────────────┼───────────────────────────────────┘
                       │
              ┌────────▼────────┐
              │ Your Provider   │  ← You build this
              │  validate()     │
              │  scrapeListings()│
              │  scrapeJobDetails()│
              └─────────────────┘
```

## Installation

```bash
npm install @jobo/scraper-sdk
```

## Modules

| Import Path                 | Description                                                 |
| --------------------------- | ----------------------------------------------------------- |
| `@jobo/scraper-sdk`         | Core types, interfaces, errors, `defineProvider()` helper   |
| `@jobo/scraper-sdk/runtime` | Base classes, HTML parsing, pagination & data helpers       |
| `@jobo/scraper-sdk/build`   | Compiler for bundling provider projects                     |
| `@jobo/scraper-sdk/testing` | Test runner for exercising providers against live endpoints |

## Quick Start

### 1. Project Structure

```
my-ats-scraper/
├── src/
│   ├── manifest.json      ← Provider metadata
│   └── provider.ts        ← Scraper implementation
├── package.json
└── tsconfig.json
```

### 2. Manifest (`src/manifest.json`)

```json
{
  "providerId": "greenhouse",
  "displayName": "Greenhouse",
  "category": "split",
  "transport": "api",
  "urlPatterns": [
    "https://boards.greenhouse.io/**",
    "https://*.greenhouse.io/**"
  ],
  "companyIdFromUrl": "https://boards.greenhouse.io/:companyId",
  "runtimeConfig": {
    "maxConcurrentDetails": 5,
    "delayBetweenRequestsMs": 200
  }
}
```

### 3. Provider Implementation

#### Full-Details Scraper (API-based)

When the ATS API returns all job details in the listing response:

```typescript
import {
  defineProvider,
  ScraperContext,
  ListingsResult,
  PaginationCursor,
  JobDetails,
} from "@jobo/scraper-sdk";
import { currentPage, pageCursor, buildMeta } from "@jobo/scraper-sdk/runtime";

interface ApiJob {
  id: number;
  title: string;
  content: string;
  location: { name: string };
  departments: { name: string }[];
  absolute_url: string;
  updated_at: string;
  metadata: { name: string; value: string }[];
}

interface ApiResponse {
  jobs: ApiJob[];
  meta: { total: number; per_page: number };
}

export default defineProvider({
  validate(url) {
    const match = url.match(/boards\.greenhouse\.io\/(\w+)/);
    return {
      isValid: !!match,
      companyId: match?.[1],
      baseUrl: match
        ? `https://boards-api.greenhouse.io/v1/boards/${match[1]}`
        : undefined,
    };
  },

  async scrapeListings(
    ctx: ScraperContext,
    cursor?: PaginationCursor,
  ): Promise<ListingsResult> {
    const page = currentPage(cursor);
    const res = await ctx.http.get<ApiResponse>(
      `${ctx.baseUrl}/jobs?page=${page}&per_page=50`,
    );

    const listings: JobDetails[] = res.data.jobs.map((job) => ({
      externalId: String(job.id),
      title: job.title,
      description: job.content,
      listingUrl: job.absolute_url,
      applyUrl: `${job.absolute_url}#app`,
      locations: [{ text: job.location.name }],
      postedAt: job.updated_at,
      meta: buildMeta(
        ["departments", job.departments.map((d) => d.name)],
        ...job.metadata.map((m) => [m.name, m.value] as [string, string]),
      ),
    }));

    const totalPages = Math.ceil(res.data.meta.total / res.data.meta.per_page);

    return {
      listings,
      hasMore: page < totalPages,
      nextCursor: page < totalPages ? pageCursor(page + 1) : undefined,
      totalCount: res.data.meta.total,
    };
  },
});
```

#### Split Scraper (HTML-based)

When the listing page only has links and details require a secondary fetch:

```typescript
import {
  defineProvider,
  ScraperContext,
  ListingsResult,
  JobDetailsResult,
  JobListing,
} from "@jobo/scraper-sdk";
import {
  BaseSplitScraper,
  extractLinks,
  extractText,
  extractJsonLd,
  findJsonLdByType,
  resolveUrl,
  stripHtml,
  parseLocation,
  buildMeta,
} from "@jobo/scraper-sdk/runtime";

export default defineProvider({
  validate(url) {
    const match = url.match(/jobs\.example\.com\/(\w+)/);
    return {
      isValid: !!match,
      companyId: match?.[1],
      baseUrl: match ? `https://jobs.example.com/${match[1]}` : undefined,
    };
  },

  async scrapeListings(ctx: ScraperContext): Promise<ListingsResult> {
    // Fetch the listing page (static HTML, no JS needed)
    const res = await ctx.http.get<string>(`${ctx.baseUrl}/jobs`, {
      responseType: "text",
    });

    // Extract job links
    const links = extractLinks(res.data, /\/jobs\/[\w-]+$/);

    const listings: JobListing[] = links.map((link) => {
      const slug = link.split("/").pop()!;
      return {
        externalId: slug,
        applyUrl: resolveUrl(`${link}/apply`, ctx.baseUrl),
        listingUrl: resolveUrl(link, ctx.baseUrl),
      };
    });

    return { listings, hasMore: false };
  },

  async scrapeJobDetails(
    ctx: ScraperContext,
    listing: JobListing,
  ): Promise<JobDetailsResult> {
    const res = await ctx.http.get<string>(listing.listingUrl!, {
      responseType: "text",
    });

    const html = res.data;

    // Try JSON-LD first (most reliable)
    const jsonLd = extractJsonLd(html);
    const jobPosting = findJsonLdByType(jsonLd, "JobPosting");

    if (jobPosting) {
      return {
        details: {
          ...listing,
          title: jobPosting.title as string,
          description: jobPosting.description as string,
          listingUrl: listing.listingUrl!,
          locations: [
            parseLocation(
              (jobPosting.jobLocation as any)?.address?.addressLocality ?? "",
            ),
          ],
          companyName: (jobPosting.hiringOrganization as any)?.name,
          postedAt: jobPosting.datePosted as string,
          meta: buildMeta([
            "employmentType",
            jobPosting.employmentType as string,
          ]),
        },
      };
    }

    // Fallback to HTML extraction
    const title = extractText(html, "h1");
    const description = extractText(html, 'div class="job-description"');

    if (!title || !description) {
      return {
        details: null,
        error: "Could not extract job details from HTML",
      };
    }

    return {
      details: {
        ...listing,
        title,
        description,
        listingUrl: listing.listingUrl!,
        locations: [
          parseLocation(extractText(html, 'span class="location"') ?? ""),
        ],
      },
    };
  },
});
```

#### Browser-Based Scraper (JS-rendered pages)

When the ATS is a SPA that requires JavaScript execution:

```typescript
import {
  defineProvider,
  ScraperContext,
  ListingsResult,
} from "@jobo/scraper-sdk";
import {
  extractText,
  extractLinks,
  resolveUrl,
} from "@jobo/scraper-sdk/runtime";

export default defineProvider({
  validate(url) {
    return { isValid: url.includes("spa-ats.example.com") };
  },

  async scrapeListings(ctx: ScraperContext): Promise<ListingsResult> {
    // Render the page with JavaScript enabled
    const page = await ctx.browser.renderPage({
      url: `${ctx.baseUrl}/careers`,
      waitUntil: "selector",
      waitForSelector: ".job-card",
      javascript: true,
    });

    // Now parse the rendered HTML
    const links = extractLinks(page.html, /\/careers\/\d+/);

    return {
      listings: links.map((link) => ({
        externalId: link.split("/").pop()!,
        applyUrl: resolveUrl(link, ctx.baseUrl),
        listingUrl: resolveUrl(link, ctx.baseUrl),
        title: extractText(page.html, ".job-card-title") ?? undefined,
      })),
      hasMore: false,
    };
  },
});
```

## API Reference

### Core Types (`@jobo/scraper-sdk`)

#### Data Models

| Type               | Description                                                                                             |
| ------------------ | ------------------------------------------------------------------------------------------------------- |
| `JobListing`       | A job listing from a search/listing page. Minimum: `externalId` + `applyUrl`.                           |
| `JobDetails`       | Full job details. Extends `JobListing` with required `title`, `description`, `locations`, `listingUrl`. |
| `JobLocation`      | A geographic location. Minimum: `text`. Optional: `city`, `state`, `country`, `isRemote`.               |
| `MetadataMap`      | `Record<string, MetadataValue>` — arbitrary key-value metadata.                                         |
| `MetadataValue`    | `string \| number \| boolean \| null \| MetadataValue[] \| { [key: string]: MetadataValue }`            |
| `PaginationCursor` | Opaque cursor for paginated scraping. Types: `offset`, `cursor`, `page`, `url`, `custom`.               |
| `ListingsResult`   | Result of `scrapeListings()`: `{ listings, hasMore, nextCursor?, totalCount? }`.                        |
| `JobDetailsResult` | Result of `scrapeJobDetails()`: `{ details, isExpired?, error? }`.                                      |

#### Provider Interface

| Type                      | Description                                                  |
| ------------------------- | ------------------------------------------------------------ |
| `ScraperProvider`         | The main interface every scraper must implement.             |
| `ScraperProviderManifest` | Schema for `manifest.json`.                                  |
| `ScraperCategory`         | `"full"` or `"split"`.                                       |
| `ScraperTransport`        | `"api"`, `"html"`, or `"browser"`.                           |
| `ValidationResult`        | Result of `validate()`: `{ isValid, companyId?, baseUrl? }`. |
| `defineProvider(p)`       | Type-narrowing helper that returns the provider as-is.       |

#### Platform Context

| Type             | Description                                                                                              |
| ---------------- | -------------------------------------------------------------------------------------------------------- |
| `ScraperContext` | Injected into every scraper method. Contains `http`, `browser`, `log`, `baseUrl`, `companyId`, `config`. |
| `HttpClient`     | HTTP client: `request()`, `get()`, `post()`. Handles proxying and fingerprinting.                        |
| `BrowserClient`  | Browser client: `renderPage()`. Handles headless rendering with JS.                                      |
| `ScraperLogger`  | Structured logger: `debug()`, `info()`, `warn()`, `error()`.                                             |

#### Errors

| Error Class           | Code            | Retryable  | Use Case                           |
| --------------------- | --------------- | ---------- | ---------------------------------- |
| `ScraperError`        | (base)          | —          | Base class for all scraper errors. |
| `NetworkError`        | `NETWORK_ERROR` | ✓          | DNS, connection, TLS failures.     |
| `HttpError`           | `HTTP_ERROR`    | 429/5xx: ✓ | HTTP error responses.              |
| `RateLimitError`      | `RATE_LIMITED`  | ✓          | 429 Too Many Requests.             |
| `ParseError`          | `PARSE_ERROR`   | ✗          | HTML/JSON structure changed.       |
| `AuthenticationError` | `AUTH_REQUIRED` | ✗          | Login wall, CAPTCHA.               |
| `ExpiredError`        | `EXPIRED`       | ✗          | Job posting removed.               |

### Runtime Utilities (`@jobo/scraper-sdk/runtime`)

#### Base Classes

| Class                                       | Description                                                                                           |
| ------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `BaseFullDetailsScraper`                    | Abstract base for full-details scrapers. Implement `validate()` + `scrapeListings()`.                 |
| `BaseSplitScraper`                          | Abstract base for split scrapers. Implement `validate()` + `scrapeListings()` + `scrapeJobDetails()`. |
| `mergeListingWithDetails(listing, details)` | Merge a listing with partial detail data into a complete `JobDetails`.                                |

#### HTML Parsing

| Function                                        | Description                                                 |
| ----------------------------------------------- | ----------------------------------------------------------- |
| `stripHtml(html)`                               | Strip all HTML tags, decode entities, normalize whitespace. |
| `stripHtmlAndTruncate(html, maxLength)`         | Strip HTML and truncate with `...`.                         |
| `extractText(html, selector)`                   | Extract inner text of the first matching tag.               |
| `extractAttribute(html, tagPattern, attribute)` | Extract an attribute value from a tag.                      |
| `extractMetaByName(html, name)`                 | Extract `<meta name="..." content="...">`.                  |
| `extractMetaByProperty(html, property)`         | Extract `<meta property="..." content="...">` (Open Graph). |
| `extractJsonLd(html)`                           | Parse all `<script type="application/ld+json">` blocks.     |
| `findJsonLdByType(items, type)`                 | Find a JSON-LD object by `@type`.                           |
| `extractLinks(html, pattern?)`                  | Extract all `<a href>` values, optionally filtered.         |
| `resolveUrl(url, baseUrl)`                      | Resolve a relative URL against a base.                      |

#### Pagination Helpers

| Function                   | Description                                   |
| -------------------------- | --------------------------------------------- |
| `offsetCursor(offset)`     | Create an offset-based cursor.                |
| `pageCursor(page)`         | Create a page-number cursor.                  |
| `cursorPagination(cursor)` | Create an opaque cursor.                      |
| `urlCursor(url)`           | Create a URL-based cursor.                    |
| `currentPage(cursor?)`     | Parse page number from cursor (default: `1`). |
| `currentOffset(cursor?)`   | Parse offset from cursor (default: `0`).      |

#### Data Helpers

| Function                           | Description                                                 |
| ---------------------------------- | ----------------------------------------------------------- |
| `parseLocation(text)`              | Parse a location string into a `JobLocation`.               |
| `parseLocations(text, delimiter?)` | Parse multiple locations from a delimited string.           |
| `parseDate(dateStr)`               | Parse a date string into ISO 8601 (handles relative dates). |
| `buildMeta(...entries)`            | Build a metadata map, skipping null/undefined values.       |
| `asString(value)`                  | Safely extract a string from unknown data.                  |
| `asNumber(value)`                  | Safely extract a number from unknown data.                  |
| `asStringArray(value)`             | Safely extract a string array from unknown data.            |
| `deduplicateListings(listings)`    | Remove duplicate listings by `externalId`.                  |
| `extractPathSegment(url, index)`   | Extract a URL path segment by position.                     |
| `buildUrl(base, path, params?)`    | Build a URL from base + path segments + query params.       |

### Build Tools (`@jobo/scraper-sdk/build`)

| Export                             | Description                                                                      |
| ---------------------------------- | -------------------------------------------------------------------------------- |
| `compileProvider(config)`          | Compile a provider project into a deployable JS bundle.                          |
| `watchProvider(config, onChange?)` | Watch mode: recompile on file changes.                                           |
| `BuildConfig`                      | Build configuration: `projectDir`, `outDir?`, `minify?`, `sourcemap?`.           |
| `BuildResult`                      | Build result: `success`, `outputDir`, `files`, `errors`, `warnings`, `duration`. |

### Testing (`@jobo/scraper-sdk/testing`)

| Export                 | Description                                                                                     |
| ---------------------- | ----------------------------------------------------------------------------------------------- |
| `runTest(config)`      | Run a comprehensive test against a live ATS endpoint.                                           |
| `runTestSuite(config)` | Run a full `tests.json` suite against a provider.                                               |
| `TestConfig`           | Test config: `provider`, `url`, `maxPages?`, `maxDetails?`, `verbose?`.                         |
| `TestResult`           | Test result: `success`, `totalListings`, `issues`, `listings`, `detailResults`, `qualityScore`. |
| `TestSuiteResult`      | Suite result: `success`, `total`, `passed`, `failed`, `cases[]`, `durationMs`.                  |
| `TestIssue`            | A data quality issue: `severity`, `phase`, `message`, `externalId?`.                            |
| `QualityScore`         | Metadata quality score: `overall` (0-100), `breakdown`, `findings`.                             |

## Manifest Reference

| Field                                  | Type                           | Required | Description                                                                              |
| -------------------------------------- | ------------------------------ | -------- | ---------------------------------------------------------------------------------------- |
| `providerId`                           | `string`                       | ✓        | Unique identifier (e.g. `"greenhouse"`).                                                 |
| `displayName`                          | `string`                       | ✓        | Human-readable name (e.g. `"Greenhouse"`).                                               |
| `category`                             | `"full" \| "split"`            | ✓        | Whether listings contain all details or need secondary fetch.                            |
| `transport`                            | `"api" \| "html" \| "browser"` | ✓        | Primary data retrieval method.                                                           |
| `urlPatterns`                          | `string[]`                     | ✓        | Glob patterns identifying this ATS (supports `*`, `**`, `:param`).                       |
| `companyIdFromUrl`                     | `string \| string[]`           |          | URL pattern(s) for extracting `companyId` (`:param` syntax).                             |
| `jobIdFromUrl`                         | `string \| string[]`           |          | URL pattern(s) for extracting `jobId` from a job detail URL. Used by `jobId` test cases. |
| `jsProviderName`                       | `string`                       |          | Global JS variable name when bundled.                                                    |
| `runtimeConfig.maxConcurrentDetails`   | `number`                       |          | Max parallel detail fetches (split only). Default: `3`.                                  |
| `runtimeConfig.delayBetweenRequestsMs` | `number`                       |          | Min delay between requests. Default: `0`.                                                |
| `runtimeConfig.pageSettleDelayMs`      | `number`                       |          | Delay after page nav (browser only). Default: `0`.                                       |
| `runtimeConfig.maxListingsPerRun`      | `number`                       |          | Max listings per run. `0` = unlimited.                                                   |
| `runtimeConfig.maxRetries`             | `number`                       |          | Max retry attempts. Default: `3`.                                                        |

## Testing (`tests.json`)

Every provider should have a `src/tests.json` file. The test suite is run with `jobo-scraper test` (no `--url` flag). Four test types are supported:

### `urlPattern` — Offline URL matching

Checks the URL against the provider's `urlPatterns` without making any HTTP requests.

```json
{ "type": "urlPattern", "name": "Company board matches", "url": "https://jobs.ashbyhq.com/ramp", "expect": { "matches": true } }
{ "type": "urlPattern", "name": "Job detail URL matches", "url": "https://jobs.ashbyhq.com/ramp/f2ad6068-02e7-4986-967e-804ecef9e043", "expect": { "matches": true } }
{ "type": "urlPattern", "name": "Wrong domain does not match", "url": "https://jobs.lever.co/ramp", "expect": { "matches": false } }
```

### `validation` — Provider URL validation

Calls `provider.validate(url)` and checks `isValid`, `companyId`, and `baseUrl`.

```json
{ "type": "validation", "name": "Extracts companyId", "url": "https://jobs.ashbyhq.com/ramp", "expect": { "isValid": true, "companyId": "ramp", "baseUrl": "https://jobs.ashbyhq.com/ramp" } }
{ "type": "validation", "name": "Rejects non-Ashby URL", "url": "https://jobs.lever.co/ramp", "expect": { "isValid": false } }
```

| Assertion   | Description                                                   |
| ----------- | ------------------------------------------------------------- |
| `isValid`   | Required. Whether `validate()` should return `isValid: true`. |
| `companyId` | Optional. Expected `companyId` extracted from the URL.        |
| `baseUrl`   | Optional. Expected `baseUrl` returned by `validate()`.        |

### `jobId` — Job ID extraction from URL

Uses the manifest's `jobIdFromUrl` patterns to extract `jobId` and `companyId` from a job detail URL. No HTTP requests.

```json
{
  "type": "jobId",
  "name": "Extracts jobId and companyId",
  "url": "https://jobs.ashbyhq.com/ramp/f2ad6068-02e7-4986-967e-804ecef9e043",
  "expect": {
    "jobId": "f2ad6068-02e7-4986-967e-804ecef9e043",
    "companyId": "ramp"
  }
}
```

| Assertion   | Description                                           |
| ----------- | ----------------------------------------------------- |
| `jobId`     | Required. Expected job ID extracted from the URL.     |
| `companyId` | Optional. Expected company ID extracted from the URL. |

> **Requires** `jobIdFromUrl` in `manifest.json`, e.g. `"jobIdFromUrl": ["https://jobs.ashbyhq.com/:companyId/:jobId"]`

### `scrape` — Live scrape with assertions

Runs a full live scrape and validates listing/detail counts, required fields, and extracted metadata.

```json
{
  "type": "scrape",
  "name": "Ramp jobs",
  "url": "https://jobs.ashbyhq.com/ramp",
  "maxPages": 1,
  "maxDetails": 3,
  "expect": {
    "minListings": 1,
    "listingHasFields": ["externalId", "applyUrl", "listingUrl"],
    "detailHasFields": ["title", "description", "listingUrl", "applyUrl"],
    "detailMetaHasKeys": ["employment_type", "department"]
  }
}
```

| Assertion           | Description                                                             |
| ------------------- | ----------------------------------------------------------------------- |
| `minListings`       | Minimum number of listings expected.                                    |
| `listingHasFields`  | Fields that must be non-null/non-empty on **every** listing.            |
| `detailHasFields`   | Fields that must be non-null/non-empty on **every** detail.             |
| `detailMetaHasKeys` | Meta keys that must be present on **at least one** detail's `meta` map. |

## Important Rules

1. **Never make HTTP requests directly.** Always use `ctx.http` or `ctx.browser`. The platform handles proxying, fingerprinting, rate limiting, and retries.

2. **External IDs must be deterministic.** The same job must always produce the same `externalId`. This is critical for deduplication across scrape runs.

3. **Return `applyUrl` on every listing.** This is the minimum viable field — even split scrapers must provide it.

4. **Use typed errors.** Throw `HttpError`, `ParseError`, `RateLimitError`, etc. instead of generic `Error`. The platform uses error types to decide retry strategy.

5. **Metadata keys should be camelCase.** Common keys: `employmentType`, `experienceLevel`, `department`, `workplaceType`, `compensation`, `benefits`, `tags`.

## Scripts

| Command             | Description                      |
| ------------------- | -------------------------------- |
| `npm run build`     | Build the SDK with tsup.         |
| `npm run dev`       | Watch mode — rebuild on changes. |
| `npm run typecheck` | Run TypeScript type checking.    |
| `npm run clean`     | Remove the `dist/` directory.    |

## License

MIT