# Mode: scan — Portal Scanner (Offer Discovery)

Scans configured job portals, filters by title relevance, and adds new offers to the pipeline for later evaluation.

## Recommended Execution

Run as a subagent to avoid consuming main context:

```
Agent(
    subagent_type="general-purpose",
    prompt="[contents of this file + specific data]",
    run_in_background=True
)
```

## Read This Configuration

Read `portals.yml` which contains:
- `search_queries`: List of WebSearch queries with `site:` filters per portal (broad discovery)
- `tracked_companies`: Specific companies with `careers_url` for direct navigation
- `title_filter`: Positive/negative/seniority_boost keywords for title filtering
- `diversity_policy`: US-only location and company-mix coverage targets

Also read `config/profile.yml` location eligibility. The distributable defaults
target the United States only. The posting's location, remote restrictions,
work authorization, and relocation requirements are hard constraints. Diversity
targets only balance US-eligible offers.

## Diversity and eligibility invariants

1. **Use job location, not headquarters.** A US company can offer a role in India; a non-US company can advertise a US role. Extract the location and remote scope from every posting.
2. **US evidence is mandatory.** Keep a posting only when it explicitly allows work in the United States. Generic `Remote`, `Americas`, `North America`, or an unspecified location is not enough and is excluded under `unknown_location: exclude`.
3. **Best-effort US breadth.** After role and eligibility filtering, balance the shortlist across US Northeast, Midwest, South, West, nationwide/remote postings, companies, and work arrangements.
4. **No fabricated coverage.** If a US location bucket has no suitable results, report a coverage gap and the reason; never substitute a non-US or ambiguous posting.
5. **Company cap.** Respect `max_results_per_company` before filling remaining slots so one large employer cannot dominate the scan.

## Apply This Discovery Strategy (3 levels)

### Use Level 1 — Direct Geometra (PRIMARY)

For scheduled companies whose `scan_method` is `geometra` (or omitted), connect to `careers_url` with Geometra MCP (`geometra_connect({ ..., isolated: true, headless: true, browserMode: "stock", blockDetection: true, blockedSitePolicy: "manual-handoff" })`), save the returned `sessionId`, then pass it explicitly to `geometra_page_model` / `geometra_list_items`. Read all visible job listings and extract the posting metadata. Companies configured with `scan_method: websearch` run their `scan_query` at Level 3; configured APIs run at Level 2. Direct Geometra is reliable because:

- It sees the page in real time (not cached Google results).
- It works with SPAs (Ashby, Lever, Workday).
- It detects new offers instantly.
- It doesn't depend on Google indexing.

**Every company MUST have a `careers_url` in portals.yml.** If it doesn't, search for it once, save it, and use it in future scans.

### Use Level 2 — ATS / Aggregator APIs (COMPLEMENTARY)

For companies using an ATS or aggregator that exposes a public JSON/RSS API, fetch structured data directly. APIs are faster than Geometra and harder to hallucinate (the response is load-bearing — record IDs verbatim from the response, never reconstruct them). Use as a complement to Level 1.

**OpenCode WebFetch compatibility:** when fetching JSON/RSS API endpoints, do not pass `format: "json"`. OpenCode accepts `text`, `markdown`, or `html`; omit `format` or use `format: "text"` and parse JSON/RSS from the returned body.

Supported API shapes:

#### Greenhouse (JSON, per-company board)

- **Endpoint**: `https://boards-api.greenhouse.io/v1/boards/{slug}/jobs`
- **Method**: `GET` (plain, no auth)
- **Shape**: `{ jobs: [{ id, title, absolute_url, updated_at, location: { name } }, ...] }`
- **Canonical URL to record**: `https://job-boards.greenhouse.io/{slug}/jobs/{id}` — do NOT use `absolute_url` when it points to a customer-skinned front-end (see **Verify Before Marking CLOSED** below).
- **ats**: `greenhouse`

#### Ashby (JSON, per-company board)

- **Endpoint**: `https://api.ashbyhq.com/posting-api/job-board/{slug}?includeCompensation=true`
- **Method**: `GET`
- **Shape**: `{ jobs: [{ id, title, jobUrl, publishedDate, locationName, employmentType, department, team, compensation }] }`
- **Canonical URL to record**: use the returned `jobUrl` (format `https://jobs.ashbyhq.com/{slug}/{uuid}`).
- **ats**: `ashby`

#### Lever (JSON, per-company board)

- **Endpoint**: `https://api.lever.co/v0/postings/{slug}?mode=json`
- **Method**: `GET`
- **Shape**: array of postings `[{ id, text, hostedUrl, createdAt, categories: { commitment, department, location, team } }, ...]`
- **Canonical URL to record**: `hostedUrl` (format `https://jobs.lever.co/{slug}/{uuid}`).
- **ats**: `lever`

#### Workday (JSON, per-tenant + site — FINICKY)

- **Endpoint**: `https://{subdomain}.{pod}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs`
  - `subdomain` = the Workday tenant hostname prefix (e.g. `nvidia`, `salesforce`, `adobe`, `shopify`).
  - `pod` = the Workday data-center pod segment (varies: `wd1`, `wd3`, `wd5`). The hostname in `careers_url` reveals which.
  - `tenant` = repeats the company slug in the path (usually equal to `subdomain`, but not always).
  - `site` = the public site name exposed by the tenant (e.g. `NVIDIAExternalCareerSite`, `External`, `ShopifyCareerSite`). Read it from the tenant's HTML landing page if unknown.
- **Method**: `POST` with JSON body:
  ```json
  {"appliedFacets": {}, "limit": 20, "offset": 0, "searchText": ""}
  ```
- **Required headers**: `Content-Type: application/json`, `Accept: application/json`. If the response is 403, set a realistic `User-Agent` header and retry — Workday tenants selectively block data-center UAs.
- **Shape**: `{ jobPostings: [{ title, externalPath, postedOn, locationsText, bulletFields }, ...], total }`
- **Canonical URL to record**: `https://{subdomain}.{pod}.myworkdayjobs.com/{site}{externalPath}` (note: `externalPath` already starts with `/job/...` — do NOT prepend an extra `/`).
- **Pagination**: increment `offset` by `limit` (20) until `jobPostings.length < limit` or `offset >= total`.
- **ats**: `workday`
- **Fallback**: Workday APIs are brittle — tenants occasionally block POST from data-center IPs, change `site` names silently, or return empty `jobPostings` while the HTML page shows listings. If the POST fails or returns 0 jobs on a tenant that Level 1 confirmed has listings, fall back to Level 1 (Geometra scraping the `careers_url`). Treat Workday as Level 2 with a guaranteed Level 1 fallback.

#### SmartRecruiters (JSON, per-company postings)

- **Endpoint**: `https://api.smartrecruiters.com/v1/companies/{company}/postings`
- **Method**: `GET` (plain, no auth)
- **Shape**: `{ content: [{ id, name, refNumber, jobAdUrl, releasedDate, location: { city, country, remote }, company: { identifier, name }, department }], totalFound, offset, limit }`
- **Canonical URL to record**: use `jobAdUrl` when present, otherwise `https://jobs.smartrecruiters.com/{company}/{id}`.
- **Pagination**: pass `?offset=N&limit=100` (max 100). Loop until `offset + content.length >= totalFound`.
- **ats**: `smartrecruiters`

#### WeWorkRemotely (RSS, cross-company aggregator)

- **Endpoints** (one public category feed per target area; the RSS URL itself has no server-side country parameter):
  - `https://weworkremotely.com/categories/remote-programming-jobs.rss`
  - `https://weworkremotely.com/categories/remote-devops-sysadmin-jobs.rss`
  - `https://weworkremotely.com/categories/remote-product-jobs.rss`
  - `https://weworkremotely.com/categories/remote-design-jobs.rss`
  - `https://weworkremotely.com/categories/all-other-remote-jobs.rss`
- **Method**: `GET` — returns RSS 2.0 XML.
- **Shape/example**: `<item><title>Acme: Senior Platform Engineer</title><link>https://weworkremotely.com/remote-jobs/acme-senior-platform-engineer</link><pubDate>...</pubDate><region>USA Only</region></item>`.
- **Required US prefilter**: after fetching the configured RSS URL, retain an item only when `<region>` exactly matches a configured `location_filter.allowed_values` value. The US-only template permits only `USA Only`. Exclude missing regions and `Anywhere in the World`, `North America Only`, `Americas Only`, or any other value before title filtering. The full posting must still explicitly confirm US hiring scope.
- **Company/role extraction**: split `<title>` on the first `: ` — left side is company, right side is role. Fallback to the whole title as role if there is no `: `.
- **Canonical URL to record**: the `<link>` verbatim (format `https://weworkremotely.com/remote-jobs/{slug}`).
- **Cross-company note**: WeWorkRemotely is NOT per-company — it aggregates postings from hundreds of companies. Scan it via the `cross_company_feeds` section in `portals.yml`, not `tracked_companies`.
- **ats**: `wwr` (aggregator). The underlying company's ATS is unknown at scan time — downstream evaluators follow the link and re-detect.

#### RemoteOK (JSON, cross-company aggregator)

- **Endpoint**: `https://remoteok.com/api`
- **Method**: `GET` — returns a JSON array. The **first element is a legal/disclaimer object** (no `id`, has `legal`) — skip it. The remaining 100 entries are postings.
- **Required headers**: `User-Agent: Mozilla/5.0 ...` — RemoteOK returns 403 without a browser-like UA.
- **Shape** (per posting after skip): `{ id, slug, company, company_logo, position, description, tags: [string], date, epoch, url, apply_url, location, salary_min, salary_max }`
- **Canonical URL to record**: `url` (format `https://remoteok.com/remote-jobs/{id}-{slug}`).
- **Required US prefilter**: normalize the optional `location` field to ISO country code `US` using the feed's `location_filter`. Missing, generic, worldwide, continental, or otherwise ambiguous values are excluded before tag/title filtering. Do not infer US eligibility from tags, salary currency, company headquarters, or the description's word `remote`; verify the full posting after the prefilter.
- **Filtering**: RemoteOK feeds are broad — use `tags` for pre-filter (e.g. `tags` contains `"engineer"` or `"ai"`) before passing through `title_filter`.
- **Cross-company note**: same as WeWorkRemotely — configure via `cross_company_feeds`, not `tracked_companies`.
- **ats**: `remoteok` (aggregator).

### Use Level 3 — WebSearch Queries (BROAD DISCOVERY)

The `search_queries` with `site:` filters cover portals broadly (all Ashby boards, all Greenhouse boards, all Lever boards, all Workday boards). Useful for discovering NEW companies not yet in `tracked_companies`, but results may be outdated.

**Execution priority:**
1. Level 1: Geometra → rotating US-tagged direct-company pool, up to `tracked_company_scan_budget`
2. Level 2: API → scheduled US-tagged `tracked_companies` with any supported API, plus enabled cross-company feeds
3. Level 3: WebSearch → enabled queries and scheduled US-tagged company queries, with `search_location_constraint` appended

The levels are additive — all are executed, results are merged and deduplicated.

## Run This Workflow

1. **Fail-closed configuration check, then read configuration**:
   - Run `npx job-forge verify:diversity portals.yml --profile config/profile.yml`.
   - If it fails, stop before Geometra, WebFetch, WebSearch, or any other
     network request. Follow its targeted upgrade instructions; never replace
     either personal YAML file wholesale or continue with legacy defaults.
   - After it passes, read `portals.yml` and `config/profile.yml`.
2. **Read history**: `data/scan-history.tsv` → previously seen URLs
3. **Read dedup sources**: all day files in `data/applications/` + `data/pipeline.md`

4. **Build the tracked-company schedule, then run Level 1 — Geometra scan** (sequential, or ≤2 parallel via `task` subagents per Hard Limit #1 in `AGENTS.md`):
   - Include only enabled direct companies carrying one of `diversity_policy.source_company_tags` (the default is `us`). This tag schedules likely US sources but never proves that a particular posting is US-eligible.
   - Sort the pool deterministically, rotate its starting offset by UTC day-of-year, and take up to `tracked_company_scan_budget`. This gives different companies exposure across runs instead of repeatedly scanning the first entries in the file.
   - Do not scan non-US-tagged or unclassified sources in the default US-only configuration. A company may be added to the pool only after its catalog scope is verified and tagged.
   - Reuse this scheduled pool for Levels 1-3. Scan companies whose `scan_method` is `geometra` or omitted here; route `websearch` entries to Level 3 and entries with an API configuration to Level 2 as applicable.

   For each scheduled company:
   a. `geometra_connect` to the `careers_url` with `isolated: true`, `headless: true`, `browserMode: "stock"`, `blockDetection: true`, and `blockedSitePolicy: "manual-handoff"`; save its returned `sessionId`
   b. `geometra_page_model({sessionId: "<returned id>"})` or `geometra_list_items({sessionId: "<returned id>", ...})` to read all job listings
   c. If the page has filters/departments, navigate the relevant sections
   d. For each job listing extract: `{title, url, company, location, remote_scope, employment_type}`
   e. If the page paginates results, navigate additional pages
   f. Accumulate in candidates list
   g. If `careers_url` fails (404, redirect), try `scan_query` as fallback and note for URL update
   h. After evidence is captured for that company, call `geometra_disconnect({sessionId: "<returned id>", closeBrowser: true})` for that session only; never disconnect peer-worker ids

5. **Level 2 — ATS / Aggregator APIs** (WebFetch can batch freely — it's cheap and doesn't use Geometra sessions):

   **5a. Per-company APIs** — for each company in the scheduled US-tagged pool with `api:` or a supported ATS API configuration defined:
   a. WebFetch (or `fetch` for Workday, which needs POST) the API URL per the endpoint shape documented above.
   b. Extract per-posting `{title, url, company, location, remote_scope, employment_type, updated_at, ats}` plus ATS-specific IDs:
      - **Greenhouse** → also record `gh_slug`, `gh_id`. URL MUST be canonical `https://job-boards.greenhouse.io/{gh_slug}/jobs/{gh_id}` — do **NOT** use `absolute_url` when it points to a customer-skinned front-end (e.g. `pinterestcareers.com/jobs/?gh_jid=N`, `okta.com/company/careers/opportunity/N`, `samsara.com/company/careers/roles/N`, `zoominfo.com/careers?gh_jid=N`, `collibra.com/.../?gh_jid=N`, `careers.toasttab.com/jobs?gh_jid=N`, `careers.airbnb.com/positions/N`, `coinbase.com/careers/positions/N`, `instacart.careers/job/?gh_jid=N`). These customer front-ends return shells or 403 to bots and cause downstream WebFetch-based verification to wrongly mark the role CLOSED.
      - **Ashby** → record the returned `jobUrl`.
      - **Lever** → record the returned `hostedUrl`.
      - **Workday** → build URL as `https://{subdomain}.{pod}.myworkdayjobs.com/{site}{externalPath}`. If the POST fails, DROP that tenant's API attempt and fall back to Level 1 for that company — do NOT fabricate postings.
      - **SmartRecruiters** → record `jobAdUrl` (fallback: `https://jobs.smartrecruiters.com/{company}/{id}`).
      - **`updated_at`**: use `updated_at` (Greenhouse) / `publishedDate` (Ashby) / `createdAt` (Lever) / `postedOn` (Workday) / `releasedDate` (SmartRecruiters) — record for staleness detection (skip if older than 90 days, flag if older than 30).
   c. Accumulate in candidates list (dedup with Level 1). The pipeline.md entry MUST carry `| ats={type}` at the end, and for Greenhouse ALSO `| gh={gh_slug}/{gh_id}` so downstream evaluators can fall back to `https://boards-api.greenhouse.io/v1/boards/{gh_slug}/jobs/{gh_id}` when the canonical URL renders as a shell.

   **5b. Cross-company aggregator feeds** — for each feed in `cross_company_feeds` with `enabled: true`:
   a. WebFetch the RSS (WeWorkRemotely) or JSON (RemoteOK) endpoint per the shape documented above.
   b. Apply `location_filter` first. An enabled global feed without a location field/filter is invalid in the US-only configuration. `require_explicit_match: true`, `on_missing: exclude`, and `on_ambiguous: exclude` are fail-closed; log rejected rows as `skipped_location`.
   c. Parse each surviving entry to `{title, url, company, location, remote_scope, employment_type, ats, updated_at}`:
      - **WeWorkRemotely** → require `<region>USA Only</region>`; split `<title>` on the first `: ` to separate company from role; `<link>` → url; `<pubDate>` → updated_at. Do not use the company's headquarters from `<description>` as job-location evidence.
      - **RemoteOK** → skip the first element (legal disclaimer); require its `location` field to normalize explicitly to ISO `US`; then take `company`, `position`, `url`, and `date`. Missing or ambiguous `location` means exclusion, even if the description later mentions a US city.
   d. Apply the feed's `tag_filter` / `category_filter` before the global `title_filter` — aggregators have much higher volume than per-company APIs.
   e. Read and verify the full posting before assigning `location_status: eligible`, then accumulate it in the candidates list (dedup with Level 1 + 5a).

6. **Level 3 — WebSearch queries** (WebSearch is parallel-safe; batch freely):
   For each query in `search_queries` with `enabled: true`, plus each US-tagged
   scheduled company routed here by `scan_method: websearch`:
   a. Wrap the defined `query` or `scan_query` in parentheses, append `diversity_policy.search_location_constraint`, and execute WebSearch. The US clause is mandatory even when the stored query says `remote`; the full posting must still pass the location eligibility classification below.
   b. From each result extract: `{title, url, company, location, remote_scope, employment_type}` when present
      - **title**: from the result title (before " @ " or " | ")
      - **url**: result URL
      - **company**: after " @ " in the title, or extract from domain/path
   c. Accumulate in candidates list (dedup with Level 1+2)

7. **Filter by title** using `title_filter` from `portals.yml`:
   - At least 1 keyword from `positive` must appear in the title (case-insensitive)
   - 0 keywords from `negative` must appear
   - `seniority_boost` keywords give priority but are not required

8. **Classify location and eligibility**:
   - Read the full posting when the listing card or search result does not contain enough location evidence.
   - Confirm explicit US scope: a US city/state, `United States`/`USA`, or a remote statement that clearly hires in the US. A US salary band or company headquarters alone is not sufficient.
   - Normalize the posting into exactly one `diversity_policy.regions[].id` using `job_location_terms`. Prefer a specific city/state bucket over `us_nationwide` when both match.
   - Compare ISO country code `US` and the normalized US bucket against `location.eligible_work_country_codes`, `preferred_region_ids`, `remote.allowed_region_ids`, `remote.require_explicit_location_eligibility`, and `willing_to_relocate` in `config/profile.yml`.
   - Set `location_status: eligible` only after both US scope and the user's work/remote constraints pass. Classify explicit non-US postings as `ineligible`; treat missing or ambiguous scope as excluded because the template sets `unknown_location: exclude`.
   - Never write non-US, `exploratory`, `location_review`, or ambiguous postings to the pipeline in the US-only default.

9. **Deduplicate** against 3 sources (URL-exact + fuzzy company+role):

   **Layer 1 — URL-exact:**
   - `scan-history.tsv` → exact URL already seen
   - `pipeline.md` → exact URL already in pending or processed

   **Layer 2 — Company + conservative role identity (catches true reposts with new URLs):**
   - all day files in `data/applications/` → normalize company name (lowercase, strip non-alphanumeric) + a conservative scan-layer role comparison. This prompt-driven prefilter does not execute the tracker matcher. Keep the candidate whenever identity is uncertain; `dedup-tracker.mjs` and `merge-tracker.mjs` apply their shared deterministic tracker identity during settlement.
   - `scan-history.tsv` → apply the same conservative identity policy to company + title columns (not just URL). A role reposted on a new URL is a duplicate only when the normalized role identities agree.
   - `pipeline.md` → apply the same conservative identity policy to company + title in pending items that include metadata (format: `- [ ] {url} | {company} | {title}`)

   **Role identity rules:**
   - Normalize company: `company.toLowerCase().replace(/[^a-z0-9]/g, '')`
   - Preserve every normalized role word and its order; never apply global role stopwords or derivational stemming such as `engineering`→`engineer` or `management`→`manager`. Preserve semantic punctuation before tokenization: `C++`, `C#`, `F#`, and `.NET` remain distinct from `C`, `F`, and `NET`. Location or work-mode words such as `US`, `United States`, `remote`, `hybrid`, and `New York` remain identity-bearing. Do not use substring or shared generic-title overlap. Remove only clearly positional seniority prefixes; never globally erase ambiguous words such as `lead`, `staff`, or `associate`. Short specialty tokens remain significant: `UI Engineer` and `UX Engineer` are distinct; `Product Manager, Cash Platform` and `AI Product Manager, Professional Services` are distinct; `Senior Product Managers, Payments` and `Product Manager - Payment` match after safe plural and seniority normalization.
   - When a role-identity match is found but the URL is new, log it as `skipped_repost` (not `skipped_dup`) with a note referencing the original entry number.

10. **Balance the eligible, deduplicated shortlist** using `diversity_policy`:
   - Write the normalized candidate array to `/tmp/jobforge-scan-candidates-{YYYY-MM-DD}.json`, including posting-derived `country_code`, `relevance_score`, `region`, and `location_status`.
   - Run `npx job-forge balance:scan /tmp/jobforge-scan-candidates-{YYYY-MM-DD}.json portals.yml` and use its JSON result as the canonical shortlist. This command resolves the balancer shipped inside the harness for both package consumers and harness checkouts.
   - `selected` is bounded by `shortlist_size`, fills US location floors, reaches `minimum_distinct_companies` before adding extra roles from represented employers, enforces `max_results_per_company`, and treats `max_region_share` as a best-effort ceiling.
   - With `unknown_location: exclude`, `review` MUST be empty. `excluded` contains non-US, ambiguous, or malformed rows.
   - Use `summary.coverage` and `summary.gaps` verbatim in the scan summary. If the helper fails, stop before writing to the pipeline rather than silently using an unbalanced list.

11. **Write balanced results**:
   - For each `selected` offer, add to `pipeline.md` section "Pending": `- [ ] {url} | {company} | {title} | {location} | US | {region} | eligible | ats={ats}`. The `ats` value is required and must be one of `greenhouse`, `ashby`, `workable`, `lever`, `workday`, `smartrecruiters`, `wwr`, `remoteok`, `builtin`, `custom`, or `unknown`.
   - For a Greenhouse API result, also append `| gh={gh_slug}/{gh_id}` using values copied verbatim from the API. Example: `- [ ] https://job-boards.greenhouse.io/webflow/jobs/7689676 | Webflow | Lead AI Engineer | New York, NY | US | us_northeast | eligible | ats=greenhouse | gh=webflow/7689676`.
   - Do not add `review` or `excluded` offers to `pipeline.md`.
   - Record selected offers in `scan-history.tsv` with status `added`; record non-US and ambiguous offers with status `skipped_location` and a concrete reason.

12. **Offers filtered by title**: record in `scan-history.tsv` with status `skipped_title`
13. **Location-ineligible offers**: record with status `skipped_location`
14. **Duplicate offers (URL-exact)**: record with status `skipped_dup`
15. **Duplicate offers (fuzzy repost)**: record with status `skipped_repost` and note `repost of #{original_entry_num}`

## Extract Title And Company From WebSearch Results

WebSearch results come in the format: `"Job Title @ Company"` or `"Job Title | Company"` or `"Job Title — Company"`.

Extraction patterns by portal:
- **Ashby**: `"Senior AI PM (Remote) @ EverAI"` → title: `Senior AI PM`, company: `EverAI`
- **Greenhouse**: `"AI Engineer at Anthropic"` → title: `AI Engineer`, company: `Anthropic`
- **Lever**: `"Product Manager - AI @ Temporal"` → title: `Product Manager - AI`, company: `Temporal`

Generic regex: `(.+?)(?:\s*[@|—–-]\s*|\s+at\s+)(.+?)$`

## Resolve Private URLs

If a publicly inaccessible URL is found:
1. Save the JD to `jds/{company}-{role-slug}.md`
2. Classify the frozen JD through the same US eligibility gate; do not enqueue it unless it is explicitly eligible.
3. Add an eligible local artifact to pipeline.md as: `- [ ] local:jds/{company}-{role-slug}.md | {company} | {title} | {location} | US | {region} | eligible | ats={ats}`

## Scan History

`data/scan-history.tsv` tracks ALL seen URLs. The final five columns are new;
readers must continue accepting the original six-column rows for backward
compatibility:

```
url	first_seen	portal	title	company	status	location	country_code	region	location_status	note
https://...	2026-02-10	Ashby — AI PM	PM AI	Acme	added	New York, NY	US	us_northeast	eligible	-
https://...	2026-02-10	Greenhouse — SA	Junior Dev	BigCo	skipped_title	Austin, TX	US	us_south	eligible	-
https://...	2026-02-10	Ashby — AI PM	SA AI	OldCo	skipped_location	Remote (EU only)	DE	europe	ineligible	not available in the US
```

## Structured Output — Required for Downstream Dispatch

Scan mode MUST write its ranked candidate list to a file, not just return it in prose. Downstream subagents (evaluators, applyers) must read URLs from this file, not from the scan subagent's return message. This prevents any hallucinated URL or ID from propagating.

**File location**: `batch/scan-output-{YYYY-MM-DD}.md`

**Format**: one markdown table per scan run, ordered by archetype-fit rank. The file contains only the balanced, US-eligible `selected` list:

| rank | company | ats | role | location | country_code | region | location_status | gh_slug | gh_id | url | updated_at |
|------|---------|-----|------|----------|--------------|--------|-----------------|---------|-------|-----|------------|
| 1 | Webflow | greenhouse | Lead AI Engineer | New York, NY | US | us_northeast | eligible | webflow | 7689676 | https://job-boards.greenhouse.io/webflow/jobs/7689676 | 2026-04-14 |
| 2 | EverAI | ashby | Senior AI PM | Remote - United States | US | us_nationwide | eligible | - | - | https://jobs.ashbyhq.com/everai/abc-123 | 2026-04-15 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |

**`ats` values** (one of): `greenhouse`, `ashby`, `workable`, `lever`, `workday`, `smartrecruiters`, `wwr`, `remoteok`, `builtin`, `custom`, `unknown`. Every row MUST populate this column — it's what the apply subagent uses to pick the correct Gmail OTP sender query. The `wwr` and `remoteok` values identify aggregator postings whose real underlying ATS is only known after the redirect is followed — downstream evaluators re-detect and may rewrite to the underlying ATS.

Every row MUST have:
- `ats` — the ATS platform hosting the posting. Inferred from the canonical URL host (e.g. `boards-api.greenhouse.io` / `job-boards.greenhouse.io` → `greenhouse`; `jobs.ashbyhq.com` → `ashby`; `jobs.lever.co` → `lever`; `*.myworkdayjobs.com` (any `wd1`/`wd3`/`wd5` pod) → `workday`; `apply.workable.com` / `jobs.workable.com` → `workable`; `api.smartrecruiters.com` / `jobs.smartrecruiters.com` → `smartrecruiters`; `weworkremotely.com` → `wwr`; `remoteok.com` → `remoteok`; `builtin.com/jobs/` → `builtin`; company-own domains → `custom`; anything indeterminate → `unknown`).
- `location`, `country_code`, `region`, and `location_status` — copied from posting-derived classification. Every selected row must have `country_code: US`, one of the five configured US region ids, and `location_status: eligible`.
- `url` in canonical form. For Greenhouse use `https://job-boards.greenhouse.io/{gh_slug}/jobs/{gh_id}` (matching the suffix in `data/pipeline.md`). For other ATSes use the platform's native URL (do not rewrite).
- `updated_at` in `YYYY-MM-DD` form (the most recent `updated_at` in the API response, or scan date when the source has no such field).

Additional columns — REQUIRED when available, `-` (dash) when not applicable:
- `gh_slug`, `gh_id` — Greenhouse-only. Copied verbatim from the Greenhouse API response (not reconstructed). For non-Greenhouse rows, emit `-` in both columns; `ats` + `url` are sufficient.

The scan subagent's return message MUST:
- Reference the file path (so orchestrators know where to read)
- Omit the ranked URL list from prose entirely (summary counts only)

**Rationale**: in a prior run, a scan subagent returned correct IDs in `scan-history.tsv` but hallucinated plausible-looking fake IDs in its prose-form top-30 list. The orchestrator trusted prose and dispatched 30 downstream subagents against fake URLs. File-based handoff prevents this class of error. Recording `ats` at scan time (rather than having the apply subagent infer it from the URL host) saves downstream re-parsing and keeps the OTP sender lookup deterministic.

## Output Summary

```
Portal Scan — {YYYY-MM-DD}
━━━━━━━━━━━━━━━━━━━━━━━━━━
Queries executed: N
Offers found: N total
Filtered by title: N relevant
US eligible: N | Non-US/ambiguous excluded: N
Duplicates: N (already evaluated or in pipeline)
New added to pipeline.md: N

US location coverage (eligible shortlist):
  Northeast N | Midwest N | South N | West N | Nationwide/US-remote N
Distinct companies: N | Remote N | Hybrid N | On-site N
Coverage gaps: {region or mix target + reason, or "none"}

NEXT STEP RECOMMENDATION:
- Structured candidate list written to: batch/scan-output-{YYYY-MM-DD}.md
- Downstream subagents MUST read URLs from that file, not from this return message
- Run /job-forge pipeline to evaluate the new offers.
```

## Verify Before Marking CLOSED (downstream rule)

**DO NOT mark a Greenhouse offer CLOSED based on a WebFetch/Geometra result alone.** Customer-skinned careers pages serve bot-hostile shells — a 403, a navbar-only response, or a client-side-only render — and WebFetch sees "no JD" and mis-classifies as CLOSED. Known customer-skinned hosts: `pinterestcareers.com`, `okta.com`, `samsara.com`, `zoominfo.com`, `collibra.com`, `careers.toasttab.com`, `careers.airbnb.com`, `coinbase.com`, `instacart.careers`. Treat any host that is NOT `greenhouse.io` / `job-boards.greenhouse.io` / `boards-api.greenhouse.io` as customer-skinned.

**Correct verification order for any Greenhouse-sourced URL** (identified by a `| gh={slug}/{id}` suffix in `pipeline.md` or a `boards-api.greenhouse.io` / `job-boards.greenhouse.io` / `boards.greenhouse.io` host):

1. Try `https://boards-api.greenhouse.io/v1/boards/{slug}/jobs/{id}`. This is the authoritative source.
   - **200 + JSON with `title` and `content`** → offer is LIVE. Use the JSON content as the JD. Do not mark CLOSED.
   - **404** → offer is genuinely closed. Mark CLOSED.
   - **Other non-2xx** → treat as transient (network/rate-limit); retry once. If still failing, mark `**Verification: unconfirmed**` and continue evaluation from whatever text is available. Do NOT mark CLOSED.
2. Only then fall back to WebFetch of the canonical `job-boards.greenhouse.io/{slug}/jobs/{id}` URL.
3. Only then fall back to Geometra on the same canonical URL.

**Rule:** Greenhouse postings with valid `gh_slug`/`gh_id` MUST be verified via the API first. A WebFetch failure on a customer-skinned domain is NOT evidence the role is closed.

## Update careers_url

Each company in `tracked_companies` MUST have a `careers_url` — the direct URL to its job listings page. The stored URL avoids searching for it every time.

**Known patterns by platform:**
- **Ashby:** `https://jobs.ashbyhq.com/{slug}`
- **Greenhouse:** `https://job-boards.greenhouse.io/{slug}` or `https://job-boards.eu.greenhouse.io/{slug}`
- **Lever:** `https://jobs.lever.co/{slug}`
- **Workday:** `https://{subdomain}.{pod}.myworkdayjobs.com/{site}` (pod = `wd1`/`wd3`/`wd5`/..., varies by tenant data center; site is tenant-defined, e.g. `External`, `NVIDIAExternalCareerSite`)
- **SmartRecruiters:** `https://careers.smartrecruiters.com/{company}` (human-facing) / `https://api.smartrecruiters.com/v1/companies/{company}/postings` (API)
- **Custom:** The company's own URL (e.g., `https://openai.com/careers`)

**If `careers_url` doesn't exist** for a company:
1. Try the pattern for its known platform
2. If that fails, do a quick WebSearch: `"{company}" careers jobs`
3. Navigate with Geometra (`geometra_connect` with `headless: true`, `browserMode: "stock"`, `blockDetection: true`, and `blockedSitePolicy: "manual-handoff"`) to confirm it works
4. **Save the found URL in portals.yml** for future scans

**If `careers_url` returns 404 or redirect:**
1. Note in the output summary
2. Try scan_query as fallback
3. Flag for manual update

## Update portals.yml

- **ALWAYS save `careers_url`** when adding a new company
- Add new queries as interesting portals or roles are discovered
- Disable queries with `enabled: false` if they generate too much noise
- Adjust filtering keywords as target roles evolve
- Keep at least `minimum_enabled_queries` enabled for every US location bucket
- Review US location coverage in every scan summary; do not silently replace an empty bucket with a non-US or ambiguous offer
- Add companies to `tracked_companies` when you want to follow them closely
- Verify `careers_url` periodically — companies change ATS platforms