# @se-studio/site-check

Validate SE marketing sites (sitemap.xml, sitemap-unindexed.xml, llms.txt) and download all markdown files into a local directory, preserving URL path structure.

## Usage

```bash
npx @se-studio/site-check <baseUrl> [-o dir] [-H "Name: value"] [--vercel-bypass [secret]] [--check-against <devUrl>] [--check-rewrites] [--ignore-missing-child-sitemaps]
```

### Examples

```bash
# Local site
npx @se-studio/site-check http://localhost:3015

# Custom output directory
npx @se-studio/site-check http://localhost:3015 -o ./out

# Compare production vs development: use production sitemap, check pages exist on dev
npx @se-studio/site-check https://example.com --check-against https://example.localhost

# Enable rewrite check (optional; off by default)
npx @se-studio/site-check https://example.com --check-rewrites

# Vercel Deployment Protection: use env secret (preferred name)
VERCEL_PROTECTION_BYPASS_TOKEN=your-secret npx @se-studio/site-check https://preview.vercel.app --vercel-bypass

# Legacy alias still works
VERCEL_AUTOMATION_BYPASS_SECRET=your-secret npx @se-studio/site-check https://preview.vercel.app --vercel-bypass

# Vercel bypass with explicit secret
npx @se-studio/site-check https://preview.vercel.app --vercel-bypass your-secret

# Custom headers
npx @se-studio/site-check https://example.com -H "Authorization: Bearer token" -H "X-Custom: value"

# Sitemap index lists a child that does not exist locally (404) — skip it and use remaining children
npx @se-studio/site-check http://localhost:3016 --ignore-missing-child-sitemaps -o ./out
```

## What it does

1. **Validates** the site by fetching:
   - `sitemap.xml` (required) — must return 200 and contain `<loc>` entries; if it is a sitemap index, child sitemaps are fetched and must return valid urlset content (or use `--ignore-missing-child-sitemaps` to skip child URLs that return **404**)
   - `sitemap-unindexed.xml` (optional) — if 404, a warning is printed and the run continues. When it exists (returns 200), its page URLs are included in all checks (existence on dev, optional rewrite check, markdown collection).
   - `llms.txt` (optional by default) — if 404, a warning is printed and the run continues; use `--strict-discovery` to require all four discovery endpoints (`llms.txt`, `markdown-index.txt`, `cms.txt`, `site-info.md`) with content validation

2. **Collects markdown URLs** from **sitemap.xml** and **sitemap-unindexed.xml** (Option B): when either URL returns a **sitemap index** (e.g. `<sitemapindex>` with child `<sitemap><loc>...</loc></sitemap>`), the tool follows those links and collects page URLs from each child sitemap’s urlset (one level only). When the response is a normal urlset, every `<loc>` is treated as a page URL. Each page URL is then converted to a `.md` URL (path + `.md`). This ensures all indexed and unindexed pages are checked.

3. **Checks for unexpected rewrites** (only when `--check-rewrites` is set): For each page URL from the sitemaps, the tool sends a HEAD request and reads the `x-nextjs-rewritten-path` response header. If the header is present and the rewritten path is different from the requested path (after normalisation), the run **fails** (exit 1). This catches cases where the sitemap lists one URL (e.g. `/learning-hub/blog/`) but the app rewrites to another (e.g. `/articles/blog`). Rewrites that only add or remove a locale prefix are allowed when `SITE_CHECK_LOCALES` is set (see [Options](#options)). By default the rewrite check is **disabled**.

4. **Checks and downloads**: For each markdown URL, the tool fetches it. If any return non-2xx, the run **fails** (exit 1) and reports which URLs are missing or errored. Otherwise it saves each response to the output directory, preserving path structure (e.g. `blog/foo.md` → `./markdown-export/blog/foo.md`, `es-US/about.md` → `./markdown-export/es-US/about.md`).

### Compare mode (`--check-against`)

When `--check-against <devUrl>` is set, the tool runs in **compare mode**:

- **Production** (the `<baseUrl>`) is the source of the URL list: sitemap.xml and sitemap-unindexed.xml are fetched from production. Validation (and llms.txt) applies to production.
- **Development** (`<devUrl>`) is the site checked: each production sitemap page URL (including from sitemap-unindexed.xml when present) is mapped to the same path on the development origin. The tool then checks that each of those development URLs returns 2xx (HEAD). If `--check-rewrites` is set, it also runs the rewrite check against the development site. Markdown collection and download are **not** performed in compare mode.

Use this to ensure a development or staging build has the same pages as production (e.g. before release). Page title comparison is not yet implemented; a future option may add it.

## Options

| Option | Description |
|--------|-------------|
| `baseUrl` | Base URL of the site (required). In compare mode this is the **production** URL (sitemap source). Trailing slash is stripped. |
| `-o`, `--out <dir>` | Output directory for markdown files (default: `./markdown-export`). Ignored in compare mode. |
| `-H`, `--header "Name: value"` | Add a request header (repeatable). Applied to both production and development requests in compare mode. |
| `--check-against <devUrl>` | Compare mode: use production sitemap(s) but check that each page exists on development (same path on `<devUrl>`). No markdown download. |
| `--check-rewrites` | Enable checking for unexpected rewrites (x-nextjs-rewritten-path). Default: off. When set, the run fails if any sitemap page URL rewrites to a different path (locale-only rewrites allowed with `SITE_CHECK_LOCALES`). |
| `--ignore-missing-child-sitemaps` | When the root sitemap (or `sitemap-unindexed.xml`) is a **sitemap index**, any child `<loc>` that returns **404** is skipped instead of failing validation. Other non-2xx responses still fail. After skipping, at least one page URL must still be collected from the remaining children. |
| `--vercel-bypass [secret]` | Set `x-vercel-protection-bypass` for [Vercel Deployment Protection](https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/protection-bypass-automation). If `secret` is omitted, uses `VERCEL_PROTECTION_BYPASS_TOKEN` from the environment (aliases: `VERCEL_AUTOMATION_BYPASS_SECRET`, `VERCEL_BYPASS_TOKEN`). |

Headers from `-H` override the Vercel bypass header if the same name is used.

### Environment

| Variable | Description |
|----------|-------------|
| `SITE_CHECK_LOCALES` | Comma-separated list of locale path segments (e.g. `en,en-gb,de`). When set, a rewrite is considered acceptable if the only difference between the requested path and the rewritten path is a leading locale segment. If unset, any rewrite to a different path fails. |
| `PREVIEW_SITE_URL` | Vercel develop/preview deployment base URL (preview smoke tests, protected preview fetches). Trailing slash is stripped. |
| `SITEMAP_PROD_URL` | Public production site base URL (sitemap-validate scripts, smoke case curation). Trailing slash is stripped. |
| `PRODUCTION_SITE_URL` | Production site base URL for audits and tooling that need a canonical prod origin. Trailing slash is stripped. |
| `VERCEL_PROTECTION_BYPASS_TOKEN` | Vercel Deployment Protection bypass secret (Vercel → Settings → Deployment Protection). Sent as `x-vercel-protection-bypass`. Aliases: `VERCEL_AUTOMATION_BYPASS_SECRET`, `VERCEL_BYPASS_TOKEN`. |

## Production HTML audit (`@se-studio/site-check/production-audit`)

Programmatic module for live production SEO audits (used by per-app `pnpm seo:audit:production` scripts). Import from `@se-studio/site-check/production-audit`:

- `runProductionSeoAudit`, `writeProductionSeoAuditOutputs`, `buildRichResultsSpotChecks`, `getProductionAuditExitCode`

**URL sources:** `sitemap.xml` and `sitemap-unindexed.xml` (same resolver as the CLI — follows sitemap indexes). `sitemap-unindexed.xml` 404 is optional.

**Markdown checks:** Every sitemap page URL must return **200** for its `.md` mirror (`/about/` → `/about.md`, `/` → `/index.md`). Excluded: `/download/` paths and static file extensions (`.pdf`, etc.) via `isNonPageAssetUrl`. Additional exclusions: `skipMarkdownPathPrefixes` in config.

**Discovery endpoints:** Production audit validates all four discovery routes (`llms.txt`, `markdown-index.txt`, `cms.txt`, `site-info.md`) with content checks; issues appear in `discoveryIssues` on the audit report.

**markdown-index.txt:** Fetched for a supplementary `missing_from_markdown_index` report; not used to gate `.md` existence checks.

**Exit code:** `getProductionAuditExitCode` returns `1` when any row has `severity: 'error'` (including `markdown_missing`).

## Smoke tests (`@se-studio/site-check/smoke-test`)

Curated local smoke tests for marketing sites: verify HTML and `.md` for URLs listed in **`smoke.cases.json`**, plus optional **discovery endpoint** checks (`llms.txt`, `markdown-index.txt`, `cms.txt`, `site-info.md`) via the `discovery` block (see skill `se-marketing-sites-smoke-test-setup`).

Optional **`cmsIntegrity`** in `smoke.cases.json` adds a **local-only** Contentful article-link check (parallel with `pnpm smoke-test` server boot). Deployment / live smoke remains HTTP-only.

Requires **^2.6.1** when using `cmsIntegrity` or `pnpm smoke-test` with integrity enabled.

Import from `@se-studio/site-check/smoke-test`:

- `loadStaticSmokeConfig`, `runStaticSmokeTest`, `runStaticSmokeTestWithIntegrity`, `runPreviewStaticSmokeTest`, `runSmokeTest`
- `formatSmokeTestReport`, `formatCombinedSmokeReport`, `getSmokeTestExitCode`, `getCombinedSmokeExitCode`

**Per-app usage** (see `apps/example-empty`):

```bash
pnpm build --filter @se-studio/site-check   # once, or after package changes
cd apps/example-empty
pnpm smoke-test:run                         # dev server must be running on app port
pnpm smoke-test                             # dev server + smoke (default dev:dev) + stop
pnpm smoke-test:preview                     # curated cases against Vercel preview (no local server)
```

Add `smoke.cases.json`, `scripts/smoke-test-run.ts`, and optionally `scripts/smoke-test-preview.ts` (or use `node ../../scripts/smoke-test-preview.mjs` from the app directory). No Contentful or `cms-server` imports required. Set `SMOKE_TEST_IGNORE=true` in `.env.local` to skip the full `smoke-test` CI script.

**Preview smoke** (`runPreviewStaticSmokeTest`): reads `PREVIEW_SITE_URL` and `VERCEL_PROTECTION_BYPASS_TOKEN` from `.env.local` (or `process.env`), applies the bypass header, and runs the same curated cases against the live preview deployment. No local dev server required.

**`smoke-test-one` env:**

| Variable | Purpose |
|----------|---------|
| `SMOKE_TEST_SERVER_SCRIPT=dev:dev` | Default dev server; use `start:dev` for prod-like `next start` (runs `pnpm build` first; not portless) |
| `SMOKE_TEST_SKIP_BUILD=true` | Skip `pnpm build` when `SERVER_SCRIPT=start:dev` (use after Vercel `next build`) |
| `SMOKE_TEST_VERIFY_CACHE=true` | Run smoke twice; assert `x-nextjs-cache` on HTML (meaningful with `start:dev`, soft in `next dev`) |
| `SMOKE_TEST_AUDIT_CACHE_LOGS=true` | Pipe server/build logs and scan for 2MB cache failures, `cache skip`, and `hard refresh` |
| `SMOKE_TEST_CACHE_SKIP_SEVERITY=warn` | `warn` or `error` for cache-skip log lines (default `warn`) |
| `SMOKE_TEST_READY_PATH=/` | Path polled until the server responds |
| `SMOKE_TEST_USE_RUNNING_SERVER=false` | Force a fresh server (default `true` for local dev) |
| `SMOKE_TEST_SKIP_INTEGRITY=true` | HTTP smoke only (set by `smoke-test-one` when `cmsIntegrity` runs in parallel) |

`LOG_CMS_FETCH=1` is set when verify-cache or audit-cache-logs is enabled. The smoke HTTP client never sends `Cache-Control: no-cache`.

### CMS article-link integrity (`cmsIntegrity` in `smoke.cases.json`)

```json
{
  "cmsIntegrity": {
    "enabled": true,
    "routing": {
      "articleTypesBasePath": "/resources",
      "tagsBasePath": "/topics",
      "peopleBasePath": "/team",
      "defaultTopic": "other",
      "enablePrimaryTagPartOfSlug": true,
      "topiclessArticleTypeSlugs": ["ebooks"]
    }
  }
}
```

Map `routing` from the app’s `constants.ts` / URL rules. Use `runStaticSmokeTestWithIntegrity` in `scripts/smoke-test-run.ts` (see `apps/example-empty`).

Import from `@se-studio/site-check/cms-integrity` for standalone checks: `runArticleLinkIntegrityFromSmokeConfig`, `evaluateArticleLinkIntegrity`, `getArticleLinkIntegrityExitCode`.

### Vercel build cache (`VERCEL_FORCE_NO_BUILD_CACHE`)

Not a smoke variable — set on the **Vercel project** when `next build` fails intermittently with webpack `WasmHash` errors (often with stale remote build cache; local `pnpm build` may still pass).

```bash
vercel env add VERCEL_FORCE_NO_BUILD_CACHE preview develop --value 1 --yes
vercel env add VERCEL_FORCE_NO_BUILD_CACHE production --value 1 --yes
```

Cannot be set in `vercel.json` — Vercel restores cache before install/build commands. Tradeoff: slower cold builds, more reliable compiles.

**Deploy check** (`smoke-test-deploy-check`): post-build gate for Vercel — starts `pnpm start` without rebuilding, runs `pnpm smoke-test:run`, exits non-zero on failure. Port defaults from `smoke.cases.json`. Set `SMOKE_TEST_IGNORE=true` in Vercel env to bypass in an emergency.

```json
{
  "buildCommand": "pnpm build && pnpm smoke-test:deploy-check"
}
```

Per-app script: `"smoke-test:deploy-check": "smoke-test-deploy-check"` (or `node ../../scripts/smoke-test-deploy-check.mjs` in the monorepo).

**Cross-site develop timing** (`develop-sites-timing`): probes all client marketing sites (smoke paths from each repo’s `smoke.cases.json`) against develop and production. Runs **N sequential full passes** (default 5) over every path — one request at a time. Reads `VERCEL_PROTECTION_BYPASS_TOKEN` from each site’s `.env.local`; never hardcodes secrets.

```bash
pnpm develop-sites-timing
pnpm develop-sites-timing --site pointme --passes 3 --output /tmp/timing.json
```

| Option | Purpose |
|--------|---------|
| `--site <key>` | One site: `se2026`, `om1`, `brightline`, `brightlifekids`, `pedestal`, `headwater`, `pointme` |
| `--passes <n>` | Full passes over all paths (default `5`) |
| `--output <path>` | JSON report (default `./develop-sites-timing.json`) |
| `--delay-ms <n>` | Pause between requests |
| `--develop-only` / `--production-only` | Probe one environment |
| `--source-root <path>` | Client repos root (default `~/source`, or `SE_SOURCE_ROOT`) |

**Live deployment smoke** (`smoke-test-live` / `runLiveSmokeTest`): HTTP checks against a deployed URL — use with Vercel `repository_dispatch` (`vercel.deployment.ready`) and [Deployment Checks](https://vercel.com/docs/deployment-checks) to hold production domain aliasing until smoke passes.

**Discovery URL origins on production deploys:** Production builds often emit canonical URLs in `llms.txt` / `markdown-index.txt` (e.g. `https://www.example.com/...`) while Deployment Checks smoke the pre-alias `*.vercel.app` hostname. By default `discovery.urlOrigin` is `auto`: when every discovery URL shares one consistent HTTPS origin, **content validation** accepts that canonical origin. **Key pages spot-checks** then fetch the same paths on the smoke **deployment** base URL (not live CDN/www, which may still be the previous alias or an incomplete edge). Set `urlOrigin: "deployment"` for strict matching of URLs in the body, or `canonicalBaseUrl` to pin the expected content origin explicitly.

| Variable | Purpose |
|----------|---------|
| `DEPLOYMENT_URL` | Live deployment base URL (e.g. `github.event.client_payload.url`) |
| `SMOKE_TEST_BASE_URL` | Alias for `DEPLOYMENT_URL` |
| `PREVIEW_SITE_URL` | Fallback for local runs (same as preview smoke) |

Per-app script: `"smoke-test:live": "smoke-test-live"`. GitHub Actions: set `DEPLOYMENT_URL` from the dispatch payload, report status with `vercel/repository-dispatch/actions/status@v1`, register the check name in Vercel Deployment Checks settings.

## Exit codes

- `0` — Validation passed (sitemap.xml required; llms.txt and sitemap-unindexed.xml optional). If `--check-rewrites` was set, no unexpected rewrites. Every sitemap page returned 200 for its `.md` URL and files were saved (or in compare mode: all production sitemap pages exist on development and, if `--check-rewrites`, no unexpected rewrites).
- `1` — Usage error (missing baseUrl), validation failed (sitemap.xml), one or more sitemap pages rewrite to a different path when `--check-rewrites` was set, one or more markdown URLs returned non-2xx (pages missing or broken), or in compare mode one or more development URLs did not return 2xx. Validation does not fail when llms.txt returns 404.