# Task 1704 — data portal standing audit

**Date:** 2026-07-20 · **Lane:** cloudflare + storage-broker · data-portal observability
**Depends on:** 1689 (portal, landed), 1694 (broker R2 object ops, landed), 1810 (loop registry, landed)

## The problem

The `[data-portal]` lifeline traces one upload end to end and ends in a re-read post-condition, so
every failure that emits an event is already covered. Three conditions emit nothing:

- **An orphan object** — an object in R2 with no manifest row. Stored, invisible to ingestion
  forever, and the upload that produced it returned `200`.
- **A phantom row** — a manifest row whose object is absent. Ingestion will pull a missing file,
  and nothing says so until ingestion runs, which may be never.
- **A stalled ingestion sweep** — an ingestion that never runs logs nothing, so a climbing backlog
  and an empty one look identical.

Only a standing reconcile reveals any of them. Absence of the audit's line is itself the signal that
all three are unobservable.

## Premise corrections carried from the task file

The task brief predates three changes. All three point the same way; none change its intent.

1. **The `[storage-audit]` idiom is now `registerLoop`.** Task 1810 moved every standing loop onto
   `platform/ui/server/loop-registry.ts`, and `scripts/check-loops-registered.mjs` fails the build on
   a bare `setInterval`. "Copy the `[storage-audit]` shape" therefore means registering a loop, not
   hand-rolling two unref'd timers.
2. **Every line number in the brief is stale**, though every cited thing still exists.
   `runStorageAudit` is `storage-broker.ts:773`; the "silent when the config names no portal" idiom
   is the `hasBookingSite` check in the availability loop; `r2ObjectList`'s drain-to-exhaustion
   comment at `cf-exec.ts:443` names this audit as its own reason for existing.
3. **`reconcilePages` (Task 1728) is a closer template than the brief knew.** It is already a pure
   orphan/phantom set difference over two lists in `lib/storage-broker/src/audit.ts`, carrying the
   concurrency contract and the refusal to print `orphan=<everything>` off a failed read. This audit
   is that shape one level down: objects against rows instead of projects against records.

## Components

### 1. `reconcileDataPortal` — pure, in `lib/storage-broker/src/audit.ts`

Third reconcile beside `reconcileStorage` and `reconcilePages`. It is D1 rows against R2 objects,
which is that lib's domain, and the ui server imports the lib by source path so there is no dist
rebuild in the loop.

```ts
interface DataPortalReconcile {
  objects: number
  rows: number
  orphanObjects: string[]   // object keys with no manifest row
  phantomRows: string[]     // manifest objectKeys with no object
  oldestUningestedHrs: number | null   // null renders as `na`
}

function reconcileDataPortal(
  objects: Array<{ key: string }>,
  rows: Array<{ objectKey: string; ingested: number; uploadedAt: string }>,
  nowMs: number,
): DataPortalReconcile
```

`nowMs` is injected so the age arithmetic is deterministic under test rather than clock-dependent.

`oldestUningestedHrs` is computed from rows with `ingested === 0` only, and is `null` when there are
none. **An `uploadedAt` that will not parse is excluded from the minimum rather than allowed to
produce `NaN`** — that is the `snapshotAgeMs=NaN` lesson from the availability loop, where a truthy
but non-ISO value slipped past a `??` guard and logged `NaN`. If exclusion leaves no usable row, the
result is `null`.

### 2. `parseManifestRows` — pure, exported from the route module

`cf.d1Query` shells `npx wrangler d1 execute --json` and is typed `unknown`. This function parses
that envelope into the row shape above and **throws on anything it does not recognise**.

Throwing rather than returning `[]` is the load-bearing decision. Zero rows against a populated
bucket reads as "every object is an orphan" — a fabricated defect count off a failed read, which is
precisely what `reconcilePages` refuses to print. A throw lands in the d1-degraded branch, where the
line says the read failed instead of asserting a number it does not have.

### 3. `runDataPortalAudit(root)` — the caller, in `ui/server/routes/storage-broker.ts`

Sits with `runStorageAudit` and `runPagesAudit` at the foot of the same file.

**Discovery.** Walks `data/accounts/*/data-portal.json`. A config that is absent, unparseable, or
parses but names no `portalDbName` and `bucketName` is **silent** — the `hasBookingSite` idiom. An
account with no portal is not a fault and must not produce a line.

**It returns before minting a house token when no account has a portal.** Most installs have no data
portal and no house Cloudflare credential; minting unconditionally would make this loop report an
error forever on every one of them.

**Reads.** One `makeHouseCfExec(root)` for the whole run, then per account:

- rows — `d1Query(portalDbName, 'SELECT objectKey, ingested, uploadedAt FROM manifest')`
- objects — `r2ObjectList(bucketName)`

The two halves are caught **separately**, and the line is emitted unconditionally afterwards. This is
`.tasks/backlog/1351` made structural: the log is not downstream of either read, so neither read can
skip it. An audit that a data-source failure silences is the same blind spot one layer up, in a task
that exists because failures emit nothing.

**Degradation.** Each line carries exactly what is still measurable:

| Failure | Line still carries |
|---|---|
| R2 read fails | `rows=N oldestUningestedHrs=N`, remainder `na`, `degraded=r2` |
| D1 read fails | `objects=N`, remainder `na`, `degraded=d1` |
| Both fail | all `na`, `degraded=d1+r2` |

`oldestUningestedHrs` surviving an R2 failure is not incidental: the backlog age is computable from
the manifest alone, and it is the one of the three conditions that a half-broken audit can still
report.

**Line format.** Counts only:

```
[data-portal-audit] account=<id> objects=N rows=N orphanObjects=N phantomRows=N oldestUningestedHrs=N
```

Healthy is `orphanObjects=0 phantomRows=0` with a bounded age. **Any non-zero orphan or phantom count
is a defect, not noise.**

`reconcilePages` reports orphan and phantom *names*, on the grounds that a count says something is
wrong and a name says what to fix. This audit deliberately does not, and the pure function returning
the keys does not change that: the keys here are `<ownerId>/<filename>`, real people's filenames,
and the journal is not where they belong. The task's success criterion specifies counts, and the
audit owning its own ownership discipline is called out in the brief as load-bearing precisely
because the in-process path holds the account-wide credential and bypasses `objectGate` entirely.

### 4. Arming, in `ui/server/index.ts`

```ts
registerLoop({
  name: 'data-portal-audit',
  intervalMs: 300_000,
  firstRunDelayMs: 50_000,
  run: runDataPortalAuditSafe,
})
```

Same interval as the storage and pages audits. The first run is offset past storage (20s), pages
(35s) and google-account (45s) so a boot does not fire four token mints in the same instant.

**Degradation is not an error.** A degraded run returns a note (`portals=N degraded=M`) and the
registry records `ok`, because the audit did the job it could and said so. A genuine throw — no
house credential where a portal exists, an unreachable registry — is logged and **rethrown**, the
`runStorageAuditSafe` shape, so /activity shows a fresh Last run against a stale Last OK.

## Testing

**Unit, `lib/storage-broker/src/__tests__/audit.test.ts`** — an object with no row counts one
`orphanObjects`; a row with no object counts one `phantomRows`; a matched pair counts neither;
`oldestUningestedHrs` comes from the oldest `ingested=0` row; `ingested=1` rows are excluded from it;
`na` when there are no uningested rows; an unparseable `uploadedAt` does not yield `NaN`.

**Envelope, same run** — `parseManifestRows` on a well-formed wrangler payload, and a throw on a
malformed one. The throw case is what stops a parse failure reading as an empty manifest.

**Route, new `ui/server/__tests__/data-portal-audit.test.ts`** — modelled on
`storage-broker-object-routes.test.ts`: `// @vitest-environment node`, `vi.mock` the neo4j store,
partially mock the lib via `importOriginal`, stub `makeHouseCfExec` with a fake carrying
`r2ObjectList` and `d1Query`. Cases: two accounts, one with a portal, yields one line; a D1 throw
still emits a line carrying `degraded=d1`; an R2 throw emits `degraded=r2` with `rows` still counted;
a malformed D1 payload degrades rather than reporting zero rows; a config naming no portal is silent;
no portal anywhere mints no token.

**Mutation check** — reverting the reconcile to compare in one direction only must fail the opposite
direction's test.

## Scope boundaries

**In scope:** the reconcile and its tests, the caller, the arming, the route-level tests, and
`.docs/data-portal.md` § Observability, which currently states the audit is not built.

**Out of scope:** the device-side ingestion sweep (unbuilt, agent-side); any change to the portal
template or the gate (1689, landed); escalation beyond a log line — `.tasks/backlog/1351` records
that a non-zero count escalating nowhere is its own problem, and that is lane-wide, not this task's.