{ "schemaVersion": 1, "sensitiveFieldsExcluded": true, "generatedAt": "1970-01-01T00:00:00.000Z", "packageName": "@happyvertical/smrt-jobs", "packageVersion": "0.43.5", "sourceManifestPath": "dist/manifest.json", "agentDocPath": "AGENTS.md", "sourceHashes": { "manifest": "5c035ac62991cbb5a2228ce5bbffd677fcbb53cdfae4286127f6f0c9b1a6a4f7", "packageJson": "682e176927de0880e3eef3f1096d27e4ffe1ea901b798c1253cb7ddd37125679", "agents": "f9b10be5ad5a560f3ce4c47646eb546918bb9c775f8d0986d29756c9bca6e234" }, "exports": [ ".", "./manifest", "./manifest.json", "./playground", "./runner", "./svelte", "./ui", "./worker-liveness-thread" ], "dependencies": { "@happyvertical/jobs": "catalog:", "@happyvertical/logger": "catalog:", "@happyvertical/smrt-config": "workspace:*", "@happyvertical/smrt-core": "workspace:*", "@happyvertical/smrt-tenancy": "workspace:*", "@happyvertical/smrt-types": "workspace:*", "@happyvertical/smrt-ui": "workspace:*", "@happyvertical/sql": "catalog:", "@happyvertical/utils": "catalog:", "@happyvertical/smrt-vitest": "workspace:*", "@sveltejs/package": "^2.5.8", "@sveltejs/vite-plugin-svelte": "^7.1.2", "@types/node": "24.13.2", "svelte": "^5.56.4", "svelte-check": "^4.7.1", "typescript": "5.9.3", "vite": "8.1.4", "vitest": "4.1.10" }, "smrtDependencies": [ "@happyvertical/smrt-config", "@happyvertical/smrt-core", "@happyvertical/smrt-tenancy", "@happyvertical/smrt-types", "@happyvertical/smrt-ui", "@happyvertical/smrt-vitest" ], "sdkDependencies": [ "@happyvertical/jobs", "@happyvertical/logger", "@happyvertical/sql", "@happyvertical/utils" ], "tags": [], "risks": [], "objects": [ { "name": "ForgeDelivery", "qualifiedName": "@happyvertical/smrt-jobs:ForgeDelivery", "collection": "forgedeliveries", "tableName": "_smrt_forge_deliveries", "packageName": "@happyvertical/smrt-jobs", "extends": "SmrtObject", "fields": [ { "name": "tenantId", "type": "text", "required": true, "columnType": "UUID" }, { "name": "provider", "type": "text", "required": true, "columnType": "TEXT" }, { "name": "deliveryId", "type": "text", "required": true, "columnType": "TEXT" }, { "name": "installationKey", "type": "text", "required": false, "columnType": "TEXT" }, { "name": "repositoryKey", "type": "text", "required": false, "columnType": "TEXT" }, { "name": "eventName", "type": "text", "required": true, "columnType": "TEXT" }, { "name": "payload", "type": "json", "required": false, "columnType": "JSON" }, { "name": "status", "type": "text", "required": true, "columnType": "TEXT", "default": "pending" }, { "name": "attempts", "type": "integer", "required": true, "columnType": "INTEGER", "default": 0 }, { "name": "maxAttempts", "type": "integer", "required": true, "columnType": "INTEGER", "default": 5 }, { "name": "receivedAt", "type": "datetime", "required": true, "columnType": "TIMESTAMP" }, { "name": "nextAttemptAt", "type": "datetime", "required": true, "columnType": "TIMESTAMP" }, { "name": "leaseOwner", "type": "text", "required": false, "columnType": "TEXT" }, { "name": "leaseToken", "type": "text", "required": false, "columnType": "TEXT" }, { "name": "leaseExpiresAt", "type": "datetime", "required": false, "columnType": "TIMESTAMP" }, { "name": "lastError", "type": "text", "required": false, "columnType": "TEXT" }, { "name": "completedAt", "type": "datetime", "required": false, "columnType": "TIMESTAMP" }, { "name": "replayCount", "type": "integer", "required": true, "columnType": "INTEGER", "default": 0 } ], "relationships": [], "methods": [], "tenant": { "scoped": true, "mode": "required", "field": "tenantId" }, "conflictColumns": [ "tenant_id", "provider", "delivery_id" ], "surfaces": [], "relationshipFeatures": [ "uuidColumns" ], "tags": [], "risks": [] }, { "name": "ForgeProjectionCheckpoint", "qualifiedName": "@happyvertical/smrt-jobs:ForgeProjectionCheckpoint", "collection": "forgeprojectioncheckpoints", "tableName": "_smrt_forge_projection_checkpoints", "packageName": "@happyvertical/smrt-jobs", "extends": "SmrtObject", "fields": [ { "name": "tenantId", "type": "text", "required": true, "columnType": "UUID" }, { "name": "projection", "type": "text", "required": true, "columnType": "TEXT" }, { "name": "subjectKey", "type": "text", "required": true, "columnType": "TEXT" }, { "name": "observationVersion", "type": "integer", "required": true, "columnType": "INTEGER" }, { "name": "deliveryId", "type": "text", "required": true, "columnType": "TEXT" }, { "name": "observedAt", "type": "datetime", "required": true, "columnType": "TIMESTAMP" } ], "relationships": [], "methods": [], "tenant": { "scoped": true, "mode": "required", "field": "tenantId" }, "conflictColumns": [ "tenant_id", "projection", "subject_key" ], "surfaces": [], "relationshipFeatures": [ "uuidColumns" ], "tags": [], "risks": [] }, { "name": "ForgeDeliveryCollection", "qualifiedName": "@happyvertical/smrt-jobs:ForgeDeliveryCollection", "collection": "forgedeliveries", "tableName": "_smrt_forge_deliveries", "packageName": "@happyvertical/smrt-jobs", "extends": "SmrtCollection", "fields": [], "relationships": [], "methods": [ "accept", "claimReady", "replay" ], "methodSignatures": [ { "name": "accept", "async": true, "params": [ "input: AcceptForgeDeliveryInput", "audit?: ForgeProjectionAuditHook" ], "returns": "Promise" }, { "name": "claimReady", "async": true, "params": [ "options: ClaimForgeDeliveryOptions", "audit?: ForgeProjectionAuditHook" ], "returns": "Promise" }, { "name": "replay", "async": true, "params": [ "id: string", "audit?: ForgeProjectionAuditHook" ], "returns": "Promise" } ], "surfaces": [], "relationshipFeatures": [ "uuidColumns" ], "tags": [], "risks": [] }, { "name": "SmrtJobEvent", "qualifiedName": "@happyvertical/smrt-jobs:SmrtJobEvent", "collection": "smrtjobevents", "tableName": "_smrt_job_events", "packageName": "@happyvertical/smrt-jobs", "extends": "SmrtObject", "fields": [ { "name": "tenantId", "type": "text", "required": false, "columnType": "UUID" }, { "name": "jobId", "type": "foreignKey", "required": true, "related": "SmrtJob", "columnType": "UUID" }, { "name": "type", "type": "text", "required": true, "columnType": "TEXT", "default": "log" }, { "name": "level", "type": "text", "required": true, "columnType": "TEXT", "default": "info" }, { "name": "stage", "type": "text", "required": false, "columnType": "TEXT" }, { "name": "progress", "type": "integer", "required": false, "columnType": "INTEGER" }, { "name": "message", "type": "text", "required": true, "columnType": "TEXT", "default": "" }, { "name": "data", "type": "json", "required": false, "columnType": "JSON" }, { "name": "createdAt", "type": "datetime", "required": true, "columnType": "TIMESTAMP" } ], "relationships": [ { "name": "jobId", "type": "foreignKey", "required": true, "related": "SmrtJob", "columnType": "UUID" } ], "methods": [ "toCursor" ], "methodSignatures": [ { "name": "toCursor", "returns": "string" } ], "tenant": { "scoped": true, "mode": "optional", "field": "tenantId" }, "conflictColumns": [ "tenant_id", "slug", "context" ], "surfaces": [ { "kind": "cli", "name": "smrtjobevent_list", "operation": "list", "objectName": "@happyvertical/smrt-jobs:SmrtJobEvent" }, { "kind": "cli", "name": "smrtjobevent_get", "operation": "get", "objectName": "@happyvertical/smrt-jobs:SmrtJobEvent" } ], "relationshipFeatures": [ "foreignKey", "uuidColumns" ], "tags": [], "risks": [] }, { "name": "SmrtJobEventCollection", "qualifiedName": "@happyvertical/smrt-jobs:SmrtJobEventCollection", "collection": "smrtjobevents", "tableName": "_smrt_job_events", "packageName": "@happyvertical/smrt-jobs", "extends": "SmrtCollection", "fields": [], "relationships": [], "methods": [ "append", "cleanup", "initialize", "latestProgressByJobIds", "listByJob", "listSinceCursor" ], "methodSignatures": [ { "name": "append", "async": true, "params": [ "input: SmrtJobEventData" ], "returns": "Promise" }, { "name": "cleanup", "async": true, "params": [ "options: object" ], "returns": "Promise" }, { "name": "initialize", "async": true, "returns": "Promise" }, { "name": "latestProgressByJobIds", "async": true, "params": [ "jobIds: string[]", "options?: object" ], "returns": "Promise>" }, { "name": "listByJob", "async": true, "params": [ "jobId: string", "options?: ListJobEventsOptions" ], "returns": "Promise" }, { "name": "listSinceCursor", "async": true, "params": [ "options?: any" ], "returns": "Promise" } ], "surfaces": [], "relationshipFeatures": [ "uuidColumns" ], "tags": [], "risks": [] }, { "name": "SmrtJob", "qualifiedName": "@happyvertical/smrt-jobs:SmrtJob", "collection": "smrtjobs", "tableName": "_smrt_jobs", "packageName": "@happyvertical/smrt-jobs", "extends": "SmrtObject", "fields": [ { "name": "tenantId", "type": "text", "required": false, "columnType": "UUID" }, { "name": "queue", "type": "text", "required": true, "columnType": "TEXT", "default": "default" }, { "name": "objectType", "type": "text", "required": true, "columnType": "TEXT" }, { "name": "objectId", "type": "text", "required": false, "columnType": "TEXT" }, { "name": "method", "type": "text", "required": true, "columnType": "TEXT" }, { "name": "args", "type": "json", "required": false, "columnType": "JSON" }, { "name": "runAt", "type": "datetime", "required": true, "columnType": "TIMESTAMP" }, { "name": "priority", "type": "integer", "required": true, "columnType": "INTEGER", "default": 50 }, { "name": "status", "type": "text", "required": true, "columnType": "TEXT", "default": "pending" }, { "name": "attempts", "type": "integer", "required": true, "columnType": "INTEGER", "default": 0 }, { "name": "maxAttempts", "type": "integer", "required": true, "columnType": "INTEGER", "default": 3 }, { "name": "timeout", "type": "integer", "required": true, "columnType": "INTEGER", "default": 300000 }, { "name": "timeoutBehavior", "type": "text", "required": true, "columnType": "TEXT", "default": "fail" }, { "name": "startedAt", "type": "datetime", "required": false, "columnType": "TIMESTAMP" }, { "name": "completedAt", "type": "datetime", "required": false, "columnType": "TIMESTAMP" }, { "name": "lastError", "type": "text", "required": false, "columnType": "TEXT" }, { "name": "resultPointer", "type": "text", "required": false, "columnType": "TEXT" }, { "name": "taskId", "type": "text", "required": false, "columnType": "TEXT" }, { "name": "taskOwnerId", "type": "text", "required": false, "columnType": "TEXT" }, { "name": "taskResult", "type": "json", "required": false, "columnType": "JSON" }, { "name": "taskInputRequests", "type": "json", "required": false, "columnType": "JSON" }, { "name": "taskInputResponses", "type": "json", "required": false, "columnType": "JSON" }, { "name": "retryStrategy", "type": "json", "required": false, "columnType": "JSON" }, { "name": "workerId", "type": "text", "required": false, "columnType": "TEXT" }, { "name": "workerHeartbeat", "type": "datetime", "required": false, "columnType": "TIMESTAMP" } ], "relationships": [], "methods": [ "cancel", "getDescription", "retry", "save" ], "methodSignatures": [ { "name": "cancel", "async": true, "returns": "Promise" }, { "name": "getDescription", "returns": "string" }, { "name": "retry", "async": true, "returns": "Promise" }, { "name": "save", "async": true, "returns": "Promise" } ], "tenant": { "scoped": true, "mode": "optional", "field": "tenantId" }, "conflictColumns": [ "tenant_id", "slug", "context" ], "surfaces": [ { "kind": "cli", "name": "smrtjob_list", "operation": "list", "objectName": "@happyvertical/smrt-jobs:SmrtJob" }, { "kind": "cli", "name": "smrtjob_get", "operation": "get", "objectName": "@happyvertical/smrt-jobs:SmrtJob" }, { "kind": "cli", "name": "smrtjob_retry", "operation": "retry", "objectName": "@happyvertical/smrt-jobs:SmrtJob" }, { "kind": "cli", "name": "smrtjob_cancel", "operation": "cancel", "objectName": "@happyvertical/smrt-jobs:SmrtJob" } ], "relationshipFeatures": [ "uuidColumns" ], "tags": [], "risks": [] }, { "name": "SmrtJobCollection", "qualifiedName": "@happyvertical/smrt-jobs:SmrtJobCollection", "collection": "smrtjobs", "tableName": "_smrt_jobs", "packageName": "@happyvertical/smrt-jobs", "extends": "SmrtCollection", "fields": [], "relationships": [], "methods": [ "claimReady", "cleanup", "countInFlightForTenant", "enqueueJob", "initialize", "listByStatus", "listReady", "stats" ], "methodSignatures": [ { "name": "claimReady", "async": true, "params": [ "options: ClaimReadyOptions" ], "returns": "Promise" }, { "name": "cleanup", "async": true, "params": [ "options: object" ], "returns": "Promise" }, { "name": "countInFlightForTenant", "async": true, "params": [ "tenantId: string | null" ], "returns": "Promise" }, { "name": "enqueueJob", "async": true, "params": [ "data: SmrtJobData", "options?: EnqueueJobOptions" ], "returns": "Promise" }, { "name": "initialize", "async": true, "returns": "Promise" }, { "name": "listByStatus", "async": true, "params": [ "status: JobStatus | JobStatus[]", "options?: object" ], "returns": "Promise" }, { "name": "listReady", "async": true, "params": [ "options?: object" ], "returns": "Promise" }, { "name": "stats", "async": true, "params": [ "queue?: string" ], "returns": "Promise" } ], "surfaces": [], "relationshipFeatures": [ "uuidColumns" ], "tags": [], "risks": [] }, { "name": "SmrtWorker", "qualifiedName": "@happyvertical/smrt-jobs:SmrtWorker", "collection": "smrtworkers", "tableName": "_smrt_workers", "packageName": "@happyvertical/smrt-jobs", "extends": "SmrtObject", "fields": [ { "name": "workerId", "type": "text", "required": true, "columnType": "TEXT" }, { "name": "pid", "type": "integer", "required": false, "columnType": "INTEGER" }, { "name": "hostname", "type": "text", "required": false, "columnType": "TEXT" }, { "name": "startedAt", "type": "datetime", "required": false, "columnType": "TIMESTAMP" }, { "name": "heartbeatAt", "type": "datetime", "required": false, "columnType": "TIMESTAMP" }, { "name": "leaseExpiresAt", "type": "datetime", "required": false, "columnType": "TIMESTAMP" }, { "name": "status", "type": "text", "required": true, "columnType": "TEXT", "default": "running" } ], "relationships": [], "methods": [], "conflictColumns": [ "worker_id" ], "surfaces": [], "relationshipFeatures": [ "uuidColumns" ], "tags": [], "risks": [] }, { "name": "SmrtWorkerCollection", "qualifiedName": "@happyvertical/smrt-jobs:SmrtWorkerCollection", "collection": "smrtworkers", "tableName": "_smrt_workers", "packageName": "@happyvertical/smrt-jobs", "extends": "SmrtCollection", "fields": [], "relationships": [], "methods": [ "assertReady", "expireWorker", "freshLeaseWorkerKeys", "pruneExpired", "registerWorker", "renewLease", "tableReady" ], "methodSignatures": [ { "name": "assertReady", "async": true, "returns": "Promise" }, { "name": "expireWorker", "async": true, "params": [ "workerKey: string" ], "returns": "Promise" }, { "name": "freshLeaseWorkerKeys", "async": true, "returns": "Promise>" }, { "name": "pruneExpired", "async": true, "params": [ "graceMs: number" ], "returns": "Promise" }, { "name": "registerWorker", "async": true, "params": [ "input: RegisterWorkerInput" ], "returns": "Promise" }, { "name": "renewLease", "async": true, "params": [ "workerKey: string", "leaseTtlMs: number" ], "returns": "Promise" }, { "name": "tableReady", "async": true, "returns": "Promise" } ], "surfaces": [], "relationshipFeatures": [ "uuidColumns" ], "tags": [], "risks": [] } ], "surfaces": [ { "kind": "cli", "name": "smrtjobevent_list", "operation": "list", "objectName": "@happyvertical/smrt-jobs:SmrtJobEvent" }, { "kind": "cli", "name": "smrtjobevent_get", "operation": "get", "objectName": "@happyvertical/smrt-jobs:SmrtJobEvent" }, { "kind": "cli", "name": "smrtjob_list", "operation": "list", "objectName": "@happyvertical/smrt-jobs:SmrtJob" }, { "kind": "cli", "name": "smrtjob_get", "operation": "get", "objectName": "@happyvertical/smrt-jobs:SmrtJob" }, { "kind": "cli", "name": "smrtjob_retry", "operation": "retry", "objectName": "@happyvertical/smrt-jobs:SmrtJob" }, { "kind": "cli", "name": "smrtjob_cancel", "operation": "cancel", "objectName": "@happyvertical/smrt-jobs:SmrtJob" } ], "prompts": [], "relationshipsV2": { "foreignKeyFields": 1, "crossPackageRefFields": 0, "junctionCollections": 0, "hierarchicalObjects": 0, "polymorphicAssociations": 0, "uuidColumns": 14 }, "agentDoc": "# @happyvertical/smrt-jobs\n\nBackground job execution with persistent queue, scheduling, and fluent builder API.\n\n## Architecture\n\n```\nSmrtObject.bg('method') → SmrtJob (in _smrt_jobs) → TaskRunner picks up → executes via ObjectRegistry\nAgentSchedule (cron) → ScheduleRunner creates SmrtJob → TaskRunner executes → ScheduleRunner updates\nTaskRunner.start() → SmrtWorker lease (in _smrt_workers) → recovery keys on worker liveness, not heartbeat\n```\n\n## SmrtJob\n\nPersistent in `_smrt_jobs`. Fields: `queue` (default), `objectType`, `objectId`, `method`, `args`, `runAt`, `priority` (higher=sooner), `status`, `attempts`/`maxAttempts`, `timeout` (default 5min), `retryStrategy`, `workerId` (the owning runner's incarnation key), `workerHeartbeat` (telemetry only — no longer gates recovery).\n\nStatus: `pending → running → completed/failed/cancelled`.\n\n## TaskRunner\n\nPolling-based execution engine. Config: `concurrency` (5), `pollInterval` (1s), `heartbeatInterval` (30s, telemetry only), `leaseTtlMs` (30s), `leaseTickMs` (10s), `shutdownTimeout` (30s).\n\n1. `start()` calls `assertReady()` (fail fast if `_smrt_workers` unmigrated), registers a seeded `SmrtWorker` lease, and adds its worker key to the process-global live set — all **before** polling\n2. Polls `claimReady()` to atomically claim pending jobs (`runAt <= NOW`, ordered by `priority DESC, runAt ASC, created_at ASC, id ASC`)\n3. Claim sets `status='running'`, `workerId=`, heartbeat/start timestamps, and increments `attempts`\n4. Resolves class via `ObjectRegistry.getClass(objectType)`. Object-bound jobs\n pass `objectId` to the constructor and let `initialize()` perform the single\n canonical hydration. Runner-owned `db`, `id`, and hydration options cannot be\n overridden by agent config; missing persisted targets fail before method\n dispatch. Jobs without `objectId` construct a static-like instance\n5. **Internal args**: `_agentConfig` and `_scheduleId` stripped from args before calling method\n6. Calls the requested method on the initialized instance\n7. Terminal/retry writes are **conditional** (`WHERE worker_id=? AND status='running'`) so a recovered row is never stomped\n8. Retry: uses strategy from `@happyvertical/jobs`, schedules future `runAt` on failure\n9. Events: `job:started`, `job:completed`, `job:failed`, `job:retrying`, `runner:started/stopped`\n\nA rejection from `processJob` can never escape the poll loop: the caller attaches `.catch(e => emit('runner:error', e))` and the error path (`handleJobError`) is itself try/caught, so a failure-path write that rejects is surfaced as a `runner:error` event instead of crashing the worker with an unhandled rejection.\n\n## Timeouts & at-least-once\n\n**Execution is at-least-once, never exactly-once. Make job handlers idempotent.**\n\nA job `timeout` only races the handler's promise — JavaScript cannot preempt an already-running function. When a handler exceeds its timeout:\n\n- The job is **failed terminally and NOT auto-retried** (a timed-out handler is still running; re-queueing it would let a second worker run a concurrent duplicate). This narrows, but does not eliminate, the overlap window — the orphaned handler keeps running until it returns on its own.\n- The orphaned handler's eventual terminal write is dropped by the ownership guard (`WHERE worker_id=? AND status='running'`), so it cannot resurrect the failed row — but any **side effects** it performs (external API calls, writes to other tables) still happen.\n- Key any non-idempotent work by `context.job.jobId` or a caller-supplied idempotency key.\n\n`timeoutBehavior` (persisted + shown in the UI) is now honored:\n\n- **`'fail'`** (default): on timeout the job fails (and, per above, is not retried).\n- **`'warn'`**: the handler is **not** raced against the timeout — it runs to completion, and at the deadline the runner logs a warning and emits a `timeout-warning` job event. A slow-but-successful handler still completes successfully.\n- **`'kill'`**: treated identically to `'fail'`. In-process JavaScript has no thread interruption, so a true \"kill\" of a running handler is impossible without worker isolation; this value is honest that it only fails the job row, it does not stop the handler. Prefer `'fail'` unless you specifically want the label.\n\n## Worker liveness & recovery (#1474)\n\nRecovery keys on **worker-process liveness**, never per-job heartbeat freshness (a CPU-bound synchronous handler used to starve the heartbeat and false-recover its own running jobs).\n\n- **`SmrtWorker` / `_smrt_workers`**: one lease row per runner *incarnation* (`workerId` is per-incarnation unique via `createWorkerKey`, so a restart never looks like it still owns the previous crash's jobs). `leaseExpiresAt` is a `datetime` (an integer epoch-ms column overflows `int4`/`INT32` on Postgres/DuckDB). Stage 1 writes/compares it against the host clock — the same approach the old heartbeat recovery used, so it's no more skew-sensitive than the code it replaced.\n- **Process-global live set** (`worker-liveness.ts`, `globalThis.__smrtLiveWorkers`): checked synchronously, so it can't be starved by a blocked loop. Covers all same-process topologies.\n- **Off-loop ticker** (`worker-liveness-ticker.ts` + `worker-liveness-thread.ts`): for engines a second connection can reach (Postgres, file-backed SQLite — `offLoopEligible()`), `start()` spawns a `node:worker_threads` ticker that renews the lease on its own thread, so a CPU-bound synchronous handler on the main loop can't starve it. In-memory SQLite / DuckDB, a thread-spawn failure, a start-handshake timeout, or the thread dying mid-run all fall back to main-loop renewal; the in-process live set keeps same-process correct regardless. The worker entry is a separate build entry resolved via `import.meta.resolve('@happyvertical/smrt-jobs/worker-liveness-thread')`.\n- **Recovery rule** (both runners): a `running` job is orphaned iff its worker is *not alive* = not in the live set **and** no fresh `_smrt_workers` lease. The live set takes precedence over a stale lease. TaskRunner also never recovers a job in its own `activeJobs`. If `_smrt_workers` is absent, recovery skips lease checks (never mass-recovers). Recovery is swept at most once per lease tick, and terminal/recovery writes use `RETURNING id` (not `rowCount`, which DuckDB/JSON adapters always report as ≥1).\n- **Lease clock**: the lease is compared against the host clock (same as the old heartbeat; fine with NTP + a 30s TTL). A dead process stops renewing and the lease expires within its TTL — that is how recovery detects death. (Instant cross-process detection via Postgres session advisory locks was prototyped on `@happyvertical/sql`'s `acquireSession()` but deferred — treating a free lock as proof-of-death false-recovers any worker legitimately in main-loop fallback mode.)\n\n## ScheduleRunner\n\nPolls `_smrt_agent_schedules` every 60s for due entries. Creates SmrtJob with `queue='agents'`, `priority=75`. Wires to TaskRunner events for completion/failure tracking. Slot reconciliation keys on worker liveness (it has no in-process active-job set, so the lease/live-set is its whole mechanism).\n\nThe job is enqueued **before** `next_run` is advanced and `running_count` is incremented: a transient enqueue failure (tenant-cap hit, DB blip) therefore leaves `next_run` and `status='active'` untouched so the same due slot is retried on the next poll, rather than losing the slot and disabling the schedule.\n\n`next_run` is always recomputed from *now*, never from the previous `next_run` — this is **fire-once-forward**: runs that came due while the runner was down are not caught up, only the next future occurrence fires. There is no missed-run backfill.\n\nCustom cron parser: 5-field (minute hour dom month dow). `*`, ranges, lists, steps supported. **Not timezone-aware**: cron fields are matched against the **server's local time** (the parser uses `getHours`/`getDate`/`getDay`/… local accessors), so `0 0 * * *` fires at local midnight on the host, not at 00:00 UTC. Deploy runners in a known timezone (e.g. `TZ=UTC`) for UTC semantics. A per-schedule timezone option is a possible future enhancement.\n\n## JobBuilder — Fluent API\n\n```typescript\nconst handle = await doc.background('analyze', { detailed: true })\n .delay('5m').priority('high').retries(5).queue('analysis').timeout(600000).enqueue();\n\nawait handle.wait({ timeout: 60000, pollInterval: 100 }); // polling-based\n```\n\n## Durable forge projections\n\n`ForgeDeliveryCollection` owns the provider-neutral delivery inbox in\n`_smrt_forge_deliveries`; `ForgeProjectionRuntime` owns lease/retry/dead-letter\ntransitions and monotonic checkpoints in\n`_smrt_forge_projection_checkpoints`.\n\n- Inbox identity is `(tenant_id, provider, delivery_id)`.\n- Tenant-facing accept/replay requires ambient tenant context. Worker claim is\n cross-tenant, then runtime restores the captured context before observation.\n- Projection callbacks must write through `ForgeProjectionContext.db`; the\n application projection, checkpoint, and inbox completion share one\n transaction.\n- Observation identity is tracker-independent\n `(projection, subjectKey, version)`. Equal or older versions are acknowledged\n without reapplying.\n- Lease-token conditions guard every terminal/failure write. Expired final\n attempts become replayable `dead_letter` rows, and persisted errors are\n redacted.\n- Generated API, CLI, and MCP surfaces stay disabled; replay is an explicit\n tenant-scoped operator action.\n\n`bg()` is shorthand: `await doc.bg('analyze', args)` → enqueues immediately, returns JobHandle.\n\n## withBackgroundJobs(Class)\n\nMixin that adds `bg()` and `background()` to any SmrtObject. Uses WeakMap for collection caching per DB instance.\n\n## Background policy (`background-policy.ts`, S5 audit #1402)\n\nThree opt-in guards, all owned **and enforced here**. Other packages apply the\neligibility marker (`reports`, `support`, `fields`, the MCP conformance\nfixture), but nothing outside this package acts on it — `TaskRunner` is the only\nreader. In particular `@happyvertical/smrt-agents` has no reference to any of\nthese guards and does not depend on this package, so marking a method does not\nchange what the agents runtime will dispatch.\n\n- `clampRetries()` / `MAX_JOB_RETRIES` (25): a requested retry count above the\n ceiling is clamped, not rejected.\n- `assertWithinTenantCreationCap(tenantId, current, cap)` /\n `DEFAULT_TENANT_JOB_CAP` (10 000): throws `TenantJobCapExceededError` when the\n tenant's count of non-terminal jobs is already at or above the cap. Applied in\n `SmrtJobCollection.enqueueJob()` (`smrt-job.ts`), which is the single choke\n point for the builder and `ScheduleRunner`; a `null` tenant and a `cap <= 0`\n both skip the check. Deliberately not serialized — concurrent enqueues can\n overshoot by the number of in-flight creators (see the method's own comment).\n- `@backgroundEligible()` / `markBackgroundEligible()` /\n `isBackgroundEligibleMethod()`: an allowlist of methods `TaskRunner` may\n dispatch, enforced at exactly one call site (`runner.ts`, after the method\n lookup). **Restrictive, not enabling** — a class with no marked methods allows\n any of its methods, and the first mark makes the set exhaustive, excluding\n every sibling method. Adding the decorator to one method of an existing class\n is therefore a behaviour change for the rest of it.\n\n## Retention (#2375)\n\n`_smrt_jobs` and `_smrt_job_events` are append-only in practice — `cleanup()`\nexisted but nothing ever called it, and job events (one row per log line and\nprogress tick) had no prune path at all. `src/retention.ts` contributes both to\nthe framework retention sweep in `@happyvertical/smrt-core`.\n\n- `TaskRunner` starts a sweeper by default. `retention: false` opts out;\n `retention: { intervalMs, policy, jobs }` tunes it. The **first sweep is one\n interval after `start()`, never at start** — a crash-looping worker must not\n become a delete loop, and a short-lived runner must exit having deleted\n nothing. `stop()` only clears the timer — it does **not** unregister the job\n tasks. The package entry point (`index.ts`) registers them unconditionally\n on import, so \"this process loaded `@happyvertical/smrt-jobs`\" is the\n contract that contributes them, not \"a sweeper happens to be running\";\n `unregisterJobRetentionTasks()` is the explicit opt-out for callers that\n want a clean registry (tests, teardown).\n- Windows (`DEFAULT_JOB_RETENTION`): completed/cancelled 7 days, failed 30\n days, events 30 days, 10 000 job rows per sweep. Events deliberately outlive\n the jobs they describe, so a job row removed at 7 days still has a readable\n log for another three weeks. `SmrtJobEvent.jobId` is therefore the explicit\n same-package archival exception: `@foreignKey('SmrtJob', { constraint:\n false })` retains runtime relationship metadata and its index without a\n physical database constraint that would block parent pruning.\n- `cleanup()` counts before deleting (`rowCount` is unreliable across engines)\n and honours `dryRun`, which is what makes `smrt db:prune --dry-run` an exact\n preview. Its `(status, completed_at)` predicate is indexed by\n `ensureJobsSystemTableCompatibility()`, which this collection runs on every\n `initialize()` — bootstrap cannot do it, because `_smrt_jobs` does not exist\n yet when bootstrap runs.\n- `cleanup({})` with no cutoff returns 0. It must never be read as \"delete\n everything\".\n\n## Gotchas\n\n- **Cron not timezone-aware**: cron fields match the server's **local** time, not UTC (set `TZ` for UTC); no missed-run catch-up (fire-once-forward)\n- **At-least-once execution**: a timeout cannot preempt a running handler; timed-out jobs fail without retry but the handler keeps running — make handlers idempotent (see \"Timeouts & at-least-once\")\n- **No dead letter queue**: failed jobs stay in DB with `status='failed'` — manual intervention\n- **Forge deliveries are not SmrtJobs**: they use their own replayable\n dead-letter inbox and projection checkpoint contract. Do not enqueue provider\n deliveries as generic background jobs.\n- **Result storage**: `resultPointer` is just a string — app must implement result backend\n- **Lazy builder**: `background()` returns builder — nothing happens until `enqueue()`\n- **wait() is polling**: JobHandle.wait() polls DB every 100ms (configurable)\n- **Migrate before start()**: `TaskRunner.start()` throws if `_smrt_workers` is missing — run `smrt db:migrate` after upgrading. Tables are never created at runtime.\n- **Recovery is liveness-based**: don't reintroduce heartbeat-threshold recovery; a blocked event loop must not look dead (see Worker liveness section, #1474)\n" }