# Startup and notification semantics (0.6.3)

## Submit first, explain later

Minimal changes + necessary validation → pin code snapshot and atomic Run record → return queued receipt → worker gates → computation. Long STATE.yaml/hypothesis prose, presentation reports and optional remote tracking are not submission prerequisites. Batch independent checks; reuse context and fork protocols. Required CPU validation belongs in a dependency Run when appropriate, not in an LLM wakeup chain.

Keep owner/session, commit, argv/config including seed, checkpoint identity, explicit GPU IDs/resource policy, log/event and termination policy fixed before submission. PRL records argv and checksums of referenced config/code; the experiment must actually use the declared seed. No snapshot/identity or safety check is skipped for speed.

Optional PRL MLflow initialization starts after successful process spawn and cannot hold a GPU lease at finalization. Required MLflow initialization runs in the worker before computation; failure produces a failed Run and critical exit notification rather than an ambiguous launch exception. Application-owned W&B/MLflow calls are outside PRL's startup guarantee. Result Markdown is best effort; the local Run record and log are authoritative.

## Timings are not interchangeable

Inspect `performance` on demand:

| Field | Meaning |
|---|---|
| launch_prepare_ms | Submission preparation through minimum record preparation, including Git, snapshot, command/config checksums and storage preflight |
| git_checkpoint_ms, snapshot_ms | Subsets of submission preparation |
| worker_boot_ms | Record creation to worker entry |
| dependency_wait_ms | Time sleeping while the parent is not ready (the parent's CPU tests are not PRL compute overhead) |
| gpu_wait_ms | GPU acquisition probes plus busy retry sleeps |
| checkpoint_validation_ms | Final stability, checksum and registration validation after GPU acquisition |
| checksum_ms, checkpoint_hash_count | Checkpoint content hash time/count, subset of validation |
| snapshot_validation_ms | Final executable snapshot HEAD and referenced-file checksums |
| process_spawn_ms | Child spawn request to successful Node spawn event |
| created_to_process_start_ms | Run creation to successful process spawn, inclusive of gates |
| first_update_ms | Process spawn to first explicitly declared update/progress match; absent if unobserved |
| tracking_init_ms | Required pre-compute or optional post-spawn tracking initialization |

These include nested/overlapping measurements; do not sum them all. They do not measure Agent time before `runLaunch`, trainer internals, or all system scheduler costs. Use session timestamps to evaluate Agent pre-submission overhead. `queued` is a receipt, `started_at` means successful process spawn, and first update requires an explicit signal:

```yaml
- id: first_update
  type: log.regex
  pattern: 'global_step=1(?:\s|$)'
  milestone: first_update
  max_matches: 1
  actions: [record, wake_agent]
```

A progress watchdog's first matching line can also establish first-update timing; choose a pattern that really means an update.

## Checkpoints and GPU retries

No checkpoint content reads occur while retrying a busy GPU. After acquisition, mutable paths are content-validated once, followed by the existing GPU safety recheck before computation. Invalid checksums or queued cancellation release the lease and prevent launch. Tests instrument actual `fs.createReadStream` calls, not just reported counters.

Existing artifact registrations point at ordinary mutable files. A `durable_complete` label is not an immutability guarantee; these paths are rehashed against the registered checksum, even if size/mtime are unchanged. This release does not introduce an immutable object store or bypass checks on alleged immutable paths. External storage must keep checkpoint bytes immutable after validation; unmanaged writers and unmanaged GPU jobs remain outside PRL's cooperative guarantees.

Tools/CLI accept `mode: shared`, `min_free_memory_mib`, `memory_safety_mib` (CLI `--mode`, `--min-free-memory-mib`, `--memory-safety-mib`). Both GPU probes require the minimum plus safety margin. Shared mode only permits unmanaged foreign GPU processes; PRL leases remain mutually exclusive. Legacy `allow_foreign_processes` and `min_free_memory_mb` remain supported.

## Notification delivery and ACK

Each notification has a stable ID. Lifecycle: matched → queued → sending → delivered → acknowledged. `sending` is persisted before calling Pi; `delivered` means transport accepted the send, not that the Agent has inspected it. Transport errors retry with backoff. A per-session process lock serializes drains; critical OOM/exit notices have unlimited retries independent of ordinary quotas.

Busy Pi sessions retain notifications in PRL's durable queue. At idle/settled, notifications for one Run are batched into one Agent message. PRL does not enqueue a fresh copy every retry interval in Pi's non-cancellable follow-up queue. The input handler rechecks ACK, ownership and supersession just before delivery, filtering duplicate IDs. Session-persisted user messages restore the seen-ID set after reload. Each ID should cause at most one normal visible delivery/Agent action in a live or resumed session.

Inspection acknowledges notifications belonging to events present in the inspected Run snapshot, including queued or in-flight notices; a newly arriving event is not silently ACKed. A separate atomic ACK tombstone is authoritative even if a late writer rewrites old YAML state. Notification delivery/ACK do not rewrite the worker's lifecycle/events record, avoiding lost OOM/exit updates.

PRL cannot provide transactional exactly-once execution across an arbitrary host crash between input acceptance, message persistence and Agent side effects. Transport delivery remains at-least-once in that crash window; receipts and ACKs make normal retries/reloads idempotent. Keep follow-up actions idempotent too. Pre-upgrade messages without notification IDs cannot be perfectly matched retroactively. No active worker or installed training package is hot-patched by this release.

## Regression commands

`npm test` builds source and runs CPU-only fixtures, mocked `nvidia-smi`, a mocked MLflow HTTP server and Pi event handlers. No training GPUs or production Runs are used. `npm run release:check` additionally reviews the package manifest. This is functional regression coverage, not a claim of a measured speedup on a production training Run.