# ADR 0003: Reliability and operability hardening

## Status

Proposed — follow-up to ADR 0002 after T1–T10 implementation.

## Context

ADR 0002 removes destructive drain/buffer behavior and makes receive semantics safer, but moves complexity into lifecycle ownership, shared state, locks, crash recovery, and operational diagnosis. Real-Pi validation also depends on external model-provider availability, which can obscure bridge failures with provider `429` errors.

Known accepted residuals:

- kill-9 can orphan a watch child until its bounded recovery path runs;
- persist-before-inject has a small at-least-once/skip window;
- resume/fork can race on first message;
- state writes do not promise fsync-level power-loss durability;
- real-Pi dogfood depends on provider quota.

## Decision

Harden reliability and operability in separate, measurable slices. Do not change notify-pull semantics or add another receive buffer.

### 1. Lifecycle invariants

Keep owner transitions behind `claim`, `release`, `startWatch`, and `stopWatch`. Assert these invariants in tests and diagnostics:

- at most one owner per `${root}:${me}`;
- no watcher runs without ownership;
- secondary sessions cannot mutate or inject;
- owner release is conditional on full identity;
- `activeId` is either null or points to an actionable pending message.

### 2. Shared lock/state helper

Use one bounded lock helper for ownership and migration. It must provide:

- structured retry and wait diagnostics;
- stale PID/identity takeover;
- atomic state replacement;
- cleanup of sibling temporary files;
- bounded failure rather than indefinite wait.

No new lock implementation may be added outside this helper.

### 3. Startup reconciliation

On owner startup and takeover, reconcile stale local state:

- remove or supersede stale ownership markers;
- validate `activeId` against current envelopes;
- preserve valid `processedIds` while dropping malformed entries;
- report orphaned or unrecoverable artifacts through status diagnostics.

Reconciliation must be idempotent and must not inject bodies.

### 4. Deterministic validation layers

Separate bridge correctness from model-provider availability:

- component tests use fake AMQ/transport implementations;
- process tests use real Node child processes and isolated roots;
- Pi tool-contract smokes use deterministic fake-agent or fixed provider mode;
- real-Pi/provider dogfood remains opt-in and reports provider failures separately from bridge assertions.

A provider `429` must never be reported as an AMQ transport failure.

### 5. Diagnostics and metrics

Extend status/diagnostic output with:

- owner or secondary mode;
- root, self, peer, and primary target;
- watcher state and restart/backoff count;
- pending count and `activeId`;
- last watch/lock/migration error;
- stale-owner takeovers and lock wait duration.

Use structured fields where possible. Avoid logging message bodies by default.

### 6. Performance discipline

Optimize only after correctness evidence:

- minimize shared-state writes to meaningful transitions;
- retain atomic rename and bounded lock scope;
- preserve processed-ID cap and dedupe semantics;
- measure watcher restart churn, lock contention, and state-write frequency before changing behavior.

## Non-goals

- No replacement of maildir read/unread truth with another inbox database.
- No implicit reply/send target selection from handshake discovery.
- No unbounded retry loop.
- No fsync guarantee unless explicit durability requirements justify its cost.
- No requirement that CI call an external LLM provider.

## Rollout

1. Add invariant and reconciliation tests.
2. Consolidate lock diagnostics and failure reporting.
3. Add deterministic Pi/tool-contract smoke mode.
4. Add `/amq-bridge diagnose` or equivalent status detail.
5. Measure contention and watcher churn.
6. Apply performance changes only where measurements show need.

Each slice gets independent review and remains backward-compatible with sidecar/echo-peer transports.

## Acceptance criteria

- Concurrent owner tests demonstrate one owner and zero duplicate injectors.
- Kill/restart tests recover stale ownership without duplicate injection.
- Malformed state/lock artifacts recover with bounded, observable behavior.
- CI runs bridge and process validation without external provider access.
- Provider failures are classified separately in real-Pi reports.
- Status exposes enough data to diagnose root, identity, owner mode, watcher, pending, and active-message problems.
- No regression in ADR 0002 tests, sidecar compatibility, or explicit-ID tool contracts.

## Trade-offs

This adds diagnostics and test infrastructure without changing core receive semantics. It increases implementation surface temporarily, but reduces lifecycle debugging cost and prevents provider availability from masking transport regressions. Performance work is deliberately deferred until measurements justify complexity.

## Defect evidence policy

Every bug fix must include a durable bug-log entry, a regression test that reproduces the failure, a fix commit referencing the evidence, and validation output. Real-peer or user-discovered defects additionally require a postmortem covering impact, root cause, detection gap, corrective action, and residual risk. A fix is not complete when code passes existing tests alone.