# Cockpit Recovery Runtimes and agents fail independently: a container or VM can be perfectly healthy while its agent's registration with the executor has dropped (idle transport drops, executor restarts, soft-locks). Cockpit's recovery surface is built around that distinction. ## Stale agents: `agent unreachable` When a runtime is **running** but its agent is not registered (or its session backends report unavailable), Inventory and Sessions keep the row visible with the health state **`agent unreachable`** instead of hiding it — and expose **Reconnect**. Reconnect is rendered for running rows of runtime kind `docker`, `container`, `vm`, `qemu`, or `kvm` whose agent is missing. Host targets are deliberately excluded from the button (see [Host runtimes](#host-runtimes) below). A Reconnect never creates a replacement instance and never destroys the running runtime — it only attempts to restore the missing agent registration. Agentic Sandbox v2026.8.3 changes the managed Linux-container identity boundary to credential-free UDS control with a unique control UID and workload UID `10001`. Containers created by older releases must be reported as requiring recreation; reconnecting or restarting one does not establish the new boundary. Cockpit preserves the executor's legacy/recreation-required posture rather than silently labeling an existing container secure-default. See the [v2026.8.3 qualification](./qualifications/agentic-sandbox-v2026.8.3.md). ## What the Bridge tries, in order `POST /api/instances/:id/reconnect` walks executor-owned recovery first, then local-development fallbacks only when policy allows them: 1. **Executor-owned reconnect** — the sandbox's own reconnect endpoints (v2 admin first, then legacy candidates). If the executor handles it, done. 2. **Docker/container fallback** — disabled by default. Set `AIWG_COCKPIT_LOCAL_DOCKER_FALLBACK=1` for local development to allow `docker exec agent-reconnect`, which SIGHUPs the in-container agent so it re-registers in place without restarting the container. Requires sandbox images that ship the `agent-reconnect` helper (agentic-sandbox v2026.7.5+); on failure the response says to repull/rebuild the image. Without the opt-in, Cockpit returns 409 `local_docker_fallback_disabled`. 3. **VM fallback** (new in 2026.7, roctinam/aiwg#1778) — for `vm`/`qemu`/`kvm` instances the Bridge delivers the same SIGHUP **through the libvirt guest-agent channel**: ```text virsh qemu-agent-command guest-exec → pkill -HUP -x agent-client ``` The sandbox VM images bake qemu-guest-agent for exactly this kind of in-guest exec, and the agent handles SIGHUP as reconnect-in-place on every runtime, so no VM image change is needed. The libvirt domain name is the instance's launch name. This fallback is automatic on Linux. On non-Linux hosts, set `AIWG_COCKPIT_LOCAL_LIBVIRT_FALLBACK=1` for explicit local development; otherwise Cockpit returns 409 `local_libvirt_fallback_disabled`. Requirements: the Bridge host needs `virsh` access to the domain, and the guest-agent channel must be up — a 502 names whichever is missing. If none of the paths apply, the 409 response spells out the manual host-side command (`pkill -HUP -x agent-client`). ## What survives a reconnect **Session survival depends on the agent version inside the runtime, not on Cockpit:** - **agentic-sandbox 2026.7.8+ agents**: transport reconnect is state-preserving — the agent no longer kills tracked workloads on stream loss, all session types (managed multiplexer *and* direct pty / headless tasks) survive, and output produced while disconnected buffers and flushes after re-register. Server-side reconcile is the sole kill authority. - **Older agents** (pre-2026.7.8): only sessions backed by a **detached multiplexer** (managed tmux) survive and are re-adopted; the agent SIGTERMs everything else as part of its reconnect cycle. This was the root cause chain fixed upstream in agentic-sandbox#633 (VM idle drop) and agentic-sandbox#634 (kill-on-reconnect). A VM keeps its baked agent binary until the image is rebuilt or the VM is reprovisioned — so a running fleet can mix both behaviors. If sessions vanish after a reconnect, check the agent version in that runtime before suspecting Cockpit. On older agents, prefer **managed** session backends for long-running work. Cockpit does not require a page refresh after a transient Bridge or executor drop. The global status changes to **Reconnecting…**, retains last-known counts with an explicit stale-state tooltip, retries the Bridge/executor path with bounded backoff, and pulses every mounted live-data view when both REST and the SSE event stream recover. Inventory continues its own polling, Sessions uses bounded retry, and terminal WebSockets reconnect independently. After the status returns to **Bridge live**, attach from Sessions as usual. ## Host runtimes Host targets don't get a Reconnect button — the agent runs directly on the Bridge host, so the recovery lever is the host-runtime daemon itself. Start Cockpit with the daemon when working host targets: ```bash AIWG_COCKPIT_START_HOST_DAEMON=1 npm run cockpit:up ``` The Inventory host-daemon column shows the daemon's status and the documented operator command when action is needed; Cockpit never starts or installs the daemon silently. Manual agent recovery on a host is `pkill -HUP -x agent-client`. ## Destroy `DELETE /api/instances/:id` is defensive about executor/runtime state skew: - Tries the executor's v2 and legacy destroy surfaces. - For Docker rows, local `docker rm -f ` reconciliation is disabled by default and enabled only by `AIWG_COCKPIT_LOCAL_DOCKER_FALLBACK=1`. With the opt-in, Cockpit can reconcile after admin success or when the admin surface reports instance-not-found; without it, those cases return 409 `local_docker_fallback_disabled` and preserve the executor boundary. - An already-removed target reports success with an `already_gone` marker instead of failing the operator's intent. - A stopped Docker row can use local Docker removal only under the same explicit fallback opt-in; otherwise Cockpit reports the disabled fallback instead of crossing the executor boundary. ## Audit Every recovery action lands in the local audit log: `instance.reconnect.requested`, `instance.destroy.requested`, with targets and results (secrets redacted). See [Trust & Security → Audit log](./trust-and-security.md#audit-log). ## See also - [Sessions](./sessions.md) — backends and what "managed" buys you - upstream: roctinam/agentic-sandbox#633 (VM idle-drop root cause), roctinam/agentic-sandbox#634 (session survival across reconnect) ## Managed-Docker identity upgrades An existing container without executor-reported control/workload identity evidence must be recreated. Reconnect and restart do not retrofit mounts, peer-credential mappings, UID separation, or cleared capability boundaries. Destroy the old managed container through the executor, then launch a new one. If a Docker startup profile is rejected for raw credential references, use the sandbox credential proxy or choose a VM runtime; Cockpit will not silently downgrade transport or materialize the credential in the container.