---
name: Sensorium Self-Repair
triggers:
  - debug sensorium
  - fix sensorium
  - repair sensorium
  - sensorium crashing
  - sensorium is crashing
  - sensorium down
  - sensorium broken
  - mcp crashing
  - mcp is dead
  - supervisor loop
  - restart loop
  - emergency repair
  - sensorium не працює
  - сенсоріум крешить
replaces_orchestrator: true
---

# Sensorium Self-Repair Skill

You are repairing **Sensorium itself** (this MCP server + its .NET supervisor) — a live production system the operator depends on. Move deliberately: **establish ground truth from logs and processes before changing anything**, distinguish real crashes from false-death loops, fix the root cause, validate, then ship. Do NOT guess. Do NOT restart blindly.

## Golden rules (read first)

- **Root cause over symptoms.** A restart "fixes" nothing if the bug recurs. Find the exact file:line and the commit that introduced it.
- **Verify file contents via terminal, never trust cached reads.** The `read_file` tool can serve a STALE copy (it served an old `package.json` version during a real incident). For anything you will edit or reason about critically, confirm with `git show HEAD:<path>` or `Get-Content <path>`.
- **Never trust a subagent's described diff.** Subagents have hallucinated multi-hundred-line diffs for files that were clean. Always confirm with `git diff` / `git status` in a terminal yourself.
- **Phantom working-tree changes = stale VS Code editor buffers.** Files silently reverting to OLD versions on disk (a committed fix undone, `package.json` version rolled back) is caused by VS Code `files.autoSave` + `files.hotExit` writing a stale editor buffer to disk — NOT an agent, NOT you editing manually. It's the same mechanism that "clobbers" AI-edited files mid-session. Confirm: `git status`/`git diff` shows a stale revert; check `%APPDATA%\Code\Backups\**` for hot-exit backups whose mtime is *after* your last save (⇒ an unreloaded window is holding a stale buffer). Fix permanently: set `"files.autoSave": "off"` + `"files.hotExit": "off"` in User settings, then `Revert File` on the stale tabs and reload the window. Until the window reloads, the risk persists for it.
- **Watch for concurrent autonomous agents.** A Sensorium worker may be committing to this repo *while you work*. Re-check `git HEAD`, branch, and `git log` before and after edits; a fix you were about to write may already be committed. (But a spontaneous working-tree revert is usually the VS-Code-buffer issue above, not an agent — verify file mtimes: an agent write shows a recent mtime, a buffer flush may not.)
- **Git safety:** stage explicit paths only. NEVER `git add -A/./-u`, `git commit -a`. NEVER commit `tmp_*`, dumps, screenshots, or binaries.
- **Deploy ≠ push.** Pushing to git does not update the running system (see "Ship & deploy").

## Key facts & locations

- **Repo:** the workspace root (e.g. `c:\src\remote-copilot-mcp`). **Runtime data dir:** `~/.remote-copilot-mcp` (`$HOME`/`%USERPROFILE%`).
- **Logs** under `~/.remote-copilot-mcp/logs/`:
  - `mcp/server.log` — MCP app log (rotated ~5 MB).
  - `mcp/mcp-stderr-YYYYMMDD.log` — captured MCP stderr (fatal errors, startup banners, `[wait]`/`[telegram]` activity).
  - `supervisor/supervisor-YYYYMMDD.log` — supervisor restart/health decisions (very noisy with Telegram `getUpdates`; filter it out). **Timezone caveat:** MCP logs (`server.log`, `mcp-stderr`) are **UTC** (`...Z`); the supervisor log was historically **LOCAL**, but a `UtcTimestampFormatter` change can switch it to UTC — always cross-reference one shared event (e.g. "MCP server is ready" vs the node startup banner) to confirm which, before correlating timestamps.
- **Process model:** the .NET supervisor process `sensorium-supervisor` supervises a Node MCP server (**HTTP port 3847**) + watcher (**3848**). MCP start command is either `node dist/index.js` (local build) or `npx -y sensorium-mcp@latest` (published). Workers are `claude`/`codex`/`copilot` processes whose command line references `.remote-copilot-mcp` — **a process check that queries only `node`/`sensorium-supervisor` will MISS every worker and falsely report "no threads running." Always include `claude,codex,copilot`.**
- **`~/.remote-copilot-mcp/server.pid`** holds `{"pid":N}` — should match the port-3847 owner. A mismatch is the signature of a false-death loop.
- **Windows: `git` is often NOT on PATH.** Locate it (`%LOCALAPPDATA%\Programs\Git\cmd\git.exe` or `C:\Program Files\Git\cmd\git.exe`) and call it by full path.
- **Prior root causes are logged in memory:** read `/memories/repo/critical-fixes-log.md`, `/memories/repo/keeper-stuck-detection-bug.md`, `/memories/repo/sensorium-architecture.md` before theorizing — the bug may be known.

## Triage workflow

1. **Ground truth first.** `git rev-parse --abbrev-ref HEAD`, `git log --oneline -20`, `git status --short`. Correlate the symptom's start time with recent commits (a bad push is the usual cause).
2. **Real crash vs. false-death loop.** In `supervisor-YYYYMMDD.log`, count `MCP server process is dead — restarting` events and their cadence. Then in `mcp-stderr-YYYYMMDD.log` check whether ONE process kept serving across those times (continuous `[wait]`/`[telegram]` activity, message-ids incrementing, **no startup banner between restarts**). Continuous serving ⇒ the server never actually died ⇒ **false-death loop** (see failure modes), not a code crash.
3. **Runtime state.** Enumerate node processes + port owners; compare `server.pid` to the port-3847 owner:
   ```powershell
   Get-Process node,sensorium-supervisor -EA SilentlyContinue | Select Id,ProcessName,StartTime
   Get-NetTCPConnection -LocalPort 3847,3848 -State Listen -EA SilentlyContinue
   Get-Content ~/.remote-copilot-mcp/server.pid
   ```
   `server.pid` ≠ port owner (but port owner alive) confirms a stale-pid false-death loop.
4. **Find the real fatal error.** In `mcp-stderr-YYYYMMDD.log` search the crash window for `uncaughtException|unhandledRejection|Error:|EADDRINUSE|ECONNREFUSED|Cannot find module|out of memory`. In `supervisor` log, filter noise: `Get-Content <log> | Where-Object { $_ -notmatch 'getUpdates|api.telegram|processing HTTP|response headers' }`.

## Known failure modes

- **False-death restart loop (supervisor).** Health loop declared the MCP dead from a stale `server.pid` without an HTTP cross-check, then spawned a duplicate that couldn't bind 3847 → died → `server.pid` repointed to the dead duplicate → loop every health interval, while the real server kept serving. Fix pattern: probe HTTP (`IsServerReadyAsync`) before killing — restart only when PID dead **and** port silent (`SupervisorWorker.cs`).
- **Cascade restart after MCP restart.** In-memory `spawnedThreads` empties → `isThreadRunning()` false for all → keeper restarts everything. Fix: rebuild tracking from PID files on startup.
- **Keeper "stuck" kills keepAlive threads.** `wait_for_instructions` long-poll didn't refresh the per-thread heartbeat → keeper declared stuck → killed. Fix: write `writeThreadHeartbeat()` in the poll loop.
- **Duplicate thread spawns.** Keeper checks the root threadId but `start_thread` creates a worker with a different id → always "not running" → re-spawns. Fix: track the worker id from the `start_thread` response.
- **Copilot process dies / no memory.** `copilot.bat` (VS Code wrapper) spawned detached dies silently — prefer `copilot.exe`; and a missing `thread_id` column crashed bootstrap. See critical-fixes-log.
- **Reads but no reply — Claude worker answers into invisible stdout.** 👀 is set on message *consume*, not on reply (so 👀 ≠ answered). With Claude tool search active, reply tools (`report_progress`/`send_voice`/`send_message_to_thread`) are deferred, so a conversational agent emits answers as plain `text` → its stdout transcript, never delivered. Signature: `server.log` `Read N messages … Processing…` with no following delivery `tool_use`. First rule out routing (operator's topic must equal the thread's `thread_registry.telegram_topic_id`; inbound `resolveThreadForTopic`/outbound `resolveTelegramTopicId` are symmetric). Fix: `ENABLE_TOOL_SEARCH=false` so tools load upfront, then **respawn the worker** (see Deploy).
- **Self-update port-handoff race (real but often MIS-attributed — do not over-chase).** In theory a new node self-registers its pid asynchronously, so `server.pid` can transiently name a dead pid while HTTP is healthy. BUT the self-update poller is gated behind a ~10-min min-uptime **AND** `remoteVersion !== currentVersion`, so at the latest published version it is a silent no-op. Before blaming self-update, confirm it is ACTUALLY firing: look for `[self-update] Channel version changed` / `Closing HTTP server` and a non-empty `update-spawn.log`; a `[self-update] Deferring — uptime …s < min 600s` line proves it is NOT updating. In practice the recent restart loops were **event-loop freeze** (heavy startup work) and **network outages to Telegram**, not self-update.
- **False `hung-mcp-restart` from a stale probe socket.** The supervisor's liveness probe (`OPTIONS /mcp`, `McpClient.IsServerReadyAsync`) reused a **dead keep-alive socket** from the pooled `HttpClient` after a prior restart, so it failed at the connection level (~2s) while the server was healthy and serving `POST /mcp` — `HealthFailThresh` such failures → needless restart + `hung-mcp-restart` maintenance flag. Tell-tale: `report_progress`/tool calls succeed within seconds of the `not responding to HTTP` verdict, AND a **freshly spawned replacement also fails the probe for ~2 min** while `POST /mcp` returns 200. Fix: force a fresh connection on the probe (`req.Headers.ConnectionClose = true`) and cross-check thread activity before killing. The same stale-probe defect also flapped supervisor `/ready` 503 (per-request live probe, no hysteresis) → debounced with a 15s last-good window.
- **Full briefing after a supervisor-triggered restart = missing reconnect snapshot.** `start_session` gives a *lightweight* reconnect only if the maintenance flag OR the `active-sessions.json` snapshot is present. Self-update writes the snapshot in-process; but supervisor force-restarts (`taskkill /F` for hung/dead/orphan/cmd) deliver **no signal**, so the graceful `shutdown()` snapshot never runs, and the supervisor's `POST /api/prepare-shutdown` used to be a **404 no-op**. Result: reconnecting workers that still have context get a full re-briefing (token/time waste, mass-reconnect freeze risk). Symptom: `preexisting but no reconnect signal (maintenance=false, snapshot=false) — full briefing` + `[reconnect-snapshot] Miss`. Fix: implement `POST /api/prepare-shutdown` to write the snapshot (one endpoint covers all force-restart paths).
- **Post-restart reconnect fragility.** The `hung-mcp-restart` maintenance response makes the AGENT run a *blocking* PowerShell poll of supervisor `/ready` (8848), then manually `start_session`. While in that poll the worker is blind to the operator, and the only recovery net is the keeper's **~1h** stuck threshold (by design: `wait_for_instructions` long-polls refresh the heartbeat, so 1h avoids false-killing idle workers). A worker that never cleanly reconnects looks frozen for up to an hour — **respawn it to recover fast**.

## Emergency stop & restart

Kill in this order so nothing auto-respawns; be **targeted** (never blanket-kill `node`/`claude`/`copilot` — the operator has an interactive coding session):

1. **Supervisor first** (`sensorium-supervisor`) — otherwise it restarts the MCP you kill.
2. **MCP server + watcher** — the Node processes owning ports 3847/3848 or whose command line matches `sensorium-mcp` / `remote-copilot-mcp`.
3. **Worker agents** — only `claude`/`codex`/`copilot` whose command line contains `.remote-copilot-mcp`.

```powershell
Get-CimInstance Win32_Process |
  Where-Object { $_.CommandLine -match 'remote-copilot-mcp|sensorium-mcp' -or $_.Name -eq 'sensorium-supervisor.exe' } |
  Select-Object ProcessId,Name,CommandLine   # ENUMERATE and confirm BEFORE killing
```

Then `Stop-Process -Id <pid> -Force` in the order above. Verify ports 3847/3848 are free afterward. Do NOT delete lock/PID/data files unless the operator asked. If a process is unkillable (`Access denied`), report it — the operator must intervene.

## Fix, validate, ship

- **Validate before pushing:**
  - TypeScript: `npx tsc --noEmit` (must exit 0).
  - Supervisor: `dotnet test supervisor-dotnet/tests/Sensorium.Supervisor.Tests.csproj`.
  - Node tests: `npm test` — **requires Node ≥ 21** (`node --test` glob). CI matrix runs Node 22/24.
- **Expert review** the change (dispatch a review subagent; verify security: command injection, path traversal, secrets).
- **Ship:** `npm version patch --no-git-tag-version` → commit explicit paths (`fix(...)` then `chore(release): x.y.z`) → `git push origin main`.
- **CI:** confirm green — `gh run watch <run-id> --repo <owner>/<repo> --exit-status`. Fix and re-push until green.
- **Deploy at runtime (push alone does NOT deploy):**
  - **MCP server** running via `npx sensorium-mcp@latest` picks up a new version only after **`npm publish`** (the watcher auto-updates). A local `node dist/index.js` needs `npm run build`. **Already-running workers keep their old spawn behavior even after the server updates — respawn them to apply spawn-side fixes.** `ENABLE_TOOL_SEARCH=false` is the right knob (Claude ≥2.1.7), but an `env` block in `~/.claude/settings.json` overrides the spawn env, and the spawn copies that file wholesale into the per-thread `CLAUDE_CONFIG_DIR` (dragging in the operator's own MCP servers).
  - **Supervisor** (.NET) needs a **rebuild + reinstall via `Install-Sensorium.ps1`** to apply supervisor code changes — a running supervisor keeps its old binary in memory.

## Field notes (hard-won — check these before deep code work)

- **"No threads connected" / a thread won't respond is usually NOT a dead thread.** First confirm liveness, don't assume: heartbeats in `~/.remote-copilot-mcp/heartbeats/{threadId}` fresh (age < ~5s) + a live `claude` worker per `pids/{threadId}.pid` = the thread IS alive. If threads are alive but the operator can't reach them, the break is the **Telegram delivery path** or the agent replying into stdout (see failure modes), not the thread.
- **The Claude worker's stream-json transcript is ground truth for agent behavior** — `~/.remote-copilot-mcp/logs/threads/<Name>_<threadId>_<YYYY-MM-DD>.json` (NDJSON; file-fd stdout+stderr, survives server restarts). Per `type:"assistant"` block: `text` = invisible to operator, `tool_use` of a delivery tool = actually sent. `type:"rate_limit_event"` = Claude 5-hour cap (`resetsAt` = epoch secs); `subtype:"api_retry"` = transient API errors. Fresh MCP heartbeat + **stale transcript mtime** = worker idle in the long-poll (normal), not frozen.
- **`[dispatcher] Poll error: fetch failed` (repeating every ~6s) + `.NET` supervisor `SocketException 10054 "connection forcibly closed by remote host"` = a NETWORK outage to `api.telegram.org`, not a code bug.** On laptops this correlates with Wi-Fi drops / Modern Standby / power-source flapping (see Windows Event Log `Kernel-Power` 105/506/507). Test reachability directly: `Resolve-DnsName api.telegram.org`, `Test-NetConnection api.telegram.org -Port 443`, `Invoke-WebRequest https://api.telegram.org -Method Head`. If those pass but the MCP still shows no successful polls, the dispatcher may not have recovered from the socket reset → **restart the MCP** to re-establish polling.
- **Responsive vs frozen vs exited — three different states, check explicitly.** `Invoke-WebRequest -Method Options http://127.0.0.1:3847/{health,ready,mcp}` returning **204** + `Get-Process -Id <pid>` `Responding=True` = event loop ALIVE (so any "silence" is a stuck sub-loop like the Telegram poller, not a freeze). No HTTP answer but port still LISTENing = frozen. Port CLOSED (`connection refused`) = process EXITED.
- **Use the lifecycle instrumentation (shipped 3.0.79+).** On a death, `server.log` gets `[lifecycle] process exit code=N uptime=Ns`; a real OS signal logs `[lifecycle] received signal ...`; a native fatal error writes `report*.json` in `logs/mcp/`; an internal self-kill attempt logs `[process] killProcessTree pid=… caller=…` or `[process] REFUSED kill … self-protection`. **`exit code=0 uptime=0-2s` markers are BENIGN** — short-lived launcher/duplicate processes that found the port taken and bailed cleanly, not the real server dying.
- **Windows Event Log is decisive for native crash vs external kill.** No `Application Error` (Event 1000) for `node.exe` at the death time ⇒ **not** a native crash (e.g. not `better_sqlite3`). No `[lifecycle]`/`[shutdown]`/`[fatal]` line either ⇒ the process was hard-killed externally (`taskkill /F` / `TerminateProcess`, which bypass signal handlers and WER) — audit the internal kill paths (`killProcessTree caller=…`) and any second supervisor doing `KillByPort`.
- **Event-loop freeze death-spiral.** Heavy synchronous startup work (better-sqlite3 consolidation, narrative backfill, per-thread `start_session` bootstrap embeddings across many reconnecting threads) blocks the loop past the supervisor's ~3s HTTP readiness probe → supervisor kills+respawns → the fresh process re-runs the same work → re-freeze. Mitigations: persist a per-day marker so consolidation/rotation run **once per day** across restarts (`src/data/daily-marker.ts`); defer/chunk heavy startup work off the readiness window.

## After an incident

Record the root cause, the fix, the commit hash, and the version in `/memories/repo/critical-fixes-log.md` (append, don't rewrite) so the next repair starts from knowledge, not zero.