# WO-50: Recover honestly from missing embeddings

Status: repository repair complete; delivered as `a3375c63` to `origin/main`. Requested Nomic re-pull completed and passive server recovery observed. Publication and live acceptance of source changes remain with the user.

## Field evidence and operational remedy

The Ollama broker recorded 102 missing-model errors for `nomic-embed-text:latest` between 14:45 and 15:09:55 PDT on September 5, 2026. Omnius is configured for `http://localhost:11434`, with no embedding model override in the running process. The errors return HTTP 404 from `/api/embed`. This reduces semantic retrieval; stored text and lexical retrieval are still available. The exact caller of every request in the bursts has not been established.

At the user's explicit request, `POST /api/pull` for the exact tag returned `success` at 15:19:57 PDT. Subsequent `/api/tags` and `/api/show` metadata confirmed `nomic-embed-text:latest`, embedding capability, 137M/F16, 768 dimensions, digest `0a109f422b47e3a30ba2b10eca18548e944e8a23073ee3f3e947efcf3c45e59f`. No inference or model-load probe was sent; no services were restarted. Successful installation metadata does not establish live embedding execution.

Passive journal follow-up through 15:25:50 PDT then recorded 51 `/api/embed` HTTP 200 responses and zero embedding 404s after the pull. The broker logged Nomic lanes ready at 15:22:07 and 15:22:12 for the existing workload. This is observed server recovery, not an agent-issued inference test or proof that every earlier missing embedding job was repaired. Capture and SHA-256 manifest: `/tmp/omnius-nomic-repull-20260905/`.

## Confirmed source defects and repair ownership

- `packages/memory/src/embeddings.ts`: a single missing-model response triggers an implicit pull and recursive retry; batch requests lack bounded availability backoff. Replace this with exact endpoint/model scoped availability caching and independent cancellation. Preserve the null failure contract, validate vectors, and honor configured model identity without substituting models. Model checks must match exact tags.
- `packages/memory/src/memoryIngestion.ts`: jobs currently complete even when no embedding is stored. Use existing durable retry/backoff for failed embedding work; retain raw text and lexical search.
- `packages/indexer/src/ollamaEmbeddings.ts`: any failed batch causes per-input retries and fabricated zero vectors. Report generation failure truthfully, reject malformed vectors, and limit legacy API fallback to an actual unsupported endpoint.
- Matching hermetic tests in memory/indexer, plus dependent builds and relevant retrieval tests.

## Acceptance

- [x] Missing-model bursts make bounded requests, with no implicit downloads or recursive retries.
- [x] Exact endpoint/model identity, TTL recovery and independent cancellation are tested.
- [x] Configuration honors the selected model; availability checks cannot match a different tag by prefix.
- [x] Failed generation cannot fabricate vectors or complete embedding jobs; text remains recoverable.
- [x] Compatibility handling cannot fan one model failure into one request per input.
- [x] Record verification and scoped commit/push.

## Separate chat-server incident

This embedding defect did not cause the observed router HTTP 503. At 14:43:43.457 PDT the broker logged an incomplete `/api/chat` read; at 14:43:43.469 its fresh retry hit connection refused. At 14:43:44.583 it reported `lane-121` stopped unexpectedly. The lane had been ready since 14:25:42 for `robit/qwen3.8-27b-obliterated-e03:27b` on GPU UUID `GPU-170a99ee-850f-2182-1050-4e8d3c87b6b0`. The broker served chat again at 14:44:09.532, and the main run subsequently produced its answer.

No OOM kill, GPU Xid, segfault or coredump was found in the narrow incident journal. The external broker discards child stdout/stderr and omits PID/exit status in its prune warning (`/usr/local/libexec/ollama-unify-gpu-negotiator`, child spawn and `_prune_dead_lanes_locked`). Its exact exit mechanism remains unknown. An external broker follow-up needs retained child stderr and return-code logging to distinguish crash, signal and deliberate exit; this Omnius repair does not claim to fix that unknown cause. Hashed journal captures are under `/tmp/omnius-ollama-failure-aki8oz5k/`; the earlier incident window is under `/tmp/omnius-ollama-early-failure-_nsbctcb/`.

## Review corrections and existing data limits

Review added regression coverage for preserving a custom embedding set through immediate failure and queued retry, for blank environment overrides, for numerically unusable indexer vectors, and for diagnostics that omit provider payloads and URL credentials.

The 30-second availability cooldown is process-local and bounded to 128 endpoint/model identities. It expires after an operator installs the model. These changes do not requeue historical jobs already incorrectly marked complete, re-embed existing vectors, or migrate a stored vector space when the embedding model changes. Restoring the same Nomic model does not require choosing a substitute embedding space.

The availability key is not a guarantee of stored vector-space isolation. A separate existing model-change limitation spans `UnifiedMemoryStore.searchVector` (dimension/set filters without query model), `EpisodeStore.search/setEmbedding` (no persisted model identity), and `ProceduralMemoryStore.searchSemantic` (stored model not used for comparison). Any future model-migration repair must preserve model identity from generation through storage and query together: unified callers in execution `memory-search.ts` and CLI `routes-v1.ts`; episode callers in `agenticRunner.ts` and `telegram-reflection-corpus.ts`; procedural callers in `proceduralMemoryStore.ts`. It needs exact-tag/unknown-model tests in the corresponding store and caller suites and lexical fallback for incompatible vectors. No partial schema migration or automatic re-embedding was attempted in this missing-model repair.

## Final verification

- `OMNIUS_SQLITE_TESTS=1 pnpm --dir packages/memory exec vitest run`: 665 passed across all 55 suites, zero skips. Includes sequential/concurrent missing-model bursts, exact identities and expiry, independent cancellation, malformed vectors, custom-set retry, raw-text retrieval, and blank environment overrides. Log: `/tmp/omnius-wo50-memory-full-final.log`.
- `pnpm --dir packages/indexer exec vitest run`: 186 passed across all 13 suites. The 39 provider cases cover missing-model single-request failure, genuine legacy-route compatibility, no fabricated vectors, exact dimensions/cardinality, numeric overflow/underflow, configuration, and redacted timeout/cancellation errors. Log: `/tmp/omnius-indexer-embedding-all.log`.
- Both package typechecks/builds passed. A subsequent clean rebuild of every workspace package passed after deleting stale build metadata, without running the publish bundler or changing the user's publish staging. Log: `/tmp/omnius-wo49-50-clean-build.log`.
- After that rebuild, actual Telegram authored-answer integration and the reflection corpus/maintenance suites passed all 15 checks (three repeat WO-49 cases and twelve dependent integration cases). Log: `/tmp/omnius-wo49-50-cli-integration.log`.
- All 17 required daemon transport/client/script cases passed, including split streams, delayed errors, subprocess failures and managed Stop. Log: `/tmp/omnius-wo49-frontend-transport.log`.

All inference and Telegram HTTP in repository checks was mocked. The sole model operation on the live server was the user-requested Nomic pull; subsequent live embedding success was observed passively in the journal. No live test inference, service restart, installed-package replacement, Telegram sends or publication occurred.
