# Multimodal Identity Memory Implementation Tracker

Goal: make voice, text, image, Telegram reply context, CLIP-like embeddings, zettelkasten links, and social profiles converge on one evidence-based identity substrate.

## Scope

- [ ] Route Telegram, TUI, GUI, voice, and API media ingestion through one identity/evidence layer.
- [ ] Keep private/public/terminal/gui scopes explicit on every assertion and retrieval path.
- [ ] Represent uploader/sender, depicted person, speaker, reply target, media asset, and message as separate graph atoms.
- [ ] Preserve CLIP/image, CLIP/text, face, speaker, transcript, and aligned embeddings with explicit vector-space metadata.
- [ ] Use agentic structured identity extraction later for "this is X" style assertions; avoid hard-coded naming heuristics.
- [ ] Feed durable interaction/profile facts into social memory and zettelkasten instead of isolated JSON-only stores.

## Code Anchors

- `MMID-001` - central multimodal identity/evidence service.
- `MMID-002` - API memory ingest uses shared orchestrator DB names.
- `MMID-003` - Telegram media ingest sends structured source/scope/sender/message metadata.
- `MMID-004` - embedding-space handling stores OpenCLIP-compatible vectors as `clipEmbedding`.
- `MMID-005` - scoped graph atoms for sender, message, media asset, reply target, and identity assertion.
- `MMID-006` - dependency/worker path for image/audio/text embedding providers.
- `MMID-007` - Telegram/TUI tool exposure and context injection.
- `MMID-008` - backward-pass validation checklist and tests.
- `MMID-009` - scoped automatic prior-enrolled visual identity association and recall injection.
- `MMID-010` - unified `.omnius/episodes.db` / `.omnius/knowledge.db` paths for identity, reflection, and API media ingest.
- `MMID-011` - natural chronology support for explicit identity-before-image assertions and unknown-face steering.

## Implementation Checklist

- [x] Create this tracking document before code changes.
- [x] Add multimodal identity service in `@omnius/memory`.
- [x] Extend graph relation vocabulary for identity evidence.
- [x] Make `/v1/memory/ingest` use `episodes.db` and `knowledge.db`.
- [x] Replace legacy `labels`-as-person behavior with structured sender/media/assertion handling.
- [x] Store supplied image/text CLIP vectors in `clipEmbedding`.
- [x] Store supplied speaker/audio vectors in native `embedding` with vector-space metadata.
- [x] Pass Telegram source scope, sender, message id, media id, caption, and reply context into ingest.
- [x] Keep legacy ingest clients compatible.
- [x] Add tests for visual media not misbinding uploader as depicted person.
- [x] Add tests for explicit identity assertion creating scoped `named_as` / `depicts` evidence.
- [x] Add tests for audio voice sample linking to sender without global leakage.
- [x] Add structured JSON output mode for `visual_memory identify` so association code does not parse display text.
- [x] Add scoped visual identity association service that records prior enrolled face matches as graph evidence.
- [x] Inject scoped visual identity recall into Telegram and TUI image ingress.
- [x] Return visual identity recall metadata from `/v1/memory/ingest` for GUI/API callers.
- [x] Wire GUI attachment upload into scoped media ingest and prepend returned identity context to the next chat turn.
- [x] Move identity/reflection stores onto the same `.omnius/episodes.db` and `.omnius/knowledge.db` files used by the orchestrator and API.
- [x] Add `identity_memory action='stage_identity'` for explicit "next image/media is <name>" chronology without regex parsing.
- [x] Apply pending same-scope visual identity assertions when a later image arrives and face enrollment succeeds.
- [x] Add unknown-face context when structured detection sees a face but no enrolled identity matches.
- [x] Run focused builds/tests.
- [x] Backward pass: verify each item above against code anchors and test coverage.

## Backward Pass Verification

- `MMID-001`: implemented `MultimodalIdentityService` and exported it from `@omnius/memory`.
- `MMID-002`: API ingest, API memory search/entities, route-v1 memory stores, embedding workers, and `multimodal_memory` bridge now use `episodes.db` / `knowledge.db`.
- `MMID-003`: Telegram media ingest now sends structured source/scope/sender/message/reply/media payloads.
- `MMID-004`: OpenCLIP-compatible vectors are stored via `setClipEmbedding`; speaker/audio vectors are stored as native embeddings.
- `MMID-005`: tests cover distinct sender, media, message, reply/identity evidence graph atoms.
- `MMID-006`: provider hooks are used when configured; full CLIP/ECAPA dependency install is opt-in via `OMNIUS_INSTALL_FULL_EMBED_DEPS=1`.
- `MMID-007`: identity memory remains an agent-facing tool; explicit name/enrollment requests are tool calls, not regex captures.
- `MMID-008`: verified with `pnpm -r build`, `pnpm --filter omnius test -- tests/identity-memory-tool.test.ts tests/visual-identity-association.test.ts`, `pnpm --filter omnius test -- tests/telegram-reflection-corpus.test.ts tests/telegram-reflection-extraction.test.ts`, `pnpm --filter omnius test -- tests/telegram-bot-api-10.test.ts`, and `git diff --check`. The visual association tests now cover prior-match recall, cross-scope isolation, pending next-image application, and unknown-face steering.
- `MMID-009`: `associateVisualIdentityFromImage` calls `visual_memory identify` with `format=json`, commits `same_person_candidate` / `depicts` evidence for prior enrolled matches, and formats same-scope recall context.
- `MMID-010`: `IdentityMemoryTool`, Telegram reflection corpus/extraction, API ingest, memory menus, and orchestrator memory now converge on `.omnius/episodes.db` / `.omnius/knowledge.db`.
- `MMID-011`: `stageVisualIdentityAssertion` records explicit pending next-image identity evidence; later image ingress consumes it only in the same scope/session and only after face enrollment succeeds. Unknown face detection produces a model-facing prompt to ask the user instead of guessing.

## Follow-Up Work After First Slice

- [ ] Add agentic structured identity assertion extraction from reply/message/media context.
- [x] Wire `visual_memory` face enrollment and prior-enrolled recognition into the identity graph.
- [ ] Extend the same structured association path to taught object CLIP recognition.
- [ ] Wire ECAPA speaker embeddings into Telegram voice/audio ingest.
- [x] Add retrieval context injection for scoped visual identity graph neighborhoods.
- [ ] Mirror scoped behavior/personality facts into `SocialMemoryStore`.
- [ ] Expose scoped Telegram tools for identity recall/enrollment with privacy guards.