# Investigation and Production Support

> A `remits-cli` skill reference. **Load this when** you are investigating live behavior: which record trail to read, how to correlate a processing chain, and the production support flows.
>
> The table of contents below carries **real line numbers** (`- L84  Some Heading`), resolved when
> this file is installed, so they are never stale. Read the head, pick your sections, and offset-read
> only those. The entry text is the heading verbatim, so it also greps.

## Table of Contents

- [The Investigation Model](#the-investigation-model)
  - [Which tool reads which record](#which-tool-reads-which-record)
  - [Correlation keys](#correlation-keys)
  - [Reading a record's `content` — persisted context, not a memory dump](#reading-a-records-content--persisted-context-not-a-memory-dump)
  - [HTTP audits](#http-audits)
  - [AI activity](#ai-activity)
  - [Node Reference Table](#node-reference-table)
  - [Runtime node and `localMode`](#runtime-node-and-localmode)
- [Production Support Workflow](#production-support-workflow)
  - [Investigation Strategy](#investigation-strategy)
  - [Presenting Findings](#presenting-findings)
  - [Verifying a Production Issue Fix](#verifying-a-production-issue-fix)

## The Investigation Model

Front stage — Schemas, Readers, Actions, Embeddables, Rules, HtmlTemplates, Agents, Tests, and the Firestore
**documents** that hold business data — is described in `platform-overview.md`. This section is only about
the **back-stage lifecycle records** an investigation actually reads, and how to read them.

### Which tool reads which record

The model — Documents (Firestore business data) vs Objects and their record trail (MySQL), linked by
`object_id` — is in `platform-overview.md` → *Objects, Documents, and the Record Trail*. What matters
here is the tool and the filterable fields:

| Record | Query it with | Fields worth filtering on |
|---|---|---|
| document | `mcp_firestore_search` | any schema field, plus `account_id`, `object_id`, `_lastModifiedAt` |
| `object` | `mcp_record_listing` / `mcp_record_view` | `status`, `type`, `name`, `referenceId` |
| `object_log` | `mcp_record_listing` / `mcp_record_view` | `type`, `description`, `content`, `threadGroupingId` |
| `event` | `mcp_record_listing` / `mcp_record_view` | `action`, `status`, `eventDate`, `threadGroupingId` |
| `alert` | `mcp_record_listing` / `mcp_record_view` | `status`, `type`, `active`, `threadGroupingId` |
| user activity session | `mcp_user_activity` | `userId`, `accountId`, `sessionKey`, `focusedOnly`, `traceId` pivots |

`mcp_record_listing` finds candidates when you do not know the id; `mcp_record_view` opens an exact one;
`mcp_object_activity` returns one Object's whole timeline in order.

**Check the data lane before concluding a record does not exist** — `--data-mode test` and `prod` read
different lanes (see `platform-overview.md` → *Test and Production Data Lanes*).

### Correlation keys

- **`object_id`** — links documents to records. Pivot `mcp_firestore_search` → `mcp_object_activity`.
- **`threadGroupingId`** — groups every record and log line from one processing chain, and is also the
  request's trace id. Pivot into `mcp_system_logs` and `mcp_performance_trace`.
- **`node`** — resolves to `serviceName` + `region` for log queries (table below).
- **`sessionKey`** — salted user-activity session key. Pivot `mcp_user_activity` `sessions` → `story`;
  each beat then carries a `traceId` / `threadGroupingId` for trace and log investigation.
- **`sessionId`** — pivots into persisted AI activity via `mcp_ai_session_search`.

### Reading a record's `content` — persisted context, not a memory dump

`content` on an `object`, `object_log`, `event`, or `alert` is the **sanitized snapshot** the platform
deliberately kept of what the producing component knew at that workflow step. It is not everything that was
in memory.

- `Object.content` — ingestion/request/file context when the Object was created or updated.
- `ObjectLog.content` — the logging component's point-in-time view.
- `Event.content` — the scheduling component's view, plus the explicit event options.
- `Alert.content` — the raising component's view, plus the explicit alert body.

**Missing or `[REDACTED]` does not mean the component never had it.** Before persisting, the platform drops
non-serializable objects, strips `requestBody` / `params` / `token` recursively, summarizes very large
strings, prunes oversized maps and collections, and redacts secret-looking keys (`password`, `secret`,
`authorization`, `cookie`, `clientSecret`, …) plus any Account/User schema field marked `sensitive: true`.
When explaining a record, distinguish what the workflow *had* at runtime from what the platform *kept*.

**Provenance travels in the context itself:**

- **`source_bcd`** — the immediate component that produced this record, e.g. `[Action:18] DataPlus Invoice Posting`
- **`upstream_source_bcd`** — the prior workflow hop, when the context was inherited from one

**Never interpret `content` in isolation.** Pair it with the record type, `source_bcd`, any
`upstream_source_bcd`, the `object_id` timeline, the `threadGroupingId` chain, and the producing
component's source. The goal is not to find a suspicious record — it is to explain **why the context looks
exactly the way it does relative to the workflow step that produced it**.

### HTTP audits

Raw inbound (Reader) and outbound (`rest(...)`) HTTP is persisted to Firestore — but **only for accounts
with `Account.enableHttpAudits`**. When it is off, the absence of audit documents proves nothing.

Audits live in **monthly** collections, not one global collection:

```
http-audits/http-audits-YYYY-MM/entries
```

Start with the month the request ran in, and check the adjacent month if the run may have crossed a
boundary. Query them with `mcp_firestore_search` like any other collection; the filters worth reaching for
are `direction` (`INBOUND` / `OUTBOUND`), `success`, `request.method`, `request.path`, `component.name`, and
`response.statusCode`. Some paths are deliberately excluded from persistence, so a missing audit is not
proof a call was never made.

Reach for audits when the question is *what exact request went out, what came back, and did this component
actually make the call* — then use the component source plus the payload to place the fault in request
formation, the partner's response, or downstream processing. Full captured shape, websocket behavior, and
embeddable patterns: `features/http-audits.md` (`mcp_get_guide`).

### AI activity

Every `ai()` call and Agent turn is persisted. Two ways in:

- **`mcp_ai_session_search`** — search groupings and open their detail. A grouping spans every session
  sharing one grouping id: the agent turns plus its guardrail and internal `ai()` calls. Use the
  **map → open** flow described under its `tool-reference.md` entry, never a whole-detail dump.
- **`ai_request_response([sessionId: id])`** — from inside component code, loads the stored
  request/response history for a session (`first: true` / `last: true` for one record). This is also how
  you replay a stored provider response in a test.

If a document or agent state already carries a session id, that is a direct pivot into persisted AI
activity.

**Before tuning any prompt, read the session.** `features/ai-session-investigation.md` owns the method —
per-turn forensics, what each layer proves, and the rule that what a tool **PRODUCED** is not necessarily
what the model **CONSUMED**. `features/ai-strategy.md` owns what to change once you know. Changing a prompt
before reading the persisted request/response is guessing.

### Node Reference Table

| Node Name | Service Name | Region |
|---|---|---|
| remitsAdmin-east5 | remits | us-east5 |
| remitsActions | remits-actions | us-east1 |
| remitsAdmin | remits | us-east1 |

`mcp_system_logs` accepts `node` directly and resolves it automatically.

### Runtime node and `localMode`

The deployed `remits` service in `us-east5` (`remitsAdmin-east5`) runs with the platform setting
`localMode=true`. If someone says "localModel" in this context, confirm they mean this `localMode`
setting. Operationally, immediate async follow-on work stays on the same Cloud Run service/node instead of
being sharded to `remits-actions`:

- Pub/Sub-style follow-on messages are handled locally after commit.
- Near-immediate tasks are handled locally when `localMode` is enabled. Future scheduled tasks still use
  Cloud Tasks.
- Local worker hops preserve the run context, including staged-source resolution, data mode,
  `threadGroupingId`, and trace correlation.
- Durable boundaries such as async HTTP ingress and Events carry that same run context across the queue.

For investigations on the default deployed host (`https://remits-529558023549.us-east5.run.app`), do not
assume "async" means `remitsActions` / `us-east1`. Start with `node:"remitsAdmin-east5"` and the
`threadGroupingId`; pivot to `remitsActions` only when the Event delivery envelope, log line, or returned
node says the work actually ran there.

This does not change the data-lane rule: a non-null `TestMode` can exist only to carry branch/staged-source
resolution. Data isolation is decided by CLI `--data-mode`: a branch-scoped `--data-mode prod` run is still
prod data, while `--data-mode test` remains isolated test data.

## Production Support Workflow

Switch to prod mode for investigations:

```bash
remits-cli data-mode set prod
```

### Investigation Strategy

Before starting an investigation outside the confirmed current repo:
1. Read `~/.remits-cli/account-repos.json`
2. Switch to the best local repo candidate
3. Read that repo's `account-info.json`
4. Confirm whether you are in `CLIENT`, `PLATFORM`, or `PRODUCT` context
5. Then continue with the investigation flow below

For generic tool/action probes in a split-world workstream, keep the account roles explicit. From a forked or
subscriber repo, use `--account-id <repo/scope>` for the checkout account, `--as-account <execution>` when
component resolution should run as a client/subscriber, and `--target-account <target>` for the account whose
records the tool operates on. Then read the printed `World:` block and saved response file; it names the
repo/scope account, execution account, target account, component owner, branch/workspace, data lane,
component branch, staging lane and source signature when known.

Use `activity inspect --scope related` when evidence spans accounts that are not in one hierarchy branch.
Always pass `--workspace`, `--branch` or `--workstream` with it: those NARROW the view to the accounts and
evidence carrying that workstream, and without one, `related` returns the whole subtree plus component owners
with an arbitrary limit-bounded slice of their evidence. The response echoes the applied `filters`.

`workstream status --workspace <workspace>` reviews the local actor-scoped files that belong to one proof
campaign. The workstream id is stamped onto every verification packet world, so the same id works on both
sides:

```bash
remits-cli workstream status --workstream <id>
remits-cli activity inspect --scope related --workstream <id>
```

**Document-First** (most common — user reports a data issue):
1. `mcp_account_view` — understand the account's schemas and components.
2. `mcp_firestore_search` — find the document, capture its `object_id`.
3. If the issue involves inbound or outbound HTTP behavior and the account has `enableHttpAudits`, query `http-audits/http-audits-YYYY-MM/entries` with `mcp_firestore_search`.
4. `mcp_object_activity` — scan the timeline for warnings, errors, unexpected events.
5. `mcp_record_listing` — search or filter alerts, events, object logs, or objects when you need to find the suspicious record first.
6. `mcp_record_view` — drill into suspicious entries for full content.
7. `mcp_ai_session_search` — if the workflow involves AI, inspect session groupings, prompts, tool definitions, and responses in human-readable form.
8. `mcp_user_activity` — for "user X is slow right now" reports, list sessions by `userId`/`accountId`, open the session story, and use the returned beat pivots.
9. `mcp_performance_trace` — for slow/sluggish reports, open the beat `traceId` with `action:"trace"`; use `action:"slowest"` when you only have a broad time window.
10. `mcp_system_logs` — correlate via `threadGroupingId` for raw log context when the trace needs supporting log lines.
11. If no local checkout exists for the responsible implementation account, use
    `mcp_component_view`/`mcp_component_grep` to explain how the component works. If the repo exists on
    this machine, inspect the branch/files there instead; the remote component tools are fallback and
    live-DB comparison surfaces, not the starting point for source comprehension.

**Slow / sluggish user report:**

1. Resolve the reporting user/account with `mcp_account_user_admin` if you only have an email/name.
2. Call `mcp_user_activity` with `action:"sessions"` and `userId` or `accountId`.
3. Open the likely row with `action:"story"` and inspect beat labels, status, `ms`, `node`, and `traceId`.
4. Open slow or failed beat pivots with `mcp_performance_trace` before querying raw logs.
5. Use the returned `mcp_system_logs` pivot only when the trace needs surrounding log lines.
6. If there is no live session, call `mcp_user_activity` `action:"watch"` for the user/account, ask for reproduction, then read `sessions`/`story` again. Focused sessions retain sanitized request detail and emit archived `REMITS_ACTIVITY` log lines.

**Error or Alert Investigation:**
1. `mcp_record_listing` — search by alert type, content, error text, action, status, `threadGroupingId`, or other exact-match record properties when you do not yet know the record ID.
2. `mcp_record_view` — inspect the chosen record/event/alert/object in full once you have its ID.
3. Use `object_id` + `threadGroupingId` to pull full timeline and logs.
4. Cross-check Firestore document state.
5. Identify `source_bcd` and any `upstream_source_bcd`.
6. Explain the record in terms of the workflow step that produced it, not as a generic JSON blob.
7. If the context looks missing, redacted, or truncated, consider sanitization rules before concluding data was never present.
8. If AI behavior is part of the symptom, use `mcp_ai_session_search` and compare the persisted session content against the Agent component implementation and `features/ai-support.md`.

**Stuck / failed / recovered Event:**

Do **not** open the Action source first. The platform records each attempt's delivery envelope — which
queue delivered it, which delivery attempt this was, and how long it was ever allowed to run — and
classifies the failure for you.

```bash
remits-cli tool --name mcp_event_diagnostics --input '{"accountId":49,"eventId":18838}' --data-mode prod
```

The same classifier is available from a Test or any component as `eventDiagnostics(18838)`.

**Read `classification` before anything else** — only `APPLICATION_FAILURE` means the bug is in the
component. The full classification table, what each `abandonmentCause` implies, and the returned `pivots`
are under **`mcp_event_diagnostics`** in `tool-reference.md`. One thing to check every time:
`delivery.deliveryAttempt` above `1` means Cloud Tasks had **already** retried this event, so any
non-idempotent side effect may have run more than once — look for duplicate records before concluding the
component "ran twice for no reason".

Full detail: `features/observability.md` and `features/events-builder-guide.md` (`mcp_get_guide`).

### Presenting Findings

Users are not engineers. When reporting investigation results:
- Lead with what happened in plain language.
- Show the evidence (document values, timeline events, log excerpts).
- Explain why it happened if you can determine the cause.
- Recommend what to do next — in terms the user can act on.

### Verifying a Production Issue Fix

When a bug is reported from production, use this pattern:

1. Investigate the live issue in **prod mode** and identify the exact affected document IDs, collection names, account IDs, and component path.
2. Make the code change in the owning `PLATFORM` or `PRODUCT` repo when the defect is in shared implementation.
3. Verify in **test mode**, not prod.
4. Prefer a **Test component** when the behavior can be asserted programmatically, because that creates a durable regression suite and lets you explicitly construct the necessary data, operations, and assertions.
5. Use `remits-cli token` plus `playwright-cli` when the proof is visual or interaction-driven.
6. If useful, create or update a dedicated embeddable "playground" in test mode to reproduce the scenario in a controlled way.

Do not move production customer data into another account's test collection as a routine verification strategy. If you cannot verify with a Test component, Playwright flow, or controlled test-mode embeddable, explain the gap clearly instead of improvising with live production validation.