# Session Debug Scenario

A debugging flow for an App developer who wants to chat with an agent and observe every detail in real-time.

---

## Step-by-Step User Flow

### 1. Session Initialization
- [ ] User selects an agent to chat with
- [ ] User initiates a new chat session (or resumes an existing one)
- [ ] App receives session ID and initial session metadata (agent config, model, mode)

### 2. Pre-Message State
- [ ] User views current session state (message count, total tokens used so far)
- [ ] User views current context window size / remaining budget
- [ ] User views agent's available tools list for this session

### 3. Send Message
- [ ] User types and sends a message
- [ ] App shows the message was received and processing started
- [ ] App shows timestamp of message submission

### 4. Real-Time Processing Observation
- [ ] App receives notification that LLM call is starting
- [ ] App shows which model is being called
- [ ] App shows the input token count for this LLM call
- [ ] App receives streaming tokens as the LLM responds (if streaming enabled)
- [ ] App receives notification that LLM call completed
- [ ] App shows output token count for this LLM call
- [ ] App shows cache hit/miss token count
- [ ] App shows total latency for this LLM call

### 5. Tool Call Detection (Loop)
- [ ] App receives notification that LLM wants to call tool(s)
- [ ] App shows tool name(s) about to be called
- [ ] App shows tool input arguments (JSON)
- [ ] App shows tool call ID for correlation

### 6. Tool Execution Observation
- [ ] App receives notification that tool execution started
- [ ] App shows tool execution duration in real-time (or final duration)
- [ ] App receives notification that tool execution completed
- [ ] App shows tool output/result
- [ ] App shows whether tool succeeded or failed
- [ ] App shows any tool error message if failed

### 7. Tool Result Sent Back to LLM
- [ ] App receives notification that tool results are being sent to LLM
- [ ] App shows the context size after adding tool results
- [ ] (Loop back to Step 4 if LLM makes more tool calls)

### 8. Final Response
- [ ] App receives the final assistant text response
- [ ] App shows the response content
- [ ] App shows response generation timestamp

### 9. Post-Message Statistics
- [ ] App shows total tokens used for this turn (input + output + cache)
- [ ] App shows total tool calls made in this turn
- [ ] App shows total LLM round-trips in this turn
- [ ] App shows total duration for the entire turn
- [ ] App shows cumulative session token usage

### 10. Session History Access
- [ ] User can view full message history for the session
- [ ] Each message shows its role (system/user/assistant/tool)
- [ ] Each message shows its token count
- [ ] Each message shows its timestamp
- [ ] Tool messages show the original tool_call_id they respond to

### 11. Session Metadata Access
- [ ] User can view session creation time
- [ ] User can view session last activity time
- [ ] User can view total messages in session
- [ ] User can view total tokens consumed by session
- [ ] User can view model used in session

### 12. Error Handling Visibility
- [ ] App shows if LLM call failed (rate limit, timeout, etc.)
- [ ] App shows retry attempts if any
- [ ] App shows if session hit token budget limit
- [ ] App shows if session hit iteration limit (for task mode)

### 13. Context Management Visibility
- [ ] App shows when context compaction is triggered
- [ ] App shows before/after token counts for compaction
- [ ] App shows what was compacted (summarized)

---

## API Analysis

### Step 1: Session Initialization
| Requirement | API Provides | Gap |
|-------------|--------------|-----|
| Select agent | `GET /agents`, `GET /agents/:name` | ✓ Full agent config including modes, tools |
| Create session | `POST /agents/:name/chat` (implicit) | ⚠️ No explicit `POST /sessions` |
| Resume session | `POST /agents/:name/chat` with `sessionId` | ✓ Works |
| Get session metadata | `GET /sessions/:id` | ✓ Returns agent_name, mode, status, timestamps |

### Step 2: Pre-Message State
| Requirement | API Provides | Gap |
|-------------|--------------|-----|
| Session state | `GET /sessions/:id` | ✓ Has message_count, status, timestamps |
| Context window size | — | ❌ Not exposed (no token count per session) |
| Available tools | `GET /agents/:name` modes.chat.tools | ⚠️ Indirect - must parse agent config |

### Step 3: Send Message
| Requirement | API Provides | Gap |
|-------------|--------------|-----|
| Send message | `POST /agents/:name/chat` | ✓ |
| Processing started | — | ❌ Sync call - no "started" notification |
| Timestamp | — | ⚠️ Not in response, must use client time |

### Step 4: Real-Time Processing Observation
| Requirement | API Provides | Gap |
|-------------|--------------|-----|
| LLM call starting | `eventBus.emit('chat.event')` internal | ❌ **No SSE/WS endpoint to subscribe** |
| Model info | In session DB | ⚠️ Not in real-time events |
| Input tokens | — | ❌ Not in real-time |
| Streaming | — | ❌ **Not implemented** |
| LLM call completed | `eventBus` internal | ❌ Not exposed |
| Output tokens | `tokenUsage` in final response | ⚠️ Only at end |
| Cache tokens | `tokenUsage.cache` in response | ⚠️ Only at end |
| Latency | — | ❌ **Not tracked** |

### Step 5: Tool Call Detection
| Requirement | API Provides | Gap |
|-------------|--------------|-----|
| Tool call notification | `eventBus.emit('chat.tool')` | ❌ Not exposed to clients |
| Tool name | In internal event | ❌ Not exposed |
| Tool arguments | In internal event (full JSON) | ❌ Not exposed |
| Tool call ID | In internal event | ❌ Not exposed |

### Step 6: Tool Execution Observation
| Requirement | API Provides | Gap |
|-------------|--------------|-----|
| Execution started | `yield { type: 'tool.start' }` | ❌ Not exposed |
| Duration | `yield { type: 'tool.end', durationMs }` | ❌ Not exposed real-time |
| Execution completed | Loop yields `tool.end` | ❌ Not exposed |
| Tool output | Stored in messages DB | ⚠️ Only after turn ends via `/sessions/:id/messages` |
| Success/failure | In `tool.end` event | ❌ Not exposed real-time |
| Error message | In `tool.end` event | ❌ Not exposed real-time |

### Step 7: Tool Result to LLM
| Requirement | API Provides | Gap |
|-------------|--------------|-----|
| Results sent notification | — | ❌ No event for this |
| Context size | — | ❌ Not exposed |

### Step 8: Final Response
| Requirement | API Provides | Gap |
|-------------|--------------|-----|
| Response content | `POST /chat` response.message.content | ✓ |
| Timestamp | — | ❌ Not in response |

### Step 9: Post-Message Statistics
| Requirement | API Provides | Gap |
|-------------|--------------|-----|
| Tokens used | `tokenUsage: { input, output, cache }` | ✓ Per-turn |
| Tool call count | `toolCalls[]` array with name, durationMs, success | ✓ |
| LLM round-trips (iterations) | — | ❌ **Not exposed** |
| Turn duration | — | ❌ **Not tracked** |
| Cumulative session tokens | — | ❌ **Not stored in session** |

### Step 10: Session History
| Requirement | API Provides | Gap |
|-------------|--------------|-----|
| Message history | `GET /sessions/:id/messages` | ✓ |
| Message role | In response | ✓ role field |
| Message tokens | — | ❌ **Not stored per message** |
| Message timestamp | `created_at` field | ✓ |
| Tool call correlation | `tool_call_id` field | ✓ |

### Step 11: Session Metadata
| Requirement | API Provides | Gap |
|-------------|--------------|-----|
| Creation time | `GET /sessions/:id` → created_at | ✓ |
| Last activity | updated_at | ✓ |
| Message count | message_count | ✓ |
| Total tokens | — | ❌ **Not stored** |
| Model | model field | ✓ |

### Step 12: Error Handling
| Requirement | API Provides | Gap |
|-------------|--------------|-----|
| LLM failure | `yield { type: 'llm.error' }` | ❌ Not exposed real-time |
| Retry attempts | In llm.error event | ❌ Not exposed |
| Token budget hit | `yield { type: 'limit.reached' }` | ❌ Not exposed real-time |
| Iteration limit | `yield { type: 'limit.reached' }` | ❌ Not exposed real-time |

### Step 13: Context Management
| Requirement | API Provides | Gap |
|-------------|--------------|-----|
| Compaction triggered | `yield { type: 'context.compacted' }` | ❌ Not exposed |
| Before/after tokens | — | ❌ Not tracked |
| What was compacted | — | ❌ Not exposed |

---

## Summary of Gaps

### 🔴 Critical Missing Features (Blocking for Debug UX)

| # | Gap | Impact | Fix Complexity |
|---|-----|--------|----------------|
| 1 | **No SSE/WebSocket endpoint** | Cannot observe tool calls in real-time | Medium - add `/sessions/:id/stream` or `/events` SSE endpoint |
| 2 | **No streaming LLM responses** | User waits for full response, no progress indicator | Medium - requires OpenAI stream:true + SSE |
| 3 | **No per-message token count** | Cannot show "this message cost X tokens" | Low - add token_count column to messages table |
| 4 | **No session total tokens** | Cannot show "session has used X tokens" | Low - add total_tokens column to sessions table |
| 5 | **No turn duration tracking** | Cannot show "this turn took X seconds" | Low - track in /chat response |

### 🟡 Important Missing Features

| # | Gap | Impact | Fix Complexity |
|---|-----|--------|----------------|
| 6 | No explicit POST /sessions | Must send first message to create session | Low - add endpoint |
| 7 | No iteration count in response | Cannot show "LLM was called N times" | Low - add to /chat response |
| 8 | No LLM latency per call | Cannot show timing breakdown | Low - track in loop |
| 9 | No timestamp in /chat response | Must use client time | Trivial |
| 10 | Tools list requires parsing agent | Inconvenient for UI | Low - add to session response |

### 🟢 Nice-to-Have

| # | Gap | Impact |
|---|-----|--------|
| 11 | Compaction visibility | Debug-only, low priority |
| 12 | Retry attempt visibility | Debug-only, low priority |
| 13 | Context window % used | Nice visualization |

---

## Recommended Implementation Order

### Phase 1: Essential Debug Data (No Real-Time)
1. Add `token_count` to messages table + track per message
2. Add `total_tokens` to sessions table + accumulate
3. Add `iterations`, `durationMs`, `timestamp` to POST /chat response
4. Add explicit `POST /sessions` endpoint

### Phase 2: Real-Time Events
5. Add `GET /events` SSE endpoint that forwards eventBus
6. Or add `GET /sessions/:id/events` for session-scoped SSE

### Phase 3: Streaming
7. Add `POST /agents/:name/chat/stream` with SSE response
8. Integrate OpenAI streaming