---
title: Debugging
description: Inspect runs, diagnose failures, and query internal state using the CLI, logs, and SQLite.
---

Three levels of inspection: CLI commands, NDJSON log files, and direct SQLite queries.

All CLI examples on this page use `bunx smithers-orchestrator ...`. The published package is `smithers-orchestrator`, not `smithers`.

## CLI Inspection

### inspect

```bash
bunx smithers-orchestrator inspect <id>
```

Shows run metadata, node statuses, approvals, and loop state.

### logs

```bash
bunx smithers-orchestrator logs <id> --tail 10
```

Shows persisted lifecycle events. Add `--follow` to keep tailing live events. For raw render frames, query `_smithers_frames` or use [renderFrame](/runtime/render-frame).

### chat

```bash
bunx smithers-orchestrator chat <id> --tail 5
```

Shows the most recent agent chat blocks for a run. Omit `<id>` to inspect the latest run.

### why

```bash
bunx smithers-orchestrator why <id>
```

Explains why a run is currently blocked or paused.

### node

```bash
bunx smithers-orchestrator node <node-id> --run-id <id>
```

Shows enriched node details for retries, tool calls, and latest output for a specific task.

### ps

```bash
bunx smithers-orchestrator ps --limit 20
bunx smithers-orchestrator ps --status running
bunx smithers-orchestrator ps --status failed
```

### graph

```bash
bunx smithers-orchestrator graph workflow.tsx --run-id <id>
bunx smithers-orchestrator graph workflow.tsx --input '{"description": "Fix bugs"}'
```

Shows task dependency structure. The second form previews the graph without running.

## NDJSON Logs

```
.smithers/executions/<runId>/logs/stream.ndjson
```

Each line is a JSON-encoded `SmithersEvent`.

```bash
tail -f .smithers/executions/<runId>/logs/stream.ndjson
```

Filter by event type:

```bash
grep '"type":"Node' .smithers/executions/<runId>/logs/stream.ndjson
grep '"type":"NodeFailed"' .smithers/executions/<runId>/logs/stream.ndjson
grep '"type":"ToolCall' .smithers/executions/<runId>/logs/stream.ndjson
```

Parse with `jq`:

```bash
tail -5 .smithers/executions/<runId>/logs/stream.ndjson | jq .
cat .smithers/executions/<runId>/logs/stream.ndjson | jq 'select(.type == "NodeFinished" or .type == "NodeFailed") | {nodeId, type}'
```

Custom log location or disable:

```bash
bunx smithers-orchestrator up workflow.tsx --log-dir ./my-logs
bunx smithers-orchestrator up workflow.tsx --log false
```

## SQLite Inspection

### Internal Tables

| Table | Purpose |
|---|---|
| `_smithers_runs` | Run metadata: status, timestamps |
| `_smithers_nodes` | Per-node state: status, iteration, attempt count |
| `_smithers_attempts` | Per-attempt: status, times, errors, JJ pointer |
| `_smithers_frames` | Render frame snapshots (XML of JSX tree) |
| `_smithers_approvals` | Approval decisions |
| `_smithers_cache` | Cached task results |
| `_smithers_tool_calls` | Tool invocations: name, args, result, duration |
| `_smithers_events` | All events with sequence numbers and JSON payloads |
| `_smithers_ralph` | Loop iteration state |

### Queries

```bash
# Run status
sqlite3 smithers.db "SELECT run_id, status, created_at_ms, updated_at_ms FROM _smithers_runs ORDER BY created_at_ms DESC LIMIT 5;"

# Node states
sqlite3 smithers.db "SELECT node_id, status, iteration FROM _smithers_nodes WHERE run_id = '<id>' ORDER BY updated_at_ms;"

# Failed attempts
sqlite3 smithers.db "SELECT node_id, attempt, status, error_message FROM _smithers_attempts WHERE run_id = '<id>' AND status = 'failed';"

# Tool calls
sqlite3 smithers.db "SELECT tool_name, arguments, duration_ms FROM _smithers_tool_calls WHERE run_id = '<id>' AND node_id = '<node-id>';"

# Events
sqlite3 smithers.db "SELECT seq, type, payload_json FROM _smithers_events WHERE run_id = '<id>' ORDER BY seq LIMIT 50;"

# Approvals
sqlite3 smithers.db "SELECT node_id, approved, decided_by, note FROM _smithers_approvals WHERE run_id = '<id>';"

# Loop state
sqlite3 smithers.db "SELECT * FROM _smithers_ralph WHERE run_id = '<id>';"

# Output tables
sqlite3 smithers.db "SELECT * FROM analysis WHERE run_id = '<id>';"
sqlite3 smithers.db "SELECT * FROM report WHERE run_id = '<id>';"
```

## Common Issues

### Stuck at "waiting-approval"

```bash
bunx smithers-orchestrator why <id>
bunx smithers-orchestrator inspect <id>
bunx smithers-orchestrator approve <id> --node <node-id>
# or: bunx smithers-orchestrator deny <id> --node <node-id>
bunx smithers-orchestrator up workflow.tsx --run-id <id> --resume
```

### Duplicate task IDs

```
Error: Duplicate Task id detected: "analyze"
```

Every `<Task>` needs a globally unique `id`. For dynamic tasks, derive IDs from unique identifiers: `id={`${item.id}:process`}`.

### Missing output rows

`ctx.output()` throws "No output found" when the task has not completed, failed, or its `id` changed between renders. Use `ctx.outputMaybe()` for conditional access:

```tsx
const result = ctx.outputMaybe("analysis", { nodeId: "analyze" });
if (result) {
  // safe to use
}
```

### Task keeps retrying

Check for schema validation errors, timeouts, or tool failures:

```bash
bunx smithers-orchestrator node <node-id> --run-id <id> --attempts --tools
grep "NodeRetrying" .smithers/executions/<runId>/logs/stream.ndjson | jq .
sqlite3 smithers.db "SELECT node_id, attempt, error_message FROM _smithers_attempts WHERE run_id = '<id>' AND status = 'failed';"
```

### Stale in-progress tasks

Tasks in-progress for over 15 minutes are auto-cancelled and retried on resume:

```bash
bunx smithers-orchestrator up workflow.tsx --run-id <id> --resume
```

## Next Steps

- [Monitoring & Logs](/guides/monitoring-logs) -- Live monitoring with events and SSE.
- [Resumability](/guides/resumability) -- How resume handles stale tasks.
- [CLI Reference](/cli/overview) -- All CLI commands and options.