--- name: evaluations description: Evaluate Indemn AI agents — create rubrics and test sets, run evaluations, analyze results, and export reports. Use for any evaluation, testing, or quality assessment task. --- # Evaluations Complete evaluation workflow for Indemn AI agents. Covers rubric creation, test set design, evaluation execution, result analysis, and reporting. **Prerequisite:** `indemn whoami` must succeed. The `indemn` CLI must be installed and authenticated. --- ## Mode Detection Check the FIRST LINE of the user's message: **HEADLESS** -- Execute autonomously without asking questions. Optimize for completion. **INTERACTIVE** -- Get explicit approval before every action that creates, modifies, or triggers anything. Approval gates: before creating anything, after creation (review?), before triggering a run, before modifying resources. --- ## Evaluation Harness Overview The harness tests conversational AI agents against quality standards through two scoring layers: | Layer | Source | Applies To | Purpose | |-------|--------|------------|---------| | **Success Criteria** | Embedded in each test item | That specific item | Did the agent handle THIS situation correctly? | | **Rubric Rules** | Separate rubric document | ALL items universally | Does the agent meet baseline behavioral standards? | An item passes only if BOTH layers pass. A high-severity rubric rule failure causes the entire item to fail. ### Three Test Types | Type | Input | What It Tests | |------|-------|---------------| | `single_turn` | A single message (new conversation, no prior context) | Specific directive compliance, edge cases | | `scenario` | Persona + initial message + max turns | End-to-end workflow completion (chat agents) | | `voice_simulation` | Persona + initial message + max turns (via LiveKit) | End-to-end voice workflow | --- ## The Directive Allocation Workflow **Run this BEFORE creating rubric or test set.** This is the core method for deciding what goes where. ### Step 1: Fetch Bot Context ```bash indemn eval bot-context --json ``` Extract the system prompt, tools, and knowledge bases. ### Step 2: Parse All Directives Identify every ALWAYS/NEVER/MUST/CRITICAL directive from the system prompt. List them all. ### Step 3: Apply the 3-Message Universal Applicability Test For EACH directive, mentally test it against these three messages: 1. **Greeting:** "Hello, I'm interested in wedding insurance" 2. **Off-topic:** "Can you help me with my taxes?" 3. **Utility:** "I need to correct my phone number" Ask: "Would I check this directive against ALL THREE of these messages?" ### Step 4: Classify Each Directive | Classification | Rule | Where It Goes | |---------------|------|---------------| | **UNIVERSAL** | Passes the 3-message test | Rubric rule | | **WORKFLOW-SPECIFIC** | Only applies in certain situations | Test set scenario success criteria | | **AMBIGUOUS** | Unclear whether universal | Default to scenario criteria | ### Step 5: Produce Allocation Table ``` | Directive | Classification | Destination | |-----------|---------------|-------------| | "ALWAYS maintain professional tone" | UNIVERSAL | Rubric rule | | "MUST collect DOB before quoting" | WORKFLOW-SPECIFIC | Scenario criteria | | "NEVER provide medical advice" | UNIVERSAL | Rubric rule | ``` ### Step 6: Create Artifacts Using the Allocation - Create the rubric using ONLY universal directives (see Rubric Creation below) - Create the test set mapping workflow-specific directives as scenario success criteria (see Test Set Creation below) --- ## Rubric Creation Rubrics define the agent's universal behavioral contract -- rules checked against EVERY response. ### Hard Cap: 3-6 Rules NEVER exceed 6 rules. A tight rubric with precise universal rules is better than a broad one with overlapping or situational rules. If you find yourself writing more than 6, you are including workflow-specific rules that belong in the test set. ### Process 1. **Fetch bot context** (if not already done): `indemn eval bot-context --json` 2. **Extract universal rules from the system prompt:** - Persona and tone -- How must the agent communicate in every response? - Universal prohibitions (NEVER/DO NOT) -- Things the agent must never do in ANY response - Universal requirements (ALWAYS/MUST) -- Things the agent must do in EVERY response 3. **Add standard quality baselines** (include each unless a prompt-derived rule already covers it): - No hallucination: Does not fabricate facts, statistics, or capabilities - Stays in scope: Operates within its defined role - No harmful content: Does not produce harmful, dangerous, or inappropriate content - Response coherence: Responses are clear, grammatically correct, and logically structured 4. **Verify each rule passes the 3-message self-check** (greeting, off-topic, utility) 5. **Write JSON and create:** ```bash indemn rubric create --name " Rubric" --agent-id --file /tmp/rubric.json ``` ### Rubric JSON Format ```json { "rules": [ { "id": "SNAKE_CASE_ID", "name": "Human-readable name", "severity": "high | medium | low", "category": "persona | instruction_compliance | safety | response_quality", "description": "What this rule checks and why it matters", "component_scope": "prompt | general | function | knowledge_base", "component_ids": null, "component_names": null, "evaluation_criteria": { "pass_conditions": ["Observable condition that indicates compliance"], "fail_conditions": ["Observable condition that indicates violation"] } } ] } ``` **component_scope values:** | Scope | Meaning | Example | |-------|---------|---------| | `prompt` | Derived from a system prompt directive | "NEVER provide medical advice" | | `general` | Standard quality/safety baseline | No hallucination, response coherence | | `function` | Informed by the agent's tool capabilities | "Does not claim capabilities it doesn't have" | | `knowledge_base` | Informed by the agent's KBs | "Grounds information rather than fabricating" | ### Severity Assignment ``` Could violation cause harm to the user? -> HIGH Does the prompt use ALWAYS/NEVER/MUST for this? -> HIGH Could violation severely damage trust? -> HIGH Does it significantly degrade response quality? -> MEDIUM Is it a tone, style, or formatting preference? -> LOW ``` Target distribution: High 20-30%, Medium 40-50%, Low 20-30%. ### Anti-Patterns - Do NOT derive rubric rules from tool descriptions -- tools inform domain understanding but tool-specific behaviors are workflow-specific by definition - Do NOT include workflow-specific rules (e.g., "collects DOB before quoting") - Do NOT write vague conditions -- every pass/fail condition must be observable from the agent's response text --- ## Test Set Creation Test sets verify that an agent can do its job -- following its workflow, using its tools, handling real-world user behavior. ### Size Guidance Aim for roughly **15 items total**. A focused test set with 12-18 precise items is far more useful than 40+ redundant ones. Consolidate items that test the same behavior. ### Process 1. **Fetch bot context** (if not already done): `indemn eval bot-context --json` 2. **Detect voice vs chat agent** -- Check for voice indicators in tools (voice, telephony, livekit, twilio, call, phone) or system prompt. If detected, use `voice_simulation` instead of `scenario` for multi-turn items. 3. **Analyze workflow depth for max_turns:** | Workflow Steps | Recommended max_turns | |---------------|----------------------| | 2-3 steps | 4-6 turns | | 4-5 steps | 6-8 turns | | 6+ steps | 8-12 turns | 4. **Create multi-turn scenarios** (primary content) for each major workflow path with varied personas: - **Happy path** -- Cooperative user following the workflow naturally - **Confused user** -- Misunderstands questions, gives unclear answers - **Resistant user** -- Pushes back on providing information - **Incomplete information** -- Gives partial answers - **Scope boundary** -- Starts in-scope, drifts out of domain 5. **Map workflow-specific directives to success criteria** on relevant scenarios 6. **Create single-turn diagnostics** for ALWAYS/NEVER directives testable standalone, edge cases, and tool outcome tests 7. **Write JSON and create:** ```bash indemn testset create --name " Test Set" --agent-id --file /tmp/testset.json ``` ### Test Set JSON: Scenario ```json { "type": "scenario", "name": "Descriptive name - what this tests", "inputs": { "persona": "You are a [type] who [traits, emotional state, behavior pattern, goal].", "initial_message": "Natural first message matching the persona", "max_turns": 8 }, "expected": { "success_criteria": ["Criterion about the conversation outcome"], "expected_outcome": "What the conversation should achieve" }, "tags": ["scenario", "category"], "priority": "high" } ``` ### Test Set JSON: Voice Simulation Identical to scenario except `"type": "voice_simulation"`. The harness routes these through LiveKit with TTS/STT. ```json { "type": "voice_simulation", "name": "Descriptive name - what this tests", "inputs": { "persona": "You are a caller who [traits]. You are on a phone call -- keep responses brief and natural.", "initial_message": "Natural first utterance", "max_turns": 6 }, "expected": { "success_criteria": ["Criterion about the conversation outcome"], "expected_outcome": "What the call should achieve" }, "tags": ["voice_simulation", "category"], "priority": "high" } ``` Voice-specific: personas should behave like phone callers (shorter utterances, less formality). max_turns generally shorter (4-6 simple, 6-8 complex). Do NOT create single_turn items for voice agents. ### Test Set JSON: Single Turn **Single-turn tests start a BRAND NEW conversation with NO prior context.** Design messages that are standalone. ```json { "type": "single_turn", "name": "Descriptive name - what this tests", "inputs": { "message": "The standalone message to send" }, "expected": { "success_criteria": ["Specific observable criterion"] }, "tags": ["category"], "priority": "high" } ``` **CRITICAL:** The `inputs` object for single-turn items contains ONLY `message`. No `max_turns`, `persona`, or `initial_message`. ### Success Criteria Rules Every criterion MUST be observable from the agent's output: - GOOD: "Agent asks for the user's date of birth before presenting a quote" - GOOD: "Response contains specific policy coverage details, not generic information" - BAD: "Agent uses the knowledge base search tool" (mechanism, not outcome) - BAD: "Agent handles the request appropriately" (vague) Rules: test ONE specific behavior, use concrete verbs ("asks for", "provides", "does not", "contains", "redirects"), 2-4 criteria per item, do NOT duplicate rubric rules as criteria. ### Session Init Limitation Scenarios start as fresh conversations with no pre-existing session state. Do NOT create items that depend on session-initialized data. Note session-dependent behaviors as coverage gaps in the completion report. ### Priority Assignment - **high**: Core workflow completion, safety, explicit prompt directives - **medium**: Secondary behaviors, edge cases, tool outcomes - **low**: Style preferences, formatting, nice-to-have behaviors --- ## Running Evaluations ### Trigger a Run ```bash indemn eval run --test-set --rubric --wait ``` The `--wait` flag polls until completion and shows a pass/fail summary. **Two models are involved in evaluations:** - **Agent model** — set via `indemn config set --model`. This is the LLM the agent uses to generate responses during the eval. - **Judge model** — set via `--eval-model` below. This is the LLM that scores the agent's responses against rubric rules and success criteria. **Key flags:** | Flag | Purpose | Notes | |------|---------|-------| | `--wait` | Poll until complete, show summary | Recommended for all runs | | `--concurrency ` | Parallel items | Use `1` for voice_simulation | | `--eval-model ` | Judge LLM (e.g., "gpt-4o") | Scores responses; does NOT change the agent's model | | `--keep` | Preserve intermediate artifacts | Useful for debugging | | `--limit ` | Run only first N items | Good for quick smoke tests | | `--component-scope ` | Filter by component scope | prompt, function, knowledge_base, general | | `--component-ids ` | Filter by component IDs | Comma-separated | | `--conversation-ids ` | Filter by conversation IDs | Comma-separated | | `--mode ` | Evaluation mode | Use "transcript" for transcript-based evaluation | ### Check Status ```bash indemn eval status [--json] ``` ### List Runs ```bash indemn eval list [--agent-id ] [--json] [--limit ] [--page ] ``` --- ## Analyzing Results ### Step 1: Resolve the Run - **UUID format** (contains hyphens, 36 chars) -- it is a `run_id`. Use directly. - **ObjectId format** (24 hex chars) -- it is an `agent_id`. Fetch the latest completed run: ```bash indemn eval list --agent-id --json --limit 5 ``` ### Step 2: Pull All Data ```bash indemn eval status --json indemn eval results --json indemn rubric get --json indemn testset get --json indemn eval bot-context --json ``` For exact response shapes, see `references/data-shapes.md`. ### Step 3: Present Run Overview ``` Run: {run_id} Agent: {bot_name} ({agent_id}) LLM: {bot_llm_provider}/{bot_llm_model} Status: {passed}/{total} items passed ({pass_rate}%) Criteria: {criteria_passed}/{criteria_total} checks passed Rubric: {rubric_rules_passed}/{rubric_rules_total} rules passed Component Scores: prompt: {score}% ({passed}/{total}) knowledge_base: {score}% ({passed}/{total}) function: {score}% ({passed}/{total}) general: {score}% ({passed}/{total}) ``` ### Step 4: Analyze Each Failed Item For every result where `passed == false`: 1. **Get the test item** -- match `test_case_id` to the test set's `items[].item_id` for name, type, inputs, success_criteria 2. **Check criteria scores** -- from `criteria_scores[]`, list which passed/failed with reasoning 3. **Check rubric scores** -- from `rubric_scores[]`, list which passed/failed with reasoning 4. **Check for divergence** -- flag when `criteria_passed != rubric_passed`. Divergence is a strong signal of an evaluation problem. 5. **Read the conversation** -- look at `output` field for the full response or transcript 6. **Cross-reference system prompt** -- check if the agent's behavior aligns with prompt instructions 7. **Classify** into one of four categories: | Category | Signal | Action | |----------|--------|--------| | **Real Agent Problem** | Both layers agree agent misbehaved, or output contradicts prompt | Investigate agent code/prompt | | **Evaluation Problem** | Criteria ambiguous/too strict, evaluator reasoning shows misunderstanding | Update test set criteria | | **Test Design Problem** | Test assumptions wrong (e.g., assumed KB lacked info it has) | Redesign test item | | **KB/Retrieval Problem** | Agent behavior correct but retrieval returned wrong content | Fix KB content or retrieval config | ### Divergence Analysis When `criteria_passed != rubric_passed` for an item, this is a first-class finding: - **Criteria FAIL + Rubric PASS** -- criteria are likely too strict or testing the wrong thing - **Criteria PASS + Rubric FAIL** -- rubric may have rules that don't apply to this scenario, or agent violated a behavioral rule despite answering correctly Always highlight divergences prominently. ### Step 5: Aggregate and Recommend Group failures by category. Prioritize fixes: - **P1 -- Evaluation fixes** (false failures mask real issues): specific criteria rewrites or rubric rule adjustments - **P2 -- Agent investigation** (real bugs): which code paths or prompt sections to examine - **P3 -- KB/Retrieval fixes**: which KBs need content updates or retrieval tuning For each recommendation, quote the current criterion/rule, explain what is wrong, and propose the exact replacement text. ### Step 6: Apply Fixes Updates create new versions (non-destructive, previous versions preserved): ```bash indemn testset update --file updated-testset.json indemn rubric update --file updated-rubric.json ``` Check version history: `indemn testset versions ` / `indemn rubric versions ` After applying fixes, re-run: ```bash indemn eval run --test-set --rubric --wait ``` --- ## Exporting Results ### Markdown Export ```bash indemn eval export [--output results.md] ``` Produces a structured markdown file with all conversations, criteria scores, and rubric scores. ### PDF Report ```bash indemn eval report [--output report.pdf] ``` Produces a branded Indemn PDF with cover page, dual scores (criteria + rubric), and per-item detail pages. --- ## Workflows ### A: New Evaluation (End-to-End) 1. Gather intent -- which agent, any focus areas 2. Fetch bot context: `indemn eval bot-context --json` 3. Run directive allocation (Steps 1-6 above) 4. Create rubric from universal directives: `indemn rubric create --name "" --agent-id --file /tmp/rubric.json` 5. Create test set with workflow-specific criteria: `indemn testset create --name "" --agent-id --file /tmp/testset.json` 6. Review (interactive: wait for approval) 7. Run: `indemn eval run --test-set --rubric --wait` (add `--concurrency 1` for voice_simulation) 8. Analyze results: `indemn eval results --json` ### B: Re-run with Existing Resources 1. List existing: `indemn rubric list --agent-id `, `indemn testset list --agent-id ` 2. Run: `indemn eval run --test-set --rubric --wait` ### C: Update Existing Resources 1. Get current: `indemn rubric get --json` / `indemn testset get --json` 2. Write updated JSON to file 3. Update: `indemn rubric update --file /tmp/updated.json` / `indemn testset update --file /tmp/updated.json` ### D: Analyze Results Only 1. Resolve run (UUID or agent-id) 2. Follow the Analyzing Results workflow above ### E: Export/Share Results 1. Markdown: `indemn eval export [--output path.md]` 2. PDF: `indemn eval report [--output path.pdf]` --- ## CLI Reference | Command | Purpose | |---------|---------| | `indemn eval bot-context [--json]` | Get agent prompt, tools, KBs, LLM config | | `indemn eval run --test-set [--rubric ] [--wait]` | Trigger an evaluation run | | `indemn eval list [--agent-id ] [--json] [--limit] [--page]` | List evaluation runs | | `indemn eval status [--json]` | Get run status and summary | | `indemn eval results [--json]` | Get per-item detailed results | | `indemn eval export [--output ]` | Export results as markdown | | `indemn eval report [--output ]` | Export results as branded PDF | | `indemn rubric create [--file ] [--name ""] [--agent-id ]` | Create rubric (--file or --name required) | | `indemn rubric get [--json] [--revision ]` | Get rubric details | | `indemn rubric list [--agent-id ] [--json] [--limit] [--page]` | List rubrics | | `indemn rubric update [--file ] [--name ""]` | Update rubric (creates new version) | | `indemn rubric delete ` | Delete rubric (requires confirmation) | | `indemn rubric versions [--json]` | List rubric version history | | `indemn testset create [--file ] [--name ""] [--agent-id ]` | Create test set (--file or --name required) | | `indemn testset get [--json] [--revision ]` | Get test set details | | `indemn testset list [--agent-id ] [--json] [--limit] [--page]` | List test sets (always use --agent-id) | | `indemn testset update [--file ] [--name ""]` | Update test set (creates new version) | | `indemn testset delete ` | Delete test set (requires confirmation) | | `indemn testset versions [--json]` | List test set version history | | `indemn config get --field prompt` | Get just the system prompt | | `indemn functions list ` | List agent's functions |