# Copyright 2026 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # https://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================= # SIMULATION EVALS # # WHY sims (not goldens): # - These flows are LLM-driven and vary naturally between runs # - The agent's exact phrasing depends on tool responses and context # - Sims test behavioral outcomes, not exact text # # KEY PATTERNS DEMONSTRATED: # 1. tags is REQUIRED — sim runner filters by --priority P0/P1/P2 # 2. response_guide includes auth credentials for the sim user to provide # 3. expectations use behavioral descriptions, not function names # (the LLM judge sees resource IDs, not display names) # 4. session_parameters left empty for variables derived by callbacks # 5. max_turns set to reasonable limit per flow complexity # 6. Multi-step sims test sequential goals in a single conversation # (e.g., authenticate → ask question → follow up with second topic) # 7. naturalness_metric is OPTIONAL and opt-in — omit it entirely and the # eval behaves exactly as before (see the last eval in this file) # ============================================================================= evals: # --- P0: Critical flows --- - name: authenticated_billing_inquiry tags: [P0, billing, CUJ-1] steps: - goal: Ask about current account balance after authenticating success_criteria: Agent authenticates the caller and provides billing information from a tool lookup response_guide: > You are a customer calling about your bill. When asked to verify your identity, provide account ID 9820598207 and customer ID 4444444. Then ask "What's my current balance?" max_turns: 10 expectations: - "The agent must verify the caller's identity before providing account details" - "The agent must call a tool to look up the account or billing information" - "The agent must provide the balance amount from the tool response, not a made-up number" session_parameters: account_id: "9820598207" customer_id: "4444444" - name: escalation_after_failed_auth tags: [P0, auth, escalation] steps: - goal: Attempt to authenticate with wrong credentials and get escalated success_criteria: Agent attempts authentication, fails, and transfers to a human agent response_guide: > You are a customer who doesn't know their account details. When asked for account ID, say "I don't have it." When asked for any other verification, give wrong information. After a few failed attempts, ask to speak to a person. max_turns: 12 expectations: - "The agent must attempt to verify the caller's identity" - "The agent must NOT reveal any account details without successful authentication" - "The agent must eventually escalate or transfer the caller to a human agent" session_parameters: {} # --- P1: Important flows --- - name: troubleshooting_flow tags: [P1, troubleshooting, CUJ-2] steps: - goal: Report a service issue and get troubleshooting help success_criteria: Agent authenticates, routes to troubleshooting, and attempts to diagnose the issue response_guide: > You are a customer with a service problem. Provide account ID 9820598207 and customer ID 4444444 when asked. Describe your issue as "my internet has been really slow since yesterday." Follow the agent's troubleshooting steps cooperatively. max_turns: 14 expectations: - "The agent must authenticate the caller before troubleshooting" - "The agent must route to troubleshooting or attempt to diagnose the issue" - "The agent must call a diagnostic or lookup tool to check for issues" session_parameters: account_id: "9820598207" customer_id: "4444444" # --- Multi-step sim: tests sequential goals in one conversation --- - name: billing_then_troubleshooting tags: [P1, billing, troubleshooting, multi-step] steps: - goal: Authenticate and ask about your current balance success_criteria: Agent verifies identity and provides account balance response_guide: > You are a customer with two questions. First, provide account ID 9820598207 and customer ID 4444444 when asked. Then ask "What's my current balance?" max_turns: 8 - goal: After getting your balance, report a service issue success_criteria: Agent acknowledges the new topic and begins troubleshooting or routes appropriately response_guide: > Now say "I also have another issue — my internet has been really slow since yesterday. Can you help with that too?" Cooperate with any troubleshooting steps the agent suggests. max_turns: 10 expectations: - "The agent must authenticate the caller before providing any account details" - "The agent must call a tool to look up billing or account information" - "The agent must provide the balance from the tool response" - "The agent must acknowledge the second topic without requiring re-authentication" - "The agent must route to troubleshooting or attempt to diagnose the service issue" session_parameters: account_id: "9820598207" customer_id: "4444444" # --- Voice sim with the OPTIONAL Naturalness Metric --- # # The naturalness_metric block is opt-in. Every eval above omits it and # behaves exactly as before: no extra Gemini call, no extra result keys. # Add it to voice evals where sounding human matters as much as task # success — Gemini then grades every agent turn as Bot-like / # Transitional / Human-like and reports an overall 1-5 score. # # Shorthand: `naturalness_metric: true` enables it with all defaults. - name: upset_caller_billing_dispute tags: [P0, voice, naturalness, CUJ-3] naturalness_metric: # false keeps the block in the file but skips grading. Default: true enabled: true # Grading model. Omit to reuse the simulation's eval_model model: "gemini-3.1-pro-preview" # Override the graded turn-level dimensions. Omit (or leave empty) for # the defaults: emotion, pacing, grammarStyle, disfluency, # lexicalVariety, concision, spokenNumbers, empathyCalibration, # scriptedness, turnTaking turn_qualities: [emotion, pacing, grammarStyle, disfluency, empathyCalibration, spokenNumbers] # Conversation-level dimensions. Defaults: personaConsistency, # repetitionAvoidance, emotionalArcTracking, conversationalFlow conversation_qualities: [personaConsistency, emotionalArcTracking, repetitionAvoidance] # Free-text rubric addendum — brand voice, locale, tone rules extra_guidance: > This caller is already angry. The agent must hold a calm, unhurried register for the rest of the call, even once the caller goes quiet and transactional. One apology only — repeated apologies read as robotic. Amounts and dates must be spoken aloud ("a hundred and fifty dollars", "Thursday the ninth"), never as 150.00 USD or 2026-04-09. # overall = turn_weight * mean(turn scores) # + (1 - turn_weight) * mean(conversation factor scores) # Default: 0.75 (clamped to 0.0-1.0) turn_weight: 0.8 # Label bands. Defaults: 2.5 and 3.75 bot_like_below: 2.5 human_like_at_or_above: 4.0 # Omit pass_threshold to keep the metric purely informational. # When set, the sim FAILS if the overall score falls below it. pass_threshold: 3.5 # Attach captured agent WAVs so tone and pacing are judged acoustically. # Requires running the sim with capture_agent_audio=True. Default: false use_audio: false # Show tool calls in the graded transcript so "let me just check # that..." is judged in context. Default: true include_tool_calls: true steps: - goal: Complain angrily about an incorrect charge and get it refunded success_criteria: Agent acknowledges the incorrect charge, looks up the account, and offers a refund response_guide: > You are a customer who has been charged twice for the same monthly plan. Open angry — "I have been billed twice and nobody told me." Provide account ID 9820598207 and customer ID 4444444 when asked, but stay annoyed. Once the agent starts helping, go quiet and transactional: short answers like "okay" and "yeah, that's right." max_turns: 12 - goal: Confirm the refund and end the call success_criteria: Agent confirms the refund amount and timing and closes the call response_guide: > Accept the first reasonable resolution offered. Do not soften your tone much — you are still mildly irritated, just cooperative. max_turns: 6 expectations: - "The agent must verify the caller's identity before discussing the account" - "The agent must call a tool to look up the account or billing history" - "The agent must state a specific refund amount and when it will land" - "The agent must not repeat the same apology or empathy line more than once" session_parameters: account_id: "9820598207" customer_id: "4444444"