# Evaluators

Evaluators define how agent output is scored. Each evaluator is a JSON file in `evals/evaluators/`.

## Supported Evaluator Types

Low-code agents support exactly four evaluator types. All four are first-class options in the Studio Web "Add evaluator" dialog. Two also have CLI-flag shortcuts; the other two are created via the UI or by hand-writing JSON in `evals/evaluators/`.

| UI label | `type` | `category` | `--type` flag | What it scores | LLM-based |
|----------|--------|-----------|---------------|----------------|-----------|
| LLM-as-a-judge: Semantic Similarity | 5 | 1 (LlmAsAJudge) | `semantic-similarity` | Whether the agent's output has the same meaning as the expected output | Yes |
| Trajectory | 7 | 3 (Trajectory) | `trajectory` | Whether the agent's reasoning path and tool usage match expected behavior | Yes |
| Exact match | 1 | 0 (Deterministic) | — | Whether the output precisely matches the expected output without variations in wording or formatting | No |
| JSON similarity | 6 | 0 (Deterministic) | — | Whether two JSON structures or values are "close enough" or share similar structure/contents | No |

How to add each type:

- **Studio Web UI** — Evaluators tab → **Create New** → Add evaluator dialog → pick any of the four. UI is the canonical surface and supports all four with no special steps.
- **CLI** — `uip agent eval evaluator add <name> --type <flag>` for `semantic-similarity` or `trajectory`. The CLI does not have a `--type` value for Exact match or JSON similarity; create those in the UI or hand-write the JSON.
- **Hand-write JSON** — drop a file in `evals/evaluators/` matching the schema below; run `uip agent refresh --output json` then `uip agent validate --output json` to confirm schema validity; reference the new `id` from your eval set's `evaluatorRefs`. Useful when you want to pin a specific model and prompt for the LLM-based types, or when you're scaffolding eval files programmatically.

### Why fewer than coded?

The coded eval reference ([coded/lifecycle/evaluations/evaluators.md](../../coded/lifecycle/evaluations/evaluators.md)) lists 13 evaluator types. Low-code supports only these four because the two surfaces use **different engines** in the SDK:

- **Coded** uses the new evaluator hierarchy (`BaseEvaluator`, eval sets carry `version: "1.0"`). 13 distinct `evaluatorTypeId` strings, each with its own implementation class.
- **Low-code** uses the **legacy** evaluator hierarchy (`BaseLegacyEvaluator`, no `version` field on the eval set). The four legacy classes shipped — `LegacyLlmAsAJudgeEvaluator`, `LegacyTrajectoryEvaluator`, `LegacyExactMatchEvaluator`, `LegacyJsonSimilarityEvaluator` — are exactly what the UI exposes.

Most coded evaluator types (`contains`, `binary-classification`, `multiclass-classification`, all four `tool-call-*`, `llm-judge-output-strict-json-similarity`, `llm-judge-trajectory-simulation`) have no legacy counterpart and cannot be used on a low-code agent.

## JSON Shapes

For hand-written files, the filename can be any descriptive name (e.g. `legacy-equality.json`) — the runtime keys off `id` / `evaluatorRefs`, not the filename. The CLI-generated `evaluator-<uuid8>.json` pattern only applies to evaluators created via `uip agent eval evaluator add`.

### Exact match (`type` 1, `category` 0 — Deterministic)

No LLM. Equivalent of coded `uipath-exact-match`.

```json
{
  "fileName": "legacy-equality.json",
  "id": "<generate-uuid>",
  "name": "Equality Evaluator",
  "description": "An evaluator that judges the agent based on expected output.",
  "category": 0,
  "type": 1,
  "targetOutputKey": "*",
  "createdAt": "<iso-timestamp>",
  "updatedAt": "<iso-timestamp>"
}
```

No `prompt`/`model` required (Deterministic category bypasses the LLM checks).

### JSON similarity (`type` 6, `category` 0 — Deterministic)

Tree-based JSON comparison. No LLM. Equivalent of coded `uipath-json-similarity`.

```json
{
  "fileName": "legacy-json-similarity.json",
  "id": "<generate-uuid>",
  "name": "JSON Similarity Evaluator",
  "description": "An evaluator that compares JSON structures with tolerance for numeric and string differences.",
  "category": 0,
  "type": 6,
  "targetOutputKey": "*",
  "createdAt": "<iso-timestamp>",
  "updatedAt": "<iso-timestamp>"
}
```

### LLM-as-a-judge: Semantic Similarity (`type` 5, `category` 1 — LlmAsAJudge)

The CLI's `evaluator add --type semantic-similarity` writes a shorter prompt; hand-write the file when you want to pin a specific model and the longer 0–100 prompt:

```json
{
  "fileName": "legacy-llm-as-a-judge.json",
  "id": "<generate-uuid>",
  "name": "LLM As A Judge Evaluator",
  "description": "An evaluator that uses an LLM to judge the similarity of the actual output to the expected output",
  "category": 1,
  "type": 5,
  "prompt": "As an expert evaluator, analyze the semantic similarity of these outputs to determine a score from 0-100.\n----\nExpectedOutput:\n{{ExpectedOutput}}\n----\nActualOutput:\n{{ActualOutput}}\n",
  "targetOutputKey": "*",
  "model": "gpt-4.1-2025-04-14",
  "createdAt": "<iso-timestamp>",
  "updatedAt": "<iso-timestamp>"
}
```

### Trajectory (`type` 7, `category` 3 — Trajectory)

```json
{
  "fileName": "legacy-trajectory.json",
  "id": "<generate-uuid>",
  "name": "Trajectory Evaluator",
  "description": "An evaluator that analyzes the execution trajectory and decision sequence taken by the agent.",
  "category": 3,
  "type": 7,
  "prompt": "Evaluate the agent's execution trajectory based on the expected behavior.\n\nExpected Agent Behavior: {{ExpectedAgentBehavior}}\nAgent Run History: {{AgentRunHistory}}\n\nProvide a score from 0-100 based on how well the agent followed the expected trajectory.",
  "model": "gpt-4.1-2025-04-14",
  "targetOutputKey": "*",
  "createdAt": "<iso-timestamp>",
  "updatedAt": "<iso-timestamp>"
}
```

After hand-writing any evaluator, run `uip agent refresh --output json` then `uip agent validate --output json` to confirm the file passes schema validation. Then reference the new evaluator's `id` from your eval set's `evaluatorRefs`. Watch for: `id` collisions with existing evaluators, missing required fields, and ISO-8601 formatting on the timestamps.

## Coded-only evaluators (NOT available on low-code)

The following coded `evaluatorTypeId` strings have no legacy class — agents working on a low-code agent should not attempt to use them. Switch to a coded agent (`version: "1.0"` eval sets) if you need any of these:

`uipath-contains`, `uipath-llm-judge-output-strict-json-similarity`, `uipath-llm-judge-trajectory-simulation`, `uipath-binary-classification`, `uipath-multiclass-classification`, `uipath-tool-call-order`, `uipath-tool-call-args`, `uipath-tool-call-count`, `uipath-tool-call-output`.

## Managing Evaluators

### Add an evaluator

```bash
uip agent eval evaluator add <name> --type <type> --path <agent_dir> --output json
```

**Options:**

| Flag | Required | Description | Default |
|------|----------|-------------|---------|
| `--type <type>` | Yes | One of: `semantic-similarity`, `trajectory` | — |
| `--description <desc>` | No | Human-readable description | Auto-generated from type |
| `--prompt <prompt>` | No | Custom LLM evaluation prompt | Built-in default per type |
| `--target-key <key>` | No | Specific output key to evaluate | `*` (all keys) |
| `--path <path>` | No | Agent project directory | `.` |

**Example:**
```bash
uip agent eval evaluator add content-quality \
  --type semantic-similarity \
  --path ./my-agent \
  --output json
```

### List evaluators

```bash
uip agent eval evaluator list --path <agent_dir> --output json
```

### Remove an evaluator

```bash
uip agent eval evaluator remove <id_or_name> --path <agent_dir> --output json
```

Removing an evaluator automatically removes its references from all eval sets that reference it.

## Default Evaluators

`uip agent init` creates two default evaluators:

### Semantic Similarity (`evaluator-default.json`, `name: "Default Evaluator"`)

Compares expected vs actual output for semantic equivalence. Default prompt asks the LLM for a 0–100 score and substitutes `{{ExpectedOutput}}` and `{{ActualOutput}}`.

### Trajectory (`evaluator-default-trajectory.json`, `name: "Default Trajectory Evaluator"`)

Evaluates the agent's reasoning path against expected behavior. Default prompt asks the LLM for a 0–100 score and substitutes `{{UserOrSyntheticInput}}`, `{{SimulationInstructions}}`, `{{ExpectedAgentBehavior}}`, and `{{AgentRunHistory}}`.

Both default evaluators ship with `"model": "same-as-agent"` — this is supported and resolves to the agent's configured model at runtime. Override with an explicit model only if you need to score with a different model than the agent uses.

The runtime DTO normalizes all evaluator scores to a 0–100 scale regardless of what the prompt asks for, but mixed-scale prompts in the same eval set produce confusing intermediate values — pick one scale per eval set.

## Filename vs Name

CLI-added evaluators are saved as `evaluator-<uuid8>.json` (first 8 hex chars of the evaluator UUID). The `<name>` argument populates the `name` field inside the JSON; it does NOT shape the filename.

```bash
uip agent eval evaluator add content-quality --type semantic-similarity --path ./my-agent
# Creates: evals/evaluators/evaluator-b47e26ca.json
# JSON has: "name": "content-quality"
```

The two `evaluator-default*.json` files are written by `uip agent init`, not by `evaluator add`. Eval sets reference evaluators by `id` (UUID), not by filename or name.

## Evaluator JSON Format

```json
{
  "fileName": "evaluator-b47e26ca.json",
  "id": "b47e26ca-7a13-4c83-9ee4-039d6415fb63",
  "name": "content-quality",
  "description": "Semantic Similarity",
  "category": 1,
  "type": 5,
  "prompt": "As an expert evaluator, ... {{ExpectedOutput}} ... {{ActualOutput}} ...",
  "model": "same-as-agent",
  "targetOutputKey": "*",
  "createdAt": "2026-05-04T00:00:00.000Z",
  "updatedAt": "2026-05-04T00:00:00.000Z"
}
```

**Type and category mapping:**

| CLI Type | `type` (numeric) | `category` |
|----------|-------------------|------------|
| `semantic-similarity` | 5 | 1 (output-based) |
| `trajectory` | 7 | 3 (trajectory-based) |

## Default Prompts and Template Variables

The prompt and score scale the CLI writes when you run `evaluator add` differs from what `uip agent init` writes for the two default evaluators:

| Type | `evaluator add` default | `uip agent init` default |
|------|-------------------------|--------------------------|
| `semantic-similarity` | Asks 0–1; uses `{{ExpectedOutput}}`, `{{ActualOutput}}` | Asks 0–100; same placeholders |
| `trajectory` | Asks 0–1; uses `{{AgentRunHistory}}`, `{{ExpectedBehavior}}` | Asks 0–100; uses `{{UserOrSyntheticInput}}`, `{{SimulationInstructions}}`, `{{ExpectedAgentBehavior}}`, `{{AgentRunHistory}}` |

Two notable inconsistencies:

1. **Trajectory placeholder names**: `{{ExpectedBehavior}}` (CLI add) vs `{{ExpectedAgentBehavior}}` (init default). When editing a prompt, use the placeholders already present in that file — do not mix.
2. **Score scales**: `evaluator add` writes 0–1 prompts; `init` writes 0–100 prompts. The runtime normalizes both to 0–100 in the result DTO, but the LLM judge actually returns whatever the prompt asks for. Mixed-scale eval sets are hard to read; pick one and rewrite the prompts you don't want.

## Custom Prompts

Pass `--prompt` to override the default. Use only the placeholders listed above for the chosen `--type`; unknown placeholders are passed through to the LLM as literal text.

```bash
uip agent eval evaluator add strict-match \
  --type semantic-similarity \
  --prompt 'Score 0-100 how closely {{ActualOutput}} matches {{ExpectedOutput}}. Return JSON {"score": N, "reason": "..."}.' \
  --path ./my-agent --output json
```

## What `uip agent validate` Checks

Validate runs schema migration, which enforces the following on every file in `evals/evaluators/`:

**Required fields:** `fileName`, `id`, `name`, `description`, `category`, `type`, `targetOutputKey`, `createdAt`, `updatedAt`. Missing field → `Required field "<field>" is missing`.

**Category ↔ type compatibility:**

| Category | Name | Allowed `type` | Additional requirements |
|----------|------|----------------|-------------------------|
| `0` | Deterministic | `1`, `6` | — |
| `1` | LlmAsAJudge | `5` | `prompt` and `model` required |
| `3` | Trajectory | `7` | `prompt` and `model` required |

Category `2` (`AgentScorer`) exists in the SDK enum but is reserved/unused — do not write it manually.

Eval sets are validated against a Zod schema. The CLI surfaces the offending file path, JSON path, and message — fix and re-run validate.

## Runtime Errors (Eval Worker)

These errors surface only after `uip agent eval run start` — `uip agent validate` does NOT catch them. They come from the cloud eval worker (`python-eval-worker/workflows/eval/activities.py`) and the SDK's `EvaluatorFactory`.

| Error string | Trigger | Fix |
|--------------|---------|-----|
| `Evaluator '<id>' is an LLM-based evaluator but 'model' is not set in its evaluatorConfig. Specify a valid model name (e.g. 'claude-haiku-4-5-20251001').` | Evaluator JSON has empty/missing `model` (and is not `same-as-agent`). The worker fail-fasts before calling the LLM gateway. | Set `model` in the evaluator JSON to a model available in your tenant, or set `"model": "same-as-agent"` and ensure `agent.json` has a model. |
| `'same-as-agent' model option requires agent settings. Ensure agent.json contains valid model settings.` | Evaluator uses `"same-as-agent"` but `agent.json` has no resolvable model. | Set `model` in `agent.json`, or override the evaluator with an explicit model. |

**Pre-empt locally:** before upload, run

```bash
uip agent eval evaluator list --path ./my-agent --output json --output-filter '[?model==`""` || model==null]'
```

to find any LLM evaluator without an explicit model. (Switch to `--output-filter '[?model==`"same-as-agent"`]'` if you want to flag those that depend on `agent.json`.)

## Anti-patterns

- **Don't reference an evaluator by filename.** Eval sets reference evaluators by UUID (`id`).
- **Don't pass `--type` in PascalCase.** Only `semantic-similarity` and `trajectory` are accepted.
- **Don't assume `evaluator add` mirrors `init`'s prompts.** They differ for trajectory; check the resulting JSON before reusing template variables in your own scoring tooling.
- **Don't delete an evaluator file by hand.** Use `uip agent eval evaluator remove` so `evaluatorRefs` in eval sets are cleaned up automatically.
- **Don't copy evaluator JSON across projects without regenerating UUIDs.** `id` collisions silently corrupt cross-project resolution.
- **Don't try to add a coded-only evaluator type to a low-code agent.** Anything starting with `uipath-tool-call-*`, `uipath-binary-classification`, `uipath-multiclass-classification`, `uipath-contains`, `uipath-llm-judge-output-strict-json-similarity`, or `uipath-llm-judge-trajectory-simulation` has no legacy class and the eval worker will not load it. If you need one of these, the agent must be coded, not low-code.
- **Don't hand-write a category/type combination outside the validate matrix.** Validate accepts cat 0 → types {1, 6}, cat 1 → type {5}, cat 3 → type {7}. Anything else fails schema migration.