> Discover all available pages from the documentation index: https://mastra.ai/llms.txt

# createScorer

Mastra provides a unified `createScorer` factory that allows you to define custom scorers for evaluating input/output pairs. You can use either native JavaScript functions or LLM-based prompt objects for each evaluation step. Custom scorers can be added to Agents and Workflow steps.

## How to create a custom scorer

Use the `createScorer` factory to define your scorer with a name, description, and optional judge configuration. Then chain step methods to build your evaluation pipeline. You must provide at least a `generateScore` step.

**Prompt object steps** are step configurations expressed as objects with `description` + `createPrompt` (and `outputSchema` for `preprocess`/`analyze`). These steps invoke the judge LLM. **Function steps** are plain functions and never call the judge.

```typescript
import { createScorer } from '@mastra/core/evals'

const scorer = createScorer({
  id: 'my-custom-scorer',
  name: 'My Custom Scorer', // Optional, defaults to id
  description: 'Evaluates responses based on custom criteria',
  type: 'agent', // Optional: for agent evaluation with automatic typing
  judge: {
    model: myModel,
    instructions: 'You are an expert evaluator...',
  },
})
  .preprocess({
    /* step config */
  })
  .analyze({
    /* step config */
  })
  .generateScore(({ run, results }) => {
    // Return a number
  })
  .generateReason({
    /* step config */
  })
```

## `createScorer` options

**id** (`string`): Unique identifier for the scorer. Used as the name if name is not provided.

**name** (`string`): Name of the scorer. Defaults to id if not provided.

**description** (`string`): Description of what the scorer does.

**judge** (`object`): Optional judge configuration for LLM-based steps.

**judge.model** (`LanguageModel`): The LLM model instance to use for evaluation.

**judge.instructions** (`string`): System prompt/instructions for the LLM.

**judge.jsonPromptInjection** (`boolean`): When true, inject the JSON schema into the prompt instead of using the provider's native response\_format API. Set this for models that don't support native structured output (e.g. some Groq Llama models) to avoid a wasted 400 call.

**judge.inputProcessors** (`Processor[]`): Input processors applied to the internal judge agent before its messages reach the model (e.g. redaction, validation).

**judge.outputProcessors** (`Processor[]`): Output processors applied to the internal judge agent's output before it is returned (e.g. moderation, transformation).

**judge.errorProcessors** (`Processor[]`): Error processors for judge models that use Mastra’s current generation API. These implement processAPIError and can inspect LLM API rejections and signal a retry, e.g. StreamErrorRetryProcessor. Legacy model adapters use generateLegacy() and do not run error processors.

**judge.maxProcessorRetries** (`number`): Maximum number of times error processors can retry one judge generation. The runtime defaults to 10 when errorProcessors are configured without this value. Set it explicitly to bound the retry budget.

**type** (`string`): Type specification for input/output. Use 'agent' for automatic agent types. For custom types, use the generic approach instead.

**prepareRun** (`(run: ScorerRun) => ScorerRun | Promise<ScorerRun>`): Transform the scorer run data before the pipeline executes. Use this to filter messages, limit context size, or drop fields the scorer doesn't need. The \`filterRun()\` utility creates this function from declarative options. Can be async.

This function returns a scorer builder that you can chain step methods onto. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.

The judge only runs for steps defined as **prompt objects** (`preprocess`, `analyze`, `generateScore`, `generateReason` in prompt mode). If you use function steps only, the judge is never called and there is no LLM output to inspect. In that case, any score/reason must be produced by your functions.

When a prompt-object step runs, its structured LLM output is stored in the corresponding result field (`preprocessStepResult`, `analyzeStepResult`, or the value consumed by `calculateScore` in `generateScore`).

## Retry judge requests

Use the existing judge `errorProcessors` configuration to retry transient failures inside a failed judge request. This doesn't retry the scorer workflow, trace target, batch item, score write, or a completed scorer step.

`@mastra/core` `1.49.0` doesn't include scorer error-processor configuration. Upgrade to a version with scorer processor support, or backport that focused change, before using this configuration.

The following example uses one bounded retry budget. Set the processor `maxRetries` and `judge.maxProcessorRetries` to the same value. Keep the internal judge agent's model retries at the default of `0` so model retries don't multiply processor attempts.

```typescript
import { createScorer } from '@mastra/core/evals'
import { StreamErrorRetryProcessor } from '@mastra/core/processors'

const isTransientNetworkError = (error: unknown) =>
  error instanceof Error && /ECONNRESET|ETIMEDOUT|socket hang up/i.test(error.message)

const retryProcessor = new StreamErrorRetryProcessor({
  maxRetries: 2,
  maxRetryAfterMs: 30_000,
  delayMs: ({ retryCount }) => Math.min(1_000 * 2 ** retryCount, 30_000),
  matchers: [isTransientNetworkError],
  retryUnknownErrors: false,
})

export const responseQuality = createScorer({
  id: 'response-quality',
  description: 'Scores response quality',
  judge: {
    model: myModel,
    instructions: 'Return a score and concise reason.',
    errorProcessors: [retryProcessor],
    maxProcessorRetries: 2,
  },
})
  .generateScore({
    description: 'Score the response quality.',
    createPrompt: ({ run }) => `Score: ${run.output}`,
  })
  .generateReason({
    description: 'Explain the score.',
    createPrompt: () => 'Explain the score.',
  })
```

With this configuration, a failed request has at most three provider attempts: the initial request plus two processor retries. If `generateScore` completes and `generateReason` receives a retryable failure, only `generateReason` retries.

`StreamErrorRetryProcessor` honors retryable provider metadata and narrow custom matchers. It keeps `retryUnknownErrors` disabled by default, so authentication, invalid-request, and context-length errors fail immediately unless you explicitly match them. It bounds `Retry-After` values to `30_000` milliseconds by default; use `maxRetryAfterMs` to change that cap.

Avoid adding outer scorer or workflow retries. Avoid combining a nonzero model retry setting with this processor unless you intentionally accept additional attempts.

### Override retries for one step

A step's `judge` configuration overrides scorer-level judge fields. Processor arrays replace the scorer-level arrays. Omit `maxProcessorRetries` in the step configuration to inherit the scorer-level numeric cap.

Coordinated processor retries require a judge model that uses Mastra’s current generation API. Legacy model adapters call [`generateLegacy()`](https://mastra.ai/reference/agents/generateLegacy), bypass error processors, and use that API's separate AI SDK `maxRetries` default of `2`.

## Type safety

You can specify input/output types when creating scorers for better type inference and IntelliSense support:

### Agent Type Shortcut

For evaluating agents, use `type: 'agent'` to automatically get the correct types for agent input/output:

```typescript
import { createScorer } from '@mastra/core/evals'

// Agent scorer with automatic typing
const agentScorer = createScorer({
  id: 'agent-response-quality',
  description: 'Evaluates agent responses',
  type: 'agent', // Automatically provides ScorerRunInputForAgent/ScorerRunOutputForAgent
})
  .preprocess(({ run }) => {
    // run.input is automatically typed as ScorerRunInputForAgent
    const userMessage = run.inputData.inputMessages[0]?.content
    return { userMessage }
  })
  .generateScore(({ run, results }) => {
    // run.output is automatically typed as ScorerRunOutputForAgent
    const response = run.output[0]?.content
    return response.length > 10 ? 1.0 : 0.5
  })
```

### Custom Types with Generics

For custom input/output types, use the generic approach:

```typescript
import { createScorer } from '@mastra/core/evals'

type CustomInput = { query: string; context: string[] }
type CustomOutput = { answer: string; confidence: number }

const customScorer = createScorer<CustomInput, CustomOutput>({
  id: 'custom-scorer',
  description: 'Evaluates custom data',
}).generateScore(({ run }) => {
  // run.input is typed as CustomInput
  // run.output is typed as CustomOutput
  return run.output.confidence
})
```

### Built-in Agent Types

- **`ScorerRunInputForAgent`** - Contains `inputMessages`, `rememberedMessages`, `systemMessages`, and `taggedSystemMessages` for agent evaluation
- **`ScorerRunOutputForAgent`** - Array of agent response messages

Using these types provides autocomplete, compile-time validation, and better documentation for your scoring logic.

## Trace scoring with agent types

When you use `type: 'agent'`, your scorer is compatible for both adding directly to agents and scoring traces from agent interactions. The scorer automatically transforms trace data into the proper agent input/output format:

```typescript
const agentTraceScorer = createScorer({
  id: 'agent-trace-length',
  description: 'Evaluates agent response length',
  type: 'agent',
}).generateScore(({ run }) => {
  // Trace data is automatically transformed to agent format
  const userMessages = run.inputData.inputMessages
  const agentResponse = run.output[0]?.content

  // Score based on response length
  return agentResponse?.length > 50 ? 0 : 1
})

// Register with Mastra for trace scoring
const mastra = new Mastra({
  scorers: {
    agentTraceScorer,
  },
})
```

## Step method signatures

### preprocess

Optional preprocessing step that can extract or transform data before analysis.

**Function Mode:** Function: `({ run, results }) => any`

**run.input** (`any`): Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. \[{ role: 'user', content: 'hello world' }]. If the scorer is used in a workflow, this will be the input of the workflow.

**run.output** (`any`): Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.

**run.runId** (`string`): Unique identifier for this scoring run.

**run.requestContext** (`object`): Request Context from the agent or workflow step being evaluated (optional).

**results** (`object`): Empty object (no previous steps).

Returns: `any`\
The method can return any value. The returned value will be available to subsequent steps as `preprocessStepResult`.

**Prompt Object Mode:**

**description** (`string`): Description of what this preprocessing step does.

**outputSchema** (`StandardJSONSchemaV1`): Standard JSON Schema for the expected output of the preprocess step.

**createPrompt** (`function`): Function: ({ run, results }) => string. Returns the prompt for the LLM.

**judge** (`object`): (Optional) LLM judge for this step (can override main judge). See Judge Object section.

### analyze

Optional analysis step that processes the input/output and any preprocessed data.

**Function Mode:** Function: `({ run, results }) => any`

**run.input** (`any`): Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. \[{ role: 'user', content: 'hello world' }]. If the scorer is used in a workflow, this will be the input of the workflow.

**run.output** (`any`): Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.

**run.runId** (`string`): Unique identifier for this scoring run.

**run.requestContext** (`object`): Request Context from the agent or workflow step being evaluated (optional).

**results.preprocessStepResult** (`any`): Result from preprocess step, if defined (optional).

Returns: `any`\
The method can return any value. The returned value will be available to subsequent steps as `analyzeStepResult`.

**Prompt Object Mode:**

**description** (`string`): Description of what this analysis step does.

**outputSchema** (`StandardJSONSchemaV1`): Standard JSON Schema for the expected output of the analyze step.

**createPrompt** (`function`): Function: ({ run, results }) => string. Returns the prompt for the LLM.

**judge** (`object`): (Optional) LLM judge for this step (can override main judge). See Judge Object section.

### `generateScore`

**Required** step that computes the final numerical score.

**Function Mode:** Function: `({ run, results }) => number`

**run.input** (`any`): Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. \[{ role: 'user', content: 'hello world' }]. If the scorer is used in a workflow, this will be the input of the workflow.

**run.output** (`any`): Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.

**run.runId** (`string`): Unique identifier for this scoring run.

**run.requestContext** (`object`): Request Context from the agent or workflow step being evaluated (optional).

**results.preprocessStepResult** (`any`): Result from preprocess step, if defined (optional).

**results.analyzeStepResult** (`any`): Result from analyze step, if defined (optional).

Returns: `number`\
The method must return a numerical score.

**Prompt Object Mode:**

**description** (`string`): Description of what this scoring step does.

**outputSchema** (`StandardJSONSchemaV1`): Standard JSON Schema for the expected output of the generateScore step.

**createPrompt** (`function`): Function: ({ run, results }) => string. Returns the prompt for the LLM.

**judge** (`object`): (Optional) LLM judge for this step (can override main judge). See Judge Object section.

When using prompt object mode, you must also provide a `calculateScore` function to convert the LLM output to a numerical score:

**calculateScore** (`function`): Function: ({ run, results, analyzeStepResult }) => number. Converts the LLM's structured output into a numerical score.

### `generateReason`

Optional step that provides an explanation for the score.

**Function Mode:** Function: `({ run, results, score }) => string`

**run.input** (`any`): Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. \[{ role: 'user', content: 'hello world' }]. If the scorer is used in a workflow, this will be the input of the workflow.

**run.output** (`any`): Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.

**run.runId** (`string`): Unique identifier for this scoring run.

**run.requestContext** (`object`): Request Context from the agent or workflow step being evaluated (optional).

**results.preprocessStepResult** (`any`): Result from preprocess step, if defined (optional).

**results.analyzeStepResult** (`any`): Result from analyze step, if defined (optional).

**score** (`number`): Score computed by the generateScore step.

Returns: `string`\
The method must return a string explaining the score.

**Prompt Object Mode:**

**description** (`string`): Description of what this reasoning step does.

**createPrompt** (`function`): Function: ({ run, results, score }) => string. Returns the prompt for the LLM.

**judge** (`object`): (Optional) LLM judge for this step (can override main judge). See Judge Object section.

All step functions can be async.