# Workflows
> Durable Effect workflows in Voltro — what they are, when to use them, and the current runtime boundaries.
---
## Overview
_Durable Effect workflows in Voltro — what they are, when to use them, and the current runtime boundaries._
A **workflow** in Voltro is a long-running Effect program that can survive process restarts. Durable work is expressed with `workflow({...})`, checkpointed `step({...})` calls, durable `sleep({...})`, optional external `awaitSignal(...)` waits, and synchronous `awaitUpdate(...)` messages.
The implementation is `@effect/workflow` + `@effect/cluster`. No Restate, Temporal, Inngest, BullMQ, or separate worker runtime is required.
Live — start a 3-step durable workflow and watch its timeline stream as each step
completes (durable-execution UI no other framework ships):
```tsx
const run = useAction('app', 'demo.runPipeline')
const { runId } = await run.run({})
{/* live step timeline */}
```
## What's in this section
- [Defining workflows](/docs/workflows/definition) — `workflow`, `payload`, `success`, default execute factories, and checkpointed steps
- [Retries & failure handling](/docs/workflows/retries) — per-step retries, failed runs, retry controls, and compensation
- [Sleep & wakeups](/docs/workflows/sleep) — `sleep`, schedules, and `awaitSignal`
- [Clustering](/docs/workflows/cluster) — multi-instance workflow distribution and schedule coordination
- [Flow control](/docs/workflows/flow-control) — durable queues, bounded concurrency, and rate limits
- [Versioning](/docs/workflows/versioning) — definition versions, compatibility metadata, and patch markers
- [Debugging & inspection](/docs/workflows/debugging) — dashboard, inspect endpoints, CLI, tables, and tests
## When to use a workflow
| You have... | Use |
|---|---|
| Multi-step job that must survive a deploy | Workflow |
| External side effects that need retry/resume semantics | Workflow |
| Human approval or webhook callback in the middle of work | Workflow + `awaitSignal` |
| External command that must validate and return a result | Workflow + `awaitUpdate` |
| Domain event that should fan out to durable work | `triggerWorkflow(...)` + `ctx.events.publish(...)` |
| One-off durable delay inside an operation | Workflow + `sleep` |
| Recurring job such as "daily at 3am" | [Schedule](/docs/scheduling/overview) (`*.cron.tsx`) |
| Atomic database write | Mutation |
| Request-scoped HTTP/email/payment call | Action |
| Progressive server-to-client output | Stream |
| Request-scoped LLM chat | Agent or stream |
The wedge: if the operation must survive the process, use a workflow. If it is just a short request, keep it in a mutation, action, query, or stream.
## A minimal workflow
A workflow is split into two files paired by basename — a **descriptor**
(`*.workflow.tsx`, browser-safe, imports `@voltro/workflow/define`) that the
codegen pulls into the client rpcGroup, and a **server executor**
(`*.workflow.server.tsx`) that holds the database/AI/cluster imports:
```tsx
// apps/api/workflows/notes.summarise.workflow.tsx — descriptor (browser-safe)
import { workflow } from '@voltro/workflow/define'
import { Schema } from 'effect'
export const SummariseNote = workflow({
name: 'notes.summarise',
payload: { noteId: Schema.String },
success: Schema.Struct({ summary: Schema.String }),
idempotencyKey: ({ noteId }) => `notes.summarise:${noteId}`,
})
```
```tsx
// apps/api/workflows/notes.summarise.workflow.server.tsx — executor (server-only)
import { step } from '@voltro/workflow'
import { Effect, Schema } from 'effect'
import type { AppContext } from '@voltro/runtime'
const NoteRow = Schema.Struct({ id: Schema.String, body: Schema.String })
const buildExecute = (ctx: AppContext) =>
({ noteId }: { noteId: string }) =>
Effect.gen(function* () {
const note = yield* step({
name: 'load-note',
input: { noteId },
success: NoteRow,
execute: Effect.tryPromise(
() => ctx.store.select('notes').where('id', noteId).one(),
).pipe(Effect.flatMap(Schema.decodeUnknown(NoteRow)), Effect.orDie),
})
const summary = yield* step({
name: 'summarise-with-llm',
input: { noteId },
success: Schema.String,
execute: summariseWithLlm(note.body),
})
yield* step({
name: 'save-summary',
input: { noteId },
success: Schema.Void,
execute: Effect.tryPromise(
() => ctx.store.update('notes', noteId, { summary }),
).pipe(Effect.asVoid, Effect.orDie),
})
return { summary }
})
export default buildExecute
```
`step({...})` is the checkpoint boundary. A plain `yield* someEffect` composes Effect logic, but it is not automatically recorded as a workflow step. Put external I/O, database writes, and expensive work inside `step`.
Note what the store calls do NOT do: no `as never`. Reach for the fluent builder (`ctx.store.select(...)`) rather than hand-building a query descriptor, and decode the untyped `Row` into the step's `success` schema with `Schema.decodeUnknown` instead of asserting it. `.one()` fails with the typed `NoRowFound` when the note is missing (or when more than one matches), so there is no `rows[0]` to null-check.
## Starting a workflow
Every discovered `*.workflow.tsx` is emitted into `rpcGroup.generated.ts` as a unary RPC. The payload and error schemas come from the workflow, but the RPC success type is always a `WorkflowRunHandle`. Calling it starts durable work and returns that handle immediately:
```tsx
import {
useWorkflow,
useWorkflowRun,
useWorkflowRunEvents,
useWorkflowRuns,
useWorkflowRunSteps,
useWorkflowUpdate,
} from '@voltro/client'
const summarise = useWorkflow<{ noteId: string }>(
'app',
'notes.summarise',
)
const run = await summarise.start({ noteId })
const { run: live } = useWorkflowRun('app', run.id)
```
`run.id` is the durable execution id and can be passed to `useWorkflowRun(...)`. The hook subscribes to Voltro's built-in reactive workflow-run query, so the UI updates when `_voltro_workflow_runs` changes. Waiting for the success payload is explicit: use `ctx.workflows.wait(...)` on the server when another server-side operation really must block; otherwise render the live run row and let the workflow finish in the background.
For app-level job centers and detail pages, use the rest of the workflow hook family:
```tsx
const { runs } = useWorkflowRuns('app', { tag: 'notes.summarise', limit: 25 })
const { steps } = useWorkflowRunSteps('app', live?.id)
const { events } = useWorkflowRunEvents('app', live?.id)
const approve = useWorkflowUpdate('app')
await approve.update({ id: live!.id }, 'approve', { decision: true })
```
Those hooks are backed by reactive framework tables too, so status, checkpointed steps, timers, signals, and update events flow through Voltro subscriptions instead of polling.
From server code, use `ctx.workflows`:
```ts
export default async ({ noteId }, ctx) => {
await ctx.store.update('notes', noteId, { summaryStatus: 'queued' })
const run = await ctx.workflows.start('notes.summarise', { noteId })
return { runId: run.id }
}
```
Inside a mutation, `ctx.workflows.start(...)` is transaction-aware: Voltro computes the deterministic run handle immediately, queues the actual start, and only launches the workflow after the mutation commits. If the mutation rolls back, the workflow never starts. Actions and incoming webhooks start immediately after their own verification/idempotency work has succeeded.
Workflow controls use the same facade:
```ts
await ctx.workflows.signal({ id: runId }, 'approval', { approved: true })
const approved = await ctx.workflows.update({ id: runId }, 'approve', { decision: true })
await ctx.workflows.cancel('notes.summarise', executionId)
await ctx.workflows.resume('notes.summarise', executionId)
const snapshot = await ctx.workflows.query('notes.summarise', executionId)
```
Inside a workflow body, use `ctx.workflows.child(...)` for hierarchical work. Voltro records the parent execution id in `_voltro_workflow_start_contexts` before submitting the child run, so a different cluster runner still sees the same lineage when it starts executing the child.
```ts
const child = yield* Effect.promise(() =>
ctx.workflows.child('orders.shipOne', { orderId }, {
parentClosePolicy: 'cancel',
}),
)
const childSnapshot = yield* Effect.promise(() => ctx.workflows.wait(child))
```
`parentClosePolicy` controls what happens when the parent closes. `cancel` is the default and interrupts an open child. `terminate` also interrupts the child and records the decision as a hard parent-close action. `abandon` leaves the child running. The dashboard shows child runs, their parent execution id, and the chosen policy.
Inside the workflow body, `awaitUpdate(...)` receives the tracked message, validates the payload, and records either `update-completed` with the result or `update-failed` with the validation/handler error:
```tsx
import { awaitUpdate } from '@voltro/workflow'
import { Effect, Schema } from 'effect'
const result = yield* awaitUpdate(ctx, {
name: 'approve',
schema: Schema.Struct({ decision: Schema.Boolean }),
success: Schema.Struct({ accepted: Schema.Boolean }),
handle: ({ decision }) => Effect.succeed({ accepted: decision }),
})
```
## Event-triggered workflows
Use `*.trigger.ts` files when durable work should start from domain events instead of a direct RPC or action call.
```ts
// apps/api/triggers/user.signup.trigger.ts
import { triggerWorkflow } from '@voltro/runtime'
export default triggerWorkflow<{ userId: string; plan: string }, { userId: string }>({
event: 'user.signup',
workflow: 'onboarding.start',
filter: (event) => event.data.plan !== 'free',
payload: (event) => ({ userId: event.data.userId }),
idempotencyKey: (event) => event.id,
})
```
Then emit the event from a mutation, action, incoming webhook, schedule, or workflow:
```ts
yield* ctx.events.publish(userSignedUp, { userId }, { plan })
```
Voltro records the event in `_voltro_workflow_events`, records each trigger delivery in `_voltro_workflow_event_deliveries`, and starts matching workflows with `source: event:`. Emits inside mutations are post-commit safe: if the mutation rolls back, the workflow fan-out is not launched.
Use `event: '*'` for audit-style wildcard triggers that should see every emitted domain event. The local devtools and Voltro Cloud workflow dashboards mirror domain events and delivery rows live in the Events tab, including filtered/skipped deliveries, failed fan-out, and the spawned execution id.
## Runtime boundaries
- The workflow executor receives an `AppContext` built for the run. Starts from RPC handlers, mutations, actions, schedules, verified incoming handlers, and inspect tooling record a `source` such as `workflow-rpc`, `app-context`, `schedule:`, `incoming:`, or `inspect`. Request starts also carry the starter trace; authenticated request starts carry the resolved subject into the run record and executor context.
- `ctx.events.publish(...)` is available when the app declares workflow event triggers. Triggered workflow starts carry the starter subject/trace and a `source` of `event:`.
- Put tenant/user ids that the workflow must enforce into the payload. That keeps business authorization deterministic across retries, resumes, and future cross-replica handoff.
- Child workflows should be started with `ctx.workflows.child(...)`. Parent lineage and parent-close policy are persisted before the child start, so they survive cross-runner execution.
- External incoming handlers created with `defineIncomingWebhook(...)` receive `context.workflows`, so a verified webhook can start, signal, or update a workflow after signature and idempotency checks.
- Operational controls such as retry, cancel, suspend, resume, signal injection, and tracked updates also live on the dashboard, `voltro workflows ...`, and the inspect endpoints.
## What is intentionally different from Temporal
- **Polyglot workers.** Voltro workflows are TypeScript/Effect.
- **A separate workflow service.** This is intentional: the engine is mounted by the framework alongside your API.
- **Automatic code-version isolation.** A long-lived run resumed after a deploy executes the current workflow code. Use workflow `version`, `compatibleWith`, and `patches` metadata for deploy safety.
What you gain: workflows live beside queries, mutations, actions, streams, schedules, schema, and plugins. The same codegen and inspect surfaces see them.
## Where to go next
- [Defining workflows](/docs/workflows/definition)
- [Retries & failure handling](/docs/workflows/retries)
- [Sleep & wakeups](/docs/workflows/sleep)
- [Flow control](/docs/workflows/flow-control)
- [Versioning](/docs/workflows/versioning)
- [Debugging & inspection](/docs/workflows/debugging)
---
## Defining workflows
_The *.workflow.tsx convention, workflow schemas, execute factories, checkpointed steps, child workflows, and replay rules._
A workflow file is discovered by suffix: `*.workflow.tsx`. It exports one `workflow({...})` definition and a default factory that receives `AppContext` and returns the workflow executor.
## Anatomy
```tsx
// apps/api/workflows/notes.summarise.workflow.tsx
import { workflow, step } from '@voltro/workflow'
import { Effect, Schema } from 'effect'
import type { AppContext } from '@voltro/runtime'
export const SummariseNote = workflow({
name: 'notes.summarise',
payload: { noteId: Schema.String },
success: Schema.Struct({ summary: Schema.String }),
idempotencyKey: ({ noteId }) => `notes.summarise:${noteId}`,
})
const buildExecute = (ctx: AppContext) =>
({ noteId }: { noteId: string }, _executionId: string) =>
Effect.gen(function* () {
const note = yield* step({
name: 'load-note',
input: { noteId },
success: Schema.Struct({ id: Schema.String, body: Schema.String }),
execute: Effect.tryPromise(() => loadNote(ctx.store, noteId)),
})
const summary = yield* step({
name: 'summarise-with-llm',
input: { noteId },
success: Schema.String,
execute: summariseWithLlm(note.body),
})
yield* step({
name: 'save-summary',
input: { noteId },
success: Schema.Void,
execute: Effect.tryPromise(() => saveSummary(ctx.store, noteId, summary)),
})
return { summary }
})
export default buildExecute
```
Two exports matter:
- `workflow({...})` — the durable definition. `payload`, `success`, optional `error`, and optional `idempotencyKey` are read by codegen and the runtime.
- `default` — a build-execute factory, `(ctx: AppContext) => (payload, executionId) => Effect`. The CLI calls `definition.toLayer(execute)` during registration.
If the default export is not a function, discovery skips the file with a "missing default-export build-execute factory" warning.
## Schema fields
```ts
export const ImportCustomers = workflow({
name: 'customers.import',
payload: {
uploadId: Schema.String,
dryRun: Schema.optional(Schema.Boolean),
},
success: Schema.Struct({
imported: Schema.Number,
skipped: Schema.Number,
}),
error: Schema.Union(InvalidCsv, ImportProviderDown),
idempotencyKey: ({ uploadId }) => `customers.import:${uploadId}`,
messages: {
signals: {
approval: Schema.Struct({ approved: Schema.Boolean }),
},
updates: {
approve: {
payload: Schema.Struct({ decision: Schema.Boolean }),
success: Schema.Struct({ accepted: Schema.Boolean }),
},
},
},
})
```
`payload` is the start input. `success` is the resolved output. `error` is the typed failure channel. `idempotencyKey` deduplicates concurrent or repeated starts with the same logical input. `messages` is optional codegen metadata; it emits `WorkflowSignals`, `WorkflowUpdates`, and `WorkflowMessages` type maps, while runtime validation still happens at `awaitSignal(...)` / `awaitUpdate(...)`.
There are exactly two message channels: `signals` (fire-and-forget) and `updates` (synchronous, with a result). A `queries` channel was declarable until 0.34.0 and never had a send path — nothing could invoke one — so it is gone. To read a run's state, write an ordinary `*.query.ts` over `_voltro_workflow_runs` / `_voltro_workflow_run_steps`; to ask a running workflow something and get an answer, use `updates`.
## Step boundaries
`step({...})` wraps `@effect/workflow`'s `Activity.make`. It is the checkpointed unit that is journaled by the workflow engine and recorded into `_voltro_workflow_run_steps`.
```ts
const customer = yield* step({
name: 'fetch-customer',
input: { customerId },
success: Customer,
execute: Effect.tryPromise(() => crm.fetchCustomer(customerId)),
})
```
The optional `input` field is not passed to the executor. It is persisted for inspection, truncated to a dashboard-safe size, and shown beside output/errors in the run timeline.
Plain Effect composition is still useful:
```ts
const normalised = normaliseCustomer(customer)
const enriched = yield* Effect.succeed(addDerivedFields(normalised))
```
But it is not a durable activity boundary by itself. If it does external I/O, writes to storage, charges a card, sends an email, or calls an LLM, put that work inside `step`.
## Deterministic replay
The workflow body can be replayed by the engine. The structure of checkpointed steps must be stable for the same payload.
OK:
```ts
const note = yield* step({ name: 'load-note', success: Note, execute: loadNote(noteId) })
if (note.archived) {
yield* step({ name: 'notify-archived', execute: notifyArchived(note.id) })
} else {
yield* step({ name: 'summarise', execute: summarise(note.body) })
}
```
The branch depends on a checkpointed value.
Not OK:
```ts
if (Math.random() > 0.5) {
yield* step({ name: 'a', execute: doA })
} else {
yield* step({ name: 'b', execute: doB })
}
```
If time, randomness, or external state influences structure, capture it in a step first:
```ts
const choice = yield* step({
name: 'choose-branch',
success: Schema.Boolean,
execute: Effect.sync(() => Math.random() > 0.5),
})
```
## Parallel steps
Steps with no data dependency run concurrently with plain `Effect.all` — no special API:
```ts
const [jira, github] = yield* Effect.all(
[
step({ name: 'fetch-jira', success: JiraIssues, execute: fetchJira(projectKey) }),
step({ name: 'fetch-github', success: GithubPrs, execute: fetchGithub(repo) }),
],
{ concurrency: 'unbounded' },
)
```
Both steps journal independently, and the durable guarantees hold across the join:
- The steps genuinely **overlap** — one is not secretly serialized behind the other.
- On a retry or an operator redrive, a parallel step that already **completed replays** from its journal; only the sibling that failed re-executes.
Both properties are pinned by a contract test against the real engine (`workflowParallelSteps.integration.test.ts`), so an engine upgrade that broke either would go red rather than quietly serializing your fan-out.
Name each parallel step distinctly — the name is the journal key, and two concurrent steps sharing one name would share one checkpoint. Deterministic-replay rules apply unchanged: the *set* of steps started must be stable for the same payload.
## Starting from the client
Codegen synthesises an RPC for each workflow. To start it from React, use `useWorkflow(...)`; the call returns a run handle immediately and the durable work continues in the workflow engine:
```tsx
import { useWorkflow, useWorkflowRun } from '@voltro/client'
const startImport = useWorkflow<{ uploadId: string }>('app', 'customers.import')
const run = await startImport.start({ uploadId })
const { run: liveRun } = useWorkflowRun('app', run.id)
```
Use `liveRun.status` and `liveRun.output` to render progress/result state. Waiting for the success payload is explicit on the server with `ctx.workflows.wait(...)`; UI code should usually subscribe to the run row instead of blocking the interaction.
## Child workflows
Inside a workflow body, start child runs through `ctx.workflows.child(...)`. The child gets its own durable run, step history, events, cancellation controls, and dashboard detail page. Voltro persists the parent execution id and parent-close policy before submitting the child start, so the relationship survives cross-runner execution in a cluster.
```ts
const child = yield* Effect.promise(() =>
ctx.workflows.child('documents.embed', { documentId }, {
parentClosePolicy: 'cancel',
}),
)
const result = yield* Effect.promise(() => ctx.workflows.wait(child))
```
Fan out with normal Effect concurrency:
```ts
const children = yield* Effect.all(
documents.map((doc) =>
Effect.promise(() =>
ctx.workflows.child('documents.embed', { documentId: doc.id }, {
parentClosePolicy: 'abandon',
}),
),
),
{ concurrency: 8 },
)
```
Parent-close policies:
| Policy | Behavior |
|---|---|
| `cancel` | Default. Interrupt open children when the parent closes. |
| `terminate` | Interrupt open children and record a hard parent-close action. |
| `abandon` | Leave the child running when the parent closes. |
The dashboard shows child runs, their parent execution id, and the selected policy. `ctx.workflows.wait(child)` accepts the run handle directly when the parent needs the child's success/failure snapshot.
## Running on a cron — `workflow({ schedule })`
A workflow whose only trigger is a clock can declare the cron on itself, instead of a separate `*.cron.tsx` file with a `workflow:` target:
```tsx
export default workflow({
name: 'reports.nightly',
payload: Schema.Struct({ day: Schema.String }),
idempotencyKey: (p) => `nightly:${p.day}`,
schedule: {
cron: '0 3 * * *',
timezone: 'Europe/Berlin',
payload: ({ scheduledAt }) => ({ day: scheduledAt.toISOString().slice(0, 10) }),
onOverlap: 'skip',
},
})
```
This is sugar, not a second scheduler: at boot it lowers into a real schedule named `workflow:` on the same coordinated cron engine every `defineSchedule` uses — same exactly-once claims, same run rows, same Schedules panel, same `voltro schedule run` / `backfill` verbs.
What the declaration adds is overlap vocabulary **about the workflow run** (Temporal Schedules' names):
- `onOverlap: 'skip'` *(default)* — a firing stands down while the previous firing's **run** is still going.
- `'buffer'` — firings serialize behind the running one; none is lost.
- `'cancelOther'` — the new firing cancels the still-running previous run (only runs this schedule started — a manually-started run of the same workflow is never touched), then starts fresh.
The synthesized firing **awaits the run to completion** — that is what makes `skip`/`buffer` bind on the run's duration rather than on the milliseconds it takes to enqueue one, and it is why the firing watchdog (`schedule.maxRuntime`) defaults to **24 hours** here instead of a plain schedule's 30 minutes. Set it above your slowest expected run. A failed run fails the firing, so a nightly job that dies every night is red in the schedule ledger, not a wall of green.
`payload` is a value or a function of the firing — a function receives `{ scheduledAt }`, so a backfilled firing computes against **its** slot, not "now". `backfill:` and everything else about missed firings work exactly as on a plain schedule — see [Overlap & backfill](/docs/scheduling/overlap-and-backfill).
If the workflow also declares a deferring control (debounce, concurrency, …), a scheduled start passes the same admission gate as any other start — a deferred firing has nothing to await and hands the run to the admission queue.
## Tenant and subject
Workflow starts persist the starter's trace id, source, parent execution id, parent-close policy — and their **identity**, never their authority — in `_voltro_workflow_start_contexts`. Whichever runner first executes the workflow loads that context before building the executor `AppContext`.
**The guarantee: identity is persisted, authority is re-resolved at resume.**
- **Identity** (type, id, `tenantId`, `metadata`) is written and read through the same stripping function the session cookie mints through. It has to survive: the tenant scope reads `tenantId`, the run row is attributed to `id`, and a plugin service resolving a per-user credential reads `metadata`. A workflow started by tenant A still acts on tenant A's rows in three days' time.
- **Authority** comes from your [`auth.resolveScopes`](/docs/authentication/strategies) on every execution attempt, with `ctx.origin === 'workflow'`. Wire no resolver and a resumed run has no scopes — fail-closed, and the same default a cookie-authenticated request has.
- **A run with no recorded caller** — a bootstrap, or one whose row aged out — runs as `SYSTEM_SUBJECT` and is not put through your resolver.
The column used to hold the whole `Subject`, scopes included. A role removed on Monday was still asserted by Thursday's resume, out of a row nothing re-validated, on a path with no request, no cookie and no expiry. Rows written by an older build are stripped on **read**, so a resumed run cannot re-assert authority that was persisted before this changed.
Still include tenant/user ids that the business process must enforce in `payload`, validate them in the first step, and scope store reads/writes deliberately. Payload data is replay-safe and makes authorization decisions auditable across retries and deploys.
## Calling an HTTP API from a step
The framework's `HttpClient` is available inside a workflow executor — the same one handlers `yield*`, with the same SSRF allowlist and the same automatic `traceparent` propagation. `yield*` it in a step:
```ts
import { HttpClient, HttpClientRequest } from '@effect/platform'
export const executor = defineWorkflowExecutor(syncInvoice, (payload) =>
Effect.gen(function* () {
const remote = yield* step({ name: 'fetch-invoice' }, () =>
Effect.gen(function* () {
const client = yield* HttpClient.HttpClient
const res = yield* client.execute(
HttpClientRequest.get(`https://billing.example.com/invoices/${payload.invoiceId}`),
)
return yield* res.json
}),
)
yield* step({ name: 'persist' }, () => database.invoices.update(payload.invoiceId, remote))
}),
)
```
Two things follow from where it sits:
- **Wrap the call in a `step`.** The result is then checkpointed, so a retry or a resume after a deploy replays the recorded response instead of calling the remote again. A bare `yield*` outside a step re-issues the request on every replay — which for a payment capture or an email send is the difference between once and several times.
- **It is the SSRF-guarded client.** Requests to internal targets are refused by the same policy handlers get; a workflow is not a way around it. Configure the allowlist once in `app.config.ts` under `http` — dev and serve read the same key, so the policy cannot differ between them.
Reaching for `fetch` instead loses both: no allowlist, no trace propagation, and nothing tying the call to the step that made it.
## Anti-patterns
- **Using `input` in `workflow({...})`.** The current API is `payload`.
- **Assuming every `yield*` is a persisted step.** Use `step({...})` for checkpoints.
- **Starting workflows through `useMutation(...)`.** Use `useWorkflow(...)`; workflows are durable runs, not optimistic writes.
- **Branching step structure on randomness or live external state.** Capture the value in a step first.
- **Long-lived workflows without payload versioning.** A run resumed after a deploy uses the current workflow code.
---
## Retries & failure handling
_Per-step retries, failed workflow runs, retry controls, compensation, and timeouts._
Workflow durability and retry are related, but not identical. The engine checkpoints completed `step({...})` activities so an interrupted run can resume without re-running successful steps. Transient failures still need an explicit retry policy around the step that can fail.
## Per-step retries
Declare a `retry:` policy on `step({...})` and the framework **enforces** it — it compiles the policy to an Effect `Schedule` and retries `execute` for you. No hand-written retry needed:
```tsx
import { workflow, step } from '@voltro/workflow'
import { Schema } from 'effect'
class ProviderDown extends Schema.TaggedError()('ProviderDown', {
message: Schema.String,
}) {}
const summary = yield* step({
name: 'summarise-with-llm',
input: { noteId },
success: Schema.String,
error: ProviderDown,
retry: {
maxAttempts: 5,
strategy: 'exponential',
baseDelay: '500 millis',
},
execute: callLlm(note.body),
})
```
Retries run inside the one step and are transparent to the durable engine — completed steps still checkpoint; the step's final outcome is recorded. `retry: { maxAttempts: 5 }` is already a good policy (exponential backoff, jittered).
### The conditions you actually want
The useful retry question is rarely "how many times" — it's *which* failures, for *how long*, and *how spread out*. The policy covers all of it:
```tsx
retry: {
maxAttempts: 5, // total attempts including the first (default 3)
strategy: 'exponential', // 'exponential' | 'fixed' | 'linear'
baseDelay: '500 millis',
maxDelay: '30 seconds', // ceiling so exponential growth can't run away
factor: 2, // exponential growth factor
jitter: true, // full jitter (default true) — anti-thundering-herd
maxElapsed: '5 minutes', // a total time BUDGET — stop retrying after this
retryableErrors: ['ProviderDown'], // retry ONLY these typed errors; others fail fast
respectRetryAfter: true, // honor a 429: retryAfter REPLACES this attempt's backoff
}
```
- **`retryableErrors`** (or a `retryable: (error) => boolean` predicate) is the important one: retry the *transient* failures, fail *fast* on the permanent ones. A `ValidationError` should never be retried; a `ProviderDown` should.
- **`maxElapsed`** is a deadline across all attempts, not another count — the right bound when "keep trying for up to 5 minutes" matters more than "try 8 times".
- **`respectRetryAfter`** uses exactly the delay a provider asked for (a `retryAfterMillis` number, or `retryAfter` in seconds, on the thrown error) as the next delay, replacing the computed backoff for that attempt; falls back to backoff when there is no hint.
Retries here run *inside* the one step attempt, so the step is recorded as a SINGLE row in `_voltro_workflow_run_steps` with its final outcome — the individual in-step retries are not separate rows. `respectRetryAfter` uses the provider's delay AS the next delay (replacing the computed backoff for that attempt), falling back to backoff when the error carries no hint. For per-attempt rows in the dashboard, or a bespoke `Schedule`, use `stepModule.retry(step({...}), schedule)` instead. Do NOT set `retry:` on a step you ALSO wrap in `stepModule.retry` / `Effect.retry`: it would retry twice.
## What to retry
| Failure kind | Recommended handling |
|---|---|
| HTTP 5xx, connection reset, provider 429 | Retry the step with backoff and jitter |
| Validation error, malformed payload | Fail the workflow |
| Permission error | Fail the workflow; fix caller or payload |
| Timeout | Convert to a typed transient error, then retry if safe |
| Cancel/suspend by operator | Do not catch unless you are deliberately cleaning up |
Keep retry windows close to the side effect. A flaky LLM call should retry inside the LLM step, not by restarting the whole workflow body.
## Failed runs
When a workflow ultimately fails, Voltro records the run as `failed`. Query failed runs from any handler, action, or mutation with the typed SDK — no raw SQL, no knowledge of internal tables:
```ts
const failed = await ctx.workflows.listRuns({
status: 'failed',
workflowName: 'notes.summarise',
})
// failed: WorkflowRunSummary[] — id, workflowName, status, payload,
// errorTag, errorMessage, startedAt, completedAt, durationMs, …
```
`listRuns(filter?)` accepts `{ workflowName / tag, status, limit, offset }` (all optional) and returns the most-recent runs first. The inspect dashboard surfaces the same data live, but the SDK is the primary path — it's typed and reactive-friendly.
There is no separate `voltro_workflow_dlq` table in the current runtime. Failed runs are the triage queue.
## Retrying a failed run
Re-run a failed run by its id from a handler, action, or mutation:
```ts
for (const run of failed) {
// Re-run against the original payload …
await ctx.workflows.retry(run.id)
// … or replay against a corrected input after fixing bad data:
await ctx.workflows.retry(run.id, { payloadOverride: { ...run.payload, retries: 1 } })
}
```
`retry(runId, options?)` resolves the workflow by its tag, re-executes it, and returns `{ executionId }` — the engine-assigned id of the fresh run. By default it uses the original run's recorded payload; pass `payloadOverride` to replay against a different input.
For ad-hoc ops, the same operation is available manually — the CLI (`voltro workflows retry `) and the dashboard's "Retry" button both delegate to the same runtime code:
```sh
voltro workflows retry wfrun_01H...
```
`retry` starts a **fresh execution** — a new execution id, an empty journal, every step runs again. That is the right tool for a short, idempotent job, or when the input itself was wrong (`payloadOverride`). For a long multi-step pipeline where re-doing steps 1…N‑1 is expensive or unsafe, you want the opposite: resume from where it failed.
## Resume from where it failed — `suspendOnFailure`
Declare `suspendOnFailure: true` on a workflow and a failure no longer becomes a terminal `failed` run — it **suspends** with the durable journal intact:
```ts
export default workflow({
name: 'billing.close-month',
payload: { orgId: Schema.String },
success: Schema.Void,
error: Schema.Unknown,
idempotencyKey: ({ orgId }) => `billing.close-month:${orgId}`,
suspendOnFailure: true, // a failure suspends (recoverable), not fails (terminal)
execute: ({ orgId }) => Effect.gen(function* () {
yield* step('snapshot-ledger', /* … */) // completed steps are journaled
yield* step('call-tax-provider', /* … */) // ← a transient 503 here …
yield* step('finalise-invoices', /* … */)
}),
})
```
When `call-tax-provider` fails, the run goes to `suspended` (not `failed`), carrying the failure reason. Fix the cause, then **resume** — the engine replays `snapshot-ledger` from the journal (it does **not** re-run) and continues from the failed step:
```sh
voltro workflows resume wfrun_01H... # re-drives from the failure point
```
```ts
await ctx.workflows.resume(run.id) // the same, from a handler
```
Because a suspended-on-failure run is **recoverable, not dead**, it shows up under `--status suspended`, NOT in the [dead-letter view](/docs/workflows/debugging) (`voltro workflows list --dead-letter`, which is failed-and-unhandled). Choose per workflow: `suspendOnFailure: true` for a long pipeline where prior work must not be redone; the default (`retry` from scratch) for short idempotent jobs.
## Re-drive a failed run — `redrive`
`suspendOnFailure` is a decision you make **before** the run. What if a run already **failed** — it's sitting in the dead-letter view — and you still want to continue it from where it died, not re-run it from scratch? That is `redrive`:
```sh
voltro workflows redrive wf_01H... # re-drive a FAILED run from the step it died on
```
```ts
await ctx.workflows.redrive(run.id) // the same, from a handler; addressed by run id
```
`redrive` re-drives the run from its durable journal: every **completed** step replays from the journal (it does **not** re-run), and only the **failed** step(s) re-execute. It is the after-the-fact counterpart to `suspendOnFailure` + `resume` — same "continue from the failure point" outcome, but for a run that already went terminal without being marked suspend-on-failure. Fix the downstream cause first, then redrive.
The three recovery tools, and when each applies:
| Tool | Use when | Journal |
|---|---|---|
| `retry` | The input was wrong, or the job is short + idempotent | Fresh execution, empty journal — every step runs again |
| `resume` | The run is `suspended` (you set `suspendOnFailure: true`, or it awaits a signal) | Continues from the failure/suspend point — completed steps replay |
| `redrive` | The run is `failed` (dead-letter) and re-running earlier steps is expensive/unsafe | Continues from the failure point — completed steps replay, failed steps re-run |
| `resume-from-step` | The run is `failed`, but the failure point is **not** the right recovery point — an earlier step ran on stale/wrong state | Rewinds to a chosen step — steps before it replay, that step + everything after re-run |
`redrive` refuses a run that isn't a not-yet-discarded failure (use `resume` for a suspended run, `retry` for a fresh execution), and it needs a durable journal — on the memory store there is nothing to re-drive, so it declines cleanly (`redriven: false` with a reason) rather than pretending. It works under `voltro serve` as well as `voltro dev`, because dead-letter recovery happens in production. A re-drive records a `run-redriven` event on the run's timeline.
### Rewind further back — `resume-from-step`
`redrive` continues from the step that **failed**. Sometimes that isn't the right place to restart: step 5 failed, but the real problem is that step 3 completed against stale external state, and re-running from the failure point would carry that bad result forward. `resume-from-step` rewinds to a step **you** choose:
```sh
voltro workflows resume-from-step wf_01H... charge-card # rewind to 'charge-card' and re-run from there
```
It resets the chosen step **and every step after it** — including steps that succeeded — so they re-execute, while the steps *before* your target replay from the journal unchanged. `redrive` is the special case where your chosen step is exactly the one that failed. Like `redrive`, it works under both `voltro serve` and `voltro dev`, refuses a run that isn't a not-yet-discarded failure, refuses an unknown step name, and declines cleanly on a run with no durable journal.
## Compensation
For saga-style workflows, model compensation explicitly with `Effect.catchAll` around the step that can fail after an earlier side effect:
```ts
yield* step({
name: 'reserve-inventory',
success: Schema.Void,
execute: reserveInventory(orderId),
})
yield* step({
name: 'charge-card',
success: Schema.Void,
execute: chargeCard(orderId).pipe(
Effect.catchAll((error) =>
releaseInventory(orderId).pipe(
Effect.zipRight(Effect.fail(error)),
),
),
),
})
```
There is no `defineSaga(...)` helper today. Compensation is ordinary Effect code, checkpointed when you put it inside a step.
## Timeouts
A hung external call is not a failure until it errors. Wrap it:
```ts
const response = yield* step({
name: 'fetch-provider',
success: ProviderResponse,
error: ProviderDown,
execute: fetchProvider(input).pipe(
Effect.timeoutFail({
duration: '30 seconds',
onTimeout: () => new ProviderDown({ message: 'provider timed out' }),
}),
),
})
```
If the operation is not idempotent, pass an idempotency key to the provider and also enforce a unique key in your own database.
## Failure observability
Every run has three inspectable layers:
- `_voltro_workflow_runs` — run status, payload, output, top-level error, subject, start source, timing, trace id, parent execution id, parent-close policy.
- `_voltro_workflow_run_steps` — each step attempt, recorded input/output/error, retry metadata, duration.
- `_voltro_workflow_run_events` — run lifecycle, timers, signals, suspend/resume/cancel events.
The dashboard and `voltro workflows show ` read these same tables through inspect endpoints.
## Anti-patterns
- **Catching errors and returning success.** Operators lose the failed run.
- **Retrying non-idempotent side effects.** Double charges and duplicate emails are workflow bugs, not retry bugs.
- **Declaring `retry:` on a step you ALSO wrap in `Effect.retry` / `stepModule.retry`.** `retry:` is enforced now — the step would retry twice. Keep one.
- **Expecting a DLQ table.** The current triage surface is failed runs plus retry controls.
---
## Sleep & wakeups
_Durable sleep, cron schedules, external signals, and how parked workflows show up in inspection._
Use `sleep({...})` from `@voltro/workflow` for durable waits inside a workflow. It wraps `@effect/workflow`'s `DurableClock.sleep` and records timer events for the dashboard.
## Basic sleep
```tsx
import { workflow, step, sleep } from '@voltro/workflow'
import { Effect, Schema } from 'effect'
const buildExecute = (ctx: AppContext) =>
({ userId }: { userId: string }) =>
Effect.gen(function* () {
yield* step({
name: 'send-welcome',
success: Schema.Void,
execute: sendEmail(ctx, userId, 'welcome'),
})
yield* sleep({ name: 'retention-delay', duration: '7 days' })
yield* step({
name: 'send-retention-check-in',
success: Schema.Void,
execute: sendEmail(ctx, userId, 'retention-check-in'),
})
})
```
The workflow is not meant to hold a worker thread for the whole delay. The durable clock parks the workflow in the engine; Voltro records `timer-set` and `timer-fired` events in `_voltro_workflow_run_events`.
## Sleep until a wall-clock time
Compute a duration, then pass it to `sleep`:
```ts
const millisUntilNextThreeUtc = () => {
const now = new Date()
const target = new Date(now)
target.setUTCHours(3, 0, 0, 0)
if (target <= now) target.setUTCDate(target.getUTCDate() + 1)
return target.getTime() - now.getTime()
}
yield* sleep({
name: 'wait-until-3am-utc',
duration: `${millisUntilNextThreeUtc()} millis`,
})
```
Use this for one-off waits that are part of a larger workflow. For recurring cadence, use schedules.
## Recurring work belongs in schedules
Do not build a forever loop with `sleep` for "run every day" jobs. Use `*.cron.tsx`:
```tsx
// apps/api/schedules/daily-rollup.cron.tsx
import { defineSchedule } from '@voltro/runtime'
export default defineSchedule({
name: 'rollup.daily',
cron: '0 3 * * *',
timezone: 'UTC',
onOverlap: 'skip',
handler: async ({ app, scheduledAt }) => {
await runRollup(app.store, scheduledAt)
},
})
```
Schedules give you timezone parsing, overlap policy, backfill policy, cluster-wide coordination, run history, and manual "Run now" controls.
## External wakeups with `awaitSignal`
Use `awaitSignal(ctx, ...)` when a workflow must wait for an outside decision, such as a human approval or webhook callback.
```tsx
import { awaitSignal, step } from '@voltro/workflow'
import { Effect, Schema } from 'effect'
const decision = yield* awaitSignal(ctx, {
name: 'approval',
schema: Schema.Struct({
approved: Schema.Boolean,
reviewerId: Schema.String,
}),
timeoutMs: 24 * 60 * 60_000,
})
if (!decision.approved) {
yield* step({
name: 'cancel-order',
success: Schema.Void,
execute: cancelOrder(orderId, decision.reviewerId),
})
}
```
Signals are matched by name inside one workflow run. The payload is decoded with the supplied schema.
### Long waits: `awaitSignalSuspending`
`awaitSignal` **polls from inside a live activity** — the run keeps its worker fiber for the whole wait. Right for an approval that lands in seconds; wrong for a wait measured in hours or days, where a thousand parked runs pin a thousand fibers. For those, use the drop-in suspending variant:
```tsx
import { awaitSignalSuspending } from '@voltro/workflow'
const decision = yield* awaitSignalSuspending(ctx, {
name: 'approval',
schema: Schema.Struct({ approved: Schema.Boolean }),
timeoutMs: 3 * 24 * 60 * 60_000, // three days — a real human-in-the-loop wait
})
```
Same call shape, same senders (`ctx.workflows.signal`, the dashboard button, the HTTP endpoint resume **both** variants), but the run returns `Suspended` while parked — the worker slot is freed and the wake lives in the engine's durable store, exactly like a long `sleep`. The timeout is durable too.
**Rule of thumb: seconds → `awaitSignal`; anything a human might sleep on → `awaitSignalSuspending`.** The framework nudges you: an `awaitSignal` declaring a timeout above five minutes logs a one-time hint (once per workflow, never per poll) naming the suspending variant. Tune or effectively silence the threshold with `workflows: { suspendSignalHintMs }` in `app.config.ts`, or `VOLTRO_WORKFLOW_SUSPEND_HINT_MS` at runtime. It is a hint only — the framework never swaps the variant under a run, because the two journal differently and a silent swap mid-history is a replay trap dressed as a favour.
## Sending signals
From the dashboard, open a running workflow and use "Send signal".
From the CLI:
```sh
voltro workflows signal wfrun_01H... --name approval --payload '{"approved":true,"reviewerId":"usr_123"}'
```
From HTTP:
```http
POST /_voltro/inspect/workflows/runs/:id/signal
Content-Type: application/json
{ "signalName": "approval", "payload": { "approved": true, "reviewerId": "usr_123" } }
```
The runtime records `signal-awaited`, `signal-sent`, and `signal-received` events in `_voltro_workflow_run_events`. The polling activity inside `awaitSignal` is checkpointed; once it receives a payload, replay returns the journaled value.
## Cancellation and suspend/resume
Use operational controls, not sentinel rows:
```sh
voltro workflows cancel wfrun_01H...
voltro workflows suspend wfrun_01H...
voltro workflows resume wfrun_01H...
```
The inspect API exposes the same actions at:
```http
POST /_voltro/inspect/workflows/runs/:id/cancel
POST /_voltro/inspect/workflows/runs/:id/suspend
POST /_voltro/inspect/workflows/runs/:id/resume
```
## Anti-patterns
- **Using `Effect.sleep` for durable workflow waits.** Use `sleep({...})` so timer events are recorded and the durable clock is used.
- **Forever-loop cron workflows.** Use `*.cron.tsx` schedules for recurring cadence.
- **Polling an external system with long sleeps.** Prefer webhooks or `awaitSignal`.
- **Sleeping inside a database transaction.** Finish the write boundary before the wait.
---
## Clustering
_How Voltro scales workflows, crons, and reactivity across multiple instances — the built-in @effect/cluster runtime, schedule claims, and per-dialect CDC._
Clustering is **built in**. There is no plugin to install and no config flag to flip: run more than one instance of your app against the same database and they form a cluster automatically. Durable workflows resume on any surviving instance, each cron fires exactly once across the whole fleet, and writes on one instance become reactive subscriptions on every other.
`@effect/cluster` is a core dependency of `@voltro/workflow` — the CLI wires it for you when `voltro dev` or `voltro start` boots an app on a SQL store. You scale **out** by running N instances; the coordination machinery is already there.
## How it's enabled
There is nothing to enable in code. The wiring is automatic and driven by the store dialect:
| Store dialect | Runner storage | Horizontal scale-out |
|---|---|---|
| `postgres`, `mysql`, `mariadb`, `mssql` | `sql` | yes — workflows resume on any instance |
| `sqlite` | `memory` | no — durable replay within one process |
| `memory` (dev default) | — | single process |
On a SQL store, the runtime composes the dialect's `SqlClient` → `@effect/cluster` `SingleRunner` → `@effect/workflow` `ClusterWorkflowEngine` into one layer and provides it to your program. `sqlite` falls back to in-process durable replay (no cross-instance scale-out) because `@effect/cluster`'s runner-lock acquisition has no SQLite branch.
You do not call the cluster layer yourself — the CLI does. The boot log tells you which mode resolved:
```text
[voltro:dev] workflow engine: cluster-sql, dialect=mariadb
```
Two settings tune this for demanding topologies: **`VOLTRO_WORKFLOW_SHARD_LOCK`** keeps cross-pod handoff correct on a Galera / Percona XtraDB cluster (see [Galera and multi-primary clusters](#galera-and-multi-primary-clusters)), and **`VOLTRO_WORKFLOW_RUNNER_STORAGE`** opts out of SQL runner storage when you don't need handoff (see [Single-process runner storage](#single-process-runner-storage)).
### The deployment contract
A real multi-instance deployment has to give each pod a routable identity. This is the entire operational surface of clustering, and it is what the framework's Helm baseline injects:
- **Routable runner host (`POD_IP`).** When a workflow that started on a now-dead pod has to resume elsewhere, the surviving pod addresses the dead pod's runner by the address it advertised. The default is `localhost`, which is unreachable cross-pod. Inject the Kubernetes downward-API `status.podIP` as `POD_IP` (or set `VOLTRO_WORKFLOW_RUNNER_HOST`) so each pod advertises its real address. The runtime warns at boot if a SQL-backed runner is still on `localhost`.
- **Fixed runner port.** `SingleRunner` otherwise picks a random port, which breaks restart-resume because the runner table still references the previous process's port. Pin it via `VOLTRO_WORKFLOW_RUNNER_PORT` (default `34000`) so a restarted pod inherits the same cluster identity.
- **Unique, stable `server_id` per pod (MariaDB CDC).** The binlog reader needs a `server_id` that is unique across the fleet and stable across reschedules — duplicate ids silently break binlog streams. The runtime derives it by hashing a per-pod replica id (`POD_NAME` → FNV-1a). With a `StatefulSet` the pod name is ordinal and deterministic.
- **Replication user (MariaDB).** The binlog CDC reader connects with a dedicated user holding `REPLICATION SLAVE, REPLICATION CLIENT`, against a server configured `binlog_format=ROW`, `binlog_row_image=FULL`, `gtid_strict_mode=ON`.
The relevant downward-API wiring, from the framework's Helm baseline (the AWB 4-replica MariaDB deployment is the worked example):
```yaml
env:
- name: POD_NAME
valueFrom: { fieldRef: { fieldPath: metadata.name } } # → stable per-pod CDC server_id
- name: POD_IP
valueFrom: { fieldRef: { fieldPath: status.podIP } } # → routable workflow-runner host
- name: VOLTRO_WORKFLOW_RUNNER_HOST
value: "$(POD_IP)"
- name: VOLTRO_WORKFLOW_RUNNER_PORT
value: "34000"
- name: DB_DIALECT
value: "mariadb"
- name: CDC
value: "1"
```
That is the whole contract: stable per-pod identity (`POD_NAME` → `server_id`), routable runner host (`POD_IP`), a pinned runner port, the shared `DB_URL`, and CDC on. No application code changes between one instance and four.
### Galera and multi-primary clusters
Multi-primary MariaDB (Galera) and Percona XtraDB are first-class **and keep full cross-pod handoff** — no proxy, no single-writer endpoint, no loss of durability. This is handled automatically; the paragraphs below are the *why* for operators who need it.
By default `@effect/cluster` coordinates which runner owns which shard with **session advisory locks** (`GET_LOCK` on MySQL/MariaDB, `pg_advisory_lock` on Postgres). Advisory locks are **node-local** — a lock held on one Galera node is invisible on the others, and Galera never replicates them. So on a Galera cluster whose app pods reach the database through a load-balancing Service (connections spread across nodes), advisory-lock coordination can't see itself: pods split-brain shard ownership, and the runner-storage bootstrap can wedge before it creates `cluster_runners` (the pod stays **un-Ready** while a shard-lock refresher errors against a `cluster_runners` table it thinks is missing).
Voltro resolves this with the **`VOLTRO_WORKFLOW_SHARD_LOCK`** setting (`auto` | `row` | `advisory`, default `auto`):
- **`auto`** — probe the live connection (`@@wsrep_on`) and switch to the row-lease path on a Galera / PXC cluster, keep advisory locks everywhere else. No operator config needed.
- **`row`** — force the row-lease path. `SqlRunnerStorage` coordinates shard ownership with a **certified conditional upsert** on the `cluster_locks` table (`INSERT … ON DUPLICATE KEY UPDATE … WHERE acquired_at < `) instead of `GET_LOCK`. Galera certifies that write across *all* nodes — the same reason the cron `advisoryLock` tier uses a claims row, not a session lock — so shard ownership is coordinated correctly no matter which node a connection lands on. Cross-pod handoff works: a dead pod's lease ages out and a surviving pod takes over the shard.
- **`advisory`** — force the classic `GET_LOCK` path (single-primary only).
The boot log reports the resolved mode, and `voltro cluster status` shows it per instance (`shard-lock=row`):
```text
[voltro:serve] shard-lock coordination: row-based (dialect=mariadb, mode=auto, wsrep/Galera cluster detected)
```
**Two honest caveats.** Row-lease failover is **expiry-based, not instant** — a dead pod's shards are reclaimed after the lease timeout (seconds), where advisory locks release the instant a connection drops. That's why `auto` keeps the faster advisory path on a single-primary server and only pays the timeout on Galera. And MySQL **Group Replication** (and other non-wsrep multi-primary topologies) aren't auto-detected — set `VOLTRO_WORKFLOW_SHARD_LOCK=row` explicitly there.
### Single-process runner storage
If a deployment genuinely does **not** need cross-pod handoff — dev, a single-region single-replica service, or one where each pod owning its own workflows is acceptable — set **`VOLTRO_WORKFLOW_RUNNER_STORAGE=memory`** to skip the SQL runner storage entirely:
```text
VOLTRO_WORKFLOW_RUNNER_STORAGE=memory
```
This is the engine `sqlite` uses. It **skips `SqlRunnerStorage`** (no `cluster_runners` / `cluster_locks`, no advisory locks) while **durability is unaffected** — run/message/reply state still persists via `SqlMessageStorage`. The tradeoff is the point: **no cross-pod handoff** — a workflow started on a now-dead pod will not resume on a surviving one. Prefer the row-lease coordination above for multi-replica deployments that need handoff; reach for `memory` only when they don't.
`VOLTRO_WORKFLOW_RUNNER_STORAGE=sql` is rejected on `sqlite` / `turso` (no advisory-lock branch there), and an unrecognized `VOLTRO_WORKFLOW_SHARD_LOCK` value fails boot loudly — a misconfiguration never silently selects a broken engine.
## Durable workflows resume on any instance
Workflow durability lives in `@effect/cluster`'s engine, not in any Voltro table. Each workflow's journal (steps, idempotency keys, signals) is persisted to the cluster's storage on the shared database. When a pod dies mid-step:
1. Its runner lease stops being renewed.
2. Another runner reclaims the dead runner's shards.
3. The reclaiming runner reads the workflow journal, fast-forwards past completed steps, and resumes from the unfinished step.
Because steps are journaled with idempotency keys, a re-executed step replays its cached result rather than running its side effects twice. This is why the `POD_IP` runner host matters: the reclaiming pod must be able to reach the workflow's runner address to take it over.
`@effect/cluster` is dialect-agnostic here — its `SqlRunnerStorage` and `SqlMessageStorage` dispatch internally via `sql.onDialectOrElse({ mssql, mysql, sqlite, orElse: postgres })`. MariaDB rides the `mysql` branch (`GET_LOCK`, `ON DUPLICATE KEY UPDATE`, native `RETURNING` since 10.5).
> **mssql only:** `@effect/cluster`'s mssql storage has two driver-level bugs the framework fixes with a patch that a plain install can't carry. Run **`voltro add mssql`** once, then `pnpm install` — see [SQL Server → Workflow cluster](/docs/database/dialects/mssql#workflow-cluster). No-op on every other dialect.
## Crons fire once cluster-wide
Schedule coordination is app-wide. On a postgres store the default is `advisoryLock`; you can opt into `cluster`, or force `single` for a one-instance deployment with `app.config.ts` (`scheduling: { coordination }`). Individual `*.cron.tsx` files stay topology-agnostic.
| `coordination` | Exactly-once mechanism | When |
|---|---|---|
| `single` | none — always fires | dev, single pod |
| `advisoryLock` | claims row in `_voltro_schedule_claims` | multi-instance, default on a SQL store |
| `cluster` | `@effect/cluster` shard ownership (`ClusterCron`) | when you already run the workflow engine and want one ownership model |
**`advisoryLock`** is the lightweight tier. Every replica computes the same deterministic `scheduledAt` from the cron expression (not its own `Date.now()`, so clock skew is irrelevant), derives a claim key `${scheduleName}@${secondBucket}`, and races to `INSERT` a row into `_voltro_schedule_claims`. The primary-key conflict makes exactly one replica win; the rest lose cleanly and skip. The second-precision bucket makes claims self-expiring — a crashed winner doesn't block the next firing, because the next firing is a new bucket and therefore a new key. This is deliberately a claims **row**, not a session-level lock, because a connection pool can hand a `GET_LOCK`/`pg_advisory_lock` connection to another query before the unlock.
**`cluster`** rides the same sharding machinery as durable workflows: `ClusterCron` fires on exactly one runner (the shard owner) and migrates ownership on runner death. The in-app timer is disarmed in this mode (`@effect/cluster` owns the clock); the firing is recorded with `coordinationOutcome: 'cluster'`. It needs a SQL store with cluster storage (everything except `sqlite`/`memory`), and degrades to `single` with a warning when that's unavailable.
Overlap policy (`onOverlap: 'skip' | 'queue' | 'parallel'`) is also cluster-aware: `skip` checks for a `status='running'` row from **any** pod, not just the local one, so a long run on pod A suppresses a new firing on pod B.
## Cross-instance reactivity (per dialect)
A write on instance A has to become a reactive subscription delta on instance B. The transport is change-data-capture, and it is **not** Postgres-only — each dialect uses its native change feed:
| Dialect | Transport | Latency |
|---|---|---|
| `postgres` | `LISTEN`/`NOTIFY` | fast |
| `mariadb` | binlog CDC (ROW image, GTID) | push-based, per-replica reader |
| `mysql` | inline emit only (no binlog CDC) | single-process |
| `sqlite` | in-process bus | single-process |
On MariaDB the binlog **is** the message bus — each replica tails it itself, resuming from its own `_voltro_cdc_offsets` row keyed by replica id. There is no Redis, NATS, or external broker. CDC is on by default for any SQL dialect (`CDC=1`); set `CDC=0` to fall back to single-process inline emit. The boot log reports the resolved transport:
```text
[voltro:dev] sql dialect resolved: mariadb — CDC: binlog CDC (ROW), RETURNING: native (INSERT/DELETE); UPDATE then SELECT
```
## Observability
- **Dashboard.** Workflow runs, steps, and schedule runs stream live to the devtools/cloud dashboard.
- **Tables.** Introspection rows live on the shared database — query them directly:
- `_voltro_workflow_start_contexts` — durable starter subject/trace/source plus parent execution id and parent-close policy, loaded by whichever runner first executes the workflow or child workflow.
- `_voltro_workflow_runs`, `_voltro_workflow_run_steps`, `_voltro_workflow_run_events` — workflow execution history (these are fire-and-forget introspection; durability itself lives in the cluster engine's own tables).
- `_voltro_schedule_runs` — every firing, with `replicaId` (which instance fired it) and `coordinationOutcome` (`single` / `wonLock` / `cluster` / …).
- `_voltro_schedule_claims` — the live exactly-once claim rows for `advisoryLock` schedules.
- **Logs and traces.** `voltro logs --tail 100` shows the resolved dialect, CDC flavor, coordinator, and runner identity at boot, plus every firing and resume. `voltro traces` correlates a workflow's steps across pods.
### `voltro cluster status`
`voltro cluster status` renders the clustering snapshot of every running api — discovered the same way as `voltro logs` / `voltro traces` (the runtime registry), then merged with the live coordination state each instance reads from the shared store:
```sh
voltro cluster status # pretty table of every instance
voltro cluster status --process myApi # restrict to one registered api
voltro cluster status --format json # machine-readable, pipe to jq
```
For each instance it shows the `replicaId`, the runner address (`host:port`), the resolved dialect and clustering tier (full / inline-only / single-process / none), the CDC flavor, the coordination kind, and — on the MariaDB CDC path — the binlog `server_id`. A SQL-backed runner advertising `localhost` / `127.0.0.1` is flagged loudly: other pods can't reach it, so cross-pod workflow resume silently breaks (inject `POD_IP` via the K8s downward API or set `VOLTRO_WORKFLOW_RUNNER_HOST`). A footer summarises the shared coordination state — active replicas seen in recent runs, current schedule claims, and in-flight schedule + workflow runs. The same view is on the dashboard's **Cluster** panel.
Quick "which instance is running what" query:
```sql
SELECT "replicaId", count(*)
FROM "_voltro_schedule_runs"
WHERE status = 'running'
GROUP BY "replicaId";
```
## The multi-dialect story
Clustering works on every SQL dialect, with one capability gradient:
- **`postgres` / `mariadb` / `mssql`** — full clustering: durable workflow resume across instances, `advisoryLock` or `cluster` crons, native cross-instance reactivity (`LISTEN`/`NOTIFY` on postgres, binlog CDC on mariadb).
- **`mysql`** — durable workflow resume and schedule coordination work; cross-instance reactivity is inline-only (no binlog CDC path), so subscriptions don't fan out across instances.
- **`sqlite`** — durable workflow **replay** within a single process (`runnerStorage: 'memory'`); no horizontal scale-out.
- **`memory`** — single-process dev store; no clustering.
The same app code runs unchanged across all of them — the runtime resolves the right runner storage, coordinator, and CDC transport from the dialect at boot.
> **`clusterPlugin()` — deliberately not built.** A convenience marker plugin (declaring an app "expects multi-instance", optionally pinning the runner host/port) was considered and rejected: clustering is already automatic for any SQL store with N instances, so there is no behaviour for such a plugin to toggle. Its only knobs — runner host/port — are env-driven (`VOLTRO_WORKFLOW_RUNNER_HOST` / `VOLTRO_WORKFLOW_RUNNER_PORT`) and injected by the deployment (K8s downward API), which is where pod identity belongs. The idea is on record but will not ship.
## Anti-patterns
- **Assuming clustering is Postgres-only.** It isn't. MariaDB is a first-class clustering target (binlog CDC), and the AWB 4-replica MariaDB deployment is the production reference. `wal_level=logical` is a Postgres detail, not a clustering requirement.
- **Leaving the runner host on `localhost` in multi-pod deployments.** Resume-on-another-pod silently breaks — pods can't reach a `localhost` runner. Inject `POD_IP`. The runtime warns about this at boot.
- **Reusing a `server_id` across MariaDB pods.** Duplicate `server_id` silently breaks binlog streams. Derive it from a stable per-pod `POD_NAME` (the framework does this for you).
- **Session-level locks for cron coordination.** `advisoryLock` uses a claims **row**, not `pg_advisory_lock`/`GET_LOCK`, precisely because pooled connections make session locks unreliable. Don't reach for session locks.
---
## Flow control
_Durable queues, bounded concurrency, and rate limits inside workflow code._
Use flow control when the workflow is durable, but the resource it touches is limited: a third-party API, a tenant import lane, a GPU job, or a webhook fan-out.
Voltro exposes the workflow engine's durable queue and rate limiter through `@voltro/workflow`:
> **This page is about primitives you call INSIDE a running workflow.** For declaring `debounce` / `singleton` / `concurrency` / `throttle` / `rateLimit` / `batch` on the workflow itself — enforced before a run exists — see [Declarative flow control](/docs/workflows/declarative-flow-control).
```ts
import {
durableQueue,
processQueue,
queueWorker,
rateLimit,
step,
workflow,
} from '@voltro/workflow'
import { Effect, Schema } from 'effect'
```
## Bounded concurrency
Define a queue once, then process items from workflow code. A worker layer controls concurrency and is backed by the same SQL/cluster workflow engine, so another runner can resume after a crash.
```ts
export const ThumbnailQueue = durableQueue({
name: 'thumbnails',
payload: {
imageId: Schema.String,
sourceUrl: Schema.String,
},
success: Schema.Struct({ thumbnailUrl: Schema.String }),
idempotencyKey: ({ imageId }) => imageId,
})
export const thumbnailWorker = queueWorker(
ThumbnailQueue,
({ imageId, sourceUrl }) =>
step({
name: 'render-thumbnail',
input: { imageId },
success: Schema.Struct({ thumbnailUrl: Schema.String }),
execute: renderThumbnail(sourceUrl),
}),
{ concurrency: 4 },
)
```
Export the worker from any discovered `*.workflow.tsx` module. `voltro dev` and `voltro serve` auto-mount branded `queueWorker(...)` layers at boot; no `app.config.ts` layer plumbing is needed.
Inside a workflow:
```ts
const result = yield* processQueue(ThumbnailQueue, {
imageId,
sourceUrl,
})
```
`processQueue(...)` is a workflow activity. The item is persisted, the workflow parks while a worker processes it, and replay returns the recorded worker result instead of re-enqueueing.
## Rate limits
Use `rateLimit(...)` before a step that talks to a constrained service:
```ts
yield* rateLimit({
name: 'stripe-write',
window: '1 minute',
limit: 100,
key: tenantId,
})
const invoice = yield* step({
name: 'create-invoice',
input: { tenantId, orderId },
execute: createStripeInvoice(orderId),
})
```
The wait uses the durable workflow clock. A deploy or worker crash does not lose the delay.
## Dashboard behavior
Queue workers and rate-limited steps still record normal workflow steps, timers, and run events. The Workflows dashboard shows the parked run, the step that is waiting, and the later continuation through the same live run/step/event subscriptions. The Flow tab groups visible runs by workflow lane, queued/running/waiting status, and start source so pressure is visible in both local devtools and Voltro Cloud.
## Rules of thumb
- Use a durable queue for **bounded concurrency** and per-resource lanes.
- Use `rateLimit(...)` for **API budgets** where excess work should wait, not fail.
- Put the limiting key in your payload or workflow state so replay is deterministic.
- Keep external I/O inside `step(...)` or the queue worker body.
---
## Declarative flow control
_debounce, singleton, concurrency, throttle, rateLimit, batch, priority, timeouts and onFailure — declared on the workflow, enforced before the run exists._
Everything on this page is declared on `workflow({...})` and enforced at the **admission boundary** — the moment `ctx.workflows.start(...)` is called, before a durable run exists.
That timing is the whole point. Once a run is enqueued, the only tools left are cancel and sleep, and neither of them un-spends the durable entity. So "run this at most once per row per fifteen minutes" cannot be a primitive you call inside the body; it has to be a property of the declaration.
> **Not the same as [Flow control](/docs/workflows/flow-control).** That page covers `durableQueue` / `processQueue` / `rateLimit` — primitives you call **inside** a running workflow to bound the work it fans out. This page is about whether the run **starts at all**. They compose: a workflow can declare `concurrency` here and still use a durable queue in its body.
## The shape
```ts
import { Schema } from 'effect'
import { workflow } from '@voltro/workflow/define'
export const tourNarration = workflow({
name: 'tourNarration',
payload: Schema.Struct({
rowId: Schema.String,
tenantId: Schema.String,
editedAt: Schema.Number,
}),
success: Schema.Void,
idempotencyKey: ({ rowId, editedAt }) => `tour:${rowId}:${editedAt}`,
debounce: { key: (p) => `tour:${p.rowId}`, period: '15 minutes' },
concurrency: { limit: 5, key: (p) => p.tenantId },
timeouts: { start: '1 hour', finish: '10 minutes' },
onFailure: 'narrationFailed',
})
```
Every `key` callback receives the workflow's own **decoded payload type**. A misspelled field is a compile error, not a key that quietly becomes the string `"undefined"` and collapses every row in your deployment into one bucket.
## `idempotencyKey` is the execution's identity — read this first
This is the single most expensive misunderstanding in the workflow API, and getting it wrong produces a design that looks right and silently stops working.
`idempotencyKey` is **not** a dedupe window. It is the execution's identity, permanently:
```ts
const a = yield* wf.execute({ id: 'same' })
const b = yield* wf.execute({ id: 'same' }) // does NOT run — replays a's result
```
After the run completes the key is **spent**. A later, genuinely new invocation under that key is a silent no-op that returns the old output. Nothing errors and nothing logs, because from the engine's point of view you asked for a run it already has.
So a key must be unique **per unit of work you want to happen**:
| | |
|---|---|
| `` `tour:${rowId}` `` | wrong if the tour can ever be re-narrated |
| `` `tour:${rowId}:${editedAt}` `` | right — every edit is a new unit of work |
### And a flow-control key is a different thing
Conflating the two is what makes "I need to re-arm a key" feel like a missing feature. It is not missing; it is two fields:
- **`idempotencyKey`** — the execution's identity. **Varies** per unit of work.
- **`debounce.key` / `singleton.key` / `concurrency.key`** — the **resource** runs compete for. **Stable**.
"One job, fifteen minutes after the last edit, latest state wins" is then the example at the top of this page: twenty edits mint twenty identities, and exactly one is ever admitted. There is no `restart: true` in this API because separating the two keys *is* the mechanism it would have been.
## debounce — collapse a burst
```ts
debounce: { key: (p) => `tour:${p.rowId}`, period: '15 minutes', timeout: '1 hour' }
```
Starts sharing a key collapse into **one** pending row. The timer resets on every arrival, and the **latest payload wins** — which is what "narrate what settled" means.
`timeout` is a hard cap measured from the **first** start in the burst. It is optional and uncapped when unset, which is a real trade: an unbroken stream of starts arriving faster than `period` defers the run forever. We do not invent a default cap and we do not warn — instead the starvation is a number you can see:
```
voltro workflows flow
```
A `waiting=` climbing past a few multiples of your `period` is the signal. Set `timeout` when you see it, or from the start if the burst is user-driven.
## singleton — one run per resource
```ts
singleton: { key: (p) => p.tenantId, mode: 'skip' } // newcomer stands down
singleton: { key: (p) => p.tenantId, mode: 'cancel' } // newcomer evicts the incumbent
```
There is no default `mode`: `'skip'` discards the incoming request and `'cancel'` discards the running one, and choosing for you would silently throw away work either way.
Under `'skip'`, `start()` returns the **incumbent's** handle — a real, pollable run. Under `'cancel'`, the incumbent is cancelled **when the replacement actually starts**, not when it is queued. That matters when you also declare `debounce`: evicting at queue time would leave the whole quiet period with the old run dead and the new one not yet begun.
## concurrency — bound what is in flight
```ts
concurrency: { limit: 5, key: (p) => p.tenantId }
```
At most `limit` runs in flight per key, **across every replica** — the count rides the shared admissions ledger, so three replicas with `limit: 5` are five runs, not fifteen. Excess starts queue and are admitted as slots free.
### `pool` — share one budget across workflows
```ts
// embeddings.workflow.tsx
concurrency: { limit: 10, pool: 'openai' }
// summarize.workflow.tsx
concurrency: { limit: 10, pool: 'openai' }
```
Without `pool`, the limit bounds *this workflow's* runs. With it, every workflow declaring the same pool name competes for **one** budget — the shape a rate-limited provider forces: five workflows that each call OpenAI must share ten slots, not hold ten each.
`key` still partitions *within* the pool: give each member `key: (p) => p.tenantId` and the shared budget applies per tenant.
Every member of a pool must declare the **same `limit`** — the boot fails otherwise. Two limits for one budget is a contradiction, and silently picking either would enforce a number somebody did not write.
## throttle vs rateLimit — late, or gone
They are mutually exclusive, and declaring both is a boot error.
```ts
throttle: { limit: 100, period: '1 minute', key: (p) => p.tenantId } // QUEUES the excess
rateLimit: { limit: 100, period: '1 minute', key: (p) => p.tenantId } // DROPS the excess
```
Reach for `throttle` when every start must eventually run. Reach for `rateLimit` when the excess is genuinely surplus and running it late is worse than not running it.
A dropped start is never silent: `start()` resolves to a handle with `status: 'dropped'` and a `retryAfterMs`, and the drop is a row in the admissions ledger with its key and reason.
`throttle` has no `burst` knob. The window is sliding, so its maximum instantaneous burst is already exactly `limit`; a separate knob could only duplicate it.
## batch — many starts, one run
```ts
export const refreshIssues = workflow({
name: 'refreshIssues',
payload: Schema.Struct({ items: Schema.Array(IssueRef) }),
success: Schema.Void,
idempotencyKey: ({ items }) => `refresh:${items.length}:${items[0]?.issueKey ?? ''}`,
batch: { item: IssueRef, key: (i) => i.tenantId, maxSize: 100, timeout: '30 seconds' },
})
await ctx.workflows.start('refreshIssues', { tenantId, issueKey: 'ABC-1' })
```
Callers start it with a **single item**; the workflow's own `payload` is the **batch** shape. That mismatch is checked at declaration time — a decode failure on the batching replica is a failure nobody is watching.
`batch.item` is therefore also what an **arriving** start is validated against. A caller's payload is judged by the item schema; the run the drainer eventually starts is judged by the workflow's own `payload`. Two schemas, because there are genuinely two shapes — and a `WorkflowPayloadError` naming `items` on a `start()` call would be the framework asking the caller for the batch it is supposed to be building.
The timeout is a **deadline**, not a quiet period: it does not reset per item, or a steady trickle would never flush. `batch` and `debounce` cannot both be declared for exactly that reason.
## priority
```ts
priority: (p) => (p.urgent ? 100 : 0)
```
Higher runs first out of the pending queue. Ties break by arrival, so an all-default deployment is FIFO rather than dialect-dependent.
> **Scope — read before porting BullMQ priority lanes.** `priority` orders **the admission queue only**: the pending rows a deferring control (debounce, batch, throttle, concurrency), a pause, or a delayed `{ at }` start has parked. A start that is admitted immediately never competes with anything — it goes straight to the engine, whatever any other start's priority says. There are no cross-workflow priority lanes, no preemption of running work, and no ordering between two starts that both found a free slot. If urgent starts must overtake normal ones, put both classes behind the same `concurrency` limit (or a shared `pool`), so every start passes through the queue that `priority` orders.
## timeouts
```ts
timeouts: { start: '1 hour', finish: '10 minutes' }
```
- **`start`** bounds how long a start may sit in the admission queue, measured from the **first** arrival in its group. A debounced run that never gets a quiet moment is a job that silently did not happen.
- **`finish`** bounds the run itself once admitted, and also tightens the crash backstop on its concurrency slot.
Both expire into the same path as an exhausted retry: the run is recorded failed and `onFailure` fires. Declaring `timeouts.start` on a workflow that cannot defer is a boot error — it would never fire.
## onFailure — the signal that replaces the sweep
```ts
onFailure: 'narrationFailed'
```
A **workflow name**, not a function. A closure cannot be journaled: the failure may be noticed by a different replica, minutes later, after the process that held it is gone.
It fires for every way a run fails to deliver — not only an exhausted retry:
- the body failed and retries are spent
- `timeouts.finish` cancelled an overrunning run
- `timeouts.start` expired a start that never got a slot
- the workflow was renamed away while starts were queued
The last two produce **no run row at all**, which is exactly why polling `listRuns({ status: 'failed' })` could never see them.
The named workflow's payload is `WorkflowFailureReport`:
```ts
export const narrationFailed = workflow({
name: 'narrationFailed',
payload: Schema.Struct({
workflow: Schema.String,
payload: Schema.Unknown,
errorTag: Schema.NullOr(Schema.String),
errorMessage: Schema.NullOr(Schema.String),
runId: Schema.NullOr(Schema.String),
executionId: Schema.NullOr(Schema.String),
reason: Schema.String,
failedAt: Schema.Number,
}),
success: Schema.Void,
idempotencyKey: ({ runId, failedAt }) => `failed:${runId ?? 'none'}:${failedAt}`,
})
```
There is no `onFailure` for an `onFailure` — a handler that fails is logged and not re-notified, because the alternative is one run per failure per level with no floor.
## encryptSteps
```ts
encryptSteps: true
```
`step({ input })` is journaled and shown in the dashboard, which is a feature and the reason people pass rich input. For a step carrying personal data it is also a **second copy** outside the `.encrypted()` boundary the governance plugin establishes for tables.
`encryptSteps` closes it, reusing the **same** cipher — one key, one rotation story:
```ts
governancePlugin({ fieldEncryption: { secret: 'VOLTRO_FIELD_ENCRYPTION_KEY' } })
```
Declaring it without that plugin configured is a **boot refusal**, not a warning. A plaintext fallback would leave the declaration reading as protection while every step input sat readable.
## cancelOn — stop live work when a correlated event arrives
```ts
workflow({
name: 'tourNarration',
payload: { rowId: Schema.String, issueKey: Schema.String },
idempotencyKey: (p) => `tour:${p.rowId}`,
cancelOn: [{
event: 'jira.issue.deleted',
schema: JiraIssueDeleted,
match: (event, payload) => event.issueKey === payload.issueKey,
}],
})
```
Both sides are typed: `event` from the entry's own `schema`, `payload` from the workflow's.
**Why it is a declaration and not a race inside the body.** You can express "stop when the issue is deleted" with `awaitEvent` and an interrupt. That works while the body is *running*. It does not work while the run is sleeping for six hours, suspended on a signal, or still sitting in the admission queue — which is the case you wanted cancellation for. The event has to reach a run whose fiber is not executing anything, and only something outside the body can do that.
So it is swept: a coordinated tick reads events published since a durable watermark, resolves each declaring workflow's live runs, and cancels the ones that correlate.
**It also discards queued starts.** Cancelling only the running one leaves a debounced or concurrency-queued duplicate to start seconds later, against the row that was just deleted — the exact outcome the declaration was meant to prevent, arriving late enough that nobody connects the two.
| Field | Meaning |
|---|---|
| `event` | The name, exactly as `ctx.events.publish` writes it |
| `schema` | Decoded before `match` runs. An event whose shape does not decode is **reported and never matched** — cancelling on an event you could not read is cancelling blind |
| `match` | Required. Write `match: () => true` if you really mean "every live run" |
| `within` | Only cancel runs started within this window before the event |
| `reason` | Recorded on the run's `run-cancelled` event; defaults to `cancelOn:` |
`match` has **no default** for the same reason `singleton.mode` has none: the omitted case is "cancel every live run of this workflow", which is a legitimate thing to want and a catastrophic thing to acquire by forgetting a line.
**A run that started *after* the event is never cancelled.** Without that rule, a sweep catching up after a deployment gap reads an hour of history and kills runs that started in the meantime — and the symptom (fresh work cancelled for no visible reason) looks nothing like its cause (a restart).
## Bulk cancel and bulk replay
A bad deploy leaves four thousand runs that must all stop, or four thousand that must all be re-driven once the downstream is fixed.
```
voltro workflows cancel-many --workflow tourNarration --reason "bad deploy"
voltro workflows cancel-many --workflow tourNarration --reason "bad deploy" --commit
voltro workflows replay-many --status failed --mode redrive --limit 200 --commit
```
Three things are deliberately stricter than the obvious design:
- **`--limit` is required and there is no "all".** The cap *is* the blast radius. `truncated` in the result says whether more matched, so "did I get all of them" stays answerable without an unbounded verb ever existing.
- **It is a dry run unless you pass `--commit`.** The default for a verb that can stop a thousand runs is the one that stops none. The dashboard panel enforces the same order: the apply button does not exist until a preview has returned a number.
- **`--reason` is required for a cancel.** It lands on every affected run's `run-cancelled` event, so "why did four thousand runs stop on the 8th" has an answer in the table an operator is already reading.
The result is per-run, not a count: `succeeded`, `failed` (with the reason for each) and `skipped` (with what made each ineligible) are three different outcomes. A bulk op that reports "4000 cancelled" while forty failed is how people learn not to trust bulk ops.
Eligibility is fixed by the verb: a cancel acts on `running` and `suspended` runs; `replay --mode redrive` on `failed` only (redrive resumes from the step that died, which only exists for a failure); `replay --mode retry` on `failed` and `cancelled`. `--mode` has no default because the two cost very different amounts.
## Delayed one-off starts — `start(..., { at })`
Run this once, later — without a schedule and without a `sleep` at the top of the body:
```ts
await ctx.workflows.start('orders.remind', { orderId }, {
at: new Date(Date.now() + 24 * 60 * 60_000), // tomorrow, this time
})
```
The start is parked as a **durable row** in the same pending queue the controls above use (`mode: 'delayed'`), and the coordinated drainer fires it when `at` arrives — it survives restarts and fires on whichever replica drains, never from an in-process timer. The handle comes back `status: 'queued'` with `deferral: { mode: 'delayed', dueAt }`.
Semantics worth knowing:
- **`at` is an absolute instant, deliberately** — not a `delay` duration. A delay is measured "from when?" (enqueue? admission? retry?) and every queueing system answers differently; an instant has no such ambiguity and composes with the schedule/backfill surfaces, which are also instant-based. A relative delay is one line: `at: new Date(Date.now() + ms)`.
- **At `at`, the start becomes an ordinary ARRIVAL.** Declared controls judge it as of that moment — a debounce collapses it into whatever window is open then, a rate cap can drop it (recorded, as always). `{ at }` delays the arrival; it never outranks a control.
- An `at` in the past starts immediately — "no earlier than" is already satisfied.
- `{ at, wait: true }` is refused: there is no result to block on for a start that exists only as a future row.
- Each `{ at }` start is its own row. Two delayed starts never collapse into one — unlike debounce, nothing about `{ at }` says the second supersedes the first. Want collapsing? That is `debounce`, and they compose.
Prefer this over `sleep` as the first step of the body when the wait precedes the work: a parked row costs one row, a sleeping run costs a durable execution the whole time.
## What a deferred start returns
`start()` no longer always returns a running handle:
```ts
const handle = await ctx.workflows.start('tourNarration', payload)
if (handle.status === 'queued') { /* handle.deferral.dueAt tells you when */ }
if (handle.status === 'dropped') { /* over a rateLimit cap; it will NOT run */ }
if (handle.status === 'skipped') { /* handle.executionId is the incumbent */ }
```
`executionId` is `null` for `queued` and `dropped`, because there is no execution and there may never be one. Inventing an id there would produce a handle that polls `status: 'unknown'` forever.
### Blocking callers
`ctx.workflows.run(...)` and `start({ wait: true })` block for the run's **result**, and a start that was collapsed into a future run has none. A workflow declaring `debounce` / `batch` / `throttle` / `concurrency` therefore **refuses** those callers with an error naming both halves. Controls with a synchronous answer — `singleton`, `rateLimit` — keep working on every path (`rateLimit` throws `WorkflowRateLimitedError`, `singleton: 'skip'` throws `WorkflowSingletonHeldError` carrying the incumbent's id).
## Pausing a workflow
```
voltro workflows pause tourNarration --reason "deploying a fix"
voltro workflows unpause tourNarration
```
A pause makes starts **collect**, never discard — so you come back to a backlog rather than a hole in the data. `unpause` drains it.
The pause is a row, so it applies fleet-wide; each replica picks it up on its next drain tick (~1 s).
## Seeing what happened
```
voltro workflows flow
voltro workflows flow --workflow tourNarration --format json
```
or `GET /_voltro/inspect/workflows/flow-control`.
This is not optional colour. A debounce that collapses nineteen starts into one is indisputably correct behaviour **and** indistinguishable from nineteen starts vanishing — unless something writes down that it happened. So every decision is a row in `_voltro_workflow_admissions`, with its key, its reason, how many starts folded into it, and how long it waited.
Three questions it answers:
| Question | Where |
|---|---|
| "Twenty edits, one run — did that work, or did I lose nineteen?" | `collapsed` |
| "Nothing has run for an hour. Stuck, or quiet?" | `waiting=` on the queued row |
| "Why did *my* run not start?" | the ledger's `outcome` + `reason` |
## Cost
A workflow that declares **no** control takes exactly the code path it took before this feature existed — no query, no branch beyond one map lookup.
A workflow that declares one pays only for that one: an undeclared control costs zero round trips. The rate/throttle window reads at most `limit` rows, which is why `limit` is a throughput knob and not somewhere to put 10⁶.
The drainer runs on **one replica per tick** through the same claim arbiter the cron scheduler uses. N replicas draining at once would each see a free slot and each take it.
### Measured admission throughput
Measured, not estimated — `node packages/cli/scripts/admission-throughput.mjs` in the framework repo drives the real gate, facade and drainer against the in-memory reference store (slope across N=500/1000/2000 starts, median of 3 sequential repeats; Apple-silicon dev machine, 2026-08):
- **no controls (passthrough): ~1 µs/start (~800,000 starts/s)** — the do-nothing path really does nothing.
- **through a concurrency gate: ~110 µs/start (~9,000 starts/s)** — dominated by the reference store's *unindexed* admission-state scan, which grows with the ledger; a real dialect serves that read from an index, but also adds its round-trips. Read this as the machinery's worst-case CPU floor, not a database benchmark.
- **durable park → drain → start: ~4 µs/row (~240,000 rows/s)** of pure machinery per queued row.
The deployed ceiling is `min(these numbers, what your database serves for the admission reads/writes)` — on any SQL dialect the database is the bound long before the gate is. That is also why there is no key-group batching in the admission path: at ~9k gated starts/s worst-case CPU, batching would add a flush boundary to a path whose bound is elsewhere.
Per **step**, the recorder adds exactly **2 fire-and-forget store writes** (insert at step start, update at settle), off the step's critical path — plus the cluster engine's own journal write, which is the durability you asked for. Hot high-step workflows can turn the introspection copy off: `workflows: { recording: 'coarse' }` — see [Debugging](/docs/workflows/debugging).
---
## Versioning
_Workflow definition versions, compatibility metadata, in-body patch markers, and the replay nondeterminism tripwire._
Long-running workflows can outlive a deploy. Voltro does not run old JavaScript forever; a resumed run executes the current code. Make that explicit by versioning the workflow definition.
```ts
import { workflow } from '@voltro/workflow'
import { Schema } from 'effect'
export const ImportCustomers = workflow({
name: 'customers.import',
payload: { uploadId: Schema.String },
success: Schema.Struct({ imported: Schema.Number }),
idempotencyKey: ({ uploadId }) => uploadId,
version: 3,
compatibleWith: [2, 3],
patches: ['split-validate-and-write'],
})
```
Voltro stores `workflowVersion` and `workflowPatches` on every `_voltro_workflow_runs` row when the run starts. The local devtools and Voltro Cloud dashboard show the version chip on run rows, so operators can spot old or incompatible runs during a deploy.
## Compatibility
`compatibleWith` documents which run versions the current code can still resume. It is enforced: a resuming run whose stored `workflowVersion` is not listed is **terminally failed** with `WorkflowVersionIncompatible`.
That makes it a blunt instrument, and the bluntness is the point to understand before you reach for it. Bump `version` and leave the old one out, and every in-flight run on the old version dies. Do not bump, and those runs replay against the changed body with no protection at all. Neither is what you usually want — which is what `patches` is for.
## Patch markers
A patch marker lets the **body itself** branch, so runs that started before a change finish on the old path while new runs take the new one. This is the middle option between "kill the in-flight runs" and "hope the replay works out".
```ts
import { patch, step, workflow } from '@voltro/workflow'
export const Charge = workflow({
name: 'billing.charge',
payload: { orderId: Schema.String },
idempotencyKey: ({ orderId }) => `billing.charge:${orderId}`,
patches: ['split-tax-calculation'],
})
export default () => (payload) =>
Effect.gen(function* () {
if (yield* patch('split-tax-calculation')) {
const net = yield* step({ name: 'net-total', execute: computeNet(payload) })
const tax = yield* step({ name: 'tax', execute: computeTax(payload) })
return net + tax
}
return yield* step({ name: 'total', execute: computeTotal(payload) })
})
```
The answer is pinned to the **run**, not to the deployed code. `patches` is stamped onto `_voltro_workflow_runs.workflowPatches` when the run starts and read back from that row on every resume, so:
- a run started **before** you added the marker answers `false` for the rest of its life, however many times it replays;
- a run started **after** answers `true`, and keeps answering `true` even if you later change the declaration.
That is what makes the branch deterministic across a redeploy. Outside a recorded workflow body — a unit test, a bare `step()` call — `patch()` is `false`, which is the pre-patch path.
### Retiring a patch
Once no run predating the marker can still be in flight, delete the old branch and remove the entry from `patches`. Runs that stamped it keep the marker on their row for the audit trail; `patch()` simply stops being called.
## The replay nondeterminism tripwire
Journal entries are keyed by step **name**, with no shape check. Edit a workflow body while runs are in flight and the engine replays the cached result for every name that still matches and freshly executes every name that does not. Nothing errors. A renamed step re-runs a side effect the run already performed; a removed step silently skips work the journal says was done.
Voltro watches for that. A run re-entering its body compares the steps it **reaches** against the steps it recorded on earlier attempts, and writes a `nondeterminism-suspected` event when they disagree:
| Finding | Means |
|---|---|
| `unreached-step` | A step this run ran on an earlier attempt that the current code never reaches — renamed, removed, or moved behind a branch. Its journaled result is orphaned. |
| `extra-step-occurrence` | A recorded step reached more times than it was ever recorded — typically a loop bound that changed under a live run. |
The comparison is set membership plus a per-name count, never a total order: concurrent steps interleave differently on every attempt, so an order check would report correct code as broken.
**It is an event, never a failure.** The run keeps going and reaches its normal outcome. The checks sit on a best-effort recorder, so a lost step-row write is enough to make one fire — a false positive that killed a run would be worse than the divergence it suspects. Treat the event as "open this run and look", not as an outage.
The tripwire covers every path that replays an existing journal, including `voltro workflows redrive`. It is off on a first body entry (there is nothing to compare) and on a run with more than `VOLTRO_WORKFLOW_REPLAY_SHAPE_LIMIT` recorded steps (default 2000), where a truncated history would manufacture its own false positives.
## Rules
- Bump `version` only when old runs genuinely cannot continue — it kills them.
- Reach for `patches` first for a body change that in-flight runs should not see.
- Keep old payload decoders inside the workflow body only while their version remains compatible.
- Prefer additive payload changes with defaults over breaking changes.
- Use the dashboard version chip during deploys to find runs that started on an older contract.
- Treat a `nondeterminism-suspected` event as a deploy that needed a patch marker and did not get one.
---
## Debugging & inspection
_The dashboard's Workflows panel, inspect endpoints, CLI controls, workflow tables, and testing helpers._
Voltro records workflow runs, step attempts, and lifecycle events into framework tables. The dashboard, CLI, and inspect endpoints all read the same data.
## What is recorded
| Table | Contents |
|---|---|
| `_voltro_workflow_start_contexts` | One row per started execution id: starter subject, trace id, source, parent execution id, parent-close policy, and creation time. Used for cross-runner context handoff. |
| `_voltro_workflow_runs` | One row per run: `id`, `tag`, `executionId`, `status`, `payload`, `workflowVersion`, `workflowPatches`, `output`, error fields, subject, start source, timing, trace id, parent execution id, parent-close policy, plus the crash-loop bookkeeping `runnerEnteredAt` + `reclaimCount`. |
| `_voltro_workflow_run_steps` | One row per step attempt: step name, attempt number, recorded input, retry metadata, output or error, duration. |
| `_voltro_workflow_run_events` | Lifecycle events: `run-started`, `run-succeeded`, `run-failed`, `run-suspended`, `run-resumed`, `run-redriven`, `run-cancelled`, `timer-set`, `timer-fired`, `signal-awaited`, `signal-sent`, `signal-received`, `update-requested`, `update-received`, `update-completed`, `update-failed`, `nondeterminism-suspected`, `run-crashlooped`, `run-stalled`. |
| `_voltro_workflow_events` | Domain events emitted through `ctx.events.publish(...)`: event id, name, payload, source, subject, trace id, occurred time. |
| `_voltro_workflow_event_deliveries` | One row per workflow trigger delivery: event id, trigger id, workflow name, execution id, idempotency key, status, error. |
Run status is `running`, `succeeded`, `failed`, `cancelled`, or `suspended`.
### Turning the step rows off — `workflows.recording: 'coarse'`
Every `step()` costs two fire-and-forget writes to `_voltro_workflow_run_steps` (insert at start, update at settle) — measured, and off the step's critical path, but on a hot workflow with many steps they dominate the table's growth. `'coarse'` skips **both**:
```ts
// app.config.ts
export default {
workflows: { recording: 'coarse' }, // default: 'full'
}
```
Runtime override without a rebuild: `VOLTRO_WORKFLOW_RECORDING=coarse` (env wins over config; anything unrecognised falls back to `'full'`).
What `'coarse'` does **not** touch, deliberately: run rows, run events (signals, timers, cancels, stall reports — everything the dead-letter view and the sweeps read), and the cluster engine's own durable journal — replay and redrive are unaffected. The cost is exactly the dashboard's step timeline: empty for runs recorded under `'coarse'`. Flip it back when you need to see inside a run.
## Dead-letter: triaging failed runs
The framework does not retry a workflow on its own (see [Retries](/docs/workflows/retries) — you compose retries inside `execute:` with `Effect.retry`). So a run that reaches `failed` is **terminal**: it is the dead-letter. `voltro workflows list --dead-letter` (or the Prometheus series `voltro_workflow_runs_total{status="failed"}` and the stale `voltro_workflow_last_success_timestamp_seconds` gauge — see [Prometheus](/docs/plugins/prometheus)) is your queue of unhandled failures.
Triage a dead-letter run one of three ways:
- **Retry** it — `voltro workflows retry ` starts a fresh execution against the original (or an overridden) payload once you've fixed the cause. Every step runs again from scratch.
- **Re-drive** it — `voltro workflows redrive ` re-drives the run from its durable journal: completed steps replay, only the failed step(s) re-execute. Use this instead of `retry` for a long pipeline where redoing steps 1…N‑1 is expensive or unsafe. It is the after-the-fact counterpart to [`suspendOnFailure` + `resume`](/docs/workflows/retries) and works under `voltro serve` too. Refuses a non-failed / already-discarded run; declines cleanly when there is no durable journal (the memory store). Records a `run-redriven` event.
- **Discard** it — `voltro workflows discard ` acknowledges the failure so it drops off the `--dead-letter` view. It is an **ack, not a re-classification**: the run stays `status: 'failed'` (the outcome + audit trail survive) and gains a `discardedAt` timestamp. `--status failed` still lists it, marked `discarded`; only `--dead-letter` hides it. Discarding a non-failed run is refused, and discarding is idempotent.
## Stuck runs and crash loops
Three failure modes are detected rather than left for whoever opens the dashboard.
**A run that stops moving.** A staleness sweep reports live runs (`running` or `suspended`) that have made no progress for longer than the threshold — default 30 minutes — writing a `run-stalled` event carrying `idleMs`, the reason (`awaiting-signal`, `suspended`, `no-progress`) and the last progress instant. It changes no run state; it is a signal, not an intervention. The classic catch is a run parked on a signal nobody ever sends.
Two things it deliberately stays quiet about, because otherwise the signal is worthless:
- A run inside a durable `sleep` / `sleepUntil` whose wake instant is still in the future. That run is waiting by design, and a seven-day timer is not a stall.
- A run already reported since its last progress. A stall is reported once and again only after the run moves and stalls afresh.
Configure it in `app.config.ts` — every field optional:
```ts
export default defineApiApp({
workflows: {
staleness: {
// Set this above your slowest single STEP, not above your longest RUN:
// a run waiting on a durable timer is already excluded. Default 30 min.
stallAfterMs: 30 * 60_000,
runPage: 200, // live runs examined per tick, oldest first
onStalled: async (run) => {
await page(`${run.tag} ${run.runId} idle ${run.idleMs}ms (${run.reason})`)
},
},
},
// How often the sweep looks. `VOLTRO_STALENESS_SWEEP_MS` overrides it.
scheduling: { stalenessSweepMs: 5 * 60_000 },
})
```
`onStalled` must not throw — a rejection is collected and reported, so one bad
pager integration cannot stop detection for every other workflow.
The sweep runs on a **coordinated** schedule, so one firing per interval
fleet-wide rather than one per replica. Unlike every other framework background
task it **never stops ticking when idle**: those disarm because an arrival wakes
them, and a run going stale writes nothing there is to wake on. That makes the
cadence an unconditional cost, which is why it defaults to five minutes rather
than one second — on a thirty-minute threshold that is 12 coordination rows an
hour instead of 3 600.
`voltro doctor` runs the same detection **once**, for the moment you are standing
in front of a deployment asking whether anything is wedged:
```
✗ stuck runs: 2 run(s) have made no progress
invoices.settle wr_01J… idle 4h awaiting-signal
report.nightly wr_01J… idle 2h no-progress
Nothing was changed — this is a SUSPICION, not a verdict. `awaiting-signal`
usually means the sender never came.
```
It is the one doctor rule that reads your DATABASE rather than your source, so
run it where the app's DB env vars are set; anywhere else it prints a named skip
rather than a clean tick. It records nothing and calls no `onStalled`, so running
it neither pages anyone nor suppresses the background sweep's next real report.
**A run that keeps killing its runner.** A step that crashes the process — OOM, a native crash — cannot be caught as an error: the shard lease expires, a surviving replica claims it, and executes the same payload. Without a ceiling that rotates around the fleet forever. Voltro counts consecutive runner deaths on the run row (`runnerEnteredAt` is set on every body entry and cleared on every clean exit; `reclaimCount` counts entries that found the previous marker still set). At `VOLTRO_WORKFLOW_MAX_RECLAIMS` (default 3) the run is parked as `suspended` with `errorTag: 'WorkflowCrashLooped'` and a `run-crashlooped` event, and the body is not entered again. The counter is consecutive — any clean re-entry resets it — so a long-lived healthy run is never parked for a crash it had months ago, and an operator resume gives it a fresh budget.
**A run replaying against edited code.** See [Versioning](/docs/workflows/versioning) for the `nondeterminism-suspected` event.
## Dashboard
When `voltro dev` is running, the Workflows panel lists recent runs, their status, start source, timing, payload, output/error, step attempts, and events. The run detail view is the fastest way to answer:
- Which step is currently running?
- Was the run started by an RPC, `ctx.workflows`, a schedule, an incoming webhook, or inspect tooling?
- Is this run a child, and will it be cancelled, terminated, or abandoned when the parent closes?
- Which attempt failed?
- What input did the step receive?
- What output or typed error did it produce?
- Did a timer or signal fire?
The panel also exposes run controls for users with the right capability: cancel, retry, suspend, resume, redrive (re-drive a failed run from its journal), discard (acknowledge a failed run), send signal, and send a tracked update.
Run filters can be saved as named views. The selected view and ad-hoc
filters are mirrored into the URL query string, so a teammate can open the
same filtered run list. The Workflows panel also includes dedicated
Incoming and Flow tabs: Incoming groups runs whose source is
`incoming:` beside incoming-sourced domain events, while Flow groups
queued/running/waiting work by workflow lane and start source.
## CLI
```sh
voltro workflows list --status running --tail 50
voltro workflows list --tag notes.summarise --format json
voltro workflows list --dead-letter # failed runs not yet discarded
voltro workflows list --statuses failed,cancelled --q orders --since 2026-08-01T00:00:00Z
voltro workflows list --id-prefix wfrun_01K # matches the run id OR the execution id
voltro workflows stats --hours 24 # bucketed activity sparkline + per-workflow totals
voltro workflows start notes.summarise --payload '{"noteId":"note_123"}'
voltro workflows show wfrun_01H...
voltro workflows retry wfrun_01H...
voltro workflows cancel wfrun_01H...
voltro workflows suspend wfrun_01H...
voltro workflows resume wfrun_01H...
voltro workflows redrive wfrun_01H... # re-drive a failed run from where it died
voltro workflows discard wfrun_01H... # acknowledge a failed run
voltro workflows signal wfrun_01H... --name approval --payload '{"approved":true}'
voltro workflows update wfrun_01H... --name approve --payload '{"decision":true}'
voltro workflows children exec_01H...
```
The CLI discovers live API processes the same way as `voltro logs` and `voltro traces`, then calls inspect endpoints.
## Inspect endpoints
```http
GET /_voltro/inspect/workflows/runs
GET /_voltro/inspect/workflows/runs?tag=notes.summarise&status=failed
GET /_voltro/inspect/workflows/runs/:id/steps
GET /_voltro/inspect/workflows/runs/:id/events
GET /_voltro/inspect/workflows/events
GET /_voltro/inspect/workflows/events/:id/deliveries
GET /_voltro/inspect/workflows/children?parentExecutionId=exec_01H...
POST /_voltro/inspect/workflows/runs/:id/cancel
POST /_voltro/inspect/workflows/runs/:id/retry
POST /_voltro/inspect/workflows/runs/:id/suspend
POST /_voltro/inspect/workflows/runs/:id/resume
POST /_voltro/inspect/workflows/runs/:id/redrive
POST /_voltro/inspect/workflows/runs/:id/discard
POST /_voltro/inspect/workflows/runs/:id/signal
POST /_voltro/inspect/workflows/runs/:id/update
```
Signal body:
```json
{
"signalName": "approval",
"payload": { "approved": true }
}
```
Update body:
```json
{
"updateName": "approve",
"payload": { "decision": true },
"timeoutMs": 30000
}
```
Retry normally uses the original payload. The inspect handler also supports payload override for operator tooling.
## SQL inspection
Recent failed runs:
```sql
SELECT id, tag, status, source, "errorTag", "errorMessage", "startedAt", "completedAt"
FROM "_voltro_workflow_runs"
WHERE status = 'failed'
ORDER BY "startedAt" DESC
LIMIT 50;
```
Slow steps:
```sql
SELECT "stepName", percentile_cont(0.95) WITHIN GROUP (ORDER BY "durationMs")
FROM "_voltro_workflow_run_steps"
WHERE status = 'succeeded'
AND "startedAt" > now() - interval '24 hours'
GROUP BY "stepName"
ORDER BY 2 DESC
LIMIT 10;
```
Events for one run:
```sql
SELECT "eventType", payload, "occurredAt"
FROM "_voltro_workflow_run_events"
WHERE "runId" = 'wfrun_01H...'
ORDER BY "occurredAt" ASC;
```
## Inspecting from code
`inspectWorkflow(idOrExecutionId, store)` assembles one run into the same shape used by the dashboard detail view:
```ts
import { inspectWorkflow } from '@voltro/workflow'
const state = await inspectWorkflow(runId, ctx.store)
```
It groups step attempts by step name, so a step that failed twice and succeeded on attempt three reports `attempts: 3` with the latest output/error surfaced.
## Testing workflows
Use `makeWorkflowRunner` from `@voltro/testing` to run a workflow in-process over the in-memory workflow engine and an in-memory recorder:
```ts
import { makeTestContext, makeWorkflowRunner } from '@voltro/testing'
import { SummariseNote } from '../workflows/notes.summarise.workflow'
import buildSummariseNote from '../workflows/notes.summarise.workflow'
test('notes.summarise retries the LLM step', async () => {
const ctx = makeTestContext()
const runner = makeWorkflowRunner({
ctx,
workflows: [{
workflow: SummariseNote as never,
execute: buildSummariseNote(ctx) as never,
}],
})
const result = await runner.start('notes.summarise', { noteId: 'n_1' })
expect(result.status).toBe('succeeded')
expect(result.steps.find((s) => s.name === 'summarise-with-llm')?.attempts).toBe(3)
})
```
The result includes `{ status, output, error, steps, runId }`. `runner.inspect(runId)` returns the assembled run later without re-running it.
## A start with the wrong payload
`ctx.workflows.start(name, payload)` validates the payload against the workflow's
`payload` schema **before** the engine sees it. A mismatch throws a
`WorkflowPayloadError` naming three things:
```
Workflow "sprint.report" was started with an invalid payload. missing required
field(s): teamId. { readonly teamId: string } └─ ["teamId"] is missing
```
The error also carries them structurally — `workflowName`, `missingFields`,
`_tag: 'WorkflowPayloadError'` — so a handler can branch on it.
This matters most where nobody is watching. A **cron** whose payload drifted from
the workflow's schema fails on every single firing; the schedule run is recorded
`failed` in `_voltro_schedule_runs` and the log line is an **error**, not a warn,
so `voltro logs --level error` and any alert wired to it see it. A nightly job
that has been dead since a refactor is the exact failure this pair of behaviours
exists to surface.
An unknown workflow name lists the registered ones, so a rename reads differently
from a deletion:
```
Unknown workflow: sprint.reports. Registered workflows: billing.run, sprint.report.
```
Validation runs on the **decoded** value, so a `Schema.Date` payload accepts a
`Date` — passing the already-domain-shaped value is correct and is not rejected.
## Anti-patterns
- **Querying old table names.** Use `_voltro_workflow_runs`, `_voltro_workflow_run_steps`, and `_voltro_workflow_run_events`.
- **Filtering for `status=dead`.** Current failed runs use `status=failed`.
- **Expecting replay-from-step APIs.** Current operator retry starts a new execution from the workflow payload; completed steps are not selectively replayed through a public API.
- **Logging giant step inputs/outputs.** Step input/output is persisted for inspection. Store references to huge blobs instead of the blob itself.
---
## Event triggers
_defineEventTrigger — bind a domain event name to a workflow. ctx.events.publish(descriptor, key, payload) fans the event out to every matching trigger with filter, payload-mapping, and idempotency._
A workflow is usually kicked off directly (`ctx.workflows.start(...)`). An **event trigger** decouples that: a `*.trigger.tsx` file binds a **declared event** to a **workflow name**, and any handler that publishes that event fans it out to every trigger listening for it. The emitter never names the workflow — "something happened" is separated from "run this workflow", so you add reactions without touching the code that emits.
```tsx
// events/orders.event.ts — the declaration both sides share
export const orderPaid = defineEvent({
name: 'order.paid',
key: Schema.Struct({ orderId: Schema.String }),
payload: Schema.Struct({ total: Schema.Number, tenantId: Schema.String }),
})
// triggers/orderPaid.trigger.tsx
import { defineEventTrigger } from '@voltro/runtime'
export default defineEventTrigger({
on: orderPaid, // the declared event to listen for
workflow: 'fulfilment.run', // the workflow to start
// Optional: skip when the predicate returns false.
filter: (e) => e.data.total > 0,
// Optional: map the event envelope → the workflow's payload (default: e.data).
payload: (e) => ({ orderId: e.data.orderId, tenantId: e.data.tenantId }),
// Optional: dedup key — two emits with the same key start the workflow once.
idempotencyKey: (e) => `fulfil:${e.data.orderId}`,
})
```
Discovery walks every `*.trigger.tsx`; the `default` export must be a `defineEventTrigger(...)` descriptor (one per file). The workflow runtime is active whenever the app has any workflow OR any trigger.
## Emitting an event
From any handler with a `ctx`, publish the declared event:
```ts
const execute = (input: { orderId: string }, ctx: AppContext) => Effect.gen(function* () {
// ... mark the order paid ...
yield* ctx.events.publish(orderPaid, { orderId: input.orderId }, {
total: order.total,
tenantId: ctx.request.subject.tenantId,
})
})
```
`publish` reaches BOTH audiences from one call: the durable one (this page — the event log plus every matching trigger) and the ephemeral one ([connected clients](/docs/data/events)). Inside a mutation both fire on COMMIT and neither on rollback, so a client and a triggered workflow cannot disagree about whether the thing happened.
`on:` reads the event NAME off the descriptor, so renaming the event moves the trigger with it. The older string form (`event: 'order.paid'`) was removed in 0.25.0 — with a string, a rename left the trigger matching nothing and the workflow simply never ran again, without an error.
The event envelope every `filter` / `payload` / `idempotencyKey` receives is:
```ts
{ id, name, data, occurredAt, source, subject?, traceId? }
```
`subject` + `traceId` are inherited from the emitting request, so a triggered workflow continues the same trace and carries the same actor.
## Matching + delivery
- **Exact + wildcard.** A trigger matches its exact `event` name; a trigger registered for `'*'` matches every event (and runs in addition to exact matches).
- **Idempotency.** Before starting, the runtime checks for an existing delivery with the same `idempotencyKey` (default `:`) — a duplicate is recorded as `status: 'skipped', reason: 'duplicate'` and the workflow is not started twice.
- **Every emit is audited.** The event itself lands in `_voltro_workflow_events`; one row per trigger lands in `_voltro_workflow_event_deliveries` with `status` (`starting → started`, or `skipped` / `failed`), the started `executionId`, and any error message. Both tables are reactive — the dashboard's Workflows view surfaces deliveries live.
## Retention — and why the delivery TTL is not just housekeeping
Both audit tables are append-only, so both are swept on a **30-day** default by the
boot retention GC (postgres): `VOLTRO_WORKFLOW_EVENTS_TTL_HOURS` and
`VOLTRO_WORKFLOW_EVENT_DELIVERIES_TTL_HOURS`. The delivery log grows faster — one
row per *trigger*, so three triggers on one event write four rows per emit.
**The delivery TTL is the deduplication window.** The idempotency check above looks
for an existing delivery row, so once a row is swept its key is no longer
deduplicated. With the default key (`:`, and `eventId` is fresh
per emit) a duplicate cannot occur and this costs nothing. It matters only when you
supply your own `idempotencyKey`: if your app can re-emit the *same* stable key
(`fulfil:order-123`) more than 30 days apart and must still be deduplicated, raise
`VOLTRO_WORKFLOW_EVENT_DELIVERIES_TTL_HOURS` past that horizon.
Unlike `_voltro_outbox`, the delivery log is **not** status-filtered — a stale
`starting` row has no requeue path and no reader, so keeping it would leave the
table unbounded for exactly the rows a crash produces.
## When to use it
Reach for an event trigger when ONE thing happening should fan out to several independent reactions, or when you want the emitter to stay ignorant of the consumers:
- `order.paid` → start fulfilment AND a receipt-email workflow AND an analytics rollup — three triggers, one `publish`.
- A mutation publishes `user.signedUp`; an onboarding workflow trigger starts the drip sequence. The signup mutation never imports the onboarding workflow.
If the handler already knows exactly which workflow to run and there's only one, call [`ctx.workflows.start(...)`](/docs/workflows/definition) directly — the trigger indirection only pays off when you're decoupling publish from react, or fanning one event out to many.