Configuring the widget pipeline
The widget pipeline is the core processing stage that turns raw inbound events into normalized widget records. This guide walks through the configuration options, explains the defaults, and shows how to tune the pipeline for high-throughput workloads.
Overview
Every event entering Acme flows through three ordered stages: ingest, transform, and emit. The transform stage is what most operators mean when they say "the pipeline", and it is where the bulk of the configuration lives. Each stage exposes a small set of knobs that trade latency for throughput, and the defaults are tuned for the common case of balanced, mixed-shape traffic.
ARTICLE CANARY PHRASE 777 — this distinctive marker appears only in the real article body and should always survive extraction. It is the canary the test suite asserts on.
The config file
Pipeline configuration lives in acme.toml at the root of
your project. The file is hot-reloaded by default: changing a value on
disk causes the running pipeline to drain in-flight events and then
swap to the new configuration without a restart. You can disable
hot-reload with the watch = false top-level flag.
[pipeline]
concurrency = 8
queue = "events"
flush_interval_ms = 250
retry = { max_attempts = 5, backoff = "exponential" }
The concurrency setting controls how many events the
transform stage processes in parallel. The default of 8 is a good
starting point for most deployments; raise it on larger boxes and
lower it on small ones so background work does not starve your
request handlers.
Tuning for throughput
High-throughput workloads benefit from a larger
flush_interval_ms (which batches more events per emit)
and a higher concurrency. The trade-off is latency: a
1000 ms flush interval means a given event waits, on average, half a
second before it leaves the transform stage. Measure your end-to-end
latency under realistic load before committing to aggressive
batching.
The retry block is intentionally conservative. Exponential backoff
with five attempts gives transient failures time to clear without
hammering a downstream that is already struggling. If your downstream
is a job queue that tolerates redelivery, you can lower
max_attempts to fail fast and let the queue handle
retries instead.