---
icon: material/pipe-valve
title: "Receiver"
---
The [Receiver](https://doc.log10x.com/apps/receiver/) acts on events before they reach the analyzer. Each log pattern gets a byte cap, and one action per service handles the lines over it: pass, offload, tier_down, compact, sample or drop.
- [:material-information-outline: **Overview**](#overview)
- [:material-cog-outline: **Configuration**](#configuration)
- [:material-compare-horizontal: **Actions & Related Apps**](#modes-related-apps)
- [:material-connection: **Integration**](#integration-deployment)
### :material-information-outline: Overview
??? tenx-overview "What is the Receiver and how does it control costs"
The Receiver applies one action per service to the events a pattern spends above its cap:
- **pass**: forward unchanged.
- **sample**: forward a rate-limited share of the lines a pattern sends over its cap, in real time at the source, before they reach the analyzer. Patterns come from automatic [message enrichment](https://doc.log10x.com/run/initialize/message/){target="_blank"}, which identifies each event type by its symbol identity and prices each event at bytes × $/GB.
- **compact**: replace repeated lines with an encoded form the destination expands at query time. In Splunk, classic dashboards keep their searches, alerts are compiled at save time, and a search in the search bar or a Dashboard Studio panel needs the app's `tx` macro written in front of its terms. Compact is lossless on Splunk (up to 256 KB per event) and on self-hosted Elasticsearch 8.17.0 or OpenSearch 2.19.0 with the expand plugin installed ([Splunk](../../apps/receiver/compact/splunk.md), [Elasticsearch](../../apps/receiver/compact/elasticsearch.md)). On managed/SaaS destinations (Datadog, CloudWatch, Azure, GCP, Sumo, managed Elasticsearch) compact leaves lines unchanged, and ClickHouse services take offload instead.
- **tier_down**: tag the lines over the cap for a cheaper tier the destination enforces (Datadog Flex, CloudWatch IA, Azure Monitor Basic/Auxiliary, or Elasticsearch's frozen tier, through an index of its own that ILM mounts there).
- **offload**: route the lines over the cap to customer-owned object storage (S3 or any S3-compatible bucket) instead of the destination. On ClickHouse and ClickStack, offload is the action that cuts cost: the lines over a service's total are kept in the account's own bucket before ClickHouse receives them ([ClickHouse FAQ](https://doc.log10x.com/faq/stacks/clickhouse/optimization/){target="_blank"}).
- **drop**: stop forwarding the lines over the cap.
Every action applies after the shipped [keep counts](https://doc.log10x.com/run/receive/rate/#error-class-lines){target="_blank"}: up to 50 ERROR, CRITICAL and FATAL lines per pattern per window pass, released in four slices across the window, even when their pattern is over its cap.
An AI agent picks the action per service through the log10x MCP (the `configure_engine` tool), which turns a target percent or monthly dollar budget into a per-pattern cap and an action for each service, carried as cap and action CSV files. A GitOps PR commits the files to the config repo, and they hot-reload on the next pull. Explicit sample rates for named patterns come from the [mute file](https://doc.log10x.com/run/receive/rate/#protection-list){target="_blank"}, a CSV keyed by pattern, committed to git and pulled the same way.
An AI agent (Claude, or a model the customer brings) drives Log10x through the log10x MCP server, which installs, configures, and queries via MCP tools. The manual steps in these docs are the same operations without an agent.
??? tenx-overview "How do budget policies work"
Configuration is a YAML file written by hand, plus the cap and action CSV files the MCP `configure_engine` tool generates, all delivered through GitOps. The [rate regulator](https://doc.log10x.com/run/receive/rate/){target="_blank"} caps the bytes each pattern ([symbol identity](https://doc.log10x.com/run/initialize/message/){target="_blank"}) may spend in its container over a rolling window. You configure:
- **A byte cap per pattern per container**, fleet-wide or per container from a cap file; the MCP derives it from a monthly dollar budget
- **A share guard**, a pattern below a minimum share of its container's volume is left alone (off as shipped)
- **Keep counts and severity floors**, an over-cap pattern passes a set count of lines per window for some levels and a set share of the others (as shipped, up to 50 lines each of ERROR, CRITICAL and FATAL, released across the window, and a share of 0 for every other level)
- **A total per service**, optional, bounding what one service passes per window across all its patterns; the lines over it take the service's action
- **An action per service** for the excess: pass (the default), offload, tier_down, compact, sample or drop
Cost is reported per pattern; the cap itself is in bytes. Caps are scoped by the [container name](https://doc.log10x.com/run/receive/rate/#containers){target="_blank"} on each node, so scaling replicas doesn't bypass them.
??? tenx-overview "What happens to logs when budget limits are reached"
The [rate regulator](https://doc.log10x.com/run/receive/rate/){target="_blank"} decides per event, on bytes:
- **At or below the pattern's cap**, every event flows through
- **Over the cap**, ERROR, CRITICAL and FATAL pass their keep count, each other level keeps its floor share, and the rest takes the container's action (pass when none is set); nothing targets the excess back down to the cap line
- **Keep counts**, ERROR, CRITICAL and FATAL pass up to 50 lines per pattern per window over the cap as shipped, released across the window, and every floor ships at 0, so DEBUG noise over the cap takes the action
The result: a noisy pattern's excess takes the action while its errors keep flowing. The Receiver exports [metrics](https://doc.log10x.com/run/output/metric/){target="_blank"} tracking what each pattern's action did, so policies can be tuned over time. Metrics for routed-away events carry the label `routeState` (for example `routeState="drop"`).
### :material-cog-outline: Configuration
??? tenx-capabilities "Can I set different budgets for different event types"
Yes. The [rate regulator](https://doc.log10x.com/run/receive/rate/){target="_blank"} tracks spend per event type using configurable field sets. You can scope budgets by:
- **[Message pattern](https://doc.log10x.com/run/initialize/message/){target="_blank"}**, each distinct event type gets its own cap
- **[K8s container name](https://doc.log10x.com/run/initialize/k8s/){target="_blank"}**, cap total spend per application, regardless of pod replica count
- **Both combined**, cap per event type per application for fine-grained control
The [keep counts](https://doc.log10x.com/run/receive/rate/#error-class-lines){target="_blank"} keep up to 50 ERROR, CRITICAL and FATAL lines per pattern per window flowing when a cap engages, released across the window; the same caps cover every level.
??? tenx-capabilities "How does the Receiver protect critical events"
The [rate regulator](https://doc.log10x.com/run/receive/rate/){target="_blank"} uses two mechanisms that protect critical events:
- **Keep counts**, ERROR, CRITICAL and FATAL lines pass up to 50 per pattern per window as shipped, released across the window, even when their pattern is over its cap; the other levels keep their floor share, 0 as shipped
- **Per-pattern caps**, the regulator engages only on a [pattern](https://doc.log10x.com/run/initialize/message/){target="_blank"} over its byte cap, and above the share guard when one is set. A low-volume pattern stays under its cap and passes through unaffected
The Receiver targets noisy, high-volume patterns, not broad categories. Security events that stay under their cap flow through untouched.
??? tenx-capabilities "How do I monitor budget consumption"
The Receiver publishes [cost metrics](https://doc.log10x.com/run/output/metric/){target="_blank"} per [event type](https://doc.log10x.com/run/initialize/message/){target="_blank"}: volume, spend rate, and per-action counts. Three ways to consume them:
- **Cost dashboards**, the metrics feed your time-series backend (Prometheus, Datadog, Elastic, and others), where budget consumption, processed volume, and cost trends per event type are charted in your own tooling
- **[Prometheus Metrics API](https://doc.log10x.com/api/metrics/){target="_blank"}**, standard REST endpoints for querying all cost and action metrics programmatically via `/query`, `/query_range`, and `/series`
- **[Native metric outputs](https://doc.log10x.com/run/output/metric/prometheus/){target="_blank"}**, export to Prometheus, Datadog, CloudWatch, or any compatible platform for custom dashboards and alerts
??? tenx-capabilities "How do I configure priority tiers for critical events"
Use `keepCounts` to pass a fixed number of lines per pattern per window for a level, and `severityFloors` to keep a share of the other levels. ERROR, CRITICAL and FATAL ship with a keep count of 50, so each keeps flowing over the cap, up to 50 lines per pattern per window, released across the window; every floor ships at 0. The floor is the fraction of an over-cap pattern's lines kept at random for that level, and it applies past the cap: a pattern over its byte cap still keeps at least this share of each level the floors cover.
**How it works:** the rate regulator engages only once a pattern has spent more bytes than the cap set for its container. From then on, each level in `keepCounts` passes up to its count per window, released across the window in `keepCountSlices` slices (4 as shipped), each event of another level is kept with the probability its floor gives, and the rest takes the container's action, `pass` unless an action file names another. A level with no floor uses `minRetentionThreshold` (`0` as shipped). The keep counts apply under mute-file entries too, so ERROR, CRITICAL and FATAL pass their keep counts even under a `0` mute.
Floor keys must match the level vocabulary the [Level Classifier](https://doc.log10x.com/run/initialize/level/){target="_blank"} emits.
**Config example:**
```yaml
# config.yaml for the Receiver
rateReceiver:
fieldNames:
- message_pattern
containerField: container
absoluteCap: 10485760 # 10 MB per pattern per container per window
keepCounts: # lines per pattern per window once over the cap
- ERROR=50
- CRITICAL=50
- FATAL=50
minRetentionThreshold: 0.1 # floor for any level not listed below
severityFloors: # share kept once over the cap, for levels without a keep count
- TRACE=0.05
- DEBUG=0.05
- INFO=0.1
- WARN=0.3
```
`absoluteCap` or a per-container cap file turns the regulator on; the shipped configuration sets no cap. For Kubernetes, the same settings are set through the [Helm chart](https://doc.log10x.com/apps/receiver/deploy/){target="_blank"}.
**Real-world scenarios:**
1. **API Gateway (errors matter most):**
- Cap: 10 MB per pattern per container per window
- ERROR keep count 50, DEBUG floor 0.05, so verbose request logs are thinned hardest and errors keep flowing
2. **Background Job Queue (low signal, high volume):**
- Cap: 2 MB per pattern per container per window
- INFO floor 0.1 for job completion messages, CRITICAL keep count 50 for job-queue-down events
3. **Kubernetes cluster (health checks vs. pod crashes):**
- Cap: 5 MB per pattern per container per window
- DEBUG floor 0.05 for health check spam, ERROR keep count 50 for pod failures and crashes
**Scope of the floor:**
- Each level's floor is read on its own, so raising WARN leaves DEBUG unchanged.
- The floor applies only to a pattern already over its cap; under the cap every event is kept whatever its level.
- The floor sets a minimum: it decides what share of the excess is kept, and the container's action decides what happens to the rest.
### :material-compare-horizontal: Actions & Related Apps { #modes-related-apps }
??? tenx-overview "When should I use the sample action vs the compact action"
sample and compact are two of the actions, each applied to the events a pattern spends above its cap.
| | sample | compact |
|---|---|---|
| **Loss** | Lossy, over the cap only the sampled share is forwarded and the rest is dropped | Lossless on supported destinations, each event replaced with a compact wire-form the expand plugin restores at query time |
| **Downstream requirements** | None | Self-hosted [Splunk](../../apps/receiver/compact/splunk.md) or [Elasticsearch](../../apps/receiver/compact/elasticsearch.md)/OpenSearch with the expand plugin installed |
| **Volume reduction** | Set by the per-pattern sample rate | Set by how repetitive the pattern is, where the destination supports it (Splunk, self-hosted Elasticsearch/OpenSearch); a no-op on managed/SaaS destinations |
| **Risk profile** | Higher, dropped events are gone; safe defaults = deny | Lower, events round-trip intact and are queryable through the expand plugin at search time |
| **Typical trigger** | A single pattern is over-budget; cap it at a sample rate | Shipping volume is the bottleneck; shrink the wire format |
??? tenx-overview "What happens to events that the Receiver filters"
The Receiver discards filtered events before they reach your log analyzer. If you want a copy of events you can fetch back later, configure your forwarder to also route a copy of events to your own S3 bucket before the Receiver acts. Each forwarder has a native mechanism for duplicating the event stream, see [per-forwarder S3 configuration](https://doc.log10x.com/apps/receiver/#pair-with-retriever).
Use [Retriever](https://doc.log10x.com/apps/retriever/) to fetch the exact stored events from your own S3 bucket on demand; it returns the stored events without rehydration or re-ingest, at your S3 storage cost.
??? tenx-overview "Can I use the Receiver with Retriever"
Yes. Configure your forwarder to duplicate events, one copy goes to your own S3 bucket, the other goes through the Receiver (filtered or compact events to your analyzer). Hard-dropped events never reach S3. Retriever fetches selected stored events from your S3 bucket on demand.
This pairs log analyzer cost control with on-demand fetch of the stored copies from your own S3 bucket. Events you route to S3 can be fetched back through Retriever on demand. Compact events stored in S3 are also queryable, Retriever expands them as it returns them.
See the [per-forwarder configuration](https://doc.log10x.com/apps/receiver/#pair-with-retriever) for Fluent Bit, Fluentd, OTel Collector, and Logstash recipes.
??? tenx-overview "Can I preview the impact before enabling actions"
Yes. The [Reporter](https://doc.log10x.com/apps/reporter/) app runs the same enrichment pipeline as the Receiver (severity classification, symbol identity, cost metrics) and only reports, leaving events unchanged. Deploy the Reporter first to see which event types incur the highest costs and what the actions would target. When the metrics confirm the expected behavior, enable the Receiver.
### :material-connection: Integration & Deployment
??? tenx-integration "Which log forwarders does the Receiver support"
The Receiver integrates with eight [log forwarders](https://doc.log10x.com/run/input/forwarder/){target="_blank"}. Six are native, and the install wizard automates each:
- [Fluent Bit](https://doc.log10x.com/run/input/forwarder/fluentbit/){target="_blank"}
- [Fluentd](https://doc.log10x.com/run/input/forwarder/fluentd/){target="_blank"}
- [Filebeat](https://doc.log10x.com/run/input/forwarder/filebeat/){target="_blank"}
- [Logstash](https://doc.log10x.com/run/input/forwarder/logstash/){target="_blank"}
- [OpenTelemetry Collector](https://doc.log10x.com/run/input/forwarder/otel-collector/){target="_blank"}
- [Vector](https://doc.log10x.com/run/input/forwarder/vector/){target="_blank"}
Splunk Universal Forwarder and Datadog Agent integrate through a Fluent Bit file-system handoff: Fluent Bit reads files from disk, the engine processes them, and the UF or agent tails the output folder.
**Deployment:** runs [in-path](https://doc.log10x.com/engine/launcher/sidecar/){target="_blank"} with your forwarder, sidecar or embedded depending on the forwarder, so it can act on events before they ship to the analyzer. Kubernetes deployment via [Helm chart](https://doc.log10x.com/apps/receiver/deploy/){target="_blank"}; the workload type follows the forwarder chart (DaemonSet for most, StatefulSet for Logstash). Setup time: ~30 minutes.
**Resource requirements:** set the heap explicitly and the container memory limit above it, for example a 512m heap in a 768Mi limit. See [Node counting](https://doc.log10x.com/faq/pricing/node-counting/) for sizing details and Kubernetes resource specs, and the [deployment guide](https://doc.log10x.com/apps/receiver/deploy/){target="_blank"} for per-forwarder configuration.
??? tenx-integration "Can the Receiver reduce Kubernetes health check volume"
Yes. Health checks, liveness probes, and pod lifecycle events are highly repetitive, so a per-pattern cap holds each one to its budget.
**How it works:**
1. Set the sample action with a budget cap for health check event types using [rate sampling](https://doc.log10x.com/run/receive/rate/){target="_blank"}
2. For the surviving events, set the [compact action](https://doc.log10x.com/apps/receiver/compact/){target="_blank"} to losslessly compact them via [templates](https://doc.log10x.com/run/transform/){target="_blank"}
3. Result: the health check patterns stay within their budget, and every pod metadata field (namespace, pod name, container name, labels) is preserved
Failed health checks classified as ERROR, CRITICAL or FATAL pass up to 50 lines per pattern per window as shipped, even when their pattern is over its cap. The setup works with [Fluentd](https://doc.log10x.com/run/input/forwarder/fluentd/){target="_blank"}, [Fluent Bit](https://doc.log10x.com/run/input/forwarder/fluentbit/){target="_blank"} and the OpenTelemetry Collector.
??? tenx-integration "How does the Receiver handle failures"
With sidecar forwarders (Fluentd, Fluent Bit, Logstash, OTel Collector, Vector), Kubernetes runs 10x as a separate container next to the forwarder and manages its lifecycle. If the 10x process exits (crash, OOM kill, panic), the container exits and the kubelet restarts it automatically via the pod's default `restartPolicy`. The forwarder keeps running while the sidecar restarts and buffers events up to its own buffer and retry limits. On the OpenTelemetry Collector route the Receiver acknowledges an export once its lines have been sent on and accepted, so the collector resends what was in flight through a Receiver restart and every line arrives, some twice ([custody run](https://github.com/log-10x/benchmarks/tree/main/clickstack-e2e-gaps){target="_blank"}). With the embedded forwarder (Filebeat), 10x runs inside the forwarder's own container, so the two share one lifecycle and restart together.
| Scenario | Behavior |
|---|---|
| 10x crash or OOM | The 10x process exits and Kubernetes restarts the sidecar container automatically. On the OpenTelemetry Collector route the Receiver acknowledges an export once its lines have been sent on and accepted, so the collector resends what was in flight and every line arrives, some twice. On the other forwarder routes the forwarder buffers events up to its own buffer limits and reconnects on restart. |
| Volume exceeds the sidecar's capacity | Forwarder buffers locally; 10x catches up. Backpressure signals the forwarder to slow down. |
| Forwarder crashes | Kubernetes restarts the forwarder container; the pod stays up and the sidecar keeps running. |
| Network interruption | The forwarder and the sidecar keep exchanging events over the pod's shared loopback network. |
| Downstream analyzer slow or unreachable | Forwarder handles retries to the destination, 10x unaffected. |
| Rollback | Single `helm uninstall`, forwarder continues unchanged, no data migration. |