--- title: "Fluentd" description: "receive and optimize events collected by Fluentd via the Fluent Forward\ \ protocol" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/forwarder/fluentd/module.yaml" icon: "simple/fluentd" --- Runs 10x Engine as a [sidecar](https://doc.log10x.com/engine/launcher/sidecar) to Fluentd for reporting, receiving, and optimizing events before they ship to their destination (Elasticsearch, Splunk, S3, Kafka, โ€ฆ). In the VM/host recipe, Fluentd and Log10x run as peer processes and exchange events over the [Fluent Forward protocol](https://docs.fluentd.org/output/forward){target="\_blank"}; works against any stock Fluentd build (td-agent, fluent-package, OSS) and the official Fluentd Helm chart on Kubernetes via a `log10x/edge-10x` sidecar container (kustomize post-renderer overlay). ## Architecture
```mermaid graph LR A["
๐Ÿ“‚ Sources
tail, http, k8s
"] --> F["
๐Ÿงช @INGEST
enrichment filters
"] F --> B["
๐Ÿ“ค out_forward
:24224
"] B --> E["
โšก 10x Engine
Report/Receive/Optimize
"] E --> C["
๐Ÿ“ฅ in_forward
:24225 โ†’ @OUTPUT
"] C --> D["
๐Ÿ“ค Destinations
ES, Splunk, S3, Kafka
"] classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8 class A input class B filter class C filter class D output class E engine class F filter ```
### Data Flow - ๐Ÿ“‚ **Sources**, Your existing Fluentd sources (`tail`, `http`, `forward`, syslog, etc.) route their events into the `@INGEST` label. - ๐Ÿงช **@INGEST**, Your enrichment filters (`kubernetes_metadata`, `record_transformer`, parsers, โ€ฆ) run here exactly once before the event is handed off to Log10x. - ๐Ÿ“ค **out\_forward** โ†’ Log10x, Fluentd forwards the enriched event to the Log10x sidecar over TCP `:24224` (or a Unix socket on Linux/macOS). - โšก **10x Engine**, The Receiver app applies rate/policy-based filtering and optionally compacts events for volume reduction. - ๐Ÿ“ฅ **in\_forward โ†’ @OUTPUT**, Processed events come back to Fluentd on `:24225` and are routed directly to the `@OUTPUT` label, which holds your destination `` blocks. Filters defined under `@INGEST` are **not** re-applied, so each event is enriched exactly once. - ๐Ÿ“ค **Destinations**, The original Fluentd tag survives the round trip, so destinations that route on `$TAG` (Splunk index, S3 path, Kafka topic, โ€ฆ) behave the same as if Log10x weren't in the path. ### What an event looks like on the way back The record structure of the original Fluentd event is preserved end-to-end, every field comes back to your `@OUTPUT` label with the same name and same position. What changes depends on the Receiver app mode: |Mode|Difference vs the event Fluentd sent in| |---|---| |Receive (default)|None. Same record.| |Receive + `symbolMessageHashField `|Adds one new field with the symbol-pattern hash (a stable identifier for the message pattern, usable as a dedup key, metric dimension, or correlation ID).| |`receiverOptimize true`|The value of the message field (`log` by default, or whatever `fluentdInputMessageField` is set to) is replaced with a compact encoded form. A separate `tenx-template` event is emitted with the template needed to decode it. All other fields stay verbatim.| |`receiverOptimize true` + `symbolMessageHashField `|Both of the above.| `symbolMessageHashField` is unset by default, which is what makes the first row true: the receive path hands the record back exactly as it arrived. The pattern hash is still computed and still rides the event inside the engine as `tenx_hash` for metrics and aggregation, it just does not reach the wire. Naming a field opts in, either as a launch argument (`tenx @run/input/forwarder/fluentd @apps/receiver symbolMessageHashField my_custom_hash`) or as an environment variable of the same name. The original Fluentd tag is carried by the Forward protocol itself and surfaces on the event as its `source` inside Log10x, used for rate-based grouping and emitted back to Fluentd as the wire tag on the return path. Internally, Log10x's Fluentd input module reads the message text from the field named by `fluentdInputMessageField` (default `log`); when the Receiver app is configured with `k8sExtractorName: fluentK8s`, the `kubernetes.*` sub-object is also materialized as enrichment fields for use by message-pattern and rate filtering. ??? tenx-keyfiles "Key Files" | File | Purpose | |------|---------| | [`stream.yaml`](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/forwarder/fluentd/stream.yaml) | Fluentd Forward input + output stream definitions | | [`conf/tenx-sidecar.conf`](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/forwarder/fluentd/conf/tenx-sidecar.conf) | Reference Fluentd config showing `@INGEST` / `@OUTPUT` label routing | ## Quickstart **1. Run Log10x:** ```bash tenx @run/input/forwarder/fluentd @apps/receiver ``` **2. Wire up your Fluentd config**, include the sidecar recipe and route your sources to `@INGEST`: ```xml title="fluentd.conf" @include "#{ENV['TENX_MODULES']}/pipelines/run/modules/input/forwarder/fluentd/conf/tenx-sidecar.conf" @type tail path /var/log/app.log tag app.logs @label @INGEST # routes the source into the sidecar @type json ``` **3. Point `@OUTPUT` at your real destinations** (the recipe defaults to `stdout` for testing): ```xml ``` For Splunk integration see the [10x for Splunk](https://doc.log10x.com/apps/receiver/compact/splunk/) documentation. For Kubernetes deployment, add the `log10x/edge-10x` sidecar on top of the official Fluentd chart via a kustomize post-renderer overlay, see the [Helm chart overlay](https://doc.log10x.com/apps/receiver/deploy/#fluentd). ## :material-wrench-outline: Config Files To configure the Fluentd module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [fluentd/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/forwarder/fluentd/config.yaml "fluentd/config.yaml"){target="\_blank"}.
Edit Online
Edit config.yaml Locally
```yaml # ๐Ÿ”ŸโŽ 'run' Fluentd receiver configuration # # Every endpoint setting below reads an environment variable first and falls # back to the shipped default, so a deployment can move a port, a host or a # socket path without editing this file. The variable name is the launch # option name in upper snake case with a TENX_ prefix, for example the # `fluentdInputPort` option reads TENX_FLUENTD_INPUT_PORT. # # To learn more see https://doc.log10x.com/run/input/forwarder/fluentd/ tenx: run # =============================== Dependencies ================================ include: - run/input/forwarder/config.yaml - run/modules/input/forwarder/fluentd # =============================== Fluentd Options ============================= fluentd: # ----------------------------- Input Options ----------------------------- input: # 'port' specifies the TCP port to listen on for events from Fluentd's # out_forward plugin. # Env var: TENX_FLUENTD_INPUT_PORT port: $=TenXEnv.get("TENX_FLUENTD_INPUT_PORT", 24224) # 'path' switches the input to a Unix domain socket instead of TCP. # When set, takes precedence over 'port'. Empty means TCP. # Env var: TENX_FLUENTD_INPUT_PATH path: $=TenXEnv.get("TENX_FLUENTD_INPUT_PATH", "") # 'messageField' is the name of the JSON field in each Fluentd record # carrying the log line text. messageField: log # ----------------------------- Output Options ---------------------------- output: # 'host' specifies the TCP host of Fluentd's in_forward source receiving # processed events from Log10x. # Env var: TENX_FLUENTD_OUTPUT_HOST host: $=TenXEnv.get("TENX_FLUENTD_OUTPUT_HOST", "127.0.0.1") # 'port' specifies the TCP port of Fluentd's in_forward source. MUST differ # from input.port or the two would collide on the same listener. # Env var: TENX_FLUENTD_OUTPUT_PORT port: $=TenXEnv.get("TENX_FLUENTD_OUTPUT_PORT", 24225) # 'path' switches the output to a Unix domain socket instead of TCP. # When set, takes precedence over 'host'/'port'. Empty means TCP. # Env var: TENX_FLUENTD_OUTPUT_PATH path: $=TenXEnv.get("TENX_FLUENTD_OUTPUT_PATH", "") # 'encodeType' is the on-wire format for the Forward record. # - 'delimited': each top-level field of the rendered record becomes its # own Forward record field, preserves the original record's structure. # - 'json': the whole record is wrapped as one stringified field. encodeType: delimited ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") the Fluentd: |Name|Description|Category| |---|---|---| |[fluentdInputPort](#fluentdinputport "TCP port to listen on for events from Fluentd's out_forward plugin")|TCP port to listen on for events from Fluentd's out\_forward plugin|Input| |[fluentdInputPath](#fluentdinputpath "Unix domain socket path to listen on (Linux/macOS)")|Unix domain socket path to listen on (Linux/macOS)|Input| |[fluentdInputMessageField](#fluentdinputmessagefield "name of the JSON field carrying the original log line")|Name of the JSON field carrying the original log line|Input| |[fluentdOutputHost](#fluentdoutputhost "TCP host of the Fluentd in_forward source receiving processed events")|TCP host of the Fluentd in\_forward source receiving processed events|Output| |[fluentdOutputPort](#fluentdoutputport "TCP port of the Fluentd in_forward source receiving processed events")|TCP port of the Fluentd in\_forward source receiving processed events|Output| |[fluentdOutputPath](#fluentdoutputpath "Unix domain socket path of the Fluentd in_forward source (Linux/macOS)")|Unix domain socket path of the Fluentd in\_forward source (Linux/macOS)|Output| |[fluentdOutputEncodeType](#fluentdoutputencodetype "output format for the Forward record. Possible values: [json, delimited]")|Output format for the Forward record. Possible values: \[json, delimited\]|Output| ### Input #### :material-menu-right-outline:**`fluentdInputPort`** TCP port to listen on for events from Fluentd's out\_forward plugin. |Type|Default|Category| |---|---|---| |String|24224|Input| TCP port where Log10x listens for events sent by Fluentd's `out_forward` plugin. Match this against the `port` in your Fluentd `` block. #### :material-menu-right-outline:**`fluentdInputPath`** Unix domain socket path to listen on (Linux/macOS). |Type|Default|Category| |---|---|---| |String|""|Input| When set, Log10x listens on a Unix domain socket instead of TCP, useful on Linux/macOS to avoid TCP overhead. Configure Fluentd's `out_forward` with `` + a matching `path`. Ignored on Windows. #### :material-menu-right-outline:**`fluentdInputMessageField`** Name of the JSON field carrying the original log line. |Type|Default|Category| |---|---|---| |String|log|Input| Name of the field in each Fluentd record that contains the actual log message text. The default (`log`) matches Fluentd's `in_tail` plugin convention for container and file logs. Set to e.g. `message` if your sources emit the log line under a different key. ### Output #### :material-menu-right-outline:**`fluentdOutputHost`** TCP host of the Fluentd in\_forward source receiving processed events. |Type|Default|Category| |---|---|---| |String|127.0.0.1|Output| Hostname or IP where the Fluentd `in_forward` source is listening for processed events from Log10x. Pairs with `fluentdOutputPort`. #### :material-menu-right-outline:**`fluentdOutputPort`** TCP port of the Fluentd in\_forward source receiving processed events. |Type|Default|Category| |---|---|---| |String|24225|Output| TCP port where the Fluentd `in_forward` source is listening for processed events from Log10x. MUST differ from the port Log10x's own Forward input listens on (default `24224`) or the two would collide on the same socket. #### :material-menu-right-outline:**`fluentdOutputPath`** Unix domain socket path of the Fluentd in\_forward source (Linux/macOS). |Type|Default|Category| |---|---|---| |String|""|Output| Optional Unix domain socket path for the Fluentd `in_forward` source. When set, takes precedence over `fluentdOutputHost` / `fluentdOutputPort`. Linux/macOS only, ignored on Windows. #### :material-menu-right-outline:**`fluentdOutputEncodeType`** Output format for the Forward record. Possible values: \[json, delimited\]. |Type|Default|Category| |---|---|---| |String|delimited|Output| How the rendered event field(s) are encoded on the outgoing Forward record. Possible values: - **delimited**: each top-level field of the rendered record becomes its own Forward record field, preserves the original record's structure on the wire. - **json**: the whole rendered record is wrapped as one stringified field. Defaults to `delimited`, which is almost always what you want.
:material-github: This module is defined in [fluentd/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/forwarder/fluentd/module.yaml "fluentd/module.yaml"){target="\_blank"}.