--- title: "Fluent Bit" description: "receive and optimize events collected by Fluent Bit via the Fluent Forward\ \ protocol" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/forwarder/fluentbit/module.yaml" icon: "simple/fluentbit" --- Runs 10x Engine as a [sidecar](https://doc.log10x.com/engine/launcher/sidecar) to Fluent Bit for reporting, receiving, and optimizing events before they ship to their destination (Elasticsearch, Splunk, S3, Kafka, …). In the VM/host recipe, Fluent Bit and Log10x run as peer processes and exchange events over the [Fluent Forward protocol](https://docs.fluentbit.io/manual/pipeline/outputs/forward){target="\_blank"} in both directions; works against any stock Fluent Bit build and the official Fluent Bit Helm chart on Kubernetes via a `log10x/edge-10x` sidecar container (`extraContainers` values overlay). ## Architecture

```mermaid graph LR A["

📂 Sources

tail, http, k8s

"] --> F["

🧪 Filters

Match app.*

"] F --> B["

📤 out_forward

:24224

"] B --> E["

⚡ 10x Engine

Receive/Optimize

"] E --> C["

📥 in_forward

:24225 Tag_Prefix tenx.

"] C --> D["

📤 Destinations

ES, Splunk, S3, Kafka

"] classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8 class A input class B filter class C filter class D output class E engine class F filter ```

### Data Flow - 📂 **Sources**, Your existing Fluent Bit inputs (`tail`, `http`, `forward`, `systemd`, …) tag events with your normal scheme (e.g. `app.*`, `k8s.*`). - 🧪 **Filters**, Your enrichment filters (`kubernetes`, `modify`, `parser`, `lua`, …) `Match` your source tags only (e.g. `Match app.*`). They run exactly once before the event is handed off to Log10x. - 📤 **out\_forward** → Log10x, A `forward` output with `Match app.*` (or whatever your source tag pattern is) ships the enriched event to the Log10x sidecar over TCP `:24224`. - ⚡ **10x Engine**, The Receiver app applies rate/policy-based filtering and optionally compacts events for volume reduction. - 📥 **in\_forward → destinations**, Processed events come back to Fluent Bit on `:24225` via a `forward` input with `Tag_Prefix tenx.`, which prepends `tenx.` to every returning tag. Your destination outputs `Match tenx.*`; the `tenx.` namespace is what keeps filters from re-firing and the `out_forward` to Log10x from looping events back. Fluent Bit has no label/scope concept, so **tag-prefix namespacing is the bypass mechanism**. - 📤 **Destinations**, Your destination outputs (`es`, `splunk`, `kafka`, `s3`, …) `Match tenx.*` and ship to the real destinations. The original tag is preserved after the prefix (`tenx.app.foo`), so destinations that route on the suffix still work. ### What an event looks like on the way back The record structure of the original Fluent Bit event is preserved end-to-end, every field comes back to your destination outputs with the same name and same position. The wire tag has a `tenx.` prefix prepended on egress (the bypass mechanism); the original tag is the suffix. What changes in the record body depends on the Receiver app mode: |Mode|Difference vs the event Fluent Bit sent in| |---|---| |Receive (default)|None. Same record. Tag is `tenx.`.| |Receive + `symbolMessageHashField `|Adds one new field with the symbol-pattern hash (a stable identifier for the message pattern, usable as a dedup key, metric dimension, or correlation ID).| |`receiverOptimize true`|The value of the message field (`log` by default, or whatever `fluentbitInputMessageField` is set to) is replaced with a compact encoded form. A separate `tenx-template` event is emitted with the template needed to decode it. All other fields stay verbatim.| |`receiverOptimize true` + `symbolMessageHashField `|Both of the above.| `symbolMessageHashField` is unset by default, which is what makes the first row true: the receive path hands the record back exactly as it arrived. The pattern hash is still computed and still rides the event inside the engine as `tenx_hash` for metrics and aggregation, it just does not reach the wire. Naming a field opts in, either as a launch argument (`tenx @run/input/forwarder/fluentbit @apps/receiver symbolMessageHashField my_custom_hash`) or as an environment variable of the same name. The original Fluent Bit tag is carried by the Forward protocol itself and surfaces on the event as its `source` inside Log10x, used for rate-based grouping and re-emitted as the wire tag on the return path (with `tenx.` prepended by the egress `forward` input). Internally, Log10x's Fluent Bit input module reads the message text from the field named by `fluentbitInputMessageField` (default `log`); when the Receiver app is configured with `k8sExtractorName: fluentK8s`, the `kubernetes.*` sub-object is also materialized as enrichment fields for use by message-pattern and rate filtering. ??? tenx-keyfiles "Key Files" | File | Purpose | |------|---------| | [`stream.yaml`](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/forwarder/fluentbit/stream.yaml) | Fluent Bit Forward input + output stream definitions | | [`conf/tenx-sidecar.conf`](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/forwarder/fluentbit/conf/tenx-sidecar.conf) | Reference Fluent Bit config showing tag-prefix bypass via `Tag_Prefix tenx.` | ## Quickstart **1. Run Log10x:** ```bash tenx @run/input/forwarder/fluentbit @apps/receiver ``` **2. Wire up your Fluent Bit config**, include the sidecar recipe and `Match` your sources with the recipe's tag conventions: ```ini title="fluent-bit.conf" @INCLUDE ${TENX_MODULES}/pipelines/run/modules/input/forwarder/fluentbit/conf/tenx-sidecar.conf [INPUT] Name tail Path /var/log/app.log Tag app.logs # any tag NOT starting with `tenx.` Parser json ``` **3. Point destination outputs at the `tenx.*` namespace** (the recipe defaults to `stdout` for testing): ```ini [OUTPUT] Name your_output_plugin Match tenx.* # only matches events that came back from Log10x # ... destination config ``` For Splunk integration see the [10x for Splunk](https://doc.log10x.com/apps/receiver/compact/splunk/) documentation. For Kubernetes deployment, add the `log10x/edge-10x` sidecar on top of the official Fluent Bit chart via an `extraContainers` values overlay, see the [Helm chart overlay](https://doc.log10x.com/apps/receiver/deploy/#fluent-bit). ## :material-wrench-outline: Config Files To configure the Fluent Bit module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [fluentbit/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/forwarder/fluentbit/config.yaml "fluentbit/config.yaml"){target="\_blank"}.

Edit Online

```yaml # 🔟❎ 'run' Fluent Bit receiver configuration # # Every endpoint setting below reads an environment variable first and falls # back to the shipped default, so a deployment can move a port, a host or a # socket path without editing this file. The variable name is the launch # option name in upper snake case with a TENX_ prefix, for example the # `fluentbitInputPort` option reads TENX_FLUENTBIT_INPUT_PORT. # # To learn more see https://doc.log10x.com/run/input/forwarder/fluentbit/ tenx: run # =============================== Dependencies ================================ include: - run/input/forwarder/config.yaml - run/modules/input/forwarder/fluentbit # ============================= Fluent Bit Options ============================ fluentbit: # ----------------------------- Input Options ----------------------------- input: # 'port' specifies the TCP port to listen on for events from Fluent Bit's # forward output plugin. # Env var: TENX_FLUENTBIT_INPUT_PORT port: $=TenXEnv.get("TENX_FLUENTBIT_INPUT_PORT", 24224) # 'path' switches the input to a Unix domain socket instead of TCP. # When set, takes precedence over 'port'. Empty means TCP. # Env var: TENX_FLUENTBIT_INPUT_PATH path: $=TenXEnv.get("TENX_FLUENTBIT_INPUT_PATH", "") # 'messageField' is the name of the JSON field in each Fluent Bit record # carrying the log line text. messageField: log # ----------------------------- Output Options ---------------------------- output: # 'host' specifies the TCP host of Fluent Bit's forward input receiving # processed events from Log10x. # Env var: TENX_FLUENTBIT_OUTPUT_HOST host: $=TenXEnv.get("TENX_FLUENTBIT_OUTPUT_HOST", "127.0.0.1") # 'port' specifies the TCP port of Fluent Bit's forward input. MUST differ # from input.port or the two would collide on the same listener. # Env var: TENX_FLUENTBIT_OUTPUT_PORT port: $=TenXEnv.get("TENX_FLUENTBIT_OUTPUT_PORT", 24225) # 'path' switches the output to a Unix domain socket of TCP. # When set, takes precedence over 'host'/'port'. Empty means TCP. # Env var: TENX_FLUENTBIT_OUTPUT_PATH path: $=TenXEnv.get("TENX_FLUENTBIT_OUTPUT_PATH", "") # 'encodeType' is the on-wire format for the Forward record. # - 'delimited': each top-level field of the rendered record becomes its # own Forward record field, preserves the original record's structure. # - 'json': the whole record is wrapped as one stringified field. encodeType: delimited ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") the Fluent Bit: |Name|Description|Category| |---|---|---| |[fluentbitInputPort](#fluentbitinputport "TCP port to listen on for events from Fluent Bit's forward output")|TCP port to listen on for events from Fluent Bit's forward output|Input| |[fluentbitInputPath](#fluentbitinputpath "Unix domain socket path to listen on (Linux/macOS)")|Unix domain socket path to listen on (Linux/macOS)|Input| |[fluentbitInputMessageField](#fluentbitinputmessagefield "name of the JSON field carrying the original log line")|Name of the JSON field carrying the original log line|Input| |[fluentbitOutputHost](#fluentbitoutputhost "TCP host of the Fluent Bit forward input receiving processed events")|TCP host of the Fluent Bit forward input receiving processed events|Output| |[fluentbitOutputPort](#fluentbitoutputport "TCP port of the Fluent Bit forward input receiving processed events")|TCP port of the Fluent Bit forward input receiving processed events|Output| |[fluentbitOutputPath](#fluentbitoutputpath "Unix domain socket path of the Fluent Bit forward input (Linux/macOS)")|Unix domain socket path of the Fluent Bit forward input (Linux/macOS)|Output| |[fluentbitOutputEncodeType](#fluentbitoutputencodetype "output format when outputFields are set. Possible values: [json, delimited]")|Output format when outputFields are set. Possible values: \[json, delimited\]|Output| ### Input #### :material-menu-right-outline:**`fluentbitInputPort`** TCP port to listen on for events from Fluent Bit's forward output. |Type|Default|Category| |---|---|---| |String|24224|Input| TCP port where Log10x listens for events sent by Fluent Bit's `forward` output. Match this against the `Port` in your Fluent Bit `[OUTPUT]` block (`Name forward`). #### :material-menu-right-outline:**`fluentbitInputPath`** Unix domain socket path to listen on (Linux/macOS). |Type|Default|Category| |---|---|---| |String|""|Input| When set, Log10x listens on a Unix domain socket instead of TCP, useful on Linux/macOS to avoid TCP overhead. Configure Fluent Bit's `forward` output with `Unix_Path` set to a matching path. Ignored on Windows. #### :material-menu-right-outline:**`fluentbitInputMessageField`** Name of the JSON field carrying the original log line. |Type|Default|Category| |---|---|---| |String|log|Input| Name of the field in each Fluent Bit record that contains the actual log message text. The default (`log`) matches Fluent Bit's `tail` input convention for container and file logs. Set to e.g. `message` if your sources emit the log line under a different key. ### Output #### :material-menu-right-outline:**`fluentbitOutputHost`** TCP host of the Fluent Bit forward input receiving processed events. |Type|Default|Category| |---|---|---| |String|127.0.0.1|Output| Hostname or IP where Fluent Bit's `forward` input is listening for processed events from Log10x. Pairs with `fluentbitOutputPort`. #### :material-menu-right-outline:**`fluentbitOutputPort`** TCP port of the Fluent Bit forward input receiving processed events. |Type|Default|Category| |---|---|---| |String|24225|Output| TCP port where Fluent Bit's `forward` input is listening for processed events from Log10x. MUST differ from the port Log10x's own Forward input listens on (default `24224`) or the two would collide on the same socket. #### :material-menu-right-outline:**`fluentbitOutputPath`** Unix domain socket path of the Fluent Bit forward input (Linux/macOS). |Type|Default|Category| |---|---|---| |String|""|Output| Optional Unix domain socket path for Fluent Bit's `forward` input. When set, takes precedence over `fluentbitOutputHost` / `fluentbitOutputPort`. Linux/macOS only, ignored on Windows. #### :material-menu-right-outline:**`fluentbitOutputEncodeType`** Output format when outputFields are set. Possible values: \[json, delimited\]. |Type|Default|Category| |---|---|---| |String|delimited|Output| Specifies how the combined output (main event field plus outputFields) is encoded when writing back to Fluent Bit. Possible values: - **json**: formats all fields as a JSON object - **delimited**: formats field values separated by the output delimiter Only takes effect when fluentbitOutputFields is set.
:material-github: This module is defined in [fluentbit/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/forwarder/fluentbit/module.yaml "fluentbit/module.yaml"){target="\_blank"}.