--- title: "Log Forwarder Inputs" description: "report, receive and optimize events collected by log forwarders" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/forwarder/module.yaml" icon: "simple/fluentbit" --- Runs the 10x Engine [in-path](https://doc.log10x.com/engine/launcher/sidecar) with your log forwarder to process collected log events _before_ they ship to output destinations (e.g., Splunk, Elasticsearch, S3). Depending on the forwarder, 10x runs as a separate `log10x/edge-10x` sidecar container added via a values or kustomize overlay (Fluentd, Fluent Bit, Logstash, OTel Collector, Vector), as an image swap to the embedded 10x variant (Filebeat, `log10x/filebeat-10x`), or as a file relay (Splunk UF, Datadog Agent); 5 of the 8 supported forwarders run the sidecar. The design enables 10x apps, the [Reporter](https://doc.log10x.com/apps/reporter/) (read-only DaemonSet alongside the forwarder) and the [Receiver](https://doc.log10x.com/apps/receiver/) (embedded or sidecar per forwarder, with pass, sample, [compact](https://doc.log10x.com/apps/receiver/#compact), tier\_down, offload, and drop actions plus read-only observation), to process events at the source while integrating with existing log forwarders (e.g., Fluentd/Bit). ### :material-toy-brick-outline: Extensibility All forwarder input [modules](https://doc.log10x.com/engine/module/) utilize core IPC I/O modules (e.g., [stdin](https://doc.log10x.com/run/input/stdin "Read events from stdin"), [Unix](https://doc.log10x.com/run/output/event/unix "Write TenXObject and template values to a log4j2 Unix domain socket appender")) as building blocks for integrating with bundled forwarders (e.g., Fluentd/Bit) and to serve as a reference for supporting additional forwarder types. ## :octicons-package-24: Modules
- :simple-fluentbit:{ .lg .middle } __Fluent Bit__ --- Receive and optimize events collected by Fluent Bit via the Fluent Forward protocol. [:octicons-arrow-right-24: More info](/run/input/forwarder/fluentbit) - :simple-fluentd:{ .lg .middle } __Fluentd__ --- Receive and optimize events collected by Fluentd via the Fluent Forward protocol. [:octicons-arrow-right-24: More info](/run/input/forwarder/fluentd) - :simple-opentelemetry:{ .lg .middle } __OpenTelemetry Collector__ --- Receive and optimize events collected by the OpenTelemetry Collector via OTLP/gRPC (both directions). [:octicons-arrow-right-24: More info](/run/input/forwarder/otel-collector) - :simple-vector:{ .lg .middle } __Vector__ --- Receive and optimize events collected by Vector via the socket sink and fluent source. [:octicons-arrow-right-24: More info](/run/input/forwarder/vector) - :simple-beats:{ .lg .middle } __Filebeat__ --- Report, receive, and optimize events collected by Filebeat forwarders. [:octicons-arrow-right-24: More info](/run/input/forwarder/filebeat) - :simple-logstash:{ .lg .middle } __Logstash__ --- Receive and optimize events collected by Logstash via newline-delimited JSON. [:octicons-arrow-right-24: More info](/run/input/forwarder/logstash) - :material-greater-than:{ .lg .middle } __Splunk UF Input__ --- Receive and optimize events before shipping to Splunk via Universal Forwarder. [:octicons-arrow-right-24: More info](/run/input/forwarder/splunkUF) - :simple-datadog:{ .lg .middle } __Datadog Agent Input__ --- Receive and optimize events before shipping to Datadog via Datadog Agent. [:octicons-arrow-right-24: More info](/run/input/forwarder/datadogAgent)
## :material-wrench-outline: Config Files To configure the Log Forwarder inputs module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [forwarder/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/forwarder/config.yaml "forwarder/config.yaml"){target="\_blank"}.
Edit Online
Edit config.yaml Locally
```yaml # 🔟❎ 'run' shared forwarder configuration # tenx: run # =============================== Dependencies ================================ include: # Shared forwarder options (mode dispatch, output encoding) - run/modules/input/forwarder/module.yaml # =============================== Mode Options ================================ # # The Receiver app picks the encoding mode based on user flags: # default → emit events verbatim (`fullText`) # receiverOptimize=true → compact-all (lossless compaction of every event) # compactReceiverLookupFile=… → compact-lookup (per-container decision) # # `symbolMessageHashField` (optional, unset by default) adds a stable # pattern-hash field alongside the encoded/fullText output: when set, encode() # and fullText are called as `encode("")` / `fullText("")`. # # Unset is the receive-path default and it means VERBATIM: the record the # forwarder gets back is the record it sent, byte for byte, with no field added. # The hash is still computed and still rides the TenXObject internally as # `tenx_hash` for metrics and aggregation, it just does not reach the wire. # `symbolMessageHashField my_custom_hash` (CLI arg or env var) opts in and names # the field. output: # outputEncodeField is the per-event output field expression, resolved once at # config-load by the `$=yield` ternary below. Written as a YAML `|-` block # scalar for readability: the engine treats the newlines/indentation between # tokens as ordinary whitespace, so this compiles identically to the original # one-liner. (Block scalars do NOT process backslash escapes, so the `\"` # below reach the expression literally, exactly what encode("field") needs.) # # The three resolved modes (the regulator marks routeState via route(action), # where action is the per-service disposition: drop/offload/tier_down/compact/ # sample/pass; this picks the output FORMAT, while `outputSoftDrop` also flips # the stream filter): # receiverOptimize=true → encoded=encode() (compact every event) # outputOffload truthy → fullText("routeState") (FULL text + marker; for S3 offload, never compacted) # outputSoftDrop truthy → encoded=isRoute("compact") ? encode("routeState") : fullText("routeState") (compact the compact-marked slice; every other action stays fullText so the forwarder routes it by routeState) # neither → fullText (receive / hard-drop) # The `symbolMessageHashField` variant (when set) passes the hash field name # into encode()/fullText() so a stable pattern-hash ships alongside the output. # # ROUTE MARKER: outputOffload and outputSoftDrop both splice the `routeState` # field onto every forwarded event so the downstream forwarder can route by the # per-service action name (offload → the customer's own S3, tier_down → the SIEM # cheap tier, compact → encoded bytes to the SIEM, pass/sample → the SIEM, drop → # suppressed). The splice writes the route NAME as a JSON STRING (e.g. # `"routeState":"offload"` / `"routeState":"pass"`), so a forwarder match must be # string-equality against the action name, NOT boolean truthiness. hard-drop # omits dropped events entirely, so it emits no marker (nothing to route). encodeField: |- $=yield TenXEnv.get("receiverOptimize") ? ("encoded=" + (TenXEnv.get("symbolMessageHashField") ? ("encode(\"" + TenXEnv.get("symbolMessageHashField") + "\")") : "encode()")) : (TenXEnv.get("outputOffload") ? (TenXEnv.get("symbolMessageHashField") ? ("fullText(\"" + TenXEnv.get("symbolMessageHashField") + "\",\"routeState\")") : "fullText(\"routeState\")") : (TenXEnv.get("outputSoftDrop") ? ("encoded=isRoute(\"compact\") ? " + (TenXEnv.get("symbolMessageHashField") ? ("encode(\"" + TenXEnv.get("symbolMessageHashField") + "\",\"routeState\")") : "encode(\"routeState\")") + " : " + (TenXEnv.get("symbolMessageHashField") ? ("fullText(\"" + TenXEnv.get("symbolMessageHashField") + "\",\"routeState\")") : "fullText(\"routeState\")")) : (TenXEnv.get("symbolMessageHashField") ? ("fullText(\"" + TenXEnv.get("symbolMessageHashField") + "\")") : "fullText"))) # outputWriteTemplates = true whenever events are encoded (compact-all or soft-drop) writeTemplates: $=yield TenXEnv.get("receiverOptimize") || TenXEnv.get("outputSoftDrop") # outputDropFilter decides whether a (possibly regulator-marked) object is written. # Resolved ONCE here so the soft/hard logic lives next to encodeField instead of # being duplicated in every output stream; streams just reference $?outputDropFilter. # (Named outputDropFilter, not outputFilter -- the latter is a built-in per-stream # engine option and would collide.) # outputSoftDrop or outputOffload truthy → "isObject" (write all; dropped events flow for routing/compaction) # unset (hard, default) → "isObject && !this.isRoute("drop")" (omit dropped objects) dropFilter: '$=yield (TenXEnv.get("outputSoftDrop") || TenXEnv.get("outputOffload")) ? "isObject" : "isObject && !this.isRoute(\"drop\")"' ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") the Log Forwarder inputs: |Name|Description| |---|---| |[outputEncodeField](#outputencodefield "the single field expression each forwarder's output stream writes per event")|The single field expression each forwarder's output stream writes per event| |[outputWriteTemplates](#outputwritetemplates "whether the forwarder emits new TenXTemplates alongside encoded events")|Whether the forwarder emits new TenXTemplates alongside encoded events| |[emit\_dropped](#emit_dropped "whether regulator-marked (`routeState=\"drop\"`) events survive the output filter")|Whether regulator-marked (\`routeState="drop"\`) events survive the output filter| |[compact\_emitted](#compact_emitted "whether all emitted events are written as `encode()` bytes or as `fullText`")|Whether all emitted events are written as \`encode()\` bytes or as \`fullText\`| |[outputDropFilter](#outputdropfilter "the filter expression each output stream uses to decide whether a marked object is written")|The filter expression each output stream uses to decide whether a marked object is written| ### :material-menu-right-outline:**`outputEncodeField`** The single field expression each forwarder's output stream writes per event. |Type|Default| |---|---| |String|fullText| Derived by the customer-facing config.yaml from `compact_emitted` (and from the legacy per-pattern `compactReceiverLookupFile` lookup when set): - **compact\_emitted=false** (default): `fullText`, every event passes through full text - **compact\_emitted=true**: `encoded=encode()`, every emitted event compacted losslessly - **compact-lookup (legacy)**: `encoded=shouldEncode() ? encode() : fullText`, per-pattern decision via the compactReceiver module (requires `compactReceiverLookupFile`) Advanced users can override directly to customize the output field expression. ### :material-menu-right-outline:**`outputWriteTemplates`** Whether the forwarder emits new TenXTemplates alongside encoded events. |Type|Default| |---|---| |String|false| True for encoding modes (compact-all, compact-lookup) so decoders can reconstruct events from templateHash + vars. False for receive mode (no encoding → no templates needed). ### :material-menu-right-outline:**`emit_dropped`** Whether regulator-marked (\`routeState="drop"\`) events survive the output filter. |Type|Default| |---|---| |String|""| Single boolean controlling whether the regulator-dropped slice reaches the forwarder. Orthogonal to `compact_emitted`. - **unset (default) = hard-drop**: outputDropFilter resolves to `isObject && !isRoute("drop")`, dropped events never leave the engine. - **truthy = emit**: outputDropFilter resolves to `isObject`, all events flow. The downstream forwarder routes by the wire-level `routeState` record field. Use truthiness (engine string `==` is identity compare): set a non-empty value (e.g. `true`) to emit dropped events, leave unset to hard-drop. ### :material-menu-right-outline:**`compact_emitted`** Whether all emitted events are written as \`encode()\` bytes or as \`fullText\`. |Type|Default| |---|---| |String|""| Single boolean controlling whether ALL emitted events are compacted on the wire. Orthogonal to `emit_dropped`. Applies uniformly: when set, both kept and dropped events (if `emit_dropped`) emit as `encode()`; when unset, both emit as `fullText`. - **unset (default) = fullText**: outputEncodeField resolves to `fullText` - **truthy = compact**: outputEncodeField resolves to `encoded=encode()` Use truthiness (engine string `==` is identity compare): set a non-empty value (e.g. `true`) for compact, leave unset for fullText. Per-pattern compact-lookup mode (via `compactReceiverLookupFile`) still overrides this when the lookup file is configured. ### :material-menu-right-outline:**`outputDropFilter`** The filter expression each output stream uses to decide whether a marked object is written. |Type|Default| |---|---| |String|isObject \&\& !isRoute("drop")| Resolved once by the customer-facing config.yaml from `emit_dropped`, so the emit/hard-drop logic lives in one place instead of being duplicated per stream: - **hard-drop** (default): `isObject && !isRoute("drop")`, dropped objects not written - **emit** (`emit_dropped` truthy): `isObject`, everything written; routing decision moves downstream to the forwarder based on the wire-level `routeState` field. Each output stream references it via `$?outputDropFilter`. Named outputDropFilter rather than outputFilter, which is a built-in per-stream engine option.
:material-github: This module is defined in [forwarder/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/forwarder/module.yaml "forwarder/module.yaml"){target="\_blank"}.