--- title: "Logstash" description: "receive and optimize events collected by Logstash via newline-delimited\ \ JSON" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/forwarder/logstash/module.yaml" icon: "simple/logstash" --- Runs 10x Engine as a [sidecar](https://doc.log10x.com/engine/launcher/sidecar) to Logstash for reporting, receiving, and optimizing events before they ship to their destination (Elasticsearch, OpenSearch, Splunk, S3, Kafka, โ€ฆ). Logstash and Log10x run as peer processes and exchange events as newline-delimited JSON over TCP, Logstash's built-in `tcp` input and output plugins with `codec => json_lines` on both legs. Works against any stock Logstash build (OSS or Elastic distribution) and the official `elastic/logstash` Helm chart with a values overlay. ## Architecture
```mermaid graph LR A["
๐Ÿ“‚ Sources
file, beats, http
"] --> F["
๐Ÿงช ingest pipeline
enrichment filters
"] F --> B["
๐Ÿ“ค tcp output
json_lines :5044
"] B --> E["
โšก 10x Engine
Report/Receive/Optimize
"] E --> C["
๐Ÿ“ฅ tcp input
json_lines :5045
"] C --> D["
๐Ÿ“ค Destinations
ES, Splunk, S3, Kafka
"] classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8 class A input class B filter class C filter class D output class E engine class F filter ```
### Data Flow - ๐Ÿ“‚ **Sources**, Your existing Logstash inputs (`file`, `beats`, `tcp`, `http`, โ€ฆ) feed into the `ingest` pipeline. - ๐Ÿงช **ingest pipeline**, Your enrichment filters (`grok`, `mutate`, `kv`, `geoip`, `date`, โ€ฆ) run here exactly once before the event is handed off to Log10x. The pipeline also stamps a `tag` field that survives the round trip and surfaces as the event's source inside Log10x. - ๐Ÿ“ค **tcp output** โ†’ Log10x, Logstash's `tcp` output plugin (`codec => json_lines`) ships the enriched event to the Log10x sidecar on TCP `:5044` (or a Unix socket on Linux/macOS via the `unix` output plugin). - โšก **10x Engine**, The Receiver app applies rate/policy-based filtering and optionally compacts events for volume reduction. - ๐Ÿ“ฅ **tcp input โ†’ destinations pipeline**, Processed events come back to Logstash on `:5045` and are picked up by a separate `destinations` pipeline whose `output` block holds your real destinations. Filters defined under `ingest` are **not** re-applied, Logstash's [multi-pipeline routing](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html){target="\_blank"} keeps each pipeline's filter chain isolated. - ๐Ÿ“ค **Destinations**, The original `tag` field survives the round trip, so destinations that route on it (Splunk index, S3 prefix, Kafka topic, โ€ฆ) behave the same as if Log10x weren't in the path. ### What an event looks like on the way back The record structure of the original Logstash event is preserved end-to-end, every field comes back to your `destinations` pipeline with the same name and same position. What changes depends on the Receiver app mode: |Mode|Difference vs the event Logstash sent in| |---|---| |Receive (default)|None. Same record.| |Receive + `symbolMessageHashField `|Adds one new field with the symbol-pattern hash (a stable identifier for the message pattern, usable as a dedup key, metric dimension, or correlation ID).| |`receiverOptimize true`|The value of the message field (`message` by default, or whatever `logstashInputMessageField` is set to) is replaced with a compact encoded form. A separate event with `tag` set to `tenx-template` is emitted with the template needed to decode it. All other fields stay verbatim.| |`receiverOptimize true` + `symbolMessageHashField `|Both of the above.| `symbolMessageHashField` is unset by default, which is what makes the first row true: the receive path hands the record back exactly as it arrived. The pattern hash is still computed and still rides the event inside the engine as `tenx_hash` for metrics and aggregation, it just does not reach the wire. Naming a field opts in, either as a launch argument (`tenx @run/input/forwarder/logstash @apps/receiver symbolMessageHashField my_custom_hash`) or as an environment variable of the same name. The `tag` field that the ingest pipeline stamps becomes the event's `source` inside Log10x, used for rate-based grouping and preserved on each event as it returns to Logstash. The message text is read from the `message` field by default (override with `logstashInputMessageField`). When the Receiver app is configured with `k8sExtractorName: fluentK8s`, the `kubernetes.*` sub-object is also lifted into pod/container metadata fields used by message-pattern and rate filtering. ??? tenx-keyfiles "Key Files" | File | Purpose | |------|---------| | [`stream.yaml`](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/forwarder/logstash/stream.yaml) | Logstash JSON socket input + output stream definitions | | [`conf/pipelines.yml`](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/forwarder/logstash/conf/pipelines.yml) | Two-pipeline driver that wires the `ingest` and `destinations` legs together | | [`conf/tenx-ingest.conf`](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/forwarder/logstash/conf/tenx-ingest.conf) | Sources, enrichment filters, and the `tcp` handoff to Log10x on `:5044` | | [`conf/tenx-destinations.conf`](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/forwarder/logstash/conf/tenx-destinations.conf) | `tcp` input on `:5045` for processed events, plus the `output` block holding your real destinations | ## Quickstart **1. Run Log10x:** ```bash tenx @run/input/forwarder/logstash @apps/receiver ``` **2. Wire up your Logstash config**, the recipe ships a `pipelines.yml` that runs the two legs side by side, an ingest leg that ends in `tcp { codec => json_lines }` and a destinations leg that starts with `tcp { codec => json_lines }`: ```yaml title="pipelines.yml" - pipeline.id: ingest path.config: "${TENX_MODULES}/pipelines/run/modules/input/forwarder/logstash/conf/tenx-ingest.conf" - pipeline.id: destinations path.config: "${TENX_MODULES}/pipelines/run/modules/input/forwarder/logstash/conf/tenx-destinations.conf" ``` **3. Point the `ingest` pipeline at your own sources** (the recipe defaults to a `file` input over `/var/log/containers/*.log`). Keep the `mutate` that stamps `tag`, Log10x reads it as the event's source, and keep the `tcp` output on `:5044`: ```ruby title="tenx-ingest.conf" input { # ... your real sources ... } filter { mutate { add_field => { "tag" => "logstash" } } } output { tcp { host => "127.0.0.1" port => 5044 codec => json_lines } } ``` **4. Point the `destinations` pipeline at your real outputs** (the recipe defaults to `stdout` for testing): ```ruby title="tenx-destinations.conf" input { tcp { host => "0.0.0.0" port => 5045 codec => json_lines } } output { # ... your real destinations ... } ``` For Kubernetes deployment via the official `elastic/logstash` Helm chart see the [Helm sidecar overlay](https://doc.log10x.com/apps/receiver/deploy/#logstash). ## :material-wrench-outline: Config Files To configure the Logstash module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [logstash/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/forwarder/logstash/config.yaml "logstash/config.yaml"){target="\_blank"}.
Edit Online
Edit config.yaml Locally
```yaml # ๐Ÿ”ŸโŽ 'run' Logstash receiver configuration # # Every endpoint setting below reads an environment variable first and falls # back to the shipped default, so a deployment can move a port, a host or a # socket path without editing this file. The variable name is the launch # option name in upper snake case with a TENX_ prefix, for example the # `logstashInputPort` option reads TENX_LOGSTASH_INPUT_PORT. # # To learn more see https://doc.log10x.com/run/input/forwarder/logstash/ tenx: run # =============================== Dependencies ================================ include: - run/input/forwarder/config.yaml - run/modules/input/forwarder/logstash # ============================== Logstash Options ============================= logstash: # ----------------------------- Input Options ----------------------------- input: # 'port' specifies the TCP port to listen on for events from Logstash's # `tcp` output with `codec => json_lines`. # Env var: TENX_LOGSTASH_INPUT_PORT port: $=TenXEnv.get("TENX_LOGSTASH_INPUT_PORT", 5044) # 'path' switches the input to a Unix domain socket instead of TCP. # When set, takes precedence over 'port'. Empty means TCP. # Env var: TENX_LOGSTASH_INPUT_PATH path: $=TenXEnv.get("TENX_LOGSTASH_INPUT_PATH", "") # 'messageField' is the name of the JSON field in each Logstash event # carrying the log line text. messageField: message # ----------------------------- Output Options ---------------------------- output: # 'host' specifies the TCP host of Logstash's `tcp` input receiving # processed events from Log10x. # Env var: TENX_LOGSTASH_OUTPUT_HOST host: $=TenXEnv.get("TENX_LOGSTASH_OUTPUT_HOST", "127.0.0.1") # 'port' specifies the TCP port of Logstash's `tcp` input. MUST differ # from input.port or the two would collide on the same listener. # Env var: TENX_LOGSTASH_OUTPUT_PORT port: $=TenXEnv.get("TENX_LOGSTASH_OUTPUT_PORT", 5045) # 'encodeType' is the on-wire format for the socket record. # - 'delimited': each top-level field of the rendered record becomes its # own emitted field, preserves the original record's structure. # - 'json': the whole record is wrapped as one stringified field. encodeType: delimited ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") the Logstash: |Name|Description|Category| |---|---|---| |[logstashInputPort](#logstashinputport "TCP port to listen on for events from Logstash's tcp output")|TCP port to listen on for events from Logstash's tcp output|Input| |[logstashInputPath](#logstashinputpath "Unix domain socket path to listen on (Linux/macOS)")|Unix domain socket path to listen on (Linux/macOS)|Input| |[logstashInputMessageField](#logstashinputmessagefield "name of the JSON field carrying the original log line")|Name of the JSON field carrying the original log line|Input| |[logstashOutputHost](#logstashoutputhost "TCP host of the Logstash tcp input receiving processed events")|TCP host of the Logstash tcp input receiving processed events|Output| |[logstashOutputPort](#logstashoutputport "TCP port of the Logstash tcp input receiving processed events")|TCP port of the Logstash tcp input receiving processed events|Output| |[logstashOutputEncodeType](#logstashoutputencodetype "output format when outputFields are set. Possible values: [json, delimited]")|Output format when outputFields are set. Possible values: \[json, delimited\]|Output| ### Input #### :material-menu-right-outline:**`logstashInputPort`** TCP port to listen on for events from Logstash's tcp output. |Type|Default|Category| |---|---|---| |String|5044|Input| TCP port where Log10x listens for events sent by Logstash's `tcp` output with `codec => json_lines`. Match this against the `port` in your Logstash `output { tcp { ... } }` block. #### :material-menu-right-outline:**`logstashInputPath`** Unix domain socket path to listen on (Linux/macOS). |Type|Default|Category| |---|---|---| |String|""|Input| When set, Log10x listens on a Unix domain socket instead of TCP, useful on Linux/macOS to avoid TCP overhead. Configure Logstash's `unix` output plugin with `codec => json_lines` and a matching `path`. Ignored on Windows. #### :material-menu-right-outline:**`logstashInputMessageField`** Name of the JSON field carrying the original log line. |Type|Default|Category| |---|---|---| |String|message|Input| Name of the field in each Logstash event that contains the actual log message text. The default (`message`) matches Logstash's convention for the `file`, `beats`, `tcp`, and `http` inputs. Set to e.g. `log` if your sources emit the log line under a different key. ### Output #### :material-menu-right-outline:**`logstashOutputHost`** TCP host of the Logstash tcp input receiving processed events. |Type|Default|Category| |---|---|---| |String|127.0.0.1|Output| Hostname or IP where the Logstash `tcp` input is listening for processed events from Log10x. Pairs with `logstashOutputPort`. #### :material-menu-right-outline:**`logstashOutputPort`** TCP port of the Logstash tcp input receiving processed events. |Type|Default|Category| |---|---|---| |String|5045|Output| TCP port where the Logstash `tcp` input is listening for processed events from Log10x. MUST differ from the port Log10x's own input listens on (default `5044`) or the two would collide on the same socket. #### :material-menu-right-outline:**`logstashOutputEncodeType`** Output format when outputFields are set. Possible values: \[json, delimited\]. |Type|Default|Category| |---|---|---| |String|delimited|Output| Specifies how the combined output (main event field plus outputFields) is encoded when writing back to Logstash. Possible values: - **json**: formats all fields as a JSON object - **delimited**: formats field values separated by the output delimiter Only takes effect when logstashOutputFields is set.
:material-github: This module is defined in [logstash/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/forwarder/logstash/module.yaml "logstash/module.yaml"){target="\_blank"}.