--- title: "Vector" description: "receive and optimize events collected by Vector via the socket sink\ \ and fluent source" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/forwarder/vector/module.yaml" icon: "simple/vector" --- Runs 10x Engine as a [sidecar](https://doc.log10x.com/engine/launcher/sidecar) to [Vector](https://vector.dev "Receive and optimize events collected by Vector via the socket sink and fluent source"){target="\_blank"} for reporting, receiving, and optimizing events before they ship to their destination (Elasticsearch, Splunk, S3, Kafka, โ€ฆ). Vector and Log10x run as peer processes, Vector sends events to Log10x via its native `socket` sink (newline-delimited JSON over TCP or Unix socket) and receives processed events back via its native `fluent` source (Fluent Forward protocol). Works against any stock Vector build (Linux/macOS/Windows) and the official `vector/vector` Helm chart with a values overlay. ## Architecture
```mermaid graph LR A["
๐Ÿ“‚ Sources
file, kubernetes_logs, journald
"] --> F["
๐Ÿงช transforms
enrichment
"] F --> B["
๐Ÿ“ค socket sink
tcp/unix โ†’ :9000
"] B --> E["
โšก 10x Engine
Receive/Optimize
"] E --> C["
๐Ÿ“ฅ fluent source
:9001 (no transforms)
"] C --> D["
๐Ÿ“ค Destinations
ES, Splunk, S3, Kafka
"] classDef input fill:#2563eb,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef filter fill:#ea580c,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef engine fill:#7c3aed,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef output fill:#16a34a,stroke:#15803d,color:#ffffff,stroke-width:2px,rx:8,ry:8 class A input class B filter class C filter class D output class E engine class F filter ```
### Data Flow - ๐Ÿ“‚ **Sources**, Your existing Vector sources (`file`, `kubernetes_logs`, `journald`, `socket`, โ€ฆ) feed events into the enrichment transforms. - ๐Ÿงช **Transforms**, Your enrichment transforms (`remap`, `filter`, `route`, โ€ฆ) run here exactly once before the event is handed off to Log10x. The recipe places them between your sources and the `tenx_in` sink, so the bypass is structural, not configured. - ๐Ÿ“ค **socket sink** โ†’ Log10x, Vector forwards the enriched event to the Log10x sidecar over TCP `:9000` (or a Unix socket on Linux/macOS) as newline-delimited JSON. - โšก **10x Engine**, The Receiver app applies rate/policy-based filtering and optionally compacts events for volume reduction. - ๐Ÿ“ฅ **fluent source**, Processed events come back to Vector on `:9001` over the Fluent Forward protocol. Only your destination sinks consume `tenx_out`; no transforms sit between them, so enrichment never re-fires. - ๐Ÿ“ค **Destinations**, Vector's destination sinks (`elasticsearch`, `splunk_hec`, `kafka`, `aws_s3`, โ€ฆ) consume `tenx_out` and ship to the real destinations. ### What an event looks like on the way back The record structure of the original Vector event is preserved end-to-end, every field comes back to your destination sinks with the same name and same position. What changes depends on the Receiver app mode: |Mode|Difference vs the event Vector sent in| |---|---| |Receive (default)|None. Same record.| |Receive + `symbolMessageHashField `|Adds one new field with the symbol-pattern hash (a stable identifier for the message pattern, usable as a dedup key, metric dimension, or correlation ID).| |`receiverOptimize true`|The value of the message field (`message` by default, or whatever `vectorInputMessageField` is set to) is replaced with a compact encoded form. A separate `tenx-template` event is emitted with the template needed to decode it. All other fields stay verbatim.| |`receiverOptimize true` + `symbolMessageHashField `|Both of the above.| `symbolMessageHashField` is unset by default, which is what makes the first row true: the receive path hands the record back exactly as it arrived. The pattern hash is still computed and still rides the event inside the engine as `tenx_hash` for metrics and aggregation, it just does not reach the wire. Naming a field opts in, either as a launch argument (`tenx @run/input/forwarder/vector @apps/receiver symbolMessageHashField my_custom_hash`) or as an environment variable of the same name. The `tag` field stamped by Vector's ingest transform (typically from `.source_type`) is carried on the Forward wire as the Fluent tag, and surfaces as the event's `source` inside Log10x, used for rate-based grouping and emitted back to Vector on the return Forward record. Internally, Log10x's Vector input module reads the message text from the field named by `vectorInputMessageField` (default `message`); when the Receiver app is configured with `k8sExtractorName: fluentK8s`, the `kubernetes.*` sub-object is also materialized as enrichment fields for use by message-pattern and rate filtering. ??? tenx-keyfiles "Key Files" | File | Purpose | |------|---------| | [`stream.yaml`](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/forwarder/vector/stream.yaml) | Vector socket input + Fluent Forward output stream definitions | | [`conf/tenx-sidecar.yaml`](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/forwarder/vector/conf/tenx-sidecar.yaml) | Reference Vector config showing the ingest sink + egress source with no return-path transforms | ## Quickstart **1. Run Log10x:** ```bash tenx @run/input/forwarder/vector @apps/receiver ``` **2. Wire up your Vector config**, start from the sidecar recipe and add your real sources + destinations: ```yaml title="vector.yaml" sources: app_logs: type: file include: [/var/log/app.log] read_from: end # Receive processed events back from Log10x tenx_out: type: fluent mode: tcp address: 127.0.0.1:9001 transforms: # Enrichment runs here exactly once, the return path skips this block. ingest: type: remap inputs: [app_logs] source: | .cluster = get_env_var("CLUSTER_NAME") ?? "unset" .tag = .source_type sinks: # Hand off to the Log10x sidecar tenx_in: type: socket inputs: [ingest] mode: tcp address: 127.0.0.1:9000 encoding: { codec: json } framing: { method: newline_delimited } # Destinations consume tenx_out (not the raw sources or `ingest`) so # enrichment never re-fires on the return path. destinations: type: console inputs: [tenx_out] encoding: { codec: json } ``` For Splunk integration see the [10x for Splunk](https://doc.log10x.com/apps/receiver/compact/splunk/) documentation. For Kubernetes deployment via the official Vector Helm chart see the [Helm sidecar overlay](https://doc.log10x.com/apps/receiver/deploy/#vector). ## :material-wrench-outline: Config Files To configure the Vector module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [vector/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/forwarder/vector/config.yaml "vector/config.yaml"){target="\_blank"}.
Edit Online
Edit config.yaml Locally
```yaml # ๐Ÿ”ŸโŽ 'run' Vector receiver configuration # # Every endpoint setting below reads an environment variable first and falls # back to the shipped default, so a deployment can move a port, a host or a # socket path without editing this file. The variable name is the launch # option name in upper snake case with a TENX_ prefix, for example the # `vectorInputPort` option reads TENX_VECTOR_INPUT_PORT. # # To learn more see https://doc.log10x.com/run/input/forwarder/vector/ tenx: run # =============================== Dependencies ================================ include: - run/input/forwarder/config.yaml - run/modules/input/forwarder/vector # =============================== Vector Options ============================== vector: # ----------------------------- Input Options ----------------------------- input: # 'port' specifies the TCP port to listen on for events from Vector's # `socket` sink (`mode: tcp`, `encoding.codec: json`, # `framing.method: newline_delimited`). # Env var: TENX_VECTOR_INPUT_PORT port: $=TenXEnv.get("TENX_VECTOR_INPUT_PORT", 9000) # 'path' switches the input to a Unix domain socket instead of TCP. # When set, takes precedence over 'port'. Empty means TCP. # Env var: TENX_VECTOR_INPUT_PATH path: $=TenXEnv.get("TENX_VECTOR_INPUT_PATH", "") # 'messageField' is the name of the JSON field in each Vector record # carrying the log line text. messageField: message # ----------------------------- Output Options ---------------------------- output: # 'host' specifies the TCP host of Vector's `fluent` source receiving # processed events from Log10x. # Env var: TENX_VECTOR_OUTPUT_HOST host: $=TenXEnv.get("TENX_VECTOR_OUTPUT_HOST", "127.0.0.1") # 'port' specifies the TCP port of Vector's `fluent` source. MUST differ # from input.port or the two would collide on the same listener. # Env var: TENX_VECTOR_OUTPUT_PORT port: $=TenXEnv.get("TENX_VECTOR_OUTPUT_PORT", 9001) # 'path' switches the output to a Unix domain socket instead of TCP. # When set, takes precedence over 'host'/'port'. Empty means TCP. # Env var: TENX_VECTOR_OUTPUT_PATH path: $=TenXEnv.get("TENX_VECTOR_OUTPUT_PATH", "") # 'encodeType' is the on-wire format for the Forward record. # - 'delimited': each top-level field of the rendered record becomes its # own Forward record field, preserves the original record's structure. # - 'json': the whole record is wrapped as one stringified field. encodeType: delimited ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") the Vector: |Name|Description|Category| |---|---|---| |[vectorInputPort](#vectorinputport "TCP port to listen on for events from Vector's socket sink")|TCP port to listen on for events from Vector's socket sink|Input| |[vectorInputPath](#vectorinputpath "Unix domain socket path to listen on (Linux/macOS)")|Unix domain socket path to listen on (Linux/macOS)|Input| |[vectorInputMessageField](#vectorinputmessagefield "name of the JSON field carrying the original log line")|Name of the JSON field carrying the original log line|Input| |[vectorOutputHost](#vectoroutputhost "TCP host of the Vector fluent source receiving processed events")|TCP host of the Vector fluent source receiving processed events|Output| |[vectorOutputPort](#vectoroutputport "TCP port of the Vector fluent source receiving processed events")|TCP port of the Vector fluent source receiving processed events|Output| |[vectorOutputPath](#vectoroutputpath "Unix domain socket path of the Vector fluent source (Linux/macOS)")|Unix domain socket path of the Vector fluent source (Linux/macOS)|Output| |[vectorOutputEncodeType](#vectoroutputencodetype "output format when outputFields are set. Possible values: [json, delimited]")|Output format when outputFields are set. Possible values: \[json, delimited\]|Output| ### Input #### :material-menu-right-outline:**`vectorInputPort`** TCP port to listen on for events from Vector's socket sink. |Type|Default|Category| |---|---|---| |String|9000|Input| TCP port where Log10x listens for events sent by Vector's `socket` sink (`mode: tcp`, `encoding.codec: json`, `framing.method: newline_delimited`). Match this against the `address` in your Vector sink. #### :material-menu-right-outline:**`vectorInputPath`** Unix domain socket path to listen on (Linux/macOS). |Type|Default|Category| |---|---|---| |String|""|Input| When set, Log10x listens on a Unix domain socket instead of TCP, useful on Linux/macOS to avoid TCP overhead. Configure Vector's `socket` sink with `mode: unix` and a matching `path`. Ignored on Windows. #### :material-menu-right-outline:**`vectorInputMessageField`** Name of the JSON field carrying the original log line. |Type|Default|Category| |---|---|---| |String|message|Input| Name of the field in each Vector record that contains the actual log message text. The default (`message`) matches Vector's convention for the `file`, `kubernetes_logs`, `journald`, and `socket` sources. Set to e.g. `log` if your sources emit the log line under a different key. ### Output #### :material-menu-right-outline:**`vectorOutputHost`** TCP host of the Vector fluent source receiving processed events. |Type|Default|Category| |---|---|---| |String|127.0.0.1|Output| Hostname or IP where Vector's `fluent` source is listening for processed events from Log10x. Pairs with `vectorOutputPort`. #### :material-menu-right-outline:**`vectorOutputPort`** TCP port of the Vector fluent source receiving processed events. |Type|Default|Category| |---|---|---| |String|9001|Output| TCP port where Vector's `fluent` source is listening for processed events from Log10x. MUST differ from the port Log10x's own socket input listens on (default `9000`) or the two would collide on the same listener. #### :material-menu-right-outline:**`vectorOutputPath`** Unix domain socket path of the Vector fluent source (Linux/macOS). |Type|Default|Category| |---|---|---| |String|""|Output| Optional Unix domain socket path for Vector's `fluent` source. When set, takes precedence over `vectorOutputHost` / `vectorOutputPort`. Linux/macOS only, ignored on Windows. #### :material-menu-right-outline:**`vectorOutputEncodeType`** Output format when outputFields are set. Possible values: \[json, delimited\]. |Type|Default|Category| |---|---|---| |String|delimited|Output| Specifies how the combined output (main event field plus outputFields) is encoded when writing back to Vector. Possible values: - **json**: formats all fields as a JSON object - **delimited**: formats field values separated by the output delimiter Only takes effect when vectorOutputFields is set.
:material-github: This module is defined in [vector/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/forwarder/vector/module.yaml "vector/module.yaml"){target="\_blank"}.