--- title: "Reporter" description: "report on costly log/trace events _before_ they ship to output" source: "https://github.com/log-10x/modules/tree/main/pipelines/reporter/app.yaml" icon: "material/play-circle-outline" --- The Reporter app is the **read-only insights arm** of the 10x pipeline. It pinpoints high-cost app/infra events **before** forwarders ship them to log analyzers, enabling storage and licensing **cost optimization**. **Deployment**: a DaemonSet alongside your forwarder, *not* a sidecar injected into it, and *not* a cloud app polling your SIEM's REST API. The reporter tails the same event stream your forwarder sees (pre-SIEM), publishes cost insight metrics, and fails independently: if the reporter goes down, your logs continue flowing to the SIEM uninterrupted. Not in the critical log path. No mutation of existing fluent-bit / fluentd / otel-collector configs. Similar deployment pattern to `datadog-agent` or `splunk-otel-collector`. The engine never moves or stores log data. Its only write is per-pattern metrics, sent to a time-series backend of the customer's choice (Prometheus, Datadog, Elastic, CloudWatch, and others). With no backend configured, metrics are collected locally and sent nowhere; fully air-gapped operation is supported. Events never leave the customer's path. Log10x is normally driven by an AI agent (Claude, or a model the customer brings) through the log10x MCP server, which installs, configures, and queries via MCP tools: deploy via `log10x_advise_install`, verify via `log10x_doctor`. To trial without deploying anything, run `log10x_poc_from_local` (local sample files) or `log10x_poc_from_siem` (a SIEM sample). The manual steps below are the same operations without an agent. ## :material-clipboard-play-outline: Setup Guide Follow the steps below. Steps that require customization link to the relevant [Config Files](#config-files) section where you can edit on github.dev or locally. ???+ tenx-bootstrap "Step 1: Install" Install the engine on the same machine as your log forwarder. The script installs the [`Runtime`](https://doc.log10x.com/engine/flavors/#runtime) native binary; the DEB and RPM packages carry either the [`Compiler`](https://doc.log10x.com/engine/flavors/#compiler) or [`Runtime (JVM)`](https://doc.log10x.com/engine/flavors/#runtime-jvm). Windows has no native build, so the PowerShell script installs `Runtime (JVM)`: - :simple-linux: [Single line script](https://doc.log10x.com/install/singleline/ "curl https://raw.githubusercontent.com/log-10x/pipeline-releases/main/install.sh | bash -s -- --flavor runtime"){data-copy="curl https://raw.githubusercontent.com/log-10x/pipeline-releases/main/install.sh | bash -s -- --flavor runtime"} | [DEB](https://doc.log10x.com/install/linux/#ubuntu-debian) | [RPM](https://doc.log10x.com/install/linux/#red-hat-centos-7) - :simple-macos: [Homebrew](https://doc.log10x.com/install/macos/ "brew install log-10x/tap/log10x"){data-copy="brew install log-10x/tap/log10x"} - :material-microsoft-windows: [PowerShell script](https://doc.log10x.com/install/win/) - :simple-docker: [Docker image](https://doc.log10x.com/install/docker/) ??? tenx-config "Step 2: Set Environment Variables" Set these environment variables before running. See [path configuration](https://doc.log10x.com/install/paths/) for details. | Variable | Description | |----------|-------------| | `TENX_MODULES` | Path to your [modules directory](https://doc.log10x.com/install/paths/#modules) | | `TENX_CONFIG` | Path to your [configuration directory](https://doc.log10x.com/install/paths/#config) | | `TENX_LICENSE_KEY` | Your Log10x license JWT ([licensing](https://doc.log10x.com/manage/license/)); unset, the engine runs on the built-in evaluation license | ```bash export TENX_MODULES=/etc/tenx/modules export TENX_CONFIG=/etc/tenx/config export TENX_LICENSE_KEY="$(cat license.jwt)" ``` For production deployments, mount the license as a file and set `TENX_LICENSE_FILE` instead. ??? tenx-forwarderinputs "Step 3: Configure Your Forwarder" === ":simple-fluentd: Fluentd" **Step 1**: Include the 10x reporter configuration: ```toml title="my-fluentd.conf" @include "#{ENV['TENX_MODULES']}/pipelines/run/modules/input/forwarder/fluentd/conf/tenx-report.conf" ``` **Step 2**: Use a `copy` directive to report on events while still sending them to output: ```xml title="my-fluentd.conf" @type tail path /path/to/file.log tag my_logs @type copy # Write to output (e.g. Elastic, Splunk) @type stdout # Report to 10x @type relabel @label @TENX ``` !!! note For large files or volume surges, set [XMX](https://www.baeldung.com/jvm-parameters#explicit-heap-memory---xms-and-xmx-options) or use [read_bytes_limit_per_second](https://docs.fluentd.org/input/tail#read_bytes_limit_per_second). === ":simple-fluentbit: Fluent-bit" **Step 1**: Include the 10x reporter configuration: ```toml title="my-fluent-bit.conf" # Nix @INCLUDE /etc/tenx/config/pipelines/run/modules/input/forwarder/fluentbit/conf/tenx-report.conf # Windows # @INCLUDE c:/program files/tenx-edge/config/pipelines/run/modules/input/forwarder/fluentbit/conf/tenx-report.conf ``` **Step 2**: The Lua filter catches all events by default. To report on a subset, update the `Match` field: ```toml title="tenx-report.conf" [FILTER] Name Lua Match * script ${TENX_MODULES}/${tenx_lua} call tenx_process ``` === ":simple-beats: Filebeat" Add the reporter processor script to your configuration: ```yaml title="my-filebeat.yml" filebeat.inputs: - type: filestream id: my-filestream-id paths: - /path/to/log processors: - script: lang: javascript file: ${TENX_MODULES}/pipelines/run/modules/input/forwarder/filebeat/script/tenx-report.js ``` === ":simple-logstash: Logstash" Add a `pipe` output to delegate events to the reporter: ```ruby title="my-logstash.conf" output { pipe { id => "tenx_pipeline" ttl => -1 # Nix command => "${TENX_BIN:/opt/tenx-edge/bin/tenx} run @run/input/forwarder/logstash/report/config.yaml @apps/reporter" # Windows # command => "${TENX_BIN:'c:\\program files\\tenx\\tenx-edge.exe'} run @run/input/forwarder/logstash/report/config.yaml @apps/reporter" } } ``` === ":simple-opentelemetry: OTel Collector" !!! note "Requires otel-collector-contrib" The OpenTelemetry Collector integration requires the **contrib** distribution for `syslogexporter` support. **Step 1**: Copy the OTel Collector configuration: ```bash cp $TENX_MODULES/pipelines/run/modules/input/forwarder/otel-collector/report/tenxNix.yaml /etc/otelcol-contrib/ ``` **Step 2**: Update the configuration to match your log sources: ```yaml title="report/tenxNix.yaml" receivers: filelog: include: - /var/log/**/*.log # Customize paths start_at: end ``` **Step 3**: Configure your final exporters (Elastic, Splunk, etc.) in the same pipeline: ```yaml title="report/tenxNix.yaml" service: pipelines: logs: receivers: [filelog, otlp] processors: [memory_limiter, batch] exporters: [syslog/tenx, elasticsearch] # Add your exporters ``` Reporter mode is **read-only**. Logs flow to your final destinations unchanged; the engine never moves or stores log data. Its only write is per-pattern metrics, sent to a time-series backend of your choice (Prometheus, Datadog, Elastic, CloudWatch, and others). === ":simple-splunk: Splunk UF" !!! note "File Relay Pattern" This integration uses a **file relay pattern**: Fluent Bit + 10x reads from Folder A, reports on events, and writes to Folder B. Splunk UF monitors Folder B and handles forwarding to Splunk indexers. Events are unchanged - 10x only observes and reports metrics. **Step 1**: Set up folder paths: ```bash export FOLDER_A=/var/log/app # App writes here export FOLDER_B=/var/log/processed # UF reads from here mkdir -p ${FOLDER_B} ``` **Step 2**: Configure Fluent Bit to read from Folder A, report, and write to Folder B: ```toml title="fluent-bit-splunk.conf" [SERVICE] Flush 1 Log_Level info [INPUT] Name tail Path ${FOLDER_A}/*.log Tag app.logs # Include 10x reporter - sends events to 10x subprocess @INCLUDE ${TENX_MODULES}/pipelines/run/modules/input/forwarder/fluentbit/conf/tenx-report.conf # Include Unix socket - receives processed events back from 10x @INCLUDE ${TENX_MODULES}/pipelines/run/modules/input/forwarder/fluentbit/conf/tenx-unix.conf # Write events (unchanged) to Folder B [OUTPUT] Name file Match * Path ${FOLDER_B} Format plain ``` **Step 3**: Configure Splunk UF to monitor Folder B: ```ini title="inputs.conf" [monitor://${FOLDER_B}] index = main sourcetype = app_logs ``` This keeps Splunk UF as the forwarder (handling buffering, retries, timeouts) while 10x reports on events inline. See the [Splunk UF module](https://doc.log10x.com/run/input/forwarder/splunkUF/) for details. === ":simple-datadog: Datadog Agent" !!! note "File Relay Pattern" This integration uses a **file relay pattern**: Fluent Bit + 10x reads from Folder A, reports on events, and writes to Folder B. Datadog Agent monitors Folder B and handles forwarding to Datadog. Events are unchanged - 10x only observes and reports metrics. **Step 1**: Set up folder paths: ```bash export FOLDER_A=/var/log/app # App writes here export FOLDER_B=/var/log/processed # DD Agent reads from here mkdir -p ${FOLDER_B} ``` **Step 2**: Configure Fluent Bit to read from Folder A, report, and write to Folder B: ```toml title="fluent-bit-datadog.conf" [SERVICE] Flush 1 Log_Level info [INPUT] Name tail Path ${FOLDER_A}/*.log Tag app.logs # Include 10x reporter - sends events to 10x subprocess @INCLUDE ${TENX_MODULES}/pipelines/run/modules/input/forwarder/fluentbit/conf/tenx-report.conf # Include Unix socket - receives processed events back from 10x @INCLUDE ${TENX_MODULES}/pipelines/run/modules/input/forwarder/fluentbit/conf/tenx-unix.conf # Write events (unchanged) to Folder B [OUTPUT] Name file Match * Path ${FOLDER_B} Format plain ``` **Step 3**: Configure Datadog Agent to monitor Folder B: ```yaml title="conf.d/tenx.d/conf.yaml" logs: - type: file path: ${FOLDER_B}/*.log service: myapp source: myapp ``` This keeps Datadog Agent as the forwarder (handling buffering, retries, metadata enrichment) while 10x reports on events inline. See the [Datadog Agent module](https://doc.log10x.com/run/input/forwarder/datadogAgent/) for details. === ":material-test-tube: Test (no forwarder)" Test the reporter without setting up a forwarder using the [Dev CLI](https://doc.log10x.com/apps/dev/). The dev app uses the [file input module](https://doc.log10x.com/run/input/file/) to read sample log files and generates cost analytics reports, allowing you to verify reporting behavior before integrating with your forwarder. **No forwarder configuration required** - provide sample log files to the file input module and skip to [Step 7](#__tabbed_6_5) to run the test. ??? tenx-symbols "Step 4: Symbol Library (optional)" Load custom [Symbol library](https://doc.log10x.com/compile/) files to transform events into typed TenXObjects. Place symbol files in the `symbolPaths` folders specified in the [symbol config](#symbols). ??? tenx-initializers "Step 5: Enrichments (optional)" Enrichments add context (severity, HTTP codes) for richer aggregation reports. **To enable:** 1. In the [app config](#mainConfig), uncomment the desired enrichment `include` entries 2. In the [initializers](#initializers) section below, configure each enrichment's settings ??? tenx-metricoutputs "Step 6: Metrics Output (optional)" Publish [aggregated](https://doc.log10x.com/run/aggregate/) metrics to time-series databases (Prometheus, Datadog). **To enable:** 1. In the [app config](#mainConfig), uncomment the desired metric output `include` entry 2. In the [metric outputs](#metricOutputs) section below, configure connection details ??? tenx-mainconfig "Step 7: Run" Start your forwarder with the updated configuration: === ":simple-fluentd: Fluentd" ```console $ fluentd -c my-fluentd.conf ``` === ":simple-fluentbit: Fluent-bit" ```console $ fluent-bit -c my-fluent-bit.conf ``` === ":simple-beats: Filebeat" ```console title="Nix/OSX" $ filebeat -c my-filebeat.yml -e 2>&1 | /opt/tenx-edge/bin/tenx run @run/input/forwarder/filebeat @apps/reporter ``` ```console title="Windows" $ filebeat -c my-filebeat.yml -e 2>&1 | "c:\program files\tenx-edge\tenx" run @run/input/forwarder/filebeat @apps/reporter ``` === ":simple-logstash: Logstash" ```console $ logstash -f my-logstash.conf ``` === ":simple-opentelemetry: OTel Collector" **Step 1**: Start Log10x Reporter first: ```console $ tenx run @run/input/forwarder/otel-collector/report @apps/reporter ``` **Step 2**: Start OTel Collector with the 10x configuration: ```console $ otelcol-contrib --config=/etc/otelcol-contrib/report/tenxNix.yaml ``` === ":simple-splunk: Splunk UF" **Step 1**: Start Fluent Bit with the 10x reporter: ```console $ fluent-bit -c fluent-bit-splunk.conf ``` **Step 2**: Start (or restart) Splunk UF to pick up Folder B: ```console $ splunk restart ``` Fluent Bit + 10x will read from Folder A, report on events (without modification), and write to Folder B. Splunk UF monitors Folder B and forwards to indexers. === ":simple-datadog: Datadog Agent" **Step 1**: Start Fluent Bit with the 10x reporter: ```console $ fluent-bit -c fluent-bit-datadog.conf ``` **Step 2**: Start (or restart) the Datadog Agent to pick up Folder B: ```console $ sudo systemctl restart datadog-agent ``` Fluent Bit + 10x will read from Folder A, report on events (without modification), and write to Folder B. Datadog Agent monitors Folder B and forwards to Datadog. === ":material-test-tube: Test (no forwarder)" Use the [Dev CLI](https://doc.log10x.com/apps/dev/) to test event processing with sample log files. **Step 1**: Provide sample log files to the [file input module](https://doc.log10x.com/run/input/file/): ```console $ cp /path/to/sample.log $TENX_CONFIG/data/sample/input/ ``` **Step 2**: Run the dev app: ```console $ tenx run @apps/dev ``` The dev app reads events from `data/sample/input/*.log` via the file input module and generates cost analytics reports without needing a forwarder. **View results in the dashboard** after running to verify reporting works correctly. ??? tenx-checklist "Step 8: Verify" Verify no errors appear in the [log file](https://doc.log10x.com/manage/logging/#log-file-location). For debugging techniques including enabling verbose logging, see [Engine Logging](https://doc.log10x.com/manage/logging/). With an agent, `log10x_doctor` confirms reporter detection and metric freshness, and `log10x_baseline` captures the starting per-pattern volume and cost. **View results in the dashboard:** Once running, view your cost analytics in the [Reporter Dashboard](https://doc.log10x.com/roi-analytics/#edge-reporter). ??? tenx-delete "Step 9: Teardown" Nothing runs in the background. Uninstall removes only what was installed. === ":simple-macos: Homebrew" ```bash brew uninstall log10x ``` === ":simple-linux: Linux" ```bash sudo rm -rf /opt/tenx-edge /etc/tenx /etc/profile.d/tenx-edge.sh ``` If installed via DEB: `sudo apt-get remove tenx-edge` If installed via RPM: `sudo yum remove tenx-edge` === ":material-microsoft-windows: Windows" Uninstall from **Settings > Apps > Installed apps**. === ":simple-docker: Docker" ```bash docker rmi log10x/pipeline-10x:1.1.39 ``` ## :material-wrench-outline: Config Files To configure the Reporter app, [:material-cog: Edit](https://doc.log10x.com/config/app/#main-config "Learn how to edit app and module configurations") these files: ??? tenx-mainconfig "Main Config" ### Main Config {.hidden-heading} The main config file loads the Reporter app's required modules. Uncomment selected modules at: [reporter/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/apps/reporter/config.yaml "reporter/config.yaml"){target="\_blank"}.
Edit Online
Edit Reporter Config Locally
```yaml # # 🔟❎ Reporter app main config # Loads the required modules for a Reporter instance. # To learn more see https://doc.log10x.com/apps/reporter # ============================ Bootstrap Runtime ============================== # Initialize the runtime. To learn more see https://doc.log10x.com/run/bootstrap tenx: run # Where the engine validates its licence (/api/v1/agent/whoami) and ships # its metrics (/api/v1/write). One endpoint drives BOTH: repointing it for # licensing also repoints metrics delivery. Unset or empty, the engine has # no address to call: metrics stay local and the licence verifies offline. backendEndpoint: $=TenXEnv.get("TENX_BACKEND_ENDPOINT", "") runtimeName: $=TenXEnv.get("TENX_RUNTIME_NAME", "myReporter") # =========================== Load App Modules ================================ # Uncomment and edit selected config.yaml files (e.g., run/input/forwarder/fluentbit) include: # ----------------------------- App settings ---------------------------------- # Load app settings: - reporter # Load gitops settings for pulling additional assets from Github. # Edit the gitops/config.yaml file with specific config # # For additional info see - https://doc.log10x.com/config/github - gitops # ---------------------------- Forwarder Input -------------------------------- # Read events via Fluent Forward protocol (port 24224 or Unix socket). # The reporter runs as a sidecar container receiving events from any forwarder. # # For additional info, see - # https://doc.log10x.com/apps/reporter/run # https://doc.log10x.com/apps/reporter/deploy # - run/input/forward # https://doc.log10x.com/run/input/forward # --------------------------- Enrich TenXObjects ------------------------------ # Uncomment and edit config files to enrich TenXObjects with calculated fields: - run/initialize/inputField # https://doc.log10x.com/run/initialize/#inputfield - run/initialize/level # https://doc.log10x.com/run/initialize/level - run/initialize/group # https://doc.log10x.com/run/initialize/group - run/initialize/message # https://doc.log10x.com/run/initialize/message - run/initialize/httpCode # https://doc.log10x.com/run/initialize/httpCode - run/initialize/lookup # https://doc.log10x.com/run/initialize/lookup - run/initialize/custom # https://doc.log10x.com/run/transform/script/object - run/initialize/k8s # https://doc.log10x.com/run/initialize/k8s # - run/initialize/geoIP # https://doc.log10x.com/run/initialize/geoIP # -------------------------- Select Metric Outputs ---------------------------- # Uncomment ONE to publish aggregated TenXSummary instances, or pass it at # launch (e.g. "@run/output/metric/prometheus/remote-write"). Metrics go to a # backend you own; the install wizard asks which one and wires it for you. # - run/output/metric/prometheus/scrape # https://doc.log10x.com/run/output/metric/prometheus/scrape # - run/output/metric/prometheus/push-gateway # https://doc.log10x.com/run/output/metric/prometheus/push-gateway # - run/output/metric/prometheus/remote-write # https://doc.log10x.com/run/output/metric/prometheus/remote-write # - run/output/metric/cloudwatch # https://doc.log10x.com/run/output/metric/cloudwatch # - run/output/metric/datadog # https://doc.log10x.com/run/output/metric/datadog # - run/output/metric/elastic # https://doc.log10x.com/run/output/metric/elastic # - run/output/metric/log # https://doc.log10x.com/run/output/metric/log ``` ??? tenx-initializers "Initializers" ### Initializers {.hidden-heading} Activate an [Object initializer](/run/initialize "Enrich TenXObjects with additional context") to enrich TenXObjects with additional context. === ":material-origin: Message" #### message {.hidden-heading} Configure the [Symbol Message Calculator](/run/initialize/message "Enrich TenXObjects with logical message symbol sequence and origin values") to enrich TenXObjects with logical message symbol sequence and origin values. Below is the default configuration from: [message/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/initialize/message/config.yaml "message/config.yaml"){target="\_blank"}.
Edit Online
Edit Symbol Message Calculator Config Locally
```yaml # 🔟❎ 'run' symbol lookup configuration # Configure a symbol origin lookup to enrich TenXObjects # To learn more see https://doc.log10x.com/run/initialize/message/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/initialize/message # ============================== Symbol Options =============================== symbol: # 'types' specifies the types of symbols that participate in the origin search. The listed # contexts are evaluated in a single pass, so list order acts as a filter rather than a precedence chain. # Supported values (case insensitive), PACKAGE, CLASS, METHOD, LOG, ENUM, CONST, TEXT, EXEC, ANY # To learn more see https://doc.log10x.com/run/transform/symbol/#contexts contexts: log,exec # 'messageField' specifies the field name to assign the message pattern to target TenXObjects: the symbol # tokens of the origin ranked highest by coverage, meaning the count of distinct library-known tokens it # explains. Span length is the last tiebreak. The origin file name itself goes to 'originField'. messageField: message_pattern # 'messageHashField' specifies the field name to assign a stable, URL-safe pattern hash (xxHash64 of the messageField value, base64url, 11 chars). Safe as a SIEM query term or forwarder filter key. # # Left unset on purpose. The hash enrichment is always computed and carried on the TenXObject under # its internal name 'tenx_hash', so metric dimensions, the aggregation key and the MCP artifacts are # unaffected. Setting this key is what names the field on the wire, and on the forwarder receive path # that is also what puts it on the record returned to the forwarder; leave it unset and the forwarder # gets its record back byte-for-byte. Uncomment and name a field to opt in: # # messageHashField: tenx_hash # 'maxLen' specifies the max char len of the output messageField maxLen: 120 # ============================== Message Options ============================== # 'negators' specifies patterns that mark a log line as boilerplate/continuation (stack trace frames, # indented detail, banners) so the message-template skips its own message-pattern compute for them. # The check fires only for templates that don't already pass the (groupSize > 1) || groupHead arms; # i.e., for single-event templates with no head signal. A non-negator line in that case (e.g., bare # "Hello World", "Heartbeat received from node-7") still gets a pattern; a negator-matching orphan # (e.g., a lone "\tat com.foo.Bar(...)") does not. message: negators: - " " # any indented line (stack frames, indented app detail, multi-line JSON, banners) - "\t" # real tab byte, for msgpack/forward inputs that preserve raw whitespace - "\\t" # literal backslash+t, for file/JSON inputs that keep escape sequences raw - "Caused by: " # Java chained exception (column 0) - "Suppressed: " # Java suppressed exception (column 0) - "--- End of " # .NET inner exception marker (column 0) - "created by " # Go goroutine creator (column 0) - "#0 " # PHP stack frame depth 0 - "#1 " # PHP stack frame depth 1 - "#2 " # PHP stack frame depth 2 - "#3 " # PHP stack frame depth 3 - "#4 " # PHP stack frame depth 4 - "#5 " # PHP stack frame depth 5 ``` === ":material-information-outline: Level" #### level {.hidden-heading} Configure the [Level Classifier](/run/initialize/level "Classify TenXObjects with a severity level") to classify TenXObjects with a severity level. Below is the default configuration from: [level/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/initialize/level/config.yaml "level/config.yaml"){target="\_blank"}.
Edit Online
Edit Level Classifier Config Locally
```yaml # 🔟❎ 'run' level classifier configuration # Configuration level classifier to enrich TenXObjects # To learn more see https://doc.log10x.com/run/initialize/level/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/initialize/level # =============================== Level Options =============================== level: # 'field' specify the field name to assign with the inferred severity level field: severity_level # 'terms' specify severity level classification, start-of-line matching # Used to identify the severity level (TRACE, DEBUG, INFO, WARN, ERROR, CRITICAL) of a log line, typically marking # the start of a log event (head). Sorted by likelihood and uniqueness. Used in a Trie for O(m) matching. terms: - 'trace=TRACE' # Common in JavaScript (Winston) or custom logs; e.g., "trace Variable x = 5" - 'Trace=TRACE' # Mixed case; e.g., "Trace Variable x = 5" - 'TRACE=TRACE' # Uppercase, less common; e.g., "TRACE Message" - 'debug=DEBUG' # Common in JavaScript (Winston) or custom logs; e.g., "debug Entering method foo" - 'Debug=DEBUG' # Mixed case; e.g., "Debug Entering method foo" - 'DEBUG=DEBUG' # Uppercase; e.g., "DEBUG Message" - 'DBG=DEBUG' # Abbreviation, less common; e.g., "DBG Entering method foo" - 'info=INFO' # Common in JavaScript (Winston) or custom logs; e.g., "info Application started" - 'Info=INFO' # Mixed case; e.g., "Info Application started" - 'INFO=INFO' # Uppercase; e.g., "INFO Message" - 'notice=INFO' # Common in custom logs or lowercase syslog; e.g., "notice Configuration updated" - 'Notice=INFO' # Mixed case; e.g., "Notice Configuration updated" - 'NOTICE=INFO' # Uppercase, syslog; e.g., "NOTICE Message" - 'warn=WARN' # Common in JavaScript (Winston) or custom logs; e.g., "warn Low disk space" - 'Warn=WARN' # Mixed case; e.g., "Warn Low disk space" - 'WARN=WARN' # Uppercase; e.g., "WARN Message" - 'warning=WARN' # Common in custom logs or verbose frameworks; e.g., "warning Low disk space" - 'Warning=WARN' # Mixed case; e.g., "Warning Low disk space" - 'WARNING=WARN' # Uppercase; e.g., "WARNING Message" - 'error=ERROR' # Common in JavaScript (Winston) or custom logs; e.g., "error Failed to connect" - 'Error=ERROR' # Mixed case; e.g., "Error Failed to connect" - 'ERROR=ERROR' # Uppercase; e.g., "ERROR Message" - 'err=ERROR' # Lowercase abbreviation; e.g., "err Failed to connect" - 'ERR=ERROR' # Uppercase abbreviation; e.g., "ERR Message" - 'TypeError=ERROR' # JavaScript, stack trace; e.g., "TypeError: undefined is not a function" - 'ReferenceError=ERROR' # JavaScript, stack trace; e.g., "ReferenceError: x is not defined" - 'ValueError=ERROR' # Python, stack trace; e.g., "ValueError: invalid literal" - 'PHP_Warning=ERROR' # PHP, warning message; e.g., "PHP Warning: Undefined variable" - 'fatal=CRITICAL' # Common in custom logs; e.g., "fatal System crash" - 'Fatal=CRITICAL' # Mixed case; e.g., "Fatal System crash" - 'FATAL=CRITICAL' # Uppercase; e.g., "FATAL Message" - 'critical=CRITICAL' # Common in custom logs; e.g., "critical System crash" - 'Critical=CRITICAL' # Mixed case; e.g., "Critical System crash" - 'CRITICAL=CRITICAL' # Uppercase; e.g., "CRITICAL Message" - 'crit=CRITICAL' # Lowercase abbreviation, syslog; e.g., "crit System crash" - 'CRIT=CRITICAL' # Uppercase abbreviation, syslog; e.g., "CRIT Message" - 'alert=CRITICAL' # Common in custom logs or syslog; e.g., "alert High CPU usage" - 'Alert=CRITICAL' # Mixed case; e.g., "Alert High CPU usage" - 'ALERT=CRITICAL' # Uppercase, syslog; e.g., "ALERT Message" - 'emerg=CRITICAL' # Common in custom logs or lowercase syslog; e.g., "emerg System unusable" - 'Emerg=CRITICAL' # Mixed case; e.g., "Emerg System unusable" - 'EMERG=CRITICAL' # Uppercase, syslog; e.g., "EMERG Message" - 'panic=CRITICAL' # Go panic, first line; e.g., "panic: runtime error: index out of range" - 'fatal_error=CRITICAL' # Go fatal error, first line; e.g., "fatal error: all goroutines are asleep" - 'thread_main_panicked_at=CRITICAL' # Rust panic, first line; e.g., "thread 'main' panicked at 'explicit panic'" - 'Segmentation_fault=CRITICAL' # C/C++ memory violation, first line; e.g., "Segmentation fault (core dumped)" - 'Aborted=CRITICAL' # C/C++ termination, first line; e.g., "Aborted (core dumped)" - 'Assertion_failed=CRITICAL' # C/C++ assertion failure, first line; e.g., "Assertion failed: x > 0" - 'PHP_Fatal_error=CRITICAL' # PHP fatal error, first line; e.g., "PHP Fatal error: Out of memory" - 'Traceback_most_recent_call_last=CRITICAL' # Python stack trace, first line; e.g., "Traceback (most recent call last)" - 'Stack_trace=CRITICAL' # Generic stack trace, first line; e.g., "Stack trace" - 'Error=CRITICAL' # JavaScript, stack trace; e.g., "Error: Something went wrong" - 'Fatal_error=CRITICAL' # PHP, stack trace; e.g., "Fatal error: Uncaught Exception" - 'Unhandled_exception=CRITICAL' # C#/Dart, stack trace; e.g., "Unhandled exception: System.NullReferenceException" - 'Exception_in_thread=CRITICAL' # Java, stack trace; e.g., "Exception in thread 'main' java.lang.RuntimeException" - 'goroutine=CRITICAL' # Go, stack trace; e.g., "goroutine 1 [running]:" - 'failed=ERROR' # ERROR-level entries in application, system, or web server logs # 'timestampPatterns' patterns for LogEvent head classification, start-of-line matching # Used to identify timestamp formats in log lines and infer their implied severity level based on the first character. # Formats are typical of Kubernetes kube-apiserver logs with varying microsecond precision. The first character # ('I', 'W', 'E', 'F') determines the severity: I=INFO, W=WARN, E=ERROR, F=CRITICAL. Sorted by precision. timestampPatterns: - "'I'MMdd HH:mm:ss.S=INFO" # Used in Kubernetes kube-apiserver logs with INFO prefix, 1-digit microsecond precision - "'I'MMdd HH:mm:ss.SS=INFO" # Used in Kubernetes kube-apiserver logs with INFO prefix, 2-digit microsecond precision - "'I'MMdd HH:mm:ss.SSS=INFO" # Used in Kubernetes kube-apiserver logs with INFO prefix, 3-digit microsecond precision - "'I'MMdd HH:mm:ss.SSSS=INFO" # Used in Kubernetes kube-apiserver logs with INFO prefix, 4-digit microsecond precision - "'I'MMdd HH:mm:ss.SSSSS=INFO" # Used in Kubernetes kube-apiserver logs with INFO prefix, 5-digit microsecond precision - "'I'MMdd HH:mm:ss.SSSSSS=INFO" # Used in Kubernetes kube-apiserver logs with INFO prefix, 6-digit microsecond precision - "'W'MMdd HH:mm:ss.S=WARN" # Used in Kubernetes kube-apiserver logs with WARNING prefix, 1-digit microsecond precision - "'W'MMdd HH:mm:ss.SS=WARN" # Used in Kubernetes kube-apiserver logs with WARNING prefix, 2-digit microsecond precision - "'W'MMdd HH:mm:ss.SSS=WARN" # Used in Kubernetes kube-apiserver logs with WARNING prefix, 3-digit microsecond precision - "'W'MMdd HH:mm:ss.SSSS=WARN" # Used in Kubernetes kube-apiserver logs with WARNING prefix, 4-digit microsecond precision - "'W'MMdd HH:mm:ss.SSSSS=WARN" # Used in Kubernetes kube-apiserver logs with WARNING prefix, 5-digit microsecond precision - "'W'MMdd HH:mm:ss.SSSSSS=WARN" # Used in Kubernetes kube-apiserver logs with WARNING prefix, 6-digit microsecond precision - "'E'MMdd HH:mm:ss.S=ERROR" # Used in Kubernetes kube-apiserver logs with ERROR prefix, 1-digit microsecond precision - "'E'MMdd HH:mm:ss.SS=ERROR" # Used in Kubernetes kube-apiserver logs with ERROR prefix, 2-digit microsecond precision - "'E'MMdd HH:mm:ss.SSS=ERROR" # Used in Kubernetes kube-apiserver logs with ERROR prefix, 3-digit microsecond precision - "'E'MMdd HH:mm:ss.SSSS=ERROR" # Used in Kubernetes kube-apiserver logs with ERROR prefix, 4-digit microsecond precision - "'E'MMdd HH:mm:ss.SSSSS=ERROR" # Used in Kubernetes kube-apiserver logs with ERROR prefix, 5-digit microsecond precision - "'E'MMdd HH:mm:ss.SSSSSS=ERROR" # Used in Kubernetes kube-apiserver logs with ERROR prefix, 6-digit microsecond precision - "'F'MMdd HH:mm:ss.S=CRITICAL" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 1-digit microsecond precision - "'F'MMdd HH:mm:ss.SS=CRITICAL" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 2-digit microsecond precision - "'F'MMdd HH:mm:ss.SSS=CRITICAL" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 3-digit microsecond precision - "'F'MMdd HH:mm:ss.SSSS=CRITICAL" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 4-digit microsecond precision - "'F'MMdd HH:mm:ss.SSSSS=CRITICAL" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 5-digit microsecond precision - "'F'MMdd HH:mm:ss.SSSSSS=CRITICAL" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 6-digit microsecond precision ``` === ":material-select-group: Group" #### group {.hidden-heading} Configure the [Group initializer](/run/initialize/group "Combine multi-line events into TenXObject group instances") to combine multi-line events into TenXObject group instances. Below is the default configuration from: [group/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/initialize/group/config.yaml "group/config.yaml"){target="\_blank"}.
Edit Online
Edit Group initializer Config Locally
```yaml # 🔟❎ 'run' event grouping configuration # Group sequences of TenXObjects to filter, aggregate and output as a single logical unit. # To learn more see https://doc.log10x.com/run/transform/group/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/initialize/group # =============================== Group Options =============================== # 'indicators' specifies a list of `value:state` pairs that determine if a log line's start marks a group head (`true`) or child (`false`). group: # 'indicators' specifies a list of strings that, when matched at the start of a log line's text, designate it as a group head. # Unmatched lines default to false, indicating they are group children (continuation-by-default). indicators: - '{' # Indicates a pretty-printed JSON/object log body (e.g., payment '{ resource: {...} }'); the opening brace heads the multi-line object so it groups as one event. Matches the log body only, grouping reads inputField, not the wrapped envelope. - '192.' # Indicates a private IP address, common in web server logs (e.g., "192.168.1.1 - - [...]") - '10.' # Indicates a private IP range, often in Kubernetes or internal network logs (e.g., "10.244.0.125 - - [...]") - '172.' # Indicates a private IP range, typical in enterprise network logs (e.g., "172.16.0.1 - - [...]") - '127.' # Indicates localhost, a common web server log initiator (e.g., "127.0.0.1 - - [...]") - 'GET ' # Indicates an HTTP GET request, marking the start of a web transaction log (e.g., "GET /index.html HTTP/1.1") - 'POST ' # Indicates an HTTP POST request, marking the start of a web transaction log (e.g., "POST /api HTTP/1.1") - 'PUT ' # Indicates an HTTP PUT request, marking the start of a web transaction log (e.g., "PUT /resource HTTP/1.1") - 'DELETE ' # Indicates an HTTP DELETE request, marking the start of a web transaction log (e.g., "DELETE /resource HTTP/1.1") - 'HEAD ' # Indicates an HTTP HEAD request, marking the start of a web transaction log (e.g., "HEAD /index.html HTTP/1.1") - 'OPTIONS ' # Indicates an HTTP OPTIONS request, marking the start of a web transaction log (e.g., "OPTIONS /api HTTP/1.1") - 'HTTP/' # Indicates an HTTP protocol version, marking the start of a web transaction log (e.g., "GET /index.html HTTP/1.1") - 'kernel:' # Indicates a Linux kernel log entry, marking the start of a system event (e.g., "kernel: [0.123456] Device initialized") - 'sshd[' # Indicates an SSH daemon log entry, marking the start of a security event (e.g., "sshd[1234]: Accepted password ...") - 'systemd[' # Indicates a systemd service log entry, marking the start of a system service event (e.g., "systemd[1]: Started service ...") - 'cron[' # Indicates a cron daemon log entry, marking the start of a scheduled task event (e.g., "cron[1234]: Running job ...") - 'syslog:' # Indicates a syslog message, marking the start of a system log event (e.g., "syslog: Message ...") - 'rsyslogd:' # Indicates an rsyslog daemon log entry, marking the start of a logging system event (e.g., "rsyslogd: Log started ...") - 'auditd[' # Indicates an audit daemon log entry, marking the start of a security audit event (e.g., "auditd[1234]: Audit event ...") - 'daemon:' # Indicates a syslog daemon facility log, marking the start of a system service event (e.g., "daemon: Service started ...") - 'user:' # Indicates a syslog user facility log, marking the start of a user-related event (e.g., "user: User logged in ...") - 'local0:' # Indicates a syslog local facility log, marking the start of a custom system event (e.g., "local0: Custom message ...") - 'local1:' # Indicates a syslog local facility log, marking the start of a custom system event (e.g., "local1: Custom message ...") - 'level=' # Indicates a structured log key, marking the start of a key-value log entry (e.g., "level=info msg=Started") - 'msg=' # Indicates a structured log key, marking the start of a message log entry (e.g., "msg=Application started") - 'message=' # Indicates a structured log key, marking the start of a message log entry (e.g., "message=Application started") - 'event=' # Indicates a structured log key, marking the start of an event log entry (e.g., "event=Service startup") - 'thread=' # Indicates a structured log key, marking the start of a thread-specific log entry (e.g., "thread=main Processing ...") - 'Starting ' # Indicates the start of a process initiation log (e.g., "Starting server on port 8080") - 'Stopping ' # Indicates the start of a process termination log (e.g., "Stopping service ...") - 'Running ' # Indicates the start of a process status log (e.g., "Running task ...") - 'Listening ' # Indicates the start of a network service log (e.g., "Listening on port 8080 ...") - 'Connecting ' # Indicates the start of a connection attempt log (e.g., "Connecting to database ...") - 'Connected ' # Indicates the start of a successful connection log (e.g., "Connected to database ...") - 'Disconnected ' # Indicates the start of a disconnection log (e.g., "Disconnected from server ...") - 'Processing ' # Indicates the start of a task processing log (e.g., "Processing request ...") - 'Received ' # Indicates the start of a data reception log (e.g., "Received message ...") - 'Sent ' # Indicates the start of a data transmission log (e.g., "Sent response ...") - 'User ' # Indicates the start of a user action log (e.g., "User logged in ...") - 'Authentication ' # Indicates the start of an authentication log (e.g., "Authentication successful ...") - 'Authorized ' # Indicates the start of an authorization log (e.g., "Authorized user access ...") - 'Failed ' # Indicates the start of a failure log (e.g., "Failed login attempt ...") - 'kubelet:=true' # Indicates a Kubernetes kubelet log entry, marking the start of a node event (e.g., "kubelet: Starting kubelet") - 'pod:=true' # Indicates a Kubernetes pod log entry, marking the start of a pod event (e.g., "pod: Starting container") - 'container:=true' # Indicates a Kubernetes container log entry, marking the start of a container event (e.g., "container: Started") - 'namespace:=true' # Indicates a Kubernetes namespace log entry, marking the start of a namespace event (e.g., "namespace: Created") - 'Traceback ' # Indicates the start of a Python stack trace (e.g., "Traceback (most recent call last):") - 'File "' # Indicates a Python stack trace line (e.g., "File "/script.py", line 10") - 'Trace:' # Indicates a Node.js console trace (e.g., "Trace: Show me") - 'Error:' # Indicates a Node.js error log (e.g., "Error: Something went wrong") - 'Warning:' # Indicates a Node.js warning log (e.g., "Warning: Deprecated method") - 'Exception in thread ' # Indicates a Java exception header (e.g., "Exception in thread 'main'") - 'goroutine ' # Indicates a Go goroutine stack trace (e.g., "goroutine 1 [running]:") - 'panic:' # Indicates a Go panic (e.g., "panic: runtime error") - 'thread ' # Indicates a Rust thread panic (e.g., "thread 'main' panicked at") - 'stack backtrace:' # Indicates a Rust backtrace header (e.g., "stack backtrace:") - 'PHP Warning:' # Indicates a PHP warning (e.g., "PHP Warning: Undefined variable") - 'PHP Fatal error:' # Indicates a PHP fatal error (e.g., "PHP Fatal error: Out of memory") - 'in /' # Indicates a PHP/Ruby file path in stack trace (e.g., "in /path/to/file.php:10") - 'Exception:' # Indicates a C# exception (e.g., "Exception: Invalid operation") - 'terminate called' # Indicates a C++ termination (e.g., "terminate called after throwing an instance of 'std::exception'") - 'syslog:' # Indicates a Linux syslog message (e.g., "syslog: Message ...") - 'IN=' # Indicates an iptables firewall input log (e.g., "IN=eth0 OUT=") - 'OUT=' # Indicates an iptables firewall output log (e.g., "OUT=eth0 SRC=") - 'SRC=' # Indicates an iptables firewall source IP log (e.g., "SRC=192.168.1.1 DST=") - 'DST=' # Indicates an iptables firewall destination IP log (e.g., "DST=10.0.0.1 LEN=") - 'KafkaServer:' # Indicates a Kafka server log (e.g., "KafkaServer: Starting Kafka server") - 'Redis:' # Indicates a Redis server log (e.g., "Redis: Server initialized") ``` === ":material-web: HttpCode" #### httpCode {.hidden-heading} Configure the [HTTP Status Code Extractor](/run/initialize/httpCode "Enrich TenXObjects with an extracted HTTP status code") to enrich TenXObjects with an extracted HTTP status code. Below is the default configuration from: [httpCode/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/initialize/httpCode/config.yaml "httpCode/config.yaml"){target="\_blank"}.
Edit Online
Edit HTTP Status Code Extractor Config Locally
```yaml # 🔟❎ 'run' HTTP code classifier configuration # The HTTP code classifier enriches TenXObjects with an extracted HTTP numeric response code (e.g., 404) field. # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/initialize/httpCode # =============================== GeoIP Options =============================== httpCode: # 'field' specifies the name of the field in which to assign an extracted HTTP status code, if found field: http_code # 'precludersBefore' specifies an array of characters or patterns that, when appearing immediately before a candidate HTTP status code in tokenized logs, indicate it is not a valid HTTP code. # Used to preclude false positives like negative numbers ('-200'), floats ('.200'), or variables ('_200'). # Entries are single characters or short strings for O(1) lookup efficiency in the heuristic. precludersBefore: ['+', '$', '#', '@', '(', '[', '~', '-', '.', ':'] # 'precludersAfter' specifies an array of characters or patterns that, when appearing immediately after a candidate HTTP status code in tokenized logs, indicate it is not a valid HTTP code. # Used to preclude false positives like percentages ('200%'), floats ('200.'), or expressions ('200+'). # Entries are single characters or short strings for O(1) lookup efficiency in the heuristic. precludersAfter: ['%', ')', ']', '=', '!', '?', '+', '-', '.'] # 'keywords' specifies an array of keywords indicating HTTP-related context in log events. # Used to filter logs likely containing HTTP status codes, reducing false positives. # Keywords cover HTTP methods, common log fields, and format-specific terms from sources like Apache, Nginx, HAProxy, Squid, IIS, Spring Boot. keywords: - HTTP/ # Indicates HTTP protocol version in request lines (e.g., "GET / HTTP/1.1"). - GET # HTTP method for retrieving resources. - POST # HTTP method for submitting data. - PUT # HTTP method for updating resources. - DELETE # HTTP method for deleting resources. - HEAD # HTTP method for retrieving headers only. - OPTIONS # HTTP method for describing communication options. - TRACE # HTTP method for diagnostic purposes. - PATCH # HTTP method for partial updates. - request # Common in log fields referring to HTTP requests (e.g., "requestMethod"). - Request # Title case variant for HTTP requests - response # Common in log fields referring to HTTP responses (e.g., "responseSize"). - Response # Title case variant for HTTP responses - status # Common key for HTTP status codes (e.g., "status":200). - Status # Title case variant for HTTP status - code # Shorthand for status code (e.g., "code":404). - Code # Title case variant for status code - upstream_status # Nginx-specific for backend status codes. - Upstream_status # Title case variant for Nginx backend status - Upstream_Status # Title case variant for Nginx backend status - http_status # JSON key for HTTP status (e.g., "http_status":500). - Http_status # Title case variant for JSON key for HTTP status - Http_Status # Title case variant for JSON key for HTTP status - status_code # Alternative key for status codes (e.g., "status_code":429). - statusCode # Camel case variant for status codes (e.g., "statusCode":200) - Completed # Spring Boot DispatcherServlet log indicator (e.g., "Completed 200 OK"). - access # Indicates access logs (e.g., Apache access log). - Access # Title case variant for access logs - error # Indicates error logs containing HTTP errors. - Error # Title case variant for error logs - method # Refers to HTTP request method (e.g., "method":"GET"). - Method # Title case variant for HTTP method - path # Refers to request path or URI (e.g., "path":"/api"). - Path # Title case variant for request path - url # Refers to request URL (e.g., "url":"http://example.com"). - Url # Title case variant for request URL - URL # Title case variant for request URL - referrer # HTTP referrer header field. - referrer # Title case variant for referrer header field. - Referer # Alternative spelling for HTTP referrer - Referer # Title case variant for alternative spelling - user-agent # HTTP user-agent header field. - User-agent # Title case variant for user-agent - User-Agent # Title case variant for user-agent - latency # Common in logs for request/response time (e.g., HAProxy, ELB). - Latency # Title case variant for latency - frontend # HAProxy term for incoming connections. - Frontend # Title case variant for HAProxy frontend - backend # HAProxy term for upstream servers. - Backend # Title case variant for HAProxy backend - TCP_MISS # Squid hierarchy code prefix (e.g., "TCP_MISS/200"). - W3SVC # IIS log prefix (e.g., "W3SVC1"). - DispatcherServlet # Spring Boot servlet class for HTTP handling. # 'strictKeywords' is a short, high-signal subset of keywords used as a # structural guard during template initialization. After a candidate HTTP # status variable token is found in the template via 'validValues', the # initializer requires at least one of these strict markers to appear # within a small token window (±5) around the candidate before the # httpToken position is bound. # # Without this check, any numeric variable that happens to hold a value # in 100..599 on the first observed event gets wrongly bound as the HTTP # code position, e.g. a kafka config limit `max.request.size = 200`, a # JVM `-Dclk.tck=100`, or any framework metric that emits counts in the # low hundreds. Subsequent events at the same variable position then # carry arbitrary non-HTTP values (1, 2, 2000, ...) that get emitted as # http_code garbage. # # Entries here should be ONLY tokens that unambiguously mark an HTTP # exchange. The loose list in 'keywords' above (which includes terms # like `error`, `path`, `url`) is too broad for adjacency gating. strictKeywords: - HTTP/ # HTTP protocol version marker in request lines (e.g., "HTTP/1.1 200 OK"). - status # JSON/KV key for HTTP status (e.g., "status":200, status=500). - Status # Title case variant. - statusCode # CamelCase variant of status code key. - status_code # Snake case variant. - http_status # HTTP-prefixed status key. - Http_Status # Title case variant. - Http_status # Mixed case variant. - upstream_status # Nginx backend status key. - Upstream_Status # Title case variant. - code # JSON/KV key for response code (e.g., "code":404). - Code # Title case variant (also catches `statusCode`-style splits). - Completed # Spring Boot DispatcherServlet log phrase (e.g., "Completed 200 OK"). - DispatcherServlet # Spring Boot servlet class that always emits a status next. - TCP_MISS # Squid hierarchy code prefix (always followed by "/"). - response # HTTP response marker. - Response # Title case variant. - GET # HTTP method verbs, always appear in request-log entries with codes. - POST - PUT - DELETE - PATCH - HEAD - OPTIONS - TRACE # 'validValues' specifies an array of all standard HTTP status codes as defined by the IANA HTTP Status Code Registry and relevant RFCs (e.g., RFC 9110, RFC 6585, RFC 7231-7235, RFC 7538, RFC 7725, RFC 8297, RFC 9111). # Used to validate potential HTTP status codes in log events, ensuring only recognized codes are extracted. # Each entry includes the numeric code followed by a comment describing its meaning and purpose. validValues: - '200' # OK: Standard response for successful HTTP requests. - '304' # Not Modified: Resource has not been modified since last requested. - '404' # Not Found: Server cannot find the requested resource. - '301' # Moved Permanently: Resource has been moved permanently to a new URI. - '302' # Found: Resource temporarily located at a different URI. - '403' # Forbidden: Client does not have access rights to the content. - '400' # Bad Request: Server cannot process due to client error. - '500' # Internal Server Error: Generic server error. - '206' # Partial Content: Server is delivering only part of the resource due to a range header. - '503' # Service Unavailable: Server temporarily unable to handle request. - '401' # Unauthorized: Authentication required and has failed or not provided. - '204' # No Content: Server processed request but no content is returned. - '502' # Bad Gateway: Server received an invalid response from upstream. - '201' # Created: Request has been fulfilled, resulting in new resource creation. - '429' # Too Many Requests: Client has sent too many requests in a given time. - '308' # Permanent Redirect: Resource permanently at another URI, method unchanged. - '307' # Temporary Redirect: Resource temporarily at another URI, method unchanged. - '303' # See Other: Response can be found under a different URI using GET. - '405' # Method Not Allowed: Request method is not supported for the resource. - '406' # Not Acceptable: Server cannot produce a response matching client’s Accept headers. - '408' # Request Timeout: Server timed out waiting for the request. - '409' # Conflict: Request conflicts with current state of the resource. - '410' # Gone: Resource is permanently unavailable. - '413' # Content Too Large: Request entity is larger than server limits. - '414' # URI Too Long: Request-URI is longer than the server can handle. - '415' # Unsupported Media Type: Media format of the requested data is not supported. - '416' # Range Not Satisfiable: Requested range cannot be fulfilled. - '422' # Unprocessable Content: Request is well-formed but semantically incorrect. - '504' # Gateway Timeout: Server, as gateway, did not get timely upstream response. - '505' # HTTP Version Not Supported: Server does not support the HTTP version. - '100' # Continue: Client should continue with request. - '101' # Switching Protocols: Server is switching protocols as requested. - '202' # Accepted: Request accepted for processing, but processing is not complete. - '203' # Non-Authoritative Information: Server is a transforming proxy with non-authoritative metadata. - '205' # Reset Content: Client should reset the document view. - '300' # Multiple Choices: Indicates multiple options for the resource. - '411' # Length Required: Content-Length header is required but not provided. - '412' # Precondition Failed: Server does not meet one of the preconditions. - '417' # Expectation Failed: Server cannot meet Expect header requirements. - '421' # Misdirected Request: Request sent to server unable to produce a response. - '423' # Locked: Resource is locked (WebDAV). - '424' # Failed Dependency: Request failed due to failure of a previous request. - '425' # Too Early: Server unwilling to risk processing a request that might be replayed. - '426' # Upgrade Required: Client should switch to a different protocol. - '428' # Precondition Required: Server requires conditional request headers. - '431' # Request Header Fields Too Large: Headers exceed server limits. - '451' # Unavailable For Legal Reasons: Resource access blocked for legal reasons. - '501' # Not Implemented: Server does not support the requested functionality. - '506' # Variant Also Negotiates: Server configuration error in content negotiation. - '507' # Insufficient Storage: Server cannot store the representation (WebDAV). - '508' # Loop Detected: Server detected an infinite loop in request processing. - '510' # Not Extended: Further extensions required for request fulfillment. - '511' # Network Authentication Required: Client needs to authenticate for network access. - '102' # Processing: Server has received and is processing the request. - '103' # Early Hints: Used to return some response headers before final HTTP message. - '207' # Multi-Status: Conveys information about multiple resources in WebDAV. - '208' # Already Reported: Used inside a DAV binding to avoid enumerating bindings repeatedly. - '226' # IM Used: Server has fulfilled a request for the resource using instance-manipulations. - '305' # Use Proxy: Resource is available only through a proxy. - '306' # (Unused): Previously used for Switch Proxy, now reserved. - '402' # Payment Required: Reserved for future use, e.g., digital payments. - '407' # Proxy Authentication Required: Client must authenticate with proxy. - '418' # (Unused): Previously “I’m a teapot” (RFC 2324, joke). ``` === ":material-table-eye: Lookup" #### lookup {.hidden-heading} Configure the [File Lookup](/run/initialize/lookup "Enrich TenXObjects with lookup table values") to enrich TenXObjects with lookup table values. Below is the default configuration from: [lookup/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/initialize/lookup/config.yaml "lookup/config.yaml"){target="\_blank"}.
Edit Online
Edit File Lookup Config Locally
```yaml # 🔟❎ 'run' lookup file configuration # Configuration text lookup to enrich TenXObjects # To learn more see https://doc.log10x.com/run/initialize/lookup/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/initialize/lookup # ============================== Lookup Options =============================== lookup: # 'file' specifies the .csv/.tsv lookup table file file: $=path("pipelines/run/initialize/lookup/http.csv") # 'keyField' specifies the field whose value is query lookup key. # The calculated code field below captures the penultimate HTTP code value from a web log event. # For example, 200 for: Mar 10 12:34:56 webserver nginx: 192.168.1.10 - - [10/Mar/2025:12:34:56 +0000] "GET /index.html HTTP/1.1" 200 1024 keyField: $=yield TenXEnv.get("httpCodeField") # 'valueField' specifies the name of the result field to assign to the lookup table value valueField: http_message ``` === ":material-kubernetes: K8s" #### k8s {.hidden-heading} Configure the [k8s Context Extractor](/run/initialize/k8s "Enrich TenXObjects with k8s pod and container information") to enrich TenXObjects with k8s pod and container information. Below is the default configuration from: [k8s/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/initialize/k8s/config.yaml "k8s/config.yaml"){target="\_blank"}.
Edit Online
Edit k8s Context Extractor Config Locally
```yaml # 🔟❎ 'run' k8s lookup configuration # Configure a k8s field extractors to enrich TenXObjects # To learn more see https://doc.log10x.com/run/initialize/k8s/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/initialize/k8s # ============================== Symbol Options =============================== k8s: # 'extractorName' specifies which extractor (e.g., fluentK8s/filebeatK8s) to use for k8s metadata extraction extractorName: fluentK8s # extract k8s context using Fluent schema # 'namespaceNameField' specifies the name of the field in which to assign an extracted k8 namespace name status code, if found namespaceNameField: k8s_namespace # 'containerNameField' specifies the name of the field in which to assign an extracted k8 container name status code, if found containerNameField: k8s_container # 'podNameField' specifies the name of the field in which to assign an extracted k8 pod name. # Default empty. Pod names churn on every deploy (each restart mints a new pod ID), so # surfacing pod name as a metric-label dimension causes unbounded TSDB active-series growth. # The stable 'k8s_namespace' + 'k8s_container' enrichments above cover the common per-app # attribution case without the deploy-churn cost. To opt in to per-pod resolution, set this # to a concrete field name (e.g. 'k8s_pod'); the 'tenx_user_process' alias activates with it. podNameField: "" ``` === ":material-globe-model: GeoIP" #### geoIP {.hidden-heading} Configure the [GeoIP lookup](/run/initialize/geoIP "Geo-reference TenXObjects via their ipAddress value") to geo-reference TenXObjects via their ipAddress value. Below is the default configuration from: [geoIP/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/initialize/geoIP/config.yaml "geoIP/config.yaml"){target="\_blank"}.
Edit Online
Edit GeoIP lookup Config Locally
```yaml # 🔟❎ 'run' GeoIP lookup configuration # The GeoIP lookup geo-references TenXObjects based on their ipAddress field value. # To learn more see https://doc.log10x.com/api/js/#TenXObject+ipAddress # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/initialize/geoIP # =============================== GeoIP Options =============================== geoIP: # 'file' specifies the MaxMind GeoIP DB (.mmdb) file to load. # To learn more see https://doc.log10x.com/api/js/#TenXLookup.loadGeoIPDB file: data/run/lookup/geo.mmdb # 'countryField' specifies the field name for country geo-reference data. # Defaults to "country". Set to "" to disable country enrichment. countryField: country # Uncomment the following fields to enable additional geo-reference data as needed: # continentField: continent # subdivisionField: subdivision # cityField: city # postalField: postal # latitudeField: latitude # longitudeField: longitude ``` ??? tenx-metricoutputs "Metric Outputs" ### Metric Outputs {.hidden-heading} Activate a [Metric output](/run/output/metric "Write aggregated TenXSummary instances to metric outputs (e.g., Prometheus, Datadog)") to write aggregated TenXSummary instances to metric outputs (e.g., Prometheus, Datadog). === ":material-monitor-dashboard: Log10x" #### log10x {.hidden-heading} Configure [log10x managed Prometheus outputs](/run/output/metric/log10x "Publish TenXSummary metrics to the log10x Prometheus backend") to publish TenXSummary metrics to the log10x Prometheus backend. Below is the default configuration from: [log10x/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/metric/log10x/config.yaml "log10x/config.yaml"){target="\_blank"}.
Edit Online
Edit log10x managed Prometheus output Config Locally
```yaml # 🔟❎ 'run' 10x-hosted metrics backend output configuration # Configuration 10x-hosted metrics backend counter outputs. # To learn more see https://doc.log10x.com/run/output/metric/log10x/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/output/metric/log10x # =========================== Log10x Metrics output =========================== # Define multiple log10x metric outputs below log10xMetrics: # ---------------------------- Counter Options ---------------------------- # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. # The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance. # To learn more see https://doc.log10x.com/run/aggregate - nameField: inputName # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter. # To learn more see https://doc.log10x.com/run/aggregate/#summary-fields counterFields: - summaryVolume - summaryBytes - summaryTotals # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values. # Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields tagFields: - $=yield TenXEnv.get("enrichmentFields") # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting. # Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames tagFieldNames: - $=yield TenXEnv.get("metricFieldNames") # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output. # To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions filter: isSummary ``` === ":simple-prometheus: Scrape" #### scrape {.hidden-heading} Configure [Prometheus scrape metrics outputs](/run/output/metric/prometheus/scrape "Publish TenXSummary metrics to Prometheus") to publish TenXSummary metrics to Prometheus. Below is the default configuration from: [scrape/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/metric/prometheus/scrape/config.yaml "scrape/config.yaml"){target="\_blank"}.
Edit Online
Edit Prometheus scrape metrics output Config Locally
```yaml # 🔟❎ 'run' Prometheus metric output configuration # Configure Prometheus metric counter outputs. # To learn more see https://doc.log10x.com/run/output/metric/prometheus/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/output/metric/prometheus/scrape # ============================== Prometheus Outputs =========================== # Define multiple Prometheus metric outputs below prometheus: # --------------------------- Connection Options -------------------------- # 'port' sets the port number to listen to the Prometheus scraper on # Override via 'PROMETHEUS_SCRAPE_PORT' to avoid collisions in sidecar deployments # (default 9100 collides with node_exporter). - port: $=TenXEnv.get("PROMETHEUS_SCRAPE_PORT", 9100) # ---------------------------- Counter Options ---------------------------- # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. # The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance. # To learn more see https://doc.log10x.com/run/aggregate nameField: inputName # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter. # To learn more see https://doc.log10x.com/run/aggregate/#summary-fields counterFields: - summaryVolume - summaryBytes - summaryTotals # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values. # Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields tagFields: - $=yield TenXEnv.get("enrichmentFields") # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting. # Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames tagFieldNames: - $=yield TenXEnv.get("metricFieldNames") # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output. # To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions filter: isSummary ``` === ":simple-prometheus: Push-gateway" #### push-gateway {.hidden-heading} Configure [Prometheus Push gateway metrics outputs](/run/output/metric/prometheus/push-gateway "Publish TenXSummary metrics to Prometheus via a Push gateway") to publish TenXSummary metrics to Prometheus via a Push gateway. Below is the default configuration from: [push-gateway/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/metric/prometheus/push-gateway/config.yaml "push-gateway/config.yaml"){target="\_blank"}.
Edit Online
Edit Prometheus Push gateway metrics output Config Locally
```yaml # 🔟❎ 'run' Prometheus push gateway metric output configuration # Configures Prometheus push gateway metric counter outputs. # To learn more see https://doc.log10x.com/run/output/metric/prometheus/push-gateway/ # For the Prometheus push gateway see: https://prometheus.io/docs/practices/pushing/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/output/metric/prometheus/push-gateway # ========================== Prometheus PG Options ============================ # Define multiple Prometheus push gateway metric outputs below prometheusPG: # --------------------------- Connection Options -------------------------- # 'host' sets the address and port number of the push gateway - host: $=TenXEnv.get("PUSHGATEWAY_URL", "localhost:9091") # 'job' sets the logical job name reported to the push gateway job: $=TenXEnv.get("PUSHGATEWAY_JOB", "my-10x-job") # ---------------------------- Counter Options ---------------------------- # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. # The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance. # To learn more see https://doc.log10x.com/run/aggregate nameField: inputName # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter. # To learn more see https://doc.log10x.com/run/aggregate/#summary-fields counterFields: - summaryVolume - summaryBytes - summaryTotals # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values. # Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields tagFields: - $=yield TenXEnv.get("enrichmentFields") # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting. # Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames tagFieldNames: - $=yield TenXEnv.get("metricFieldNames") # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output. # To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions filter: isSummary # ---------------------------- Publish Options ---------------------------- # 'step' sets the interval to send metrics Prometheus PG (e.g., '1m'). # To learn more see https://micrometer.io/docs/concepts#rate-aggregation step: 5s ``` === ":simple-prometheus: Remote-write" #### remote-write {.hidden-heading} Configure [Prometheus RW metrics outputs](/run/output/metric/prometheus/remote-write "Publish TenXSummary metrics to Prometheus via the Remote-Write protocol") to publish TenXSummary metrics to Prometheus via the Remote-Write protocol. Below is the default configuration from: [remote-write/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/metric/prometheus/remote-write/config.yaml "remote-write/config.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit Prometheus RW metrics output Config Locally
```yaml # 🔟❎ 'run' Prometheus remote write metric output configuration # Configure Prometheus remote write metric counter outputs. # To learn more see https://doc.log10x.com/run/output/metric/prometheus/remote-write/ # For Prometheus RW, see https://prometheus.io/docs/concepts/remote_write_spec/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/output/metric/prometheus/remote-write # ========================== Prometheus RW Options ============================ # Define multiple Prometheus RW metric outputs below prometheusRW: # --------------------------- Connection Options -------------------------- # 'host' sets the Prometheus remote write endpoint # 'PROMETHEUS_REMOTE_WRITE_URL' works for any remote-write-compatible backend # (Grafana Cloud, Mimir, Thanos, VictoriaMetrics, Cortex). - host: $=TenXEnv.get("PROMETHEUS_REMOTE_WRITE_URL", "http://localhost:9090/api/v1/write") # 'user' for authentication user: $=TenXEnv.get("PROMETHEUS_REMOTE_WRITE_USERNAME") # (❗ EnvVar REQUIRED) # 'password' for authentication password: $=TenXEnv.get("PROMETHEUS_REMOTE_WRITE_PASSWORD") # (❗ EnvVar REQUIRED) # ---------------------------- Counter Options ---------------------------- # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. # The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance. # To learn more see https://doc.log10x.com/run/aggregate nameField: inputName # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter. # To learn more see https://doc.log10x.com/run/aggregate/#summary-fields counterFields: - summaryVolume - summaryBytes - summaryTotals # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values. # Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields tagFields: - $=yield TenXEnv.get("enrichmentFields") # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting. # Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames tagFieldNames: - $=yield TenXEnv.get("metricFieldNames") # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output. # To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions filter: isSummary # ---------------------------- Publish Options ---------------------------- # 'step' sets the interval to send metrics to Prometheus RW (e.g. '1m'). # To learn more see https://micrometer.io/docs/concepts#rate-aggregation step: 5s ``` === ":simple-datadog: Datadog" #### datadog {.hidden-heading} Configure [Datadog metrics outputs](/run/output/metric/datadog "Publish TenXSummary metrics to Datadog") to publish TenXSummary metrics to Datadog. Below is the default configuration from: [datadog/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/metric/datadog/config.yaml "datadog/config.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit Datadog metrics output Config Locally
```yaml # 🔟❎ 'run' Datadog metric output configuration # Configure Datadog metric counter outputs. # To learn more see https://doc.log10x.com/run/output/metric/datadog/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/output/metric/datadog # ============================== Datadog Options ============================== # Define multiple Datadog metric outputs below datadog: # --------------------------- Connection Options -------------------------- # 'apiKey' and 'applicationKey' set the Datadog API and app keys (required). # To learn more see: https://docs.datadoghq.com/account_management/api-app-keys/ - apiKey: $=TenXEnv.get("DD_API_KEY") # (❗ EnvVar REQUIRED) applicationKey: $=TenXEnv.get("DD_APP_KEY") # (❗ EnvVar REQUIRED) # 'uri' sets the Datadog site address to which to connect. # 'DD_SITE' is the bare site domain (e.g. 'us5.datadoghq.com', 'datadoghq.eu'), # matches the Datadog Agent convention so existing cluster secrets work as-is. # To learn more see: https://docs.datadoghq.com/getting_started/site/ uri: $="https://" + TenXEnv.get("DD_SITE", "us5.datadoghq.com") # ---------------------------- Publish Options ---------------------------- # 'descriptions' sets whether to send meter descriptions to Datadog # Turn this off to minimize the amount of data sent on each write descriptions: false # 'step' sets the interval to send metrics to CloudWatch (e.g., '1m'). # To learn more see: https://micrometer.io/docs/concepts#rate-aggregation step: 5s # ---------------------------- Counter Options ---------------------------- # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. # The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance. # To learn more see https://doc.log10x.com/run/aggregate nameField: inputName # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter. # To learn more see https://doc.log10x.com/run/aggregate/#summary-fields counterFields: - summaryVolume - summaryBytes - summaryTotals # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values. # Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields tagFields: - $=yield TenXEnv.get("enrichmentFields") # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting. # Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames tagFieldNames: - $=yield TenXEnv.get("metricFieldNames") # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output. # To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions filter: isSummary ``` === ":fontawesome-brands-aws: Cloudwatch" #### cloudwatch {.hidden-heading} Configure [AWS CloudWatch metrics outputs](/run/output/metric/cloudwatch "Publish TenXSummary metrics to AWS CloudWatch metrics") to publish TenXSummary metrics to AWS CloudWatch metrics. Below is the default configuration from: [cloudwatch/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/metric/cloudwatch/config.yaml "cloudwatch/config.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit AWS CloudWatch metrics output Config Locally
```yaml # 🔟❎ 'run' AWS CloudWatch metric output configuration # Configure AWS CloudWatch metric counter outputs. # To learn more see https://doc.log10x.com/run/output/metric/cloudwatch/ # ============================== Dependencies ================================= # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/output/metric/cloudwatch # ============================ AWS CloudWatch Options ========================= # Define multiple AWS CloudWatch metric outputs below cloudwatch: # -------------------------- Connection Options --------------------------- # 'awsAccessKeyId' defines the AWS access key (required) # To learn more see https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html - awsAccessKeyId: $=TenXEnv.get("AWS_ACCESS_KEY_ID") # (❗ EnvVar REQUIRED) # 'awsSecretKey' defines the target AWS secret key (required) # To learn more see https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html awsSecretKey: $=TenXEnv.get("AWS_SECRET_ACCESS_KEY") # (❗ EnvVar REQUIRED) # 'namespace' defines the target AWS CloudWatch namespace (required) # To learn more see https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Namespace namespace: $=TenXEnv.get("CW_NAMESPACE") # (❗ EnvVar REQUIRED) # ---------------------------- Counter Options ---------------------------- # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. # The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance. # To learn more see https://doc.log10x.com/run/aggregate nameField: inputName # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter. # To learn more see https://doc.log10x.com/run/aggregate/#summary-fields counterFields: - summaryVolume - summaryBytes - summaryTotals # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values. # Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields tagFields: - $=yield TenXEnv.get("enrichmentFields") # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting. # Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames tagFieldNames: - $=yield TenXEnv.get("metricFieldNames") # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output. # To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions filter: isSummary # 'step' sets the interval to send metrics to CloudWatch (e.g., '1m'). # To learn more see https://micrometer.io/docs/concepts#rate-aggregation step: 5s ``` === ":simple-elastic: Elastic" #### elastic {.hidden-heading} Configure [ElasticSearch metrics outputs](/run/output/metric/elastic "Publish TenXSummary metrics to ElasticSearch") to publish TenXSummary metrics to ElasticSearch. Below is the default configuration from: [elastic/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/metric/elastic/config.yaml "elastic/config.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit ElasticSearch metrics output Config Locally
```yaml # 🔟❎ 'run' ElasticSearch metric output configuration # Configure ElasticSearch metric counter outputs. # To learn more see https://doc.log10x.com/run/output/metric/elastic/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/output/metric/elastic # =============================== Elastic Options ============================= # Define multiple Elastic metric outputs below elastic: # --------------------------- Connection Options -------------------------- # 'host' sets the host to send metrics to (e.g., https://.us-central1.gcp.cloud.es.io). - host: $=TenXEnv.get("ELASTICSEARCH_HOST") # (❗ EnvVar REQUIRED) # 'apiKeyCredentials' sets base64-encoded credentials string # To learn more see: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html apiKeyCredentials: $=TenXEnv.get("ELASTIC_API_KEY") # (❗ EnvVar REQUIRED) # 'userName' sets the the Basic Authentication username if 'apiKeyCredentials' is not set. userName: $=TenXEnv.get("ELASTIC_USERNAME") # (❗ EnvVar REQUIRED) # 'password' sets a Basic Authentication password. password: $=TenXEnv.get("ELASTIC_PASSWORD") # (❗ EnvVar REQUIRED) # ---------------------------- Counter Options ---------------------------- # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. # The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance. # To learn more see https://doc.log10x.com/run/aggregate nameField: inputName # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter. # To learn more see https://doc.log10x.com/run/aggregate/#summary-fields counterFields: - summaryVolume - summaryBytes - summaryTotals # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values. # Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields tagFields: - $=yield TenXEnv.get("enrichmentFields") # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting. # Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames tagFieldNames: - $=yield TenXEnv.get("metricFieldNames") # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output. # To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions filter: isSummary # ---------------------------- Publish Options ---------------------------- # 'step' sets the interval for sending metrics to Elastic (e.g., '1m'). # To learn more see: https://micrometer.io/docs/concepts#rate-aggregation step: 5s # ----------------------------- Index Options ----------------------------- # 'elasticIndex' sets the index name to which to write metrics. index: tenx-metrics # 'autoCreateIndex' sets whether to create the index automatically if it doesn't exist. autoCreateIndex: true # 'indexDateFormat' sets the index date format used for rolling indices. # This is appended to the index name, separated by 'elasticIndexDateSeparator'. indexDateFormat: 'yyyy-MM' # 'indexDateFormat' sets the separator between the index name and the date part indexDateSeparator: '-' # 'pipeline' sets the ingest pipeline name pipeline: null # 'timestampFieldName' sets the name of the elastic timestamp field. timestampFieldName: '@timestamp' ``` === ":material-math-log: Log" #### log {.hidden-heading} Configure [Log4j2 metrics outputs](/run/output/metric/log "Publish TenXSummary metrics to a log4j2 appender in Micrometer 'simple' format") to publish TenXSummary metrics to a log4j2 appender in Micrometer 'simple' format. Below is the default configuration from: [log/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/metric/log/config.yaml "log/config.yaml"){target="\_blank"}.
Edit Online
Edit Log4j2 metrics output Config Locally
```yaml # 🔟❎ 'run' Log4j2 metric output configuration # Configure log4j2 metric counter outputs. # To learn more see https://doc.log10x.com/run/output/metric/log/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/output/metric/log # ============================== Log4j2 Options =============================== # Define multiple log4j2 metric outputs below simple: # 'loggerName' sets the log4j2 logger to which registry metric data is periodically logged # metric are logged a 'INFO' messages using micrometer's default 'simple' format. # To learn more see: https://github.com/micrometer-metrics/micrometer/blob/main/micrometer-core/src/main/java/io/micrometer/core/instrument/simple/SimpleMeterRegistry.java - loggerName: consoleOut # ---------------------------- Counter Options ---------------------------- # 'nameField' sets the TenXSummary field whose value identifies the micrometer counter to increment. # The 'inputName' field returns the name of the aggregator that produced the target TenXSummary instance. # To learn more see https://doc.log10x.com/run/aggregate nameField: inputName # 'counterFields' specifies the TenXSummary fields whose values are used to increment the counter. # To learn more see https://doc.log10x.com/run/aggregate/#summary-fields counterFields: - summaryVolume - summaryBytes - summaryTotals # 'tagFields' sets the TenXSummary fields used as the counter tag dimension values. # Defaults to https://doc.log10x.com/run/initialize/#enrichmentFields tagFields: - $=yield TenXEnv.get("enrichmentFields") # 'tagFieldNames' allows setting custom names to aggregated fields for metrics reporting. # Defaults to https://doc.log10x.com/run/initialize/#metricFieldNames tagFieldNames: - $=yield TenXEnv.get("metricFieldNames") # 'filter' sets a JavaScript expression that TenXSummaries must evaluate as truthy to write to the output. # To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions filter: isSummary # ---------------------------- Publish Options ---------------------------- # 'step' sets the frequency in which metrics are logged to 'loggerName' in # ISO-8601 format. To learn more see: # https://docs.oracle.com/javase/8/docs/api/java/time/Duration.html#parse-java.lang.CharSequence- step: PT1S ``` ## :material-tune: Advanced Config Files To configure advanced options (optional) for the Reporter app, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files: ??? tenx-bootstrap "Bootstrap" ### Bootstrap {.hidden-heading} Configure the [Pipeline Bootstrapper](/run/bootstrap "Authenticate the log10x account and launch a target pipeline") to authenticate the log10x account and launch a target pipeline. Below is the default configuration from: [bootstrap/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/bootstrap/config.yaml "bootstrap/config.yaml"){target="\_blank"}.
Edit Online
Edit Pipeline Bootstrapper Config Locally
```yaml # 🔟❎ run bootstrap configuration # This config file specifies bootstrap options for the run pipeline. # To learn more see https://doc.log10x.com/run/bootstrap # To learn more see https://doc.log10x.com/config/ tenx: run # =============================== Launch Settings ============================== # 'licenseKey' specifies a signed JWT license token used for authenticating against the 10x service licenseKey: $=TenXEnv.get("TENX_LICENSE_KEY", "NO-LICENSE") # 'licenseFile' specifies a path to a file containing the signed JWT license token. # Preferred over 'licenseKey' on production hosts: the token does not appear in 'ps' output. licenseFile: $=TenXEnv.get("TENX_LICENSE_FILE") # 'airgapped' suppresses every call to the 10x SaaS gateway: the license is verified locally against the embedded # public key, and ALL outbound calls to the 10x SaaS gateway are suppressed (no DNS, no TCP). # Use for deployments where security review forbids vendor phone-home. airgapped: $=TenXEnv.get("TENX_AIRGAPPED", "false") # 'includePaths' specifies folders on disk for resolving relative config file/folder references in addition to the working folder includePaths: [] # 'quiet' disables printing version information to the console. # quiet: true # 'jarFiles' specifies .jar files to dynamically load for use by compile, input and output API extensions. jarFiles: [] # 'metricEndpoint' specifies a Prometheus remote-write endpoint for pipeline usage and health metrics. # Unset by default. The endpoint is a self-hosted or vendor Prometheus-compatible store, never a 10x # service, and is where the MCP reads engine state back from to tune policy such as regulator rates. # metricEndpoint: https://prometheus.example.com/api/v1/write # 'disabledArgs' specifies a list of launch arguments that are disallowed from either command line or user config files. disabledArgs: [] # 'debugEnvVars' list environment variables to debug debugEnvVars: [] ``` ??? tenx-githubsync "Github Sync" ### Github Sync {.hidden-heading} Below is the default configuration from: [gitops/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/gitops/config.yaml "gitops/config.yaml"){target="\_blank"}.
Edit Online
Edit config.yaml Locally
```yaml # 🔟❎ GitOps configuration # This config file pulls symbol, config and lookup files from a remote repo to # enable centralized configuration management via GitHub. # To learn more see https://doc.log10x.com/config/github # The target 10x pipeline to run (e.g., run, compile) is specified in the config files pulled from the repo tenx: any include: - source: github options: # Env-var overrides: GH_ENABLED, GH_REPO, GH_BRANCH, GH_PATH, GH_SYNC_INTERVAL. # Defaults below are for the compact-receiver GitOps reload test: # they pull test/compact-lookup.csv from talwgx/test every 30s. # For a real deployment, set GH_ENABLED=false OR override the repo. enabled: $=TenXEnv.get("GH_ENABLED", false) # 'token' specifies an access token for accessing the Github repo. token: $=TenXEnv.get("GH_TOKEN") # 'repo' specifies the GitHub user/repo from which to pull repo: $=TenXEnv.get("GH_REPO", "talwgx/test") # 'branch' on 'repo' (if null, uses default repo default branch) branch: $=TenXEnv.get("GH_BRANCH", "main") # 'paths' specifies Glob patterns within 'repo' to pull and extract to a temp folder. # (List items don't honor $= expressions; hardcoded glob. Override repo+branch # to target a different repo layout. Compact lookup must live as test/*.csv.) paths: - test/*.csv # 'syncInterval' specifies an interval by which check to check the remote branch for updates (null to disable). # This interval works in conjunction with: https://doc.log10x.com/run/reload syncInterval: $=TenXEnv.get("GH_SYNC_INTERVAL", "30s") # 'destination' mirrors pulled content into a stable absolute folder, # after the initial pull and after every sync (atomic per-file writes, # mtime refreshed every sync). The pull cache itself lives under a # sha-addressed temp folder whose path changes per commit -- options # like rateReceiverLookupFile must point HERE, not into the cache: # GH_DEST=/tmp/policy + rateReceiverLookupFile=/tmp/policy/test/mutes.csv # The folder must be writable by the engine user: the shipped container # runs as uid 1000, so a root-level path like /policy fails the launch. # Unset = no mirror (prior behavior). Requires engine >= 1.1.69. destination: $=TenXEnv.get("GH_DEST") # --------------------------------------------------------------------------- # Alternative source: kubernetes ConfigMap # # Pulls regulator CSV / lookup files from a kube-apiserver ConfigMap. Same # polling cadence + atomic write semantics as the github lane, but the # source-of-truth lives in the cluster rather than a remote repo. Useful # for air-gapped deployments and for tooling that already writes intent # (e.g. log10x-mcp configure_engine) directly into kubernetes. # # DISABLED unless K8S_ENABLED=true on the pod. The MCP install wizard sets # it (with a seeded ConfigMap + RBAC read grant), so a wizard-installed # receiver picks up caps.csv / actions.csv / config-generation.csv without # hand-wiring. The seed matters: the first pull runs on the launch thread # and a MISSING ConfigMap aborts startup; the lookup loader also refuses a # rows-less CSV, so the seeded files each carry one no-op `tenx-seed` row. # # Auth uses the pod's mounted service-account token (re-read each poll so # BoundServiceAccountTokenVolume rotation is honored). CA cert is re-loaded # on mtime change. Atomic file replacement via temp + Files.move. - source: kubernetes options: # Env-var overrides: K8S_ENABLED, K8S_NAMESPACE, K8S_CONFIGMAP, # K8S_SYNC_INTERVAL, K8S_API_SERVER, K8S_TOKEN_PATH, K8S_CA_PATH. enabled: $=TenXEnv.get("K8S_ENABLED", false) # 'namespace' defaults to the pod's namespace (read from the mounted # service-account token volume). Override for cross-namespace pulls. namespace: $=TenXEnv.get("K8S_NAMESPACE") # 'configMap' is the ConfigMap name to pull from. configMap: $=TenXEnv.get("K8S_CONFIGMAP", "log10x-action-intent") # 'keys' is a list of glob patterns over the ConfigMap data + binaryData # keys. Only keys matching at least one glob are written to disk. keys: - "*.csv" - "*.json" # 'syncInterval', floored to 60s by the scheduler (mirrors github lane). syncInterval: $=TenXEnv.get("K8S_SYNC_INTERVAL", "30s") # 'apiServer' default is the in-cluster URL. Override for out-of-cluster # smoke tests (e.g., https://127.0.0.1:6443 via kubectl proxy). apiServer: $=TenXEnv.get("K8S_API_SERVER", "https://kubernetes.default.svc") # Paths to the projected service-account token + CA cert. Defaults are # the standard in-cluster mount points; override only for non-standard # service-account volume layouts. tokenPath: $=TenXEnv.get("K8S_TOKEN_PATH", "/var/run/secrets/kubernetes.io/serviceaccount/token") caPath: $=TenXEnv.get("K8S_CA_PATH", "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt") ``` ??? tenx-symbols "Symbols" ### Symbols {.hidden-heading} Below is the default configuration from: [symbol/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/symbol/config.yaml "symbol/config.yaml"){target="\_blank"}.
Edit Online
Edit config.yaml Locally
```yaml # 🔟❎ 'run' symbol file configuration # Loads symbol library files to transform events into well-defined TenXObjects. # To learn more see https://doc.log10x.com/run/symbol # Set the 10x pipeline to 'run' tenx: run # ============================ Symbol Options ================================= symbol: # 'paths' specifies the file/folder locations to scan for symbol library files. # To learn more see https://doc.log10x.com/run/symbol/#symbolpaths paths: - $=path("data/shared/symbols", false) - $=path("", false) literals: [] ``` ??? tenx-aggregators "Aggregators" ### Aggregators {.hidden-heading} Configure [Aggregators](/run/aggregate "Aggregate and summarize TenXObjects to publish as metrics") to aggregate and summarize TenXObjects to publish as metrics. Below is the default configuration from: [report/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/aggregate/report/config.yaml "report/config.yaml"){target="\_blank"}.
Edit Online
Edit Aggregator Config Locally
```yaml # 🔟❎ 'run' aggregator config for the reporter app. # This file configures aggregators for tracking emitted (non-dropped) events. # To learn more see https://doc.log10x.com/run/aggregate # Set the 10x pipeline to 'run' tenx: run # ============================== Aggregator Options =========================== # Multiple aggregators can be defined below aggregator: # Defines an aggregator for tracking emitted (non-dropped) events # This enables an output metric to show an accurate (lossless) metric of the # data actually emitted to downstream systems. - name: emitted_events # 'filter' sets a JavaScript expression that TenXObjects must evaluate as truthy to be aggregated. # To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions # # This filter excludes dropped events by checking isRoute("drop"), # ensuring only emitted events are tracked. # # For more info, see https://doc.log10x.com/api/js/#TenXBaseObject+dropped filter: "this.isObject && !this.isRoute(\"drop\")" # 'fields' set the names of intrinsic/extracted/calculated fields whose combined values # identify a unique row within the aggregator's in-mem table for # each object passed to it. This aggregator defaults to enrichment fields. # To learn more see https://doc.log10x.com/run/initialize/#enrichmentFields fields: - $=yield TenXEnv.get("enrichmentFields") # 'totalFields' specifies fields to extract and tally into generated TenXSummaries totalFields: [] # 'flushThreshold' sets the max number of TenXObjects whose values to upsert into # the aggregator's in-mem table. Once exceeded, the aggregator instantiates TenXSummaries # from the table rows to send to output and reset the table. flushThreshold: 10000 # 'flushInterval' sets a periodic interval, after which the aggregator # flushes the rows of its in-mem table into the pipeline as TenXSummary instances. flushInterval: 2s # 'maxCardinality' sets the maximum number of rows for the aggregator's # in-mem table. If an TenXObjects whose 'values' are not contained in the table # is passed to the aggregator, 'defaultValue' will be used for it instead # until the aggregator flushes its in-mem table after 'flushInterval' or 'flushThreshold' # are exceeded. maxCardinality: 65536 # 'defaultValue' sets a constant value that is used instead of 'values' when # 'maxCardinality' is exceeded within 'flushInterval' defaultValue: unknown ``` ??? tenx-template "Template" ### Template {.hidden-heading} Below is the default configuration from: [template/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/template/config.yaml "template/config.yaml"){target="\_blank"}.
Edit Online
Edit config.yaml Locally
```yaml # 🔟❎ 'run' template file configuration # Load TenXTemplates .json files that define the structure/schema of TenXObjects. # To learn more see https://doc.log10x.com/run/template/ # Set the 10x pipeline to 'run' tenx: run # =============================== Template =================================== template: # 'files' specifies GLOB pattern for finding JSON-encoded TenXTemplates. files: - $=path("data/templates/*.json") - $=path("data/sample/output/*.json") # 'cacheSize' controls the maximum total byte size of templates held # in the in-memory cache vs. on disk. Set to 0 to disable pruning. cacheSize: $=parseBytes("10MB") # =============================== Variable =================================== var: # 'placeholder' specifies a character to use when encoding a TenXTemplate # to signify the location of a runtime variable value. placeholder: "$" # 'maxRecurIndexes' controls the maximum number of variable values to reuse. maxRecurIndexes: 10 # =============================== Timestamp ================================== timestamp: # 'prefix' specifies a prefix for a TenXTemplate's timestamp tokens. prefix: ( # 'postfix' specifies a postfix for a TenXTemplate's timestamp tokens. postfix: ) ``` ??? tenx-transform "Transform" ### Transform {.hidden-heading} Configure the [Transform](/run/transform "Transform log and trace events into well-defined TenXObjects") to transform log and trace events into well-defined TenXObjects. === ":material-stamper: Timestamp" #### timestamp {.hidden-heading} Configure the [Timestamp parser](/run/transform/timestamp "Extract alphanumeric and epoch timestamp values from input events") to extract alphanumeric and epoch timestamp values from input events. Below is the default configuration from: [timestamp/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/transform/timestamp/config.yaml "timestamp/config.yaml"){target="\_blank"}.
Edit Online
Edit Timestamp parser Config Locally
```yaml # 🔟❎ 'run' timestamp parser configuration # Identify unix/alphanumeric timestamp structures within TenXTemplates. # https://doc.log10x.com/run/transform/timestamp/ # Set the 10x pipeline to 'run' tenx: run # ============================= Timestamp Options ============================= timestamp: # 'maxPerObject' controls the max number of timestamps to add into an TenXObject's # 'timestamp' array. Set to 0 for unlimited. maxPerObject: 0 # 'searchDirection' controls the direction(s) from with timestamps are searched for # within the object's 'text' field. Possible values: # - fromStart: search for 'maxPerObject' from the start of 'text', # limiting search to 'searchLengthLimitFromStart' characters # - fromEnd: search backward for 'maxPerObject' from the end of 'text', # limiting search to 'searchLengthLimitFromEnd' characters # - fromStartAndEnd: search for 'maxPerObject' from both the start and end of 'text' # limiting search to 'searchLengthLimitFromStart' and 'searchLengthLimitFromEnd' respectively # - none: do not parse timestamps searchDirection: fromStartAndEnd # 'searchLengthLimitFromStart' limits the number of characters to search for # timestamps from the beginning of the object's 'text' field. 0 for unlimited searchLengthLimitFromStart: 0 # 'searchLengthLimitFromStart' limits the number of characters to search for # timestamps from the end of the object's 'text' field. 0 for unlimited searchLengthLimitFromEnd: 0 # 'zone' controls the timezone for formatting timestamp epoch # values to string. The 'java.time.ZoneId.of(String zoneId)' # is used to obtain the timezone from the 'zone' value. # If set to null, the host OS timezone is used. zone: null # 'literals' contains an array of strings to treat as a part # of any timestamp candidate found when structuring an TenXObject. literals: - T - Z - I # Go INFO - E # Go ERROR - W # Go WARN # 'patterns' specifies an array of date-time formats to attempt when parsing timestamps from input events. # Timestamp formats that appear frequently within an input stream can be 'bumped' higher within the list below. patterns: # Most common formats - "'I'MMdd HH:mm:ss.S" # Used in Kubernetes kube-apiserver logs with INFO prefix, 1-digit microsecond precision. - "'I'MMdd HH:mm:ss.SS" # Used in Kubernetes kube-apiserver logs with INFO prefix, 2-digit microsecond precision. - "'I'MMdd HH:mm:ss.SSS" # Used in Kubernetes kube-apiserver logs with INFO prefix, 3-digit microsecond precision. - "'I'MMdd HH:mm:ss.SSSS" # Used in Kubernetes kube-apiserver logs with INFO prefix, 4-digit microsecond precision. - "'I'MMdd HH:mm:ss.SSSSS" # Used in Kubernetes kube-apiserver logs with INFO prefix, 5-digit microsecond precision. - "'I'MMdd HH:mm:ss.SSSSSS" # Used in Kubernetes kube-apiserver logs with INFO prefix, 6-digit microsecond precision. - "'W'MMdd HH:mm:ss.S" # Used in Kubernetes kube-apiserver logs with WARNING prefix, 1-digit microsecond precision. - "'W'MMdd HH:mm:ss.SS" # Used in Kubernetes kube-apiserver logs with WARNING prefix, 2-digit microsecond precision. - "'W'MMdd HH:mm:ss.SSS" # Used in Kubernetes kube-apiserver logs with WARNING prefix, 3-digit microsecond precision. - "'W'MMdd HH:mm:ss.SSSS" # Used in Kubernetes kube-apiserver logs with WARNING prefix, 4-digit microsecond precision. - "'W'MMdd HH:mm:ss.SSSSS" # Used in Kubernetes kube-apiserver logs with WARNING prefix, 5-digit microsecond precision. - "'W'MMdd HH:mm:ss.SSSSSS" # Used in Kubernetes kube-apiserver logs with WARNING prefix, 6-digit microsecond precision. - "'E'MMdd HH:mm:ss.S" # Used in Kubernetes kube-apiserver logs with ERROR prefix, 1-digit microsecond precision. - "'E'MMdd HH:mm:ss.SS" # Used in Kubernetes kube-apiserver logs with ERROR prefix, 2-digit microsecond precision. - "'E'MMdd HH:mm:ss.SSS" # Used in Kubernetes kube-apiserver logs with ERROR prefix, 3-digit microsecond precision. - "'E'MMdd HH:mm:ss.SSSS" # Used in Kubernetes kube-apiserver logs with ERROR prefix, 4-digit microsecond precision. - "'E'MMdd HH:mm:ss.SSSSS" # Used in Kubernetes kube-apiserver logs with ERROR prefix, 5-digit microsecond precision. - "'E'MMdd HH:mm:ss.SSSSSS" # Used in Kubernetes kube-apiserver logs with ERROR prefix, 6-digit microsecond precision. - "'F'MMdd HH:mm:ss.S" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 1-digit microsecond precision. - "'F'MMdd HH:mm:ss.SS" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 2-digit microsecond precision. - "'F'MMdd HH:mm:ss.SSS" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 3-digit microsecond precision. - "'F'MMdd HH:mm:ss.SSSS" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 4-digit microsecond precision. - "'F'MMdd HH:mm:ss.SSSSS" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 5-digit microsecond precision. - "'F'MMdd HH:mm:ss.SSSSSS" # Used in Kubernetes kube-apiserver logs with FATAL prefix, 6-digit microsecond precision. - "yyyy-MM-dd HH:mm:ss.SSSSSSZ" # Used in macOS unified logs (log show), with microsecond precision and timezone offset (e.g., -0400). - "yyyy-MM-dd HH:mm:ss" # Widely used in Java applications, databases (e.g., MySQL, PostgreSQL), and application servers (e.g., Tomcat, JBoss). - "yyyy-MM-dd'T'HH:mm:ss.SSSZ" # Common in web services, APIs, Java (with DateTimeFormatter.ISO_OFFSET_DATE_TIME), Python (with datetime.isoformat()), and systems requiring precise timestamps with timezone information. - "MMM dd HH:mm:ss" # Frequently seen in syslog, Unix-based systems, network devices (e.g., Cisco routers), and web servers (e.g., Apache, Nginx). - "MMM d HH:mm:ss" # Classic syslog (RFC 3164) on days 1-9: the day is space-padded, e.g. "Jan 3 17:38:36". - "MMM d HH:mm:ss" # Same date without the padding (strftime %b %-d), e.g. "Nov 9 12:01:01" in Thunderbird-style system logs. - "dd/MMM/yyyy:HH:mm:ss Z" # Standard syslog format, used in firewalls (e.g., Cisco ASA, Palo Alto), network monitoring tools, and Unix-based systems. - "yyyy-MM-dd HH:mm:ss,SSS" # Common in Java logging frameworks like Log4j and Logback. - "yyyy-MM-dd'T'HH:mm:ss,SSS" # Common in Java logging with ISO 8601 date-time, comma-separated milliseconds. - "MM/dd/yyyy HH:mm:ss" # Used in Windows event logs, .NET applications, and U.S.-based systems. - "EEE MMM dd HH:mm:ss yyyy" # Human-readable format used in various logs, including some web servers and application logs. - "EEE MMM dd HH:mm:ss zzz yyyy" # java.util.Date.toString() and the Unix date command, e.g. "Thu Jan 15 10:00:01 UTC 2026". Common in Java application logs and shell-script output. - "yyyy-MM-dd'T'HH:mm:ss" # Simplified ISO 8601 format, used in many modern applications and frameworks. - "HH:mm:ss" # Time-only format, used when the date is implied or provided separately, common in embedded systems and some programming languages. - "yyyyMMdd HH:mm:ss" # Compact format used in some legacy systems and batch processing logs. - "yyyyMMdd-H:m:s:SSS" # Compact date, hyphen, unpadded time, colon before the milliseconds, e.g. "20171223-22:15:29:606" or "20171224-0:0:0:234". Android HealthApp and similar mobile logs. - "yyyyMMdd-H:m:s:SS" # Same with two fraction digits, e.g. "20171223-22:30:24:28". - "yyyyMMdd-H:m:s:S" # Same with one fraction digit. - "MM.dd HH:mm:ss" # Month.day without a year, e.g. "10.30 16:49:06". Proxifier and other Windows utility logs. - "MMM dd, yyyy h:mm:ss a" # Human-readable format with 12-hour clock, used in application logs and some U.S.-based systems. - "MMM dd, yyyy hh:mm:ss a" # Human-readable format with 12-hour clock (padded hour), used in application logs and some U.S.-based systems. # Common formats with slight variations - "yyyy-MM-dd HH:mm:ss.SSS" # Extended precision format used in Java applications and databases requiring millisecond accuracy. - "dd-MMM-yyyy HH:mm:ss.SSS" # Common in Java logging frameworks (e.g., Log4j, SLF4J) and application logs requiring human-readable dates with millisecond precision. - "yyyy-MM-dd'T'HH:mm:ss.S'Z'" # Microsecond precision ISO 8601 format, 1-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.SS'Z'" # Microsecond precision ISO 8601 format, 2-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'" # Microsecond precision ISO 8601 format, 3-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.SSSS'Z'" # Microsecond precision ISO 8601 format, 4-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.SSSSS'Z'" # Microsecond precision ISO 8601 format, 5-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'" # Microsecond precision ISO 8601 format, 6-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.S'Z'" # Nanosecond precision ISO 8601 format, 1-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.SS'Z'" # Nanosecond precision ISO 8601 format, 2-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'" # Nanosecond precision ISO 8601 format, 3-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.SSSS'Z'" # Nanosecond precision ISO 8601 format, 4-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.SSSSS'Z'" # Nanosecond precision ISO 8601 format, 5-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'" # Nanosecond precision ISO 8601 format, 6-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.SSSSSSS'Z'" # Nanosecond precision ISO 8601 format, 7-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSS'Z'" # Nanosecond precision ISO 8601 format, 8-digit precision. - "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSS'Z'" # Nanosecond precision ISO 8601 format, 9-digit precision. - "yyyy-MM-dd'T'HH:mm:ssZZZZZ" # ISO 8601 with timezone offset, used in systems where timezone information is critical. - "yyyy-MM-dd HH:mm:ss.SSSZZZZZ" # Similar to above but without the 'T' separator, used in database logs and application servers. - "yyyy-MM-dd HH:mm:ss.S" # Microsecond precision format, 1-digit precision. - "yyyy-MM-dd HH:mm:ss.SS" # Microsecond precision format, 2-digit precision. - "yyyy-MM-dd HH:mm:ss.SSS" # Microsecond precision format, 3-digit precision. - "yyyy-MM-dd HH:mm:ss.SSSS" # Microsecond precision format, 4-digit precision. - "yyyy-MM-dd HH:mm:ss.SSSSS" # Microsecond precision format, 5-digit precision. - "yyyy-MM-dd HH:mm:ss.SSSSSS" # Microsecond precision format, 6-digit precision. - "MM/dd/yyyy*HH:mm:ss*SSS" # Used in some U.S.-based systems, particularly in legacy applications or specific logging frameworks. - "M/d/yyyy h:mm:ss a:SSS" # Common in systems using 12-hour time format, such as some Windows applications or older logging systems. - "M/d/yyyy hh:mm:ss a:SSS" # Common in systems using 12-hour time format with milliseconds and padded hour, such as some Windows applications. - "M/dd/yyyy hh:mm:ss a" # Similar to above, used in systems where millisecond precision is not needed. - "yyyy-MM-dd'T'HH:mm:ss.SSSX" # ISO 8601 with basic timezone offset format, used in systems that require standardized timestamps. - "yyyy-MM-dd'T'HH:mm:ss.SSSz" # ISO 8601 with timezone name, used in applications that need to display timezone information. - "yyyy-MM-dd'T'HH:mm:ss'Z'" # ISO 8601 format assuming UTC, used in systems where all times are in UTC. - "yy/MM/dd HH:mm:ss" # Two-digit year format, used in compact logs or older systems. - "MMM dd HH:mm:ss ZZZZ" # Used in some Unix-based systems and web servers, includes timezone offset. - "HH:mm:ss,SSS" # Time-only format with milliseconds, used in performance logs or systems where date is provided separately. - "yyyy-MM-dd*HH:mm:ss" # Used in some application logs where the separator is a space or asterisk. - "yyyy MMM dd HH:mm:ss.SSS" # Human-readable format with milliseconds, used in some application logs. - "dd/MMM/yyyy HH:mm:ss" # Used in European systems and some web applications. - "yyyy-MM-dd'T'HH:mm:ss.SSS''Z''" # ISO 8601 with milliseconds and literal 'Z', used in systems where 'Z' is explicitly included. - "MMM dd yyyy HH:mm:ss" # Human-readable format, used in various logs. - "yyyy-MM-dd HH:mm:ss ZZZZ" # Format with timezone offset, used in systems requiring timezone information. - "yyyy-MM-dd HH:mm:ssZZZZZ" # Similar to above, used in application logs. - "dd MMM yyyy HH:mm:ss" # European format, used in some web applications and databases. - "MMdd_HH:mm:ss" # Compact format without separators, used in file names or space-constrained logs. - "yyyy-MM-dd HH:mm:ss,SSSZZZZZ" # Java logging format with timezone offset. - "yyyyMMdd HH:mm:ss.SSS" # Compact format with milliseconds, used in some legacy systems. - "yyyy/MM/dd HH:mm:ss" # Format used in some Asian systems, particularly in Japan. - "dd/MM/yyyy HH:mm:ss" # Common in European systems, including some web applications and databases. - "MM-dd-yyyy HH:mm:ss" # U.S. format variant, used in some older systems. - "yyyyMMddHHmmss" # Compact format for file names or database timestamps, used in systems where space is a concern. - "EEE, dd MMM yyyy HH:mm:ss zzz" # RFC 1123 format, used in HTTP headers, web servers, and email systems. ``` === ":material-equal: Fields" #### fields {.hidden-heading} Configure the [Field parser](/run/transform/fields "Scan TenXTemplates for JSON and KV fields") to scan TenXTemplates for JSON and KV fields. Below is the default configuration from: [fields/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/transform/fields/config.yaml "fields/config.yaml"){target="\_blank"}.
Edit Online
Edit Field parser Config Locally
```yaml # 🔟❎ 'run' TenXTemplate field extract configuration # Configure how to extract JSON objects and KV structures from TenXTemplates. # To learn more see https://doc.log10x.com/run/transform/fields/ # Set the 10x pipeline to 'run' tenx: run # =============================== Extract Options ============================= field: # 'extract' controls whether to scan TenXTemplate for JSON # objects or key-value lists (e.g., 'X=Y'). extract: true # 'nameBreaks' controls which characters found to the left of a token # whose is a candidate for being a 'key' in a KV field formation should serve # as a terminator for the search. For example, for an object whose 'text' field # contains the following entry ',tx_result=OK' the desired key name should be 'tx_result' # and as such, the name terminator character should be ',' vs. '_' (in which case # the key would have been named 'result'). nameBreaks: ', /\{}.()[]' # 'valueBreaks' controls which characters found to the right of a token # whose is a candidate for being a 'value' in a KV field formation should serve # as a terminator for the search. For example, for an object whose 'text' field # contains the following entry: 'status=RESULT_SUCCESS' the desired KV value # should be 'RESULT_SUCCESS', and as such, the value terminator character # should be ',' vs. '_' (in which case the value would be 'RESULT'). valueBreaks: ', /\{}()[]' ``` === ":material-origin: Symbol" #### symbol {.hidden-heading} Configure the [Origin selector](/run/transform/symbol "Select the source code/binary executable origin of TenXTemplate symbol values") to select the source code/binary executable origin of TenXTemplate symbol values. Below is the default configuration from: [symbol/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/transform/symbol/config.yaml "symbol/config.yaml"){target="\_blank"}.
Edit Online
Edit Origin selector Config Locally
```yaml # 🔟❎ 'run' symbol origin configuration # Identify the origin of symbol values within TenXTemplates. # To learn more see https://doc.log10x.com/run/transform/symbol/ # Set the 10x pipeline to 'run' tenx: run # =============================== Origin Options ============================== symbol: # 'maxOrigins' controls the number of source/binary origins to list # per symbol token sequence. As a series of tokens within # a target TenXTemplate (e.g., 'ERROR', 'could not connect to "{}') may appear # in multiple source/binary files within the pipeline's loaded library. # The sorting algorithm configured below is used to select the 'maxSymbolOrigins' # topmost entries to list. maxOrigins: 64 # 'sequenceReserved' defines a list of terms to ignore when searching # for the symbol tokens that constitute an TenXTemplate's 'message' portion. # For example, for an event whose text contains: 'connect success = true', # the value 'true' will not be considered a part of the event's message, # as the 'true' value is mostly likely the result of a variable boolean state. # For more information, see: https://doc.log10x.com/api/js/#TenXObject+symbolSequence sequenceReserved: # literals - "null" - "nil" - "true" - "false" # stopwords (articles, conjunctions, prepositions) - "a" - "an" - "the" - "to" - "of" - "in" - "on" - "at" - "by" - "for" - "from" - "with" - "and" - "or" - "as" - "is" # generic log words - "log" - "logger" - "logs" - "http" - "https" # severity levels, uppercase - TRACE - DEBUG - INFO - NOTICE - WARN - WARNING - ERROR - ERR - CRITICAL - CRIT - ALERT - EMERGENCY - EMERG - FATAL - PANIC # severity levels, lowercase (engine matching is case-sensitive). # Note. The token "error" is intentionally NOT reserved here. # Verification against 3254 demo events (2026-05-20) found 1280 # signal hits where the "error" key carries the actual message # body the pattern groups; reserving it would collapse distinct # error patterns into one. # Note. The token "emergency" is also intentionally NOT reserved #, sample showed only prose hits, no envelope use observed. - "trace" - "debug" - "info" - "notice" - "warn" - "warning" - "err" - "critical" - "crit" - "alert" - "fatal" - "panic" # severity-equivalent envelope keys - "level" - "levelname" - "severity" # envelope time keys - "time" - "timestamp" - "ts" - "msg" - "message" # OpenTelemetry / collector envelope (these tokens leak from the # otelcol's Go-package logging into pattern names) - "otel" - "otelcol" - "opentelemetry" - "collector" - "contrib" - "processor" - "processors" - "receiver" - "receivers" - "exporter" - "exporters" - "pipeline" - "pipelines" # structured / envelope keys (singular + plural, engine matcher # is exact, no stem; both forms appear in real otel logs). # Note: "trace" is already reserved above under lowercase severity. - "service" - "services" - "instance" - "instances" - "component" - "components" - "signal" - "signals" - "resource" - "resources" - "scope" - "scopes" - "attribute" - "attributes" - "traces" - "span" - "spans" - "flag" - "flags" - "context" - "contexts" - "baggage" # K8s audit / Kubernetes API envelope (verified zero signal in # the 2026-05-20 sample, every occurrence was a JSON metadata # key or audit field) - "metadata" - "namespace" - "verb" - "uid" - "identity" - "protocol" - "priority" - "encoding" - "headers" - "duration" # HTTP / request envelope (verified safe, appear only as # klog field markers or JSON keys, not as message content) - "path" - "status" # generic identity keys (very common envelope tokens in JSON logs) - "id" - "name" - "kind" - "version" - "type" # source-location envelope (Go / Java / Python loggers leak these) - "caller" - "func" - "function" - "file" - "line" - "module" - "package" - "class" - "method" - "stack" - "stacktrace" # process / runtime context. # Note. The token "host" is intentionally NOT reserved, 1018 of # 1024 occurrences in the verification corpus were the content # phrase "no such host" inside DNS-failure messages; reserving # it would collapse every "no such host" pattern into one. # Note. The token "app" is intentionally NOT reserved, 42 of 44 # occurrences were the Flask logger-name discriminator in # "INFO in app:" lines; reserving it would collapse Python-app # logger structure. - "pid" - "tid" - "process" - "thread" - "hostname" - "application" # package-qualifier prefixes (Go / Java loggers embed these in # source-path tokens like `com.foo.bar.MyLogger`) - "com" - "org" - "io" - "net" # tokenizer artifacts - "t" # ----------------------------- Debug Options ------------------------------- debug: # 'symbol' debugging allows for verbose printing of the selection process # for symbol tokens from the pipeline's 10x symbol files used to produce the # results of 10x reflection functions. # For more information, see: https://doc.log10x.com/api/js/#TenXObject+symbol # 'loggerName' controls the name of the log4j logger used by origins and symbols below loggerName: "symbolDebugger" # 'origins' outputs information about the origin selection process for an TenXTemplate symbol. # For example, setting 'units' to 'foo.js' will output information about if/how it # was selected as the origin of TenXTemplate symbols. # In other words, if a symbol (e.g., 'MyClass') has 'foo.java' as the source file # from which it originated, adding 'foo.js' to 'units' will emit information # about the selection process. # Specifying '*' will emit information for all source /binary files that have # been selected as the origin of any TenXTemplate objects within the pipeline. origins: [ # '*' ] # 'symbols' logs the origin selection process for symbol tokens within an TenXTemplate. # For example, set 'symbols' to 'Could not connect to' to log how the 10x JavaScript 'symbolSequence' # determines the origin source code/binary file within the pipeline's symbol library. # Specifying '*' will emit information for all source code /binary files that have # been selected as the origin of any TenXTemplate objects within the pipeline. symbols: [ # '*' ] ``` === ":material-select-group: Group" #### group {.hidden-heading} Configure the [Group sequencer](/run/transform/group "Group and sequence TenXObjects") to group and sequence TenXObjects. Below is the default configuration from: [group/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/transform/group/config.yaml "group/config.yaml"){target="\_blank"}.
Edit Online
Edit Group sequencer Config Locally
```yaml # 🔟❎ 'run' event grouping configuration # Group sequences of TenXObjects to filter, aggregate and output as a single logical unit. # To learn more see https://doc.log10x.com/run/transform/group/ # Set the 10x pipeline to 'run' tenx: run # =============================== Group Options =============================== group: # 'filters' specify JavaScript expressions an TenXObject instance/group must # evaluate as truthy against to be written to output filters: [] # 'maxSize' defines the maximum number of TenXObjects to group # before the group is sealed and forwarded into the pipeline. # Subsequent TenXObjects can form a new group. maxSize: 20000 # 'flushTimeout' defines the max interval (e.g., 10s) to wait for # new events to be read from an input stream before it flushes any # pending TenXObjects group into the pipeline. # This mechanism is designed to avoid latencies in dispatching pending event # groups to output destinations. flushTimeout: $=parseDuration("5s") # 'async' specifies whether to sequence and group TenXObjects in a dedicated thread async: true ``` === ":octicons-cpu-24: Parallelize" #### parallelize {.hidden-heading} Configure the [Parallel processor](/run/transform/parallelize "Distribute event parsing and transformation workloads across multiple cores.") to distribute event parsing and transformation workloads across multiple cores.. Below is the default configuration from: [parallelize/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/transform/parallelize/config.yaml "parallelize/config.yaml"){target="\_blank"}.
Edit Online
Edit Parallel processor Config Locally
```yaml # 🔟❎ 'run' event parallel processing configuration # Transform log/trace events read from inputs into well-defined TenXObjects using multiple cores. # To learn more see https://doc.log10x.com/run/transform/parallelize/ # Set the 10x pipeline to 'run' tenx: run # =============================== Parallel Options ============================ parallelEvent: # 'threadPoolSize' specifies the number of threads allocated to transform events into TenXObjects concurrently. # If the value is - # - = 0: transform events into TenXObjects synchronously using their input stream calling thread. # - < 1: interpreted as a percentage of the number of available cores (e.g., 0.5 = use up to 50% of available cores) # - = 1: allocate a single dedicated to transform events. # - > 1: interpreted as a fixed number of threads (e.g., 10 = 10 threads) threadPoolSize: "0.5" # 'batchSize' specifies the maximum number of events to queue for concurrent processing before flushing. # If 'threadPoolSize' is 1, this value is unused, and events are transformed into TenXObject synchronously # to their calling input's thread. If 0, flush pending events after 'flushInterval' expires or the # source input reaches end-of-file. batchSize: 1000 # 'flushInterval' specifies the maximum wait duration before flushing queued events # If 'threadPoolSize' is 1, this value is unused, and events are transformed into TenXObject synchronously # to their input thread. If 0, no wait flush interval is applied. flushInterval: 2s # 'processingTimeout' specifies the maximum wait duration before dropping un-processed queued events # This value provides a backstop for overflowing the heap if the pipeline cannot dequeue # pending events to transform into TenXObjects. If 0, no timeout is applied. processingTimeout: 30s ```
:material-github: This app is defined in [reporter/app.yaml](https://github.com/log-10x/modules/tree/main/pipelines/reporter/app.yaml "reporter/app.yaml"){target="\_blank"}.