--- title: "Symbol Message Calculator" description: "enrich TenXObjects with logical message symbol sequence and origin values" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/initialize/message/module.yaml" icon: "material/origin" --- Extracts consistent message identifiers from log events for accurate log-to-metrics conversion and cost control. Raw log events contain high-cardinality [variable](https://doc.log10x.com/run/transform/structure/#variables) data (timestamps, IDs, values) mixed with constant, low-cardinality [symbols](https://doc.log10x.com/run/transform/structure/#symbols). The message initializer uses symbol libraries to isolate stable [message patterns](https://doc.log10x.com/run/transform/symbol/ "Select the source code/binary executable origin of TenXTemplate symbol values") from each event, enabling accurate classification of event instances by their logical type. ## :material-target: Message Extraction Every token the symbol library recognizes carries a set of candidate origins: the source unit, a file or a binary, that can emit the token, together with the enclosing scope the library recorded for it. A token emitted by many units resolves to many candidates, so ambiguity is the normal case and resolving it is the selector's job. Candidates are keyed by originating unit plus enclosing scope, so symbols sharing an origin and a scope coalesce into one candidate. The initializer identifies the core message pattern by ranking those candidates from a [TenXTemplate](https://doc.log10x.com/run/template/ "Import joint JSON schemas files to expand events into well-defined TenXObjects."), on a five-key comparator, every key descending: 1. **Prose evidence**: the width of the candidate's widest matched field that is **not inside a container**. Container depth is counted from the opens and closes preceding the field, so a JSON attribute value that quotes another component's message is excluded here even though it is prose-shaped. This key decides first. 2. **Widest matched phrase**: a per-field maximum, rather than a sum, over library-matched, multi-character tokens that are not [reserved](https://doc.log10x.com/run/transform/symbol/#symbolsequencereserved "list of terms to ignore when scanning symbol tokens"). This is the widest single library-matched phrase the origin accounts for. 3. **Distinct known tokens**: the count of distinct library-known tokens anywhere on the line that the origin explains. This is the coverage term. 4. **Character total**: the combined character length of those distinct tokens. 5. **Span length**: the run length of the selected span. This is the only run-length key, it sits last, and the comparator reaches it only when the first four keys all tie. Prose ranks ahead of bulk because text inside a container is citation, not authorship. A twenty-token error value carried as an attribute does not outrank the two-token statement it is attached to. Coverage ranks ahead of length because the more places a word appears across a code base, the less that word says about where a line came from. A word appearing in one file identifies that file; a word appearing in hundreds identifies nothing. Ranking by how many distinct known words an origin explains picks the origin that best accounts for the line. One tie-break runs after the ranking rather than as a sixth key. Where the winner ties its nearest rival on all five, the tied group yields to a candidate whose unit the line's own authorship region names. ### Evidence gate A typed selection ([`symbolContexts`](#symbolcontexts "types of symbols to search") of `log,exec`) claims an origin: this source statement wrote this line. That claim stands on prose evidence exclusively, and a candidate whose prose score is below two is rejected rather than crowned. **A line with no prose evidence gets no typed origin and keeps its raw fallback identity.** The lone `any` context asserts nothing about provenance and gates at one token instead, so thin fragments still receive an identity. ### Emission Selection picks the origin. Emission then builds the value in three stages. - **Anchor**: the winner's widest matched prose field, emitted verbatim. The message region runs forward from the anchor's own start and stops at the first container open, newline, or second timestamp; where that boundary falls inside the anchor field the region collapses back to the field. The anchor is **exempt from reserved filtering**: applying it here renames the statement, turning `no baggage found in context` into `no_found`. Timestamps, variables and repeats are still dropped. - **Suffix**: distinct non-reserved tokens from the winner's other matched fields, each token counted once however often it repeats, up to a budget of sixteen. - **Padding**: forward and backward from the emitted region while the [`symbolMaxLen`](#symbolmaxlen "max number of output chars") budget allows. Reserved filtering governs here and in the suffix, which is where a generic word is genuinely noise. The backward walk stops where the statement starts: at the nearest preceding timestamp, or at a token that closes a container or ends a line. Everything before that point is preamble, the node ids, request ids, thread and logger names a format prints ahead of the message, and it stays out of the pattern. Every stage, and the `any` fallback below, skips an **identifier fragment**: a library word whose neighbour carries a digit, directly or across a single `-`, `_` or `.`. The `C` in the node id `R21-M0-N4-C:J05-U11` and the `req` in `req-b3e2...` are library words, and skipping them keeps one statement on one identity however its ids are spelled. Detection reads only the line, so the identity stays a function of the line and the library. The [`symbolContexts`](#symbolcontexts "types of symbols to search") list filters which symbol contexts participate. Contexts are evaluated in a single pass, so list order acts as a filter rather than a precedence chain. The `inputField` parameter limits searches to specific JSON fields. Setting `inputField: log` searches only within the log field content. ### Repeatability The comparator is deterministic: every key is a content-derived integer. Three details bound that behavior. - A full five-key tie that the authorship pass also leaves undecided falls back to candidate insertion order, which holds stable for a given engine build and is not a documented ordering. - Two truncation caps can hide a true origin: [`symbolMaxOrigins`](https://doc.log10x.com/run/transform/symbol/#symbolmaxorigins "max number of source/binary origins to list per symbol") (default 64, the cap that binds at runtime) and [`maxSymbolUnitsPerToken`](https://doc.log10x.com/run/symbol/#maxsymbolunitspertoken "number of symbol units to retrieve when searching for the origin of an TenXTemplate symbol sequence") (default 128, approximate, stopping in the low 130s). - When the selected sequence comes back as a single token, the module re-runs the selection under the `any` context, which concatenates all symbol tokens in range, minus identifier fragments, and bypasses the comparator. The claim the engine supports is scoped: the same engine version, the same symbol library, the same configuration and the same event yield the same pattern. ## :material-fingerprint: Pattern identity: pattern vs template Four terms are easy to conflate. They are distinct: - **Pattern** (`symbolMessage`), the selection described above: a **subset** of representing tokens chosen from the template, not the whole line. Short and legible (e.g. `Receive ListRecommendations for product ids`). It is the unit of cost attribution. - **`pattern_hash`** (alias: `tenx_hash`), the hash of the `symbolMessage` (the [`symbolMessageHashField`](#symbolmessagehashfield "field name to assign with the hash of the symbol sequence value"), default `tenx_hash`). This is the **stable, user-facing identity** that tools and metrics key on. It is stable because it keys on the representing **subset**: it stays constant across deploys, restarts, pod renames, and format drift, and many template variants that share the representing tokens collapse to the **same** `pattern_hash`. - **Template**, the full `$`-marked structural shape of the line (every token, with variable slots marked `$`). A single pattern sits over a **set** of templates, one per format variant present in the data. - **`template_hash`**, the engine-internal fingerprint of a template's field-set. It exists only to join encoded events back to their entry in `templates.json` at decode time. It is **not** the stable identity, it is **many-to-one** with the pattern, and it should never be surfaced to a user or agent as the identifier. Use `pattern_hash` for that. ### Field names by surface An encoded event opens with a leading segment, and that segment always carries the template join key, the value a decoder uses to rebuild the original line from its `templates.json` entry. Each integration handles that field to suit its own schema: the Splunk app extracts it as `tenx_hash`, and the Elasticsearch plugin looks the key up in its `l1es_dml` template index. The pattern-level identity is a separate value, the hash of the selected pattern, written to the field named by [`symbolMessageHashField`](#symbolmessagehashfield "field name to assign with the hash of the symbol sequence value") and also defaulting to `tenx_hash`. The name therefore appears on more than one surface, carrying the join key on an encoded event in Splunk and the pattern hash on an enriched event out of the engine. The two values answer different questions: the join key says which template rebuilds this line, and the pattern hash says which pattern this line belongs to. Building on this process, here's how it applies to real events: === ":simple-opentelemetry: OTel Demo" **Kubernetes Example:** ```json { "stream": "stderr", "log": "2025-04-17 14:32:40,287 INFO [main] [recommendation_server.py:47] - Receive ListRecommendations for product ids:['L9ECAV7KIM', '0PUK6V6EV0']", "docker": { "container_id": "9c04355088aa168abb1a074b696ad15366c254602be8cbb69299e1e87d3bcffb" }, "kubernetes": { "container_name": "recommendationservice", "namespace_name": "default" } } ``` **Extracted Message:** `Receive_ListRecommendations_for_product_ids` === ":material-apache-kafka: Kafka" **Kafka Controller Event:** ```json { "stream": "stdout", "log": "[2025-08-01 22:19:30,905] INFO [controller-1-to-controller-registration-channel-manager]: Recorded new controller, from now on will use node 0.0.0.0:9093 (id: 1 rack: null) (kafka.server.NodeToControllerRequestThread)", "docker": { "container_id": "79af0d7ce5f3c159411c6a15ee2d9044f3559bd2fe1630f8a6640d4c2cc87771" }, "kubernetes": { "container_name": "kafka", "namespace_name": "default", "pod_name": "kafka-549545757c-2lmxv", "container_image": "ghcr.io/open-telemetry/demo:2.0.2-kafka" } } ``` **Extracted Message:** `channel_manager_Recorded_new_controller_from_now_on_will_use_node_id_rack` === ":simple-opensearch: OpenSearch" **OpenSearch PeerFinder Event:** ```json { "stream": "stdout", "log": "[2025-08-01T22:19:24,590][INFO ][o.o.d.PeerFinder ] [opensearch-0] setting findPeersInterval to [1s] as node commission status = [true] for local node [{opensearch-0}{N_KuFBFGRmSnettsBzOX3Q}{3XUyt5iPRMKvzuHPCaPFyg}{192.168.57.56}{192.168.57.56:9300}{dimr}{shard_indexing_pressure_enabled=true}]", "docker": { "container_id": "b6f244ebdaa72d7565b8944a1aad79cd5ac06ac767e4e603145a5e4bfd121883" }, "kubernetes": { "container_name": "opensearch", "namespace_name": "default", "pod_name": "opensearch-0", "container_image": "docker.io/opensearchproject/opensearch:2.19.0" } } ``` **Extracted Message:** `commission_status_local_node_opensearch_shard_indexing_pressure_enabled` === ":material-web: Web" **HTTP Access Log Event:** ```json { "stream": "stdout", "log": "192.168.43.96 - - [01/Aug/2025:22:21:50 +0000] \"GET /products/LensCleaningKit.jpg HTTP/1.1\" 200 101928 \"http://frontend-proxy:8080/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/133.0.0.0 Safari/537.36\"", "docker": { "container_id": "ac37d50d39857193f5d2ff92872f1c022d3b746fa721233eccd1b4aae7d26a8b" }, "kubernetes": { "container_name": "image-provider", "namespace_name": "default", "pod_name": "image-provider-58c6f8444-q4c8p", "container_image": "ghcr.io/open-telemetry/demo:2.0.2-image-provider" } } ``` **Extracted Message:** `frontend_proxy_Mozilla_X11_Linux_x86_AppleWebKit_KHTML_like_Gecko_Safari` :material-github: See the [JavaScript implementation](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/initialize/message/message-template.js){target="\_blank"} of this module on Github. ___ ## :material-rocket-launch-outline: Applications 💰 **Cost tracking**: Identifies high-volume event types consuming log budgets with the [Dev app](https://doc.log10x.com/apps/dev/) app 📈 **Cost control**: Apply intelligent filtering using the [Receiver](https://doc.log10x.com/apps/receiver/) app to prevent over-billing 🤖 **Multi-platform analytics**: Feed patterns into AIOps and monitoring systems via [metric outputs](https://doc.log10x.com/run/output/metric/ "Write aggregated TenXSummary instances to metric outputs (e.g., Prometheus, Datadog)") for Datadog, CloudWatch, and Prometheus 🔄 **Automatic adaptation**: Updates automatically with code changes using [symbol libraries](https://doc.log10x.com/compile/link/#symbol-library). No manual regex pattern configuration and maintenance ## :material-wrench-outline: Config Files To configure the Symbol Message Calculator module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [message/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/initialize/message/config.yaml "message/config.yaml"){target="\_blank"}.
Edit Online
Edit config.yaml Locally
```yaml # 🔟❎ 'run' symbol lookup configuration # Configure a symbol origin lookup to enrich TenXObjects # To learn more see https://doc.log10x.com/run/initialize/message/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/initialize/message # ============================== Symbol Options =============================== symbol: # 'types' specifies the types of symbols that participate in the origin search. The listed # contexts are evaluated in a single pass, so list order acts as a filter rather than a precedence chain. # Supported values (case insensitive), PACKAGE, CLASS, METHOD, LOG, ENUM, CONST, TEXT, EXEC, ANY # To learn more see https://doc.log10x.com/run/transform/symbol/#contexts contexts: log,exec # 'messageField' specifies the field name to assign the message pattern to target TenXObjects: the symbol # tokens of the origin ranked highest by coverage, meaning the count of distinct library-known tokens it # explains. Span length is the last tiebreak. The origin file name itself goes to 'originField'. messageField: message_pattern # 'messageHashField' specifies the field name to assign a stable, URL-safe pattern hash (xxHash64 of the messageField value, base64url, 11 chars). Safe as a SIEM query term or forwarder filter key. # # Left unset on purpose. The hash enrichment is always computed and carried on the TenXObject under # its internal name 'tenx_hash', so metric dimensions, the aggregation key and the MCP artifacts are # unaffected. Setting this key is what names the field on the wire, and on the forwarder receive path # that is also what puts it on the record returned to the forwarder; leave it unset and the forwarder # gets its record back byte-for-byte. Uncomment and name a field to opt in: # # messageHashField: tenx_hash # 'maxLen' specifies the max char len of the output messageField maxLen: 120 # ============================== Message Options ============================== # 'negators' specifies patterns that mark a log line as boilerplate/continuation (stack trace frames, # indented detail, banners) so the message-template skips its own message-pattern compute for them. # The check fires only for templates that don't already pass the (groupSize > 1) || groupHead arms; # i.e., for single-event templates with no head signal. A non-negator line in that case (e.g., bare # "Hello World", "Heartbeat received from node-7") still gets a pattern; a negator-matching orphan # (e.g., a lone "\tat com.foo.Bar(...)") does not. message: negators: - " " # any indented line (stack frames, indented app detail, multi-line JSON, banners) - "\t" # real tab byte, for msgpack/forward inputs that preserve raw whitespace - "\\t" # literal backslash+t, for file/JSON inputs that keep escape sequences raw - "Caused by: " # Java chained exception (column 0) - "Suppressed: " # Java suppressed exception (column 0) - "--- End of " # .NET inner exception marker (column 0) - "created by " # Go goroutine creator (column 0) - "#0 " # PHP stack frame depth 0 - "#1 " # PHP stack frame depth 1 - "#2 " # PHP stack frame depth 2 - "#3 " # PHP stack frame depth 3 - "#4 " # PHP stack frame depth 4 - "#5 " # PHP stack frame depth 5 ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") the Symbol Message Calculator: |Name|Description| |---|---| |[symbolContexts](#symbolcontexts "types of symbols to search")|Types of symbols to search| |[symbolMaxLen](#symbolmaxlen "max number of output chars")|Max number of output chars| |[symbolMessageField](#symbolmessagefield "field name to assign with symbol sequence value")|Field name to assign with symbol sequence value| |[symbolMessageHashField](#symbolmessagehashfield "field name to assign with the hash of the symbol sequence value")|Field name to assign with the hash of the symbol sequence value| |[symbolOriginField](#symboloriginfield "field name to assign with symbol message origin file")|Field name to assign with symbol message origin file| |[symbolSkeletonField](#symbolskeletonfield "field name for the pattern rendering hint (the message shape with one marker per identity token)")|Field name for the pattern rendering hint (the message shape with one marker per identity token)| |[messageNegators](#messagenegators "list of strings that, when matched at the start of a log line's text, mark it as low-value continuation/boilerplate that does NOT receive its own message pattern.")|List of strings that, when matched at the start of a log line's text, mark it as low-value continuation/boilerplate that does NOT receive its own message pattern.| ### :material-menu-right-outline:**`symbolContexts`** Types of symbols to search. |Type|Default| |---|---| |String|log,exec| Specifies a comma delimited list of [symbol contexts](https://doc.log10x.com/run/transform/symbol/#contexts) values. This is argument is passed as the `symbolContexts` argument value for the [symbolSequence](https://doc.log10x.com/api/js/#TenXObject+symbolSequence) function. The listed contexts are evaluated in a single pass, so list order acts as a filter rather than a precedence chain. ### :material-menu-right-outline:**`symbolMaxLen`** Max number of output chars. |Type|Default| |---|---| |Number|0| Specifies the max number of chars of the [symbolMessageField](https://doc.log10x.com/run/initialize/message/#symbolmessagefield "field name to assign with symbol sequence value"). Set 0 to unlimited. ### :material-menu-right-outline:**`symbolMessageField`** Field name to assign with symbol sequence value. |Type|Default| |---|---| |String|symbolMessage| Specifies the name of the field to assign the return value of [symbolSequence](https://doc.log10x.com/api/js/#TenXObject+symbolSequence) function when invoking it with the a symbolContext argument value of [symbolContexts](https://doc.log10x.com/run/initialize/message/#symbolcontexts "types of symbols to search"). For an example, see [symbolInputField](https://doc.log10x.com/run/initialize/message/#symbolinputfield). ### :material-menu-right-outline:**`symbolMessageHashField`** Field name to assign with the hash of the symbol sequence value. |Type|Default| |---|---| |String|""| Specifies the name of the field to assign a stable, URL-safe hash of the [symbolMessageField](https://doc.log10x.com/run/initialize/message/#symbolmessagefield "field name to assign with symbol sequence value") value, computed via [TenXString.hash](https://doc.log10x.com/api/js/#TenXString.hash). The value is an 11-character base64url identifier (xxHash64 of the UTF-8 bytes, big-endian, no padding). It is one-way and reproduces byte-for-byte outside the engine, so it is safe to use directly as a deduplication key, metric dimension, analyzer query term, or forwarder filter key. **Unset by default.** The enrichment field itself is always present on the TenXObject under the internal name `tenx_hash`, so metrics dimensions, the aggregation key and the MCP artifacts carry the pattern hash whether or not this option is set. What the option controls is the *wire* name, and with it whether the hash reaches a coupled log forwarder: on the [forwarder receive path](https://doc.log10x.com/run/input/forwarder/ "Report, receive and optimize events collected by log forwarders") the record goes back to the forwarder byte-for-byte as it arrived unless this option names a field, and then that field is spliced onto the returned record. Set it to `tenx_hash` to get the historical name, or to any other name to both rename the enrichment field and ship it, for example `symbolMessageHashField my_custom_hash`. ### :material-menu-right-outline:**`symbolOriginField`** Field name to assign with symbol message origin file. |Type|Default| |---|---| |String|| Specifies the name of the field to assign the return value of [symbolOrigin](https://doc.log10x.com/api/js/#TenXObject+symbolOrigin) function. ### :material-menu-right-outline:**`symbolSkeletonField`** Field name for the pattern rendering hint (the message shape with one marker per identity token). |Type|Default| |---|---| |String|""| When set, each template also carries a `skeleton`: the message region of the log line with `~` standing in for every token that went into the message pattern and `$` for every variable. A consumer holding the pattern and the skeleton reconstructs the original line for display — punctuation, spacing and dropped stopwords included — so a readable pattern needs no access to the logs themselves. Unset (the default) computes and emits nothing. ### :material-menu-right-outline:**`messageNegators`** List of strings that, when matched at the start of a log line's text, mark it as low-value continuation/boilerplate that does NOT receive its own message pattern. |Type|Default| |---|---| |List|\[\]| A list of strings that, when matched at the start of a log line's text, mark it as a known continuation pattern (e.g., stack trace frame, indented detail, banner) that should not receive its own message pattern. Negators have two consumers. The message template's gate evaluates them when the event is neither part of a multi-event group (groupSize \> 1) nor a group head: in that case, an event still receives a pattern unless it matches a negator. This is the "Hello World vs orphan stack frame" discriminator. The group template's head detection also consults them as a veto BETWEEN the timestamp and severity checks: a negator-prefixed line cannot be promoted to group head by severity inference alone (a token like `info` inside an object dump), so continuation fragments weld into their parent instead of minting orphan heads. Both consumers are inert until this list is configured — there is no default. - ` ` - any indented line (universal continuation signal). - ` ` - any tab-prefixed line (Java/Go stack frames etc.). - `Caused by: ` - Java chained exception (column 0). - `Suppressed: ` - Java suppressed exception (column 0). - `--- End of ` - .NET inner exception marker. - `created by ` - Go goroutine creator. - `#0 `..`#5 ` - PHP stack frames (depth 0-5).
:material-github: This module is defined in [message/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/initialize/message/module.yaml "message/module.yaml"){target="\_blank"}.