---
title: "Log10x App for Splunk"
description: "Splunk app for transparent expansion of 10x-optimized events"
source: "https://github.com/log-10x/splunk-app"
icon: "material/greater-than"

---

Search and visualize [compact](https://doc.log10x.com/run/transform/#compact) events in Splunk; the original lines of each event up to 256 KB expand back exactly at [search time](https://www.log10x.com/blog/cutting-splunk-log-storage/){target="_blank"}. This [open-source app](https://github.com/log-10x/splunk-app){target="_blank"} expands compact events at search time: classic dashboards keep their panels unchanged, a search puts the `tx` macro in front of its terms, and an alert created in the app's Compile Alert view compiles once so the scheduler runs it natively.

Use the [Reporter](https://doc.log10x.com/apps/reporter/) (DaemonSet, pre-ingestion) or the [MCP server](https://github.com/log-10x/log10x-mcp){target="_blank"}'s SIEM-sample tool (agentless, via Splunk REST API) to identify optimization opportunities in your existing Splunk data.

<div style="display: flex; justify-content: center; align-items: center; gap: 12px; flex-wrap: wrap;" markdown>
[:material-github: View on GitHub](https://github.com/log-10x/splunk-app){ .md-button target="_blank" }
</div>

---

## :material-chart-line: Analytics Dashboard

The Analytics dashboard shows compact event counts, active templates, compression ratio and storage saved. It finds compact events through the `tenx-events` macro.

<figure markdown="span">
  ![Analytics dashboard](../../../assets/splunk-app.png){ loading=lazy }
  <figcaption>The Analytics dashboard</figcaption>
</figure>

| Panel | Description |
|--------|-------------|
| **Total Encoded Events** | Compact events indexed |
| **Active Templates** | Templates in the KV Store |
| **Compression Ratio** | Original size over compact size, on the latest 5,000 events |
| **Estimated Storage Savings** | Bytes saved and percentage reduction, on the latest 10,000 events |
| **Event Volume, Last 7 Days** | Compact events per hour |
| **Top 10 Templates by Usage** | Template hashes with the most events |
| **Inflation Success Rate** | Share of the latest 1,000 events that expand |

---

## :material-lightbulb-outline: How It Works

A [compact event](https://doc.log10x.com/run/transform/#compact) carries a template hash and the event's variable values. The app reads the template hash into the field `tenx_hash`; in the engine's metrics, the label of that name holds the pattern hash. The constant words live in the KV Store, so the app puts them back at search time and re-applies the search to the expanded lines.

A word that exists only in a template is not in the index, so a plain keyword search over compact data finds the events where the word is a value and misses the others. To search by what the lines say, write the app's `tx` macro in front of the search terms, after the index and any time terms. It runs as native SPL in the search bar, Dashboard Studio panels, saved searches and alerts: it selects the compact events that can match, expands them, and applies the search to the original lines. Classic dashboards keep their panels unchanged: the browser sends each panel's search to the app, which rewrites it on the server. An alert created in the **Compile Alert** view compiles once and then runs as an ordinary saved search.

The Receiver compacts each record's log message and leaves its other fields, such as the Kubernetes namespace, pod and container, as they arrived. The forwarder sends the compact message as the Splunk event and those fields as indexed fields, so they stay searchable by name and in `tstats` without the app. The [splunk-app README](https://github.com/log-10x/splunk-app#receiver-side-configuration){target="_blank"} covers the Receiver settings the app needs.

### :material-arrow-right-bold-outline: Ingestion Flow

The Receiver [compacts](https://doc.log10x.com/run/transform/#compact) events at the edge, and Splunk ingests them into Splunk with reduced payload size:

<div style="text-align: center;">

```mermaid
graph LR
    A["<div style='font-size: 14px;'>🗜️ Compact</div><div style='font-size: 10px;'>Compact Events</div>"] --> B["<div style='font-size: 14px;'>📡 Ingest</div><div style='font-size: 10px;'>HEC</div>"]
    B --> C["<div style='font-size: 14px;'>📋 KV Store</div><div style='font-size: 10px;'>Templates</div>"]
    B --> D["<div style='font-size: 14px;'>💾 Index</div><div style='font-size: 10px;'>Encoded</div>"]

    classDef edge fill:#7c3aed88,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef ingest fill:#9333ea88,stroke:#7c3aed,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef store fill:#2563eb88,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A edge
    class B ingest
    class C,D store
```

</div>

🗜️ **Compact**: [Receiver (Compact mode)](index.md) compacts events, extracting repetitive patterns into templates

📡 **Ingest**: the forwarder sends compact events and templates to Splunk over HEC

📋 **KV Store**: [Templates](https://doc.log10x.com/run/template/) stored in `tenx_dml` collection for lookup at search time

💾 **Index**: [compact events](https://doc.log10x.com/run/transform/#compact), each a template hash and its values, with the record's metadata as indexed fields

### :material-magnify: Search Flow

Two paths reach the same rewrite, then [expand](https://doc.log10x.com/run/transform/#expand) the events:

<div style="text-align: center;">

```mermaid
graph LR
    E["<div style='font-size: 14px;'>📊 Dashboard</div><div style='font-size: 10px;'>Panel SPL</div>"] --> F["<div style='font-size: 14px;'>🪝 Hook</div><div style='font-size: 10px;'>Intercept</div>"]
    E2["<div style='font-size: 14px;'>👤 Search Bar</div><div style='font-size: 10px;'>tx</div>"] --> G
    F --> G["<div style='font-size: 14px;'>🔄 Transform</div><div style='font-size: 10px;'>Prefilter + Macro</div>"]
    G --> H["<div style='font-size: 14px;'>📖 Expand</div><div style='font-size: 10px;'>Expand</div>"]
    H --> I["<div style='font-size: 14px;'>📊 Results</div><div style='font-size: 10px;'>Full Data</div>"]

    classDef user fill:#059669,stroke:#047857,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef hook fill:#f59e0b,stroke:#d97706,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef result fill:#ea580c88,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class E,E2 user
    class F,G hook
    class H,I result
```

</div>

📊 **Dashboard**: `dashboard.js` loads on every classic dashboard in the app carrying it and points the panel's search at the app's REST endpoint, via `$.ajaxSetup`

👤 **Search Bar**: `` `tx` `` in front of the search terms runs a one-row command that reads the search and looks its words up in the templates; the same search works in Dashboard Studio panels, saved searches and alerts

🔄 **Transform**: each search word is resolved against the templates, and the SPL becomes a hash prefilter plus the `tenx-inflate` macro

📖 **Expand**: the macro joins [compact events](https://doc.log10x.com/run/transform/#compact) with templates from the KV Store

📊 **Results**: fully expanded events, with the original search re-applied so only true matches remain

**Scheduled alerts** run server-side, where the browser hook never fires. An alert created in the **Compile Alert** view, or through the `/tenx-alert` endpoint, is compiled once at save time into a native saved search (a template-hash prefilter plus the [`tenx-inflate`](https://github.com/log-10x/splunk-app) macro), so the scheduler runs an ordinary search with no per-run proxy. The **Compile Alert** view and the `/tenx-alert` endpoint handle this, and a recompile pass converts `| tx` and `| tenxsearch` alerts to native form and refreshes their prefilters as new templates appear. See [SAVE_TIME_ALERTS.md](https://github.com/log-10x/splunk-app/blob/main/SAVE_TIME_ALERTS.md){target="_blank"}.

---

## :material-rocket-launch-outline: Quickstart

Get encoded events flowing to Splunk in under 15 minutes.

??? tenx-step-deploy "Step 1: Install Splunk App"

    Clone the repository and install to your Splunk apps directory:

    ``` { .console .copy }
    git clone https://github.com/log-10x/splunk-app.git
    cp -r splunk-app/tenx-for-splunk $SPLUNK_HOME/etc/apps/
    $SPLUNK_HOME/bin/splunk restart
    ```

    In a distributed deployment, install the app on the search heads and on the indexers or heavy forwarders that parse the compact events, where it sets each event's `_time`.

    **Prerequisites:**

    | Requirement | Description |
    |-------------|-------------|
    | Splunk Enterprise | 9.4 through 10.6 (bundled Python 3.9 through 3.13) |
    | Admin Access | Required for app installation and KV Store setup |

    ??? tenx-checklist "Verify Installation"

        After restart, confirm the app appears in Splunk:

        1. Navigate to **Apps** → **Manage Apps**
        2. Search for "10x"; the list shows **Log10x App**
        3. Click the app name to open the Analytics dashboard

??? tenx-step-config "Step 2: Create HEC Tokens"

    Create two HTTP Event Collector tokens in Splunk: one for templates, one for encoded events.

    **Navigate to HEC Settings:**

    1. Go to **Settings** → **Data inputs** → **HTTP Event Collector**
    2. Click **Global Settings** and ensure HEC is **Enabled**
    3. Note the **HTTP Port Number** (default: 8088)

    **Create Templates Token:**

    | Setting | Value |
    |---------|-------|
    | Name | `tenx-templates` |
    | Source type | `tenx_dml_raw_json` |
    | Index | `tenx_dml` |
    | Enable indexer acknowledgement | Off |

    **Create Encoded Events Token:**

    | Setting | Value |
    |---------|-------|
    | Name | `tenx-encoded` |
    | Source type | `tenx_encoded` |
    | Index | Your target index |
    | Enable indexer acknowledgement | Off |

    The app recognizes compact events by their sourcetype. To use a sourcetype other than `tenx_encoded`, add the app's extraction to that sourcetype's `props.conf` stanza:

    ```ini title="props.conf"
    [my_sourcetype]
    REPORT-tenx = tenx-hash-vars-extraction
    ```

    !!! tip "Point the dashboards at these events"
        The app's Analytics and Diagnostics dashboards find compact events through the `tenx-events` search macro, which defaults to `index=* sourcetype=tenx_encoded`. Set it to your index and sourcetype under **Settings** → **Advanced search** → **Search macros**, for example `index=my_compact_index sourcetype=tenx_encoded`.

    !!! warning "Create Index First"
        If `tenx_dml` index doesn't exist, create it via **Settings** → **Indexes** → **New Index** before creating the token.

??? tenx-step-link "Step 3: Configure Forwarder"

    Set three options in the Receiver's configuration. Each keeps expansion exact:

    | Option | Value | Why |
    |--------|-------|-----|
    | `varMaxRecurIndexes` | `0` | Every variable value travels in the event |
    | `timestampZone` | `UTC` | The app renders timestamps in UTC |
    | `maxPerObject` | `1` | The app stores one timestamp format per template |

    A template that breaks the first or third rule stays compact and carries `tenx_expand_refused`, naming the reason.

    Configure your log forwarder to send encoded events and templates to Splunk.

    === ":simple-fluentbit: Fluent-bit"

        Start from the Fluent Bit overlay in [Deploy](../deploy.md), and add `receiverOptimize true varMaxRecurIndexes 0 timestampZone UTC timestampMaxPerObject 1` to the sidecar's arguments. In that overlay, set `Merge_Log Off` in the `kubernetes` filter (and remove `Keep_Log Off`), so a JSON log line stays in `log`, where the Receiver compacts it. Then replace the `[OUTPUT]` block that writes `tenx.*` to stdout: the filter below goes under `filters:` and the two outputs under `outputs:`.

        Each record returns from the Receiver with `tenx.` in front of its tag. Its `log` value is compact and its other fields are unchanged. The events output sends the compact message as the Splunk event and the Kubernetes metadata as indexed fields.

        ```toml title="filters"
        # Lift the Kubernetes metadata to top-level keys the output can name.
        [FILTER]
            Name          nest
            Match_Regex   ^tenx\.(?!tenx-template).*
            Operation     lift
            Nested_under  kubernetes
            Add_prefix    k8s_
        ```

        ```toml title="outputs"
        # Templates: one record per new template, for the KV Store.
        [OUTPUT]
            Name              splunk
            Match             tenx.tenx-template
            Host              your-splunk-host.com
            Port              8088
            Splunk_Token      YOUR_TEMPLATES_HEC_TOKEN
            event_index       tenx_dml
            event_source      fluent-bit-tenx
            event_sourcetype  tenx_dml_raw_json
            TLS               On
            TLS.Verify        On

        # Compact events: the message as the event, the metadata as indexed fields.
        [OUTPUT]
            Name              splunk
            Match_Regex       ^tenx\.(?!tenx-template).*
            Host              your-splunk-host.com
            Port              8088
            Splunk_Token      YOUR_ENCODED_EVENTS_HEC_TOKEN
            event_index       your_logs_index
            event_sourcetype  tenx_encoded
            event_key         $log
            event_field       stream $stream
            event_field       k8s_namespace_name $k8s_namespace_name
            event_field       k8s_pod_name $k8s_pod_name
            event_field       k8s_pod_id $k8s_pod_id
            event_field       k8s_container_name $k8s_container_name
            event_field       k8s_container_image $k8s_container_image
            event_field       k8s_docker_id $k8s_docker_id
            event_field       k8s_host $k8s_host
            TLS               On
            TLS.Verify        On
        ```

        A record key reaches Splunk only through `event_key` or an `event_field` line. To keep pod labels, add a second `nest` filter with `Nested_under k8s_labels` and `Add_prefix k8s_label_`, then an `event_field` line for each label you search on.

        !!! warning "Match the returning tags"
            Both outputs match tags that start with `tenx.`, the prefix the overlay's egress input adds. An output that matches the original tags sends records that never passed through the Receiver, and one that matches `tenx-template` without the prefix misses the templates, so no event expands.

    === ":simple-fluentd: Fluentd"

        Start from the Fluentd overlay in [Deploy](../deploy.md), add `receiverOptimize true varMaxRecurIndexes 0 timestampZone UTC timestampMaxPerObject 1` to the sidecar's arguments, and add the Splunk HEC output plugin to the Fluentd image (`fluent-gem install fluent-plugin-splunk-hec`). In that overlay, replace the `@OUTPUT` label in `04_outputs.conf` with the one below.

        Templates return tagged `tenx-template`. Every other record keeps its tag, with its `log` value compact and its other fields unchanged.

        ```xml title="04_outputs.conf"
        <label @OUTPUT>
          # Templates: one record per new template, for the KV Store.
          <match tenx-template>
            @type splunk_hec
            hec_host your-splunk-host.com
            hec_port 8088
            hec_token YOUR_TEMPLATES_HEC_TOKEN
            index tenx_dml
            sourcetype tenx_dml_raw_json
            source fluentd-tenx
          </match>

          # HEC rejects a whole batch that contains an empty event.
          <filter **>
            @type grep
            <exclude>
              key log
              pattern /\A\z/
            </exclude>
          </filter>

          # Lift the Kubernetes metadata to top-level keys the output can name.
          <filter **>
            @type record_transformer
            enable_ruby true
            <record>
              k8s_namespace_name ${record.dig("kubernetes", "namespace_name")}
              k8s_pod_name ${record.dig("kubernetes", "pod_name")}
              k8s_container_name ${record.dig("kubernetes", "container_name")}
              k8s_container_image ${record.dig("kubernetes", "container_image")}
              k8s_host ${record.dig("kubernetes", "host")}
            </record>
          </filter>

          # Compact events: the message as the event, the metadata as indexed fields.
          <match **>
            @type splunk_hec
            hec_host your-splunk-host.com
            hec_port 8088
            hec_token YOUR_ENCODED_EVENTS_HEC_TOKEN
            index your_logs_index
            sourcetype tenx_encoded
            <fields>
              stream
              k8s_namespace_name
              k8s_pod_name
              k8s_container_name
              k8s_container_image
              k8s_host
            </fields>
            <format>
              @type single_value
              message_key log
            </format>
          </match>
        </label>
        ```

        A record key reaches Splunk only through `message_key` or the `<fields>` list. Add a `record_transformer` line and a `<fields>` entry for each further key you search on.

??? tenx-step-dashboard "Step 4: Verify End-to-End"

    Run these SPL queries to confirm everything is working:

    **1. Check templates are arriving:**
    ```spl
    index=tenx_dml sourcetype=tenx_dml_raw_json | head 10
    ```
    :material-check-circle-outline:{ .success } Expected: JSON events with `templateHash` and `template` fields

    **2. Check KV store is populated:**
    ```spl
    | inputlookup tenx-dml-lookup | stats count
    ```
    :material-check-circle-outline:{ .success } Expected: Count > 0 (up to 5 minutes for the first templates)

    **3. Check compact events expand:**
    ```spl
    index=your_logs_index `tx` | head 10
    ```
    :material-check-circle-outline:{ .success } Expected: Full expanded events, not `~hash,var1,var2...` format

    **4. Check the metadata arrives as indexed fields:**
    ```spl
    | tstats count where index=your_logs_index sourcetype=tenx_encoded by k8s_namespace_name
    ```
    :material-check-circle-outline:{ .success } Expected: one row per namespace, with no app involved

    !!! tip "Templates indexed before the app"
        The "Consume KV" saved search stores new templates every 5 minutes. To load templates indexed before the app was installed, run this search once as an admin or power user:

        ```spl
        index=tenx_dml sourcetype=tenx_dml_raw_json earliest=-30d | sendalert tenx_dml_to_kv
        ```

        Running it again is safe: templates already stored are skipped.

??? tenx-step-search "Step 5: Search from the Search Bar"

    Write the `tx` macro in front of the search terms, after the index and any time terms:

    ```spl
    index=your_logs_index earliest=-4h `tx` "connection refused" NOT retry
    ```
    :material-check-circle-outline:{ .success } Expected: expanded events on the Events tab

    The words after `tx` are ordinary SPL, applied to the expanded lines. `NOT`, `OR`, `AND`, phrases, parenthesised groups, wildcards, field conditions including `IN (...)`, `TERM()`, `CASE()` and a trailing pipeline behave as they do on the original data. The same search works in Dashboard Studio panels, saved searches and alerts. Time modifiers go before `tx`: after it they would match no event, so the search stops with a message, as it does for a comma outside quotes. Where a deployment names its template index differently, edit the `tenx-templates` macro beside `tenx-events`.

    !!! info "Name compact indexes"
        Without `tx`, a keyword search over compact data finds only the events where the word is a value. Keep compact indexes out of users' default index sets and name them in searches.

    !!! tip "Classic dashboards need no macro"
        Classic dashboards in an app carrying `dashboard.js` keep their panel SPL unchanged. See [Expansion in Other Splunk Apps](#expansion-in-other-splunk-apps) below.

---

## :material-clipboard-check-outline: Verification Checklist

Use this checklist to diagnose issues at each stage of the pipeline.

??? tenx-checklist "Templates Arriving in Splunk"

    **Test Query:**
    ```spl
    index=tenx_dml sourcetype=tenx_dml_raw_json earliest=-1h | stats count
    ```

    | Result | Meaning | Action |
    |--------|---------|--------|
    | Count > 0 | Templates arriving | Proceed to KV store check |
    | Count = 0 | No templates received | Check forwarder config, HEC token, network connectivity |

    **Check raw event format:**
    ```spl
    index=tenx_dml sourcetype=tenx_dml_raw_json | head 1
    ```

    Expected format:
    ```json
    {"templateHash":"#ABC123xy","template":"INFO $0 - User $1 logged in from $2"}
    ```

??? tenx-checklist "KV Store Populated"

    **Test Query:**
    ```spl
    | inputlookup tenx-dml-lookup | stats count
    ```

    | Result | Meaning | Action |
    |--------|---------|--------|
    | Count > 0 | KV store populated | Proceed to expansion check |
    | Count = 0 | Templates not processed | Check "Consume KV" saved search |

    **Verify saved search configuration:**

    1. Go to **Settings** → **Searches, reports, and alerts**
    2. Find "Consume KV" in the Log10x App
    3. Verify search string includes: `index=tenx_dml sourcetype=tenx_dml_raw_json`
    4. Click **Run** to manually trigger

??? tenx-checklist "Events Expanding Correctly"

    **Test Query:**
    ```spl
    index=your_logs_index `tx` | head 5
    ```

    | Result | Meaning | Action |
    |--------|---------|--------|
    | Full expanded events | Expansion working | None |
    | `~hash,var1,var2...` format | Events not expanding | Check hash exists in KV store |
    | `{"log":"~hash,..."}` | JSON wrapping | Add `event_key $log` to forwarder |

    **Check if template hash exists:**
    ```spl
    | inputlookup tenx-dml-lookup | search pattern_hash="YOUR_HASH_FROM_EVENT"
    ```

---

## :material-alert-outline: Known Limits

The app stores each compact event whole up to 262,144 bytes (256 KB), the `TRUNCATE` value on the `tenx_encoded` sourcetype. Splunk cuts a longer event at index time and adds no marker to it; the cut drops the event's trailing values, so it expands to fewer lines than the original held. A single compact event passes that size when the Receiver groups many lines of one message into it, such as a very long multi-line stack trace. The longest event in the OpenTelemetry demo sample is 25,472 bytes.

A compact event's `_time` is the time in its original line wherever the template has a timestamp, and the time Splunk indexed it otherwise. The app sets it while events are parsed, so in a distributed deployment it goes on the indexers or heavy forwarders as well as the search heads.

**Words that live in a template need `tx`.** A plain search such as `index=app_logs error` matches a compact event only when the word is one of its values; a word that is part of the template's constant text is in the KV Store, not the event. `tx` looks each word up in the templates and selects the compact events that carry it as a value or through their template, then applies the full search to the expanded lines. A word a rendered timestamp could print (one or two digits, a year, an epoch, month and day names), a word with a character outside a-z and 0-9 (`Zürich`), and a word carried by more than 5,000 templates select nothing on their own; a `NOT`, a field comparison, or an `OR` with any of these selects every event in range. Expanding every event in range costs about 40 seconds per 150,000 compact events on one laptop, against about 1.5 to 4 seconds for a search with a usable word. A number that appears only inside a timestamp (milliseconds, a date) is not found, since the compact event holds the timestamp as an epoch. The template lookup runs inside a subsearch, under Splunk's subsearch time limit of 60 seconds by default; a dictionary of tens of thousands of templates has not been measured. A second compact sourcetype is searched once it is added to the `tenx-events` macro in the indexed form `sourcetype::name`.

**Time modifiers go before `tx`, and a comma after it goes in quotes.** The words after `tx` are the arguments of a later `search` command, where `earliest=` and `latest=` match no event and a comma outside quotes is matched literally. `tx` stops such a search with a message instead of returning nothing.

**`tx` inside another subsearch or macro.** Inside `append`, `join` or a subsearch, or wrapped in another macro, `tx` cannot read its own words. It then expands every compact event in range and applies the words after, with a warning: the same result, more slowly.

**Events and counts.** A compact event is the event the Receiver formed. A line without a timestamp of its own, such as a line of a stack trace, joins the line before it from the same source, so a stack trace is one event and a count counts events. Words joined with AND can match on different lines of one event, and `NOT` excludes the whole event.

**Sources read by an add-on stay raw.** Enterprise Security and the CIM data models read events through vendor add-ons (Windows, Palo Alto, Zeek, CrowdStrike, Okta and the like), whose extractions run on the raw event at search time, before anything in the search pipeline can expand a compact one. Keep those sourcetypes out of the Receiver. Cisco sourcetypes stay raw as well: Splunk meters them at a reduced rate only under their own sourcetype, and compacted they would meter about 1.3 times their raw cost.

**The templates index needs read access.** `tx` resolves words against `index=tenx_dml`. A role that cannot read it still finds words that are values, and finds nothing for words that are template text, with no message. Give every role that searches compact data read access to the templates index, and keep the `tenx-templates` macro and `dest_dml_index` in `tenx_config.conf` pointing at the same index.

**Replication lag.** With `replicate = true` on the collection, the expansion runs on the indexers, which read the templates from the knowledge bundle. A new template reaches the indexers with the next bundle, about a minute; after a search head restart the first bundle takes one to two minutes, and a search run before it arrives returns no compact events and no message. The bundle carries the whole collection, about 3.5 KB per template.

**Field conditions inside a grouped event.** A field in a line that the Receiver grouped under a timestamped line of its own container is read from the first record of the group only. Field conditions on the lines inside a multi-line event (a stack trace, a dump) do not match them.

---

## :material-bug-outline: Troubleshooting

??? tenx-troubleshoot "Templates Not Populating KV Store"

    **Symptom:** `| inputlookup tenx-dml-lookup` returns 0 rows even though templates exist in `tenx_dml` index.

    **Common Causes:**

    | Cause | Solution |
    |-------|----------|
    | Templates indexed before the app was installed, or while "Consume KV" was down | Run `index=tenx_dml sourcetype=tenx_dml_raw_json earliest=-30d | sendalert tenx_dml_to_kv` once |
    | Saved search disabled | Enable in **Settings** → **Searches, reports, and alerts** → "Consume KV" |
    | Templates in another index | Point the templates HEC token at `tenx_dml` |

??? tenx-troubleshoot "Events Show Raw Encoded Format"

    **Symptom:** Events display as `~#ABC123,value1,value2,value3` instead of expanded JSON/text.

    **Common Causes:**

    | Cause | Solution |
    |-------|----------|
    | Template hash not in KV store | Wait 5 minutes for "Consume KV", or run the backfill search above |
    | Hash mismatch (different data) | Verify encoded events match templates in `tenx_dml` |
    | Search ran without expansion | On a dashboard, ensure `dashboard.js` is in that app; in the search bar, write `tx` in front of the search terms |
    | A `tx` search stops with a message about time modifiers or a comma | Move `earliest=` and `latest=` before `tx`; remove the comma or put the text in quotes |

    **Debug hash lookup:**
    ```spl
    index=your_logs_index
    | rex field=_raw "~(?<hash>[^,]+)"
    | lookup tenx-dml-lookup pattern_hash as hash OUTPUT pattern
    | table _time hash pattern
    ```

??? tenx-troubleshoot "Events Wrapped in JSON"

    **Symptom:** Events arrive as `{"log":"~hash,var1,var2..."}` instead of raw encoded format.

    === ":simple-fluentbit: Fluent-bit"

        **Cause:** Missing `event_key $log` in Splunk output configuration.

        **Solution:** Add `event_key $log` to your encoded events OUTPUT section:

        ```toml hl_lines="8"
        [OUTPUT]
            Name              splunk
            Match_Regex       ^tenx\.(?!tenx-template).*
            Host              your-splunk-host.com
            Port              8088
            Splunk_Token      YOUR_TOKEN
            event_index       your_index
            event_key         $log
            TLS               On
        ```

        This tells Fluent Bit to send only the `log` field content, not wrap it in JSON.

    === ":simple-fluentd: Fluentd"

        **Cause:** Missing `<format>` block in Splunk HEC output configuration.

        **Solution:** Add `format single_value` with `message_key log` to your encoded events match:

        ```xml hl_lines="8-11"
        <match **>
          @type splunk_hec
          hec_host your-splunk-host.com
          hec_port 8088
          hec_token YOUR_TOKEN
          index your_index
          sourcetype tenx_encoded
          <format>
            @type single_value
            message_key log
          </format>
        </match>
        ```

        This tells Fluentd to send only the `log` field content, not the full JSON record.

??? tenx-troubleshoot "HEC Connection Refused or Timeout"

    **Symptom:** Forwarder logs show connection errors to Splunk HEC.

    **Diagnostic Steps:**

    1. **Test HEC endpoint:**
       ```bash
       curl -k https://your-splunk:8088/services/collector/health
       ```
       Expected: `{"text":"HEC is healthy","code":17}`

    2. **Test with token:**
       ```bash
       curl -k https://your-splunk:8088/services/collector \
         -H "Authorization: Splunk YOUR_TOKEN" \
         -d '{"event":"test"}'
       ```
       Expected: `{"text":"Success","code":0}`

    | Error | Cause | Solution |
    |-------|-------|----------|
    | Connection refused | HEC disabled | Enable in **Settings** → **Data inputs** → **HTTP Event Collector** → **Global Settings** |
    | 401 Unauthorized | Invalid token | Verify token value, check token is enabled |
    | 403 Forbidden | Token restrictions | Check token's allowed indexes and sourcetypes |
    | SSL error | Forwarder does not trust the HEC certificate | Install Splunk's CA certificate on the forwarder host; turn verification off only on a test instance |

---

## :material-code-tags: Expansion in Other Splunk Apps

`dashboard.js` loads on every classic dashboard in the app that carries it. The Log10x App ships it, so its own dashboards expand. To expand classic dashboards in another app, copy the file there:

``` { .console .copy }
cp $SPLUNK_HOME/etc/apps/tenx-for-splunk/appserver/static/dashboard.js \
   $SPLUNK_HOME/etc/apps/YOUR_APP/appserver/static/
```

Restart Splunk to load it.

!!! info "Search bar and Dashboard Studio"
    The search bar and Dashboard Studio use the `tx` macro, as in the Quickstart.

??? info "Programmatic Hook for Custom Views"

    For custom classic dashboards, include the hook in JavaScript:

    ``` { .javascript }
    require([
        "/static/app/tenx-for-splunk/javascript/search/tenx_search_hook.js"
    ], function(TenxSearchHook) {
        TenxSearchHook.execute(true);
    });
    ```

---

## :material-puzzle-outline: Components

| Component | Description |
|-----------|-------------|
| [**Search Hook**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/appserver/static/dashboard.js){target="\_blank"} | JavaScript module routing classic dashboard searches to the REST endpoint |
| [**tx Macro**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/default/macros.conf){target="\_blank"} | Written in front of a search to search compact events from the search bar, Dashboard Studio, saved searches and alerts |
| [**tenxprefilter Command**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/bin/tenx_prefilter.py){target="\_blank"} | Run once per search by `tx`; reads the search and selects the compact events to expand |
| [**Search Handler**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/bin/tenx_search_handler.py){target="\_blank"} | REST endpoint transforming SPL queries |
| [**KV Store**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/default/collections.conf){target="\_blank"} | Template patterns for event expansion |
| [**Inflate Macro**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/default/macros.conf){target="\_blank"} | SPL macro joining events with templates |
| [**Consume KV Search**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/default/savedsearches.conf){target="\_blank"} | Scheduled search storing new templates in the KV Store |
| [**Analytics Dashboard**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/default/data/ui/views/tenx_dashboard.xml){target="\_blank"} | Compression and template metrics |
| [**Diagnostics Dashboard**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/default/data/ui/views/tenx_diagnostics.xml){target="\_blank"} | Troubleshooting and verification tools |

---

<br/>:material-github: This app is open source. View on [GitHub](https://github.com/log-10x/splunk-app "Log10x App"){target="\_blank"}.
