--- title: "10x app for Splunk" description: "Splunk app for transparent expansion of 10x-optimized events" source: "https://github.com/log-10x/splunk-app" icon: "material/greater-than" --- Search and visualize [compact](https://doc.log10x.com/run/transform/#compact) events in Splunk; the original lines expand back exactly at search time. This [open-source app](https://github.com/log-10x/splunk-app){target="_blank"} expands compact events at search time: classic dashboards keep their panels unchanged, the search bar wraps a query in one command, and scheduled alerts compile once at save time so the scheduler runs them natively. Use the [Reporter](https://doc.log10x.com/apps/reporter/) (DaemonSet, pre-ingestion) or the [MCP server](https://github.com/log-10x/log10x-mcp){target="_blank"}'s SIEM-sample tool (agentless, via Splunk REST API) to identify optimization opportunities in your existing Splunk data.
[:material-github: View on GitHub](https://github.com/log-10x/splunk-app){ .md-button target="_blank" }
--- ## :material-chart-line: Analytics Dashboard The app includes a built-in analytics dashboard providing real-time visibility into optimization performance, storage savings, and ROI metrics.
![10x Analytics Dashboard](../../../assets/splunk-app.png){ loading=lazy }
10x Analytics Dashboard showing encoding metrics and ROI
| Metric | Description | |--------|-------------| | **Total Encoded Events** | Count of optimized events ingested | | **Active Templates** | Number of unique patterns in KV Store | | **Reduction Ratio** | Average reduction factor across all events | | **Storage Savings** | Estimated bytes saved and percentage reduction | | **Event Volume Over Time** | Trend comparison of encoded vs original volume | | **Top Templates by Usage** | Most frequently matched patterns | | **Expansion Success Rate** | Percentage of events successfully expanded | --- ## :material-lightbulb-outline: How It Works A [compact event](https://doc.log10x.com/run/transform/#compact) carries a template hash and the event's variable values. The constant words live in the KV Store, so the app puts them back at search time and re-applies the search to the expanded lines. Classic dashboards are rewritten in the browser and keep their panels unchanged. Splunk's search page loads no app JavaScript, so a query typed there is wrapped in the `tenxsearch` command, which runs the same rewrite server-side. Scheduled alerts compile once at save time and then run as ordinary saved searches. Compact mode encodes the entire event for Splunk, including the Kubernetes metadata, into a single compact line. Splunk bills against uncompressed ingest volume, so compacting everything maximizes license-cost reduction. The Receiver uses a different mode for Elasticsearch from the same forwarder pipeline; there, only the log message is encoded so plugin queries on metadata still work. The [splunk-app README](https://github.com/log-10x/splunk-app#receiver-side-configuration){target="_blank"} covers the Receiver setting. ### :material-arrow-right-bold-outline: Ingestion Flow Events are [compact](https://doc.log10x.com/run/transform/#compact) at the edge and ingested into Splunk with reduced payload size:
```mermaid graph LR A["
🗜️ Compact
Compact Events
"] --> B["
📡 Ingest
UF / HEC
"] B --> C["
📋 KV Store
Templates
"] B --> D["
💾 Index
Encoded
"] classDef edge fill:#7c3aed88,stroke:#6d28d9,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef ingest fill:#9333ea88,stroke:#7c3aed,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef store fill:#2563eb88,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8 class A edge class B ingest class C,D store ```
🗜️ **Compact**: [Receiver (Compact mode)](index.md) compacts events, extracting repetitive patterns into templates 📡 **Ingest**: Encoded events forwarded to Splunk via Universal Forwarder or HEC with reduced payload size 📋 **KV Store**: [Templates](https://doc.log10x.com/run/template/) stored in `tenx_dml` collection for lookup at search time 💾 **Index**: [Compact events](https://doc.log10x.com/run/transform/#compact) stored with template hash references ### :material-magnify: Search Flow Two paths reach the same rewrite, then [expand](https://doc.log10x.com/run/transform/#expand) the events:
```mermaid graph LR E["
📊 Dashboard
Panel SPL
"] --> F["
🪝 Hook
Intercept
"] E2["
👤 Search Bar
tenxsearch
"] --> G F --> G["
🔄 Transform
Prefilter + Macro
"] G --> H["
📖 Expand
Expand
"] H --> I["
📊 Results
Full Data
"] classDef user fill:#059669,stroke:#047857,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef hook fill:#f59e0b,stroke:#d97706,color:#ffffff,stroke-width:2px,rx:8,ry:8 classDef result fill:#ea580c88,stroke:#c2410c,color:#ffffff,stroke-width:2px,rx:8,ry:8 class E,E2 user class F,G hook class H,I result ```
📊 **Dashboard**: `dashboard.js` loads on every classic dashboard in the app carrying it and points the panel's search at the app's REST endpoint, via `$.ajaxSetup` 👤 **Search Bar**: `| tenxsearch searchstring="..."` runs the same rewrite server-side, and works in saved searches and the REST API too 🔄 **Transform**: each search word is resolved against the templates, and the SPL becomes a hash prefilter plus the `tenx-inflate` macro 📖 **Expand**: the macro joins [compact events](https://doc.log10x.com/run/transform/#compact) with templates from the KV Store 📊 **Results**: fully expanded events, with the original search re-applied so only true matches remain **Scheduled alerts** run server-side, where the browser hook never fires. They are instead compiled once at save time into a native saved search (a template-hash prefilter plus the [`tenx-inflate`](https://github.com/log-10x/splunk-app) macro), so the scheduler runs an ordinary search with no per-run proxy. The **10x Compile Alert** view and the `/tenx-alert` endpoint handle this, and a recompile pass migrates legacy alerts to native form and refreshes their prefilters as new templates appear. See [SAVE_TIME_ALERTS.md](https://github.com/log-10x/splunk-app/blob/main/SAVE_TIME_ALERTS.md){target="_blank"}. --- ## :material-rocket-launch-outline: Quickstart Get encoded events flowing to Splunk in under 15 minutes. ??? tenx-step-deploy "Step 1: Install Splunk App" Clone the repository and install to your Splunk apps directory: ``` { .console .copy } git clone https://github.com/log-10x/splunk-app.git cp -r splunk-app/tenx-for-splunk $SPLUNK_HOME/etc/apps/ $SPLUNK_HOME/bin/splunk restart ``` **Prerequisites:** | Requirement | Description | |-------------|-------------| | Splunk Enterprise | Version 8.x, 9.x, or 10.x (bundled Python 3.7 through 3.13) | | Admin Access | Required for app installation and KV Store setup | ??? tenx-checklist "Verify Installation" After restart, confirm the app appears in Splunk: 1. Navigate to **Apps** → **Manage Apps** 2. Search for "10x" - you should see **10x for Splunk** 3. Click the app name to open the Analytics dashboard ??? tenx-step-config "Step 2: Create HEC Tokens" Create two HTTP Event Collector tokens in Splunk - one for templates, one for encoded events. **Navigate to HEC Settings:** 1. Go to **Settings** → **Data inputs** → **HTTP Event Collector** 2. Click **Global Settings** and ensure HEC is **Enabled** 3. Note the **HTTP Port Number** (default: 8088) **Create Templates Token:** | Setting | Value | |---------|-------| | Name | `tenx-templates` | | Source type | `tenx_dml_raw_json` | | Index | `tenx_dml` | | Enable indexer acknowledgement | Off | **Create Encoded Events Token:** | Setting | Value | |---------|-------| | Name | `tenx-encoded` | | Source type | Select appropriate for your logs | | Index | Your target index | | Enable indexer acknowledgement | Off | !!! warning "Create Index First" If `tenx_dml` index doesn't exist, create it via **Settings** → **Indexes** → **New Index** before creating the token. ??? tenx-step-link "Step 3: Configure Forwarder" Configure your log forwarder to send encoded events and templates to Splunk. === ":simple-fluentbit: Fluent-bit" **Include 10x compact configuration:** ```toml title="fluent-bit.conf" # Include the 10x compact sidecar configuration @INCLUDE ${TENX_MODULES}/pipelines/run/modules/input/forwarder/fluentbit/conf/tenx-sidecar.conf ``` **Configure Splunk outputs:** ```toml title="fluent-bit.conf" # ========================= TEMPLATES OUTPUT ========================= # Routes templates to tenx_dml index for KV store population [OUTPUT] Name splunk Match tenx-template Host your-splunk-host.com Port 8088 Splunk_Token YOUR_TEMPLATES_HEC_TOKEN event_index tenx_dml event_source fluent-bit-tenx event_sourcetype tenx_dml_raw_json TLS On TLS.Verify Off # ========================= ENCODED EVENTS OUTPUT ==================== # Routes encoded log events to your target index # CRITICAL: Use 'event_key $log' to prevent JSON wrapping [OUTPUT] Name splunk Match_Regex ^(?!tenx-template).* Host your-splunk-host.com Port 8088 Splunk_Token YOUR_ENCODED_EVENTS_HEC_TOKEN event_index your_logs_index event_sourcetype your_sourcetype event_key $log TLS On TLS.Verify Off ``` !!! danger "Critical: event_key $log" Without `event_key $log`, Fluent Bit wraps events in JSON: `{"log":"~hash,var1,var2..."}`. The Splunk app cannot expand this format. Always use `event_key $log` for encoded events output. === ":simple-fluentd: Fluentd" **Prerequisites:** Install the Splunk HEC output plugin: ```bash gem install fluent-plugin-splunk-hec ``` **Include 10x compact configuration:** ```xml title="fluentd.conf" # Include the 10x compact exec_filter configuration # This launches tenx as a subprocess and handles event encoding @include "#{ENV['TENX_MODULES']}/pipelines/run/modules/input/forwarder/fluentd/conf/tenx-optimize-unix.conf" ``` **Configure Splunk outputs:** ```xml title="fluentd.conf" # ========================= TEMPLATES OUTPUT ========================= # Routes templates to tenx_dml index for KV store population @type splunk_hec host your-splunk-host.com port 8088 token YOUR_TEMPLATES_HEC_TOKEN index tenx_dml sourcetype tenx_dml_raw_json use_ssl true ssl_verify false # ========================= ENCODED EVENTS OUTPUT ==================== # Routes encoded log events to your target index # IMPORTANT: Use 'format single_value' with 'message_key log' to send # only the encoded log content, not the full JSON record @type splunk_hec host your-splunk-host.com port 8088 token YOUR_ENCODED_EVENTS_HEC_TOKEN index your_logs_index sourcetype your_sourcetype use_ssl true ssl_verify false @type single_value message_key log ``` !!! danger "Critical: format single_value" Without the `` block, Fluentd wraps events in JSON: `{"log":"~hash,var1,var2...","tag":"..."}`. The Splunk app cannot expand this format. Use `format single_value` with `message_key log` to send only the encoded log content. === ":simple-opentelemetry: OTel Collector" **Configure exporters in your OTel config:** ```yaml title="otel-collector.yaml" exporters: # Templates exporter splunk_hec/templates: endpoint: "https://your-splunk-host.com:8088/services/collector" token: "YOUR_TEMPLATES_HEC_TOKEN" index: "tenx_dml" sourcetype: "tenx_dml_raw_json" tls: insecure_skip_verify: true # Encoded events exporter splunk_hec/encoded: endpoint: "https://your-splunk-host.com:8088/services/collector" token: "YOUR_ENCODED_EVENTS_HEC_TOKEN" index: "your_logs_index" sourcetype: "your_sourcetype" tls: insecure_skip_verify: true service: pipelines: logs/templates: receivers: [tenx_templates] exporters: [splunk_hec/templates] logs/encoded: receivers: [tenx_encoded] exporters: [splunk_hec/encoded] ``` ??? tenx-step-dashboard "Step 4: Verify End-to-End" Run these SPL queries to confirm everything is working: **1. Check templates are arriving:** ```spl index=tenx_dml sourcetype=tenx_dml_raw_json | head 10 ``` :material-check-circle-outline:{ .success } Expected: JSON events with `templateHash` and `template` fields **2. Check KV store is populated:** ```spl | inputlookup tenx-dml-lookup | stats count ``` :material-check-circle-outline:{ .success } Expected: Count > 0 (may take up to 2 minutes for first templates) **3. Check compact events expand:** ```spl | tenxsearch searchstring="index=your_logs_index" | head 10 ``` :material-check-circle-outline:{ .success } Expected: Full expanded events, not `~hash,var1,var2...` format !!! tip "First Templates Take Time" The "Consume KV" saved search runs every 2 minutes. If KV store is empty, wait 2-3 minutes and check again. ??? tenx-step-search "Step 5: Search from the Search Bar" Splunk's search page loads no app JavaScript, so the hook cannot reach the search bar. Wrap the search in the `tenxsearch` command instead: ```spl | tenxsearch searchstring="index=your_logs_index sourcetype=tenx_encoded error" ``` :material-check-circle-outline:{ .success } Expected: expanded events on the Events tab Escape a quoted phrase inside the wrapper: ```spl | tenxsearch searchstring="index=your_logs_index \"connection refused\"" ``` `NOT`, `OR`, `AND`, parenthesised groups, wildcards, field conditions, inline `earliest=`/`latest=` and a trailing pipeline all behave as they do on the original data. The command also runs in saved searches, alerts and the REST API. !!! warning "A search without the command returns zero" A bare search on a compact sourcetype finds nothing and reports no matches, because the constant words are in the KV Store rather than the event. Keep compact indexes out of users' default index sets and name them so the omission is visible. !!! tip "Dashboards need no command" Classic dashboards in an app carrying `dashboard.js` keep their panel SPL unchanged. See [Expansion in Other Splunk Apps](#expansion-in-other-splunk-apps) below. --- ## :material-clipboard-check-outline: Verification Checklist Use this checklist to diagnose issues at each stage of the pipeline. ??? tenx-checklist "Templates Arriving in Splunk?" **Test Query:** ```spl index=tenx_dml sourcetype=tenx_dml_raw_json earliest=-1h | stats count ``` | Result | Meaning | Action | |--------|---------|--------| | Count > 0 | Templates arriving | Proceed to KV store check | | Count = 0 | No templates received | Check forwarder config, HEC token, network connectivity | **Check raw event format:** ```spl index=tenx_dml sourcetype=tenx_dml_raw_json | head 1 ``` Expected format: ```json {"templateHash":"#ABC123xy","template":"INFO $0 - User $1 logged in from $2"} ``` ??? tenx-checklist "KV Store Populated?" **Test Query:** ```spl | inputlookup tenx-dml-lookup | stats count ``` | Result | Meaning | Action | |--------|---------|--------| | Count > 0 | KV store populated | Proceed to expansion check | | Count = 0 | Templates not processed | Check "Consume KV" saved search | **Verify saved search configuration:** 1. Go to **Settings** → **Searches, reports, and alerts** 2. Find "Consume KV" in the 10x for Splunk app 3. Verify search string includes: `index=tenx_dml sourcetype=tenx_dml_raw_json` 4. Click **Run** to manually trigger ??? tenx-checklist "Events Expanding Correctly?" **Test Query:** ```spl index=your_logs_index | head 5 ``` | Result | Meaning | Action | |--------|---------|--------| | Full expanded events | Working correctly | Done! | | `~hash,var1,var2...` format | Events not expanding | Check hash exists in KV store | | `{"log":"~hash,..."}` | JSON wrapping | Add `event_key $log` to forwarder | **Check if template hash exists:** ```spl | inputlookup tenx-dml-lookup | search pattern_hash="YOUR_HASH_FROM_EVENT" ``` --- ## :material-bug-outline: Troubleshooting ??? tenx-troubleshoot "Templates Not Populating KV Store" **Symptom:** `| inputlookup tenx-dml-lookup` returns 0 rows even though templates exist in `tenx_dml` index. **Common Causes:** | Cause | Solution | |-------|----------| | Missing `index=` in saved search | Edit "Consume KV" search to include `index=tenx_dml` | | Time window too narrow | Increase `dispatch.earliest_time` from `-3m` to `-1h` or `-1d` | | Special characters in hash | Upgrade Splunk app - URL encoding fix required | | Saved search disabled | Enable in **Settings** → **Searches** → "Consume KV" | **Fix saved search manually:** Edit `$SPLUNK_HOME/etc/apps/tenx-for-splunk/local/savedsearches.conf`: ```ini [Consume KV] search = index=tenx_dml sourcetype=tenx_dml_raw_json dispatch.earliest_time = -1d ``` Restart Splunk or run the search manually to apply. ??? tenx-troubleshoot "Events Show Raw Encoded Format" **Symptom:** Events display as `~#ABC123,value1,value2,value3` instead of expanded JSON/text. **Common Causes:** | Cause | Solution | |-------|----------| | Template hash not in KV store | Wait 2 min for "Consume KV" to run, or trigger manually | | Hash mismatch (different data) | Verify encoded events match templates in `tenx_dml` | | Search ran without expansion | On a dashboard, ensure `dashboard.js` is in that app; in the search bar, wrap the search in `tenxsearch` | **Debug hash lookup:** ```spl index=your_logs_index | rex field=_raw "~(?[^,]+)" | lookup tenx-dml-lookup pattern_hash as hash OUTPUT pattern | table _time hash pattern ``` ??? tenx-troubleshoot "Events Wrapped in JSON" **Symptom:** Events arrive as `{"log":"~hash,var1,var2..."}` instead of raw encoded format. === ":simple-fluentbit: Fluent-bit" **Cause:** Missing `event_key $log` in Splunk output configuration. **Solution:** Add `event_key $log` to your encoded events OUTPUT section: ```toml hl_lines="8" [OUTPUT] Name splunk Match_Regex ^(?!tenx-template).* Host your-splunk-host.com Port 8088 Splunk_Token YOUR_TOKEN event_index your_index event_key $log TLS On ``` This tells Fluent Bit to send only the `log` field content, not wrap it in JSON. === ":simple-fluentd: Fluentd" **Cause:** Missing `` block in Splunk HEC output configuration. **Solution:** Add `format single_value` with `message_key log` to your encoded events match: ```xml hl_lines="8-11" @type splunk_hec host your-splunk-host.com port 8088 token YOUR_TOKEN index your_index sourcetype your_sourcetype @type single_value message_key log ``` This tells Fluentd to send only the `log` field content, not the full JSON record. ??? tenx-troubleshoot "HEC Connection Refused or Timeout" **Symptom:** Forwarder logs show connection errors to Splunk HEC. **Diagnostic Steps:** 1. **Test HEC endpoint:** ```bash curl -k https://your-splunk:8088/services/collector/health ``` Expected: `{"text":"HEC is healthy","code":17}` 2. **Test with token:** ```bash curl -k https://your-splunk:8088/services/collector \ -H "Authorization: Splunk YOUR_TOKEN" \ -d '{"event":"test"}' ``` Expected: `{"text":"Success","code":0}` | Error | Cause | Solution | |-------|-------|----------| | Connection refused | HEC disabled | Enable in **Settings** → **Data inputs** → **HTTP Event Collector** → **Global Settings** | | 401 Unauthorized | Invalid token | Verify token value, check token is enabled | | 403 Forbidden | Token restrictions | Check token's allowed indexes and sourcetypes | | SSL error | Certificate issues | Set `TLS.Verify Off` (dev) or configure proper certs (prod) | --- ## :material-code-tags: Expansion in Other Splunk Apps `dashboard.js` loads on every classic dashboard in the app that carries it. The 10x for Splunk app ships it, so its own dashboards expand. To expand classic dashboards in another app, copy the file there: ``` { .console .copy } cp $SPLUNK_HOME/etc/apps/tenx-for-splunk/appserver/static/dashboard.js \ $SPLUNK_HOME/etc/apps/YOUR_APP/appserver/static/ ``` Restart Splunk to load it. !!! info "What copying the hook does not cover" The search bar and Dashboard Studio load no app JavaScript, so they are unaffected by the copy. Wrap the search in the `tenxsearch` command there, as in the Quickstart. ??? info "Programmatic Hook for Custom Views" For custom classic dashboards, include the hook in JavaScript: ``` { .javascript } require([ "/static/app/tenx-for-splunk/javascript/search/tenx_search_hook.js" ], function(TenxSearchHook) { TenxSearchHook.execute(true); }); ``` --- ## :material-puzzle-outline: Components | Component | Description | |-----------|-------------| | [**Search Hook**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/appserver/static/dashboard.js){target="\_blank"} | JavaScript module routing classic dashboard searches to the REST endpoint | | [**tenxsearch Command**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/bin/tenxsearch.py){target="\_blank"} | Generating command running the same rewrite from the search bar, saved searches and alerts | | [**Search Handler**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/bin/tenx_search_handler.py){target="\_blank"} | REST endpoint transforming SPL queries | | [**KV Store**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/default/collections.conf){target="\_blank"} | Template patterns for event expansion | | [**Inflate Macro**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/default/macros.conf){target="\_blank"} | SPL macro joining events with templates | | [**Consume KV Search**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/default/savedsearches.conf){target="\_blank"} | Scheduled search populating KV store from templates | | [**Analytics Dashboard**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/default/data/ui/views/tenx_dashboard.xml){target="\_blank"} | Encoding metrics and ROI visualization | | [**Diagnostics Dashboard**](https://github.com/log-10x/splunk-app/blob/main/tenx-for-splunk/default/data/ui/views/tenx_diagnostics.xml){target="\_blank"} | Troubleshooting and verification tools | ---
:material-github: This app is open source. View on [GitHub](https://github.com/log-10x/splunk-app "10x for Splunk"){target="\_blank"}.