--- icon: material/check-circle-outline title: "Compatibility" --- Splunk dashboards, Universal Forwarders, testing on your own environment, HEC integration, Splunk Enterprise on-premises, and the full Splunk Cloud KV-Store pilot checklist. ??? tenx-compat "Does 10x work with my existing Splunk dashboards and queries" **Yes.** 10x preserves all Splunk field mappings and metadata. **Fields preserved:** - `_time`, `host`, `source`, `sourcetype` - `_raw`, `index` - All field extractions (props.conf, transforms.conf) - Custom metadata and tags **Functionality preserved:** - Saved searches and dashboards require zero changes - Scheduled alerts are compiled once at save time (via the **10x Compile Alert** view) so they run natively server-side - SPL queries return the same results **How optimization works:** The 10x Engine uses [template-based](https://doc.log10x.com/run/transform/){target="_blank"} encoding, not field removal. Each repeated log event is replaced by a compact reference to its template while maintaining full searchability in Splunk. ??? tenx-compat "Does this work with Universal Forwarders" **Yes.** The 10x [Receiver](https://doc.log10x.com/apps/receiver/){target="_blank"} runs as a file relay in front of [Universal Forwarders](https://doc.log10x.com/run/input/forwarder/splunkUF/){target="_blank"}: Fluent Bit with 10x reads the app's log folder, optimizes, and writes to a second folder the Universal Forwarder monitors. The [Reporter](https://doc.log10x.com/apps/reporter/){target="_blank"} deploys as a DaemonSet alongside them. Fluent Bit with 10x reads the app's logs, optimizes, and writes them to a second folder. The Universal Forwarder tails that folder with its standard `inputs.conf` and ships to the indexers over S2S. Forwarding directly to Splunk HEC is a separate option, not part of the Universal Forwarder relay. **Compatibility:** Universal Forwarder 8.x, 9.x, 10.x. No changes to Universal Forwarder configuration required. Works with existing HEC endpoints and token authentication. ??? tenx-compat "How do I test this on my Splunk environment" 1. **[Cost POC](https://doc.log10x.com/apps/mcp/)**, install the 10x MCP and ask it to run a POC on your Splunk logs. The engine runs on your own machine, results in minutes. No account, no credit card. 2. **[Reporter](https://doc.log10x.com/apps/reporter/)**, Deploy as a DaemonSet alongside your forwarder for cost visibility before logs reach Splunk. Alternatively, the [MCP server](https://github.com/log-10x/log10x-mcp){target="_blank"}'s SIEM-sample tool connects to Splunk via REST API for agentless Splunk-side analysis (no DaemonSet). 3. **[Receiver](https://doc.log10x.com/apps/receiver/deploy/)**, Deploy via Helm chart alongside your forwarder. Filter mode (sampling) or [Compact mode](https://doc.log10x.com/apps/receiver/compact/) (lossless shrink via the [10x for Splunk](https://doc.log10x.com/apps/receiver/compact/splunk/) app). ~30 min setup. 4. **[Retriever](https://doc.log10x.com/apps/retriever/)**, Route events to S3, stream selected data to Splunk on-demand. Each step is independent, start with the cost POC to see your reduction ratio, then move to production when ready. ??? tenx-compat "How does this integrate with Splunk HEC" The 10x Engine sits between your log forwarder and Splunk HEC: ``` App → Fluentd/UF → 10x Receiver (Compact mode) → Splunk HEC → Indexer ``` **Execution (Receiver):** - Runs as a file relay in front of the Universal Forwarder, or as a sidecar container with Fluentd or OTel - Filters (lossy) or compacts (lossless) logs in-memory before forwarding to HEC - Uses standard HEC token authentication - Can also forward via splunktcp to indexers (standard UF protocol) **Observability (Reporter):** - Deploys as a DaemonSet alongside your forwarder (logs never pass through it) - Emits pre-ingestion cost insight metrics to the time-series backend you configure (Prometheus, Datadog, CloudWatch, Elastic) - Fails independently, logs continue flowing to Splunk if Reporter goes down **Fetching from S3:** - [Retriever](https://doc.log10x.com/apps/retriever/){target="_blank"} routes events to your own S3 and returns the exact offloaded events to HEC on-demand **Agentless Splunk analysis:** The [MCP server](https://github.com/log-10x/log10x-mcp){target="_blank"}'s SIEM-sample tool queries Splunk REST API for cost analysis without deploying a DaemonSet. **Compatibility:** Works with Splunk Cloud Platform (all regions), Splunk Enterprise 8.x, 9.x, and 10.x. No changes to HEC configuration required. ??? tenx-compat "Works with Splunk Enterprise on-premises" **Yes.** Works with both Splunk Cloud and Splunk Enterprise on-premises (8.x, 9.x, 10.x), across single-instance, distributed, and clustered deployments. On-premises, the Receiver sidecar and Reporter DaemonSet forward to HEC or to Heavy Forwarders, using either HEC token or Heavy Forwarder S2S authentication. **Agentless Splunk analysis via [log10x-mcp](https://github.com/log-10x/log10x-mcp){target="_blank"} for on-premises:** - SIEM-sample tool queries Splunk Enterprise REST API - Requires admin or power user credentials - Runs wherever you run an MCP server (workstation, CI, infrastructure) - No data egress to external systems **[Retriever](https://doc.log10x.com/apps/retriever/){target="_blank"} for on-premises:** Works with AWS S3, Azure Blobs, and any S3-compatible [object storage](https://doc.log10x.com/run/input/objectStorage/){target="_blank"}. Returns the exact offloaded events to Splunk Enterprise HEC endpoints on-demand. ??? tenx-compat "Splunk Cloud: KV Store Setup & Pilot Checklist" The [10x for Splunk](https://github.com/log-10x/splunk-app){target="_blank"} app expands compact events at search time using a KV Store collection. Here's the complete setup and pilot validation checklist for Splunk Cloud. **Before You Start** - Admin or Power User access to your Splunk Cloud instance - Splunk Cloud supporting KV Store (all modern instances do) - Ability to create HTTP Event Collector (HEC) tokens - Two HEC tokens configured: one for templates (`tenx_dml_raw_json` sourcetype), one for encoded events **Day 1: App Installation & KV Store Setup** 1. **Install 10x for Splunk app** - [ ] Download from [GitHub](https://github.com/log-10x/splunk-app/releases){target="_blank"} - [ ] Upload via Settings > Apps > Install app from file - [ ] Restart (if prompted by Splunk Cloud) - [ ] Verify: Settings > Apps > Confirm "10x for Splunk" appears in app list 2. **Create KV Store collection** - [ ] Go to Settings > Advanced Search > Collections - [ ] Create new collection named `tenx_dml` - [ ] Schema fields (automatically generated, verify all present): ``` _key (primary key) pattern_hash (string) pattern (string) pattern_parts (array) part_0 (string) pattern_terminator (string) timestamp_format (string) ``` - [ ] Verify: `| inputlookup tenx-dml-lookup | stats count` returns 0 (empty) 3. **Create indexes for template data** - [ ] **Create `tenx_dml` index** (required for storing templates): ``` Settings → Indexes → New Index Name: tenx_dml Data type: Events Max size: 10GB (adjust based on your expected template volume) Retention: 30+ days (templates are reference data, not logs) ``` Verify: `| rest /services/data/indexes | search title="tenx_dml"` - [ ] **Optional: Create separate index for encoded events:** ``` Settings → Indexes → New Index Name: encoded_events (or your preferred name) Data type: Events Max size: Depends on log volume ``` Or use `main` index if preferred (encoded events are searchable until expansion) 4. **Verify props.conf and transforms.conf** - [ ] Check Settings > Field Extractions > Verify `tenx_encoded` sourcetype has `REPORT-tenx` extraction - [ ] Verify transforms.conf has `tenx-hash-vars-extraction` and `tenx-dml-lookup` defined - [ ] If missing, manually add via Settings > Add data > Source type settings **Phase 4: Enable HTTP Event Collector (HEC) globally** 4a. **Enable HEC globally** (required before creating tokens): - [ ] Settings → Data Inputs → HTTP Event Collector → Global Settings - [ ] Toggle "All Tokens" to **ENABLED** - [ ] Set Default Input Port: `8088` (or your custom port) - [ ] Enable SSL: **YES** (recommended for production) - [ ] Click Save - [ ] Verify: `| rest /services/data/inputs/http | search disabled=0` 4b. **Create HEC Token 1 (for templates)**: - [ ] Settings → Data Inputs → HTTP Event Collector → New Token - [ ] Name: `tenx-templates` - [ ] Source Type: `tenx_dml_raw_json` - [ ] Index: `tenx_dml` (created in step 3) - [ ] Indexes allowed: `tenx_dml` (restrict to this index only) - [ ] Disabled: NO - [ ] Click **Save Token** - [ ] Copy the token value (save it for later) 4c. **Create HEC Token 2 (for encoded events)**: - [ ] Settings → Data Inputs → HTTP Event Collector → New Token - [ ] Name: `tenx-encoded` - [ ] Source Type: `tenx_encoded` - [ ] Index: Your target index (where searchable events go) - [ ] Indexes allowed: Your target index - [ ] Disabled: NO - [ ] Click **Save Token** - [ ] Copy the token value (save it for later) **Week 1: Data Ingestion & KV Store Population** 5. **Send template data via HEC** (or via your log forwarder) - [ ] **Via curl** (for testing): ```bash SPLUNK_HOST="your-splunk-cloud.splunkcloud.com" SPLUNK_PORT="8088" HEC_TOKEN="" curl -k https://$SPLUNK_HOST:$SPLUNK_PORT/services/collector/event \ -H "Authorization: Splunk $HEC_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "event": { "templateHash": "abc123def456", "template": "User %s logged in from %s", "templateParts": ["User", "logged in from"] }, "sourcetype": "tenx_dml_raw_json", "index": "tenx_dml" }' ``` - [ ] **Via Fluentd/Fluent Bit:** Configure your forwarder output to send to this HEC endpoint with same token and sourcetype - [ ] Verify templates arriving: `index=tenx_dml sourcetype=tenx_dml_raw_json | head 10` 6. **Send encoded events via HEC** (or via your log forwarder) - [ ] **Via curl** (for testing): ```bash SPLUNK_HOST="your-splunk-cloud.splunkcloud.com" SPLUNK_PORT="8088" HEC_TOKEN="" curl -k https://$SPLUNK_HOST:$SPLUNK_PORT/services/collector/event \ -H "Authorization: Splunk $HEC_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "event": "~abc123def456,admin,192.168.1.1", "sourcetype": "tenx_encoded", "index": "main" }' ``` - [ ] **Via Fluentd/Fluent Bit:** Configure your forwarder output to send to this HEC endpoint with same token and sourcetype - [ ] Verify encoded events arriving: `index=main sourcetype=tenx_encoded | head 10` 7. **Wait for KV Store population** - [ ] The "Consume KV" saved search runs every 2 minutes (automatic) - [ ] Check: `index=_internal savedsearch_name="Consume KV" | table _time, status, result_count` (verify no errors) - [ ] Check KV store: `| inputlookup tenx-dml-lookup | stats count` (should be > 0 after 2-3 min) 8. **Monitor template consumption** - [ ] Run: `sourcetype=tenx_dml_pure | stats count` (confirms templates are searchable; this sourcetype lands in the index set by `dest_dml_index`, `main` by default) - [ ] Run: `| inputlookup tenx-dml-lookup | head 5 | table _key, pattern, timestamp_format` (verify structure) **Week 2: Expansion Validation & Performance Testing** 9. **Test basic expansion** - [ ] Run: `index=your_target_index sourcetype=tenx_encoded | head 10 | \`tenx-inflate\`` - [ ] Verify: All fields (`_raw`, `_time`, `host`, `source`, etc.) are restored to original values - [ ] Check: No `tenx_hash`, `tenx_var_*` fields remain in final output (cleanup working) 10. **Test debug mode** - [ ] Run: `index=your_target_index sourcetype=tenx_encoded | head 1 | \`tenx-inflate-debug\` | table *` - [ ] Verify: `_raw` field matches original (unencoded) log format - [ ] Check: `tenx_ts_sec` correctly detects timestamp precision (milliseconds vs nanoseconds) 11. **Test field extractions and searches post-expansion** - [ ] Run existing saved search/dashboard on expanded data - [ ] Verify: All field extractions work (extractions applied post-expansion) - [ ] Check: Alerts trigger correctly on expanded events - [ ] Compare: Results match pre-optimization historical logs (sample query on same time range) 12. **Measure search performance** - [ ] Run: `index=your_target_index sourcetype=tenx_encoded earliest=-1h | \`tenx-inflate\` | stats count` - [ ] Note: The `tenx-inflate` macro adds a per-event decode step at search time; expect a small overhead relative to searching un-encoded events - [ ] Compare: Same query without expansion vs with expansion, and validate the overhead on your own data volume **Post-Pilot: Production Deployment** 13. **Enable analytics dashboard** - [ ] Open: App launcher > 10x for Splunk > Analytics Dashboard - [ ] Verify: Shows total compact events, reduction ratio, storage savings - [ ] Check: Panels populate when opened (the dashboard runs its searches on view) 14. **Set up monitoring and alerts** - [ ] Monitor KV store size: `| inputlookup tenx-dml-lookup | stats count` (alert on growth or error against the threshold you set) - [ ] Monitor expansion failures: Check tenx app logs for errors - [ ] Optional: Set up dashboard for Splunk license impact (GB before/after) 15. **Risk Mitigation & Rollback** - [ ] **Rollback procedure:** Simply disable 10x for Splunk app: 1. Settings > Apps > 10x for Splunk > Disable 2. Re-run searches without `\`tenx-inflate\`` macro (searches work on encoded raw data until disabled) 3. KV Store collection remains; can re-enable app without data loss - [ ] **Zero data loss:** Encoded events remain in index; templates preserved in KV Store - [ ] **Retention:** Configure KV Store collection retention if needed (Settings > Collections) **Splunk Cloud Limitations & Workarounds** - **No custom Python alert actions**, Covered. App uses standard KV Store and Search hooks (no custom Python required) - **Limited app customization**, App config available in local/default folders; can override via local/ without modifying default/ - **Network egress**, All data stays within Splunk Cloud. No external calls needed after app installation - **KV Store max size**, Bounded by your Splunk KV Store limits, one entry per template. Monitor via `| inputlookup tenx-dml-lookup | stats count`. If approaching your limit, consider archiving old templates **Forwarder Configuration Examples** **Fluentd:** ```yaml @type http_buffered endpoint_url https://:8088/services/collector/event serializer json auth_type basic auth_key "Splunk " flush_interval 10s ``` **Fluent Bit:** ```ini [OUTPUT] Name http Match * Host Port 8088 URI /services/collector/event header Authorization Splunk header Content-Type application/json json_date_key timestamp Format json ``` **Universal Forwarder:** The Universal Forwarder integration uses a file relay pattern, not a custom forwarder output. Fluent Bit with the 10x sidecar reads logs from the original folder, filters and (optionally) compacts them, and writes the processed output to a second folder. The Universal Forwarder monitors that second folder with a standard `inputs.conf` and ships to the indexers over its normal S2S protocol. The UF keeps its existing configuration; no custom outputs are required. **Support & Troubleshooting** - **Templates not in KV Store:** Check saved search logs: `index=_internal savedsearch_name="Consume KV"` - **Expansion returns empty:** Verify template format in `tenx_dml_raw_json` index, ensure KV Store has matching hash - **Performance degradation:** Limit time ranges in searches; filter by `tenx_hash` before expansion for large datasets - **HEC token rejected:** Verify token is enabled and not deleted: `| rest /services/data/inputs/http` - **For detailed troubleshooting:** See [10x for Splunk Troubleshooting Guide](https://github.com/log-10x/splunk-app#troubleshooting){target="_blank"} on GitHub