---
icon: material/wrench-outline
title: "Troubleshooting"
---

Common issues after putting 10x in front of Datadog: optimized logs not appearing, the bill not dropping, and Datadog Agent problems.

### :material-wrench-outline: Troubleshooting

??? tenx-troubleshoot "10x optimized logs aren't appearing in Datadog"

    **Check these in order:**

    1. **The 10x sidecar is running.** Check the sidecar container's logs:
       ```bash
       kubectl logs <forwarder-pod> -c <10x-sidecar>   # the 10x sidecar container
       # Or: docker logs <container-id>
       # Or: systemctl status log10x
       ```

    2. **The pipeline is healthy.** Run `log10x_doctor` from your MCP client to confirm the engine is processing events and emitting metrics.

    3. **The forwarder/Agent is reading 10x output.** Confirm the Datadog Agent (or Fluent Bit) input points at the file or socket the Receiver writes to, and that the path in your config matches.

    4. **Datadog is ingesting.** Datadog → Logs → Live Tail, filter on your `source` / `service`. If it's empty, the forwarder isn't shipping.

    5. **The API key is valid.** `echo $DD_API_KEY` is non-empty, and the key is active under Organization Settings → API Keys.

    **Common causes:**

    | Problem | Solution |
    |---------|----------|
    | Agent can't read the 10x socket/file | Check file permissions and the socket/file path in your config |
    | Datadog API key invalid | Verify the key in the Datadog UI, rotate if needed |
    | Log pipeline filters out logs | Check Log Pipelines for filters blocking your service |
    | Network/firewall blocking Datadog intake | Test `curl -v https://http-intake.logs.datadoghq.com` |

??? tenx-troubleshoot "My Datadog cost didn't go down after deploying 10x"

    **Check these:**

    1. **The down-tier and offload split is happening.** Run `log10x_commitment_report` (or query your TSDB) to confirm patterns are being routed to Flex and offloaded, with non-zero volume.

    2. **Datadog usage metrics.** Datadog → Dashboards → Usage Metrics. The down-tiered slice should now show up under the Flex index, not Standard.

    3. **The Flex index is enabled and ordered first.** Datadog indexes are first-match-wins, so the Flex index must sit ahead of your catch-all index, or the down-tiered events never reach it.

    4. **Offloaded patterns reach S3, not Datadog.** Confirm the offload bucket is filling.

    **Why costs might not drop:**

    - **New services added:** if you added instrumentation or new apps, increased volume can offset the reduction
    - **Flex index not enabled or mis-ordered:** if Flex Logs isn't enabled, or the Flex index sits after the catch-all, the down-tiered slice stays in the Standard index
    - **Metric lag:** allow a day or two for usage metrics to settle
    - **Error logs not down-tiered:** if most growth is from new errors/alerts, those stay in the Standard index by design

    **Solution:** confirm the down-tier and offload split with a local cost POC on real logs and check the Flex index is enabled and ordered first. If still no change, contact support.

??? tenx-troubleshoot "Datadog Agent is crashing after deploying the 10x sidecar"

    **Check Agent logs:**

    ```bash
    tail -f /var/log/datadog/agent.log | grep -i error
    ```

    **Common causes & fixes:**

    | Error | Cause | Fix |
    |-------|-------|-----|
    | `permission denied` reading log file | 10x output file has wrong permissions | `chmod 644 /path/to/log` |
    | `connection refused` to 10x socket | Socket path doesn't exist or 10x isn't running | Verify the sidecar is running, check the path in config |
    | `out of memory` | Heap pressure under load | Raise the Receiver's container memory limit. The heap is sized from that limit, so raising the limit raises the heap. `-Xmx` has no effect on the native build. |
    | Agent reads non-standard lines | Agent is tailing the compact S3-bound output, not the Datadog-bound output | Point the Agent at the Receiver's Datadog-bound output (standard log format); the compact stream is for S3 only |

    **If the Agent still crashes:**

    1. Temporarily disable 10x (remove the sidecar) to isolate.
    2. Restart the Agent: `systemctl restart datadog-agent`.
    3. Verify the Agent is healthy: `datadog-agent status`.
    4. Re-enable 10x and check logs.