---
icon: material/currency-usd
title: "Optimization"
---

Why compact events go to S3 rather than Datadog, the cost-insight metrics 10x publishes, what the Receiver regulates, querying offloaded logs, query overhead, and how to migrate.

### :material-currency-usd: Optimization

??? tenx-costopt "Can Receiver (Compact mode) output go directly to Datadog"

    **No loss, but not Datadog-bound either. [Receiver (Compact mode)](https://doc.log10x.com/apps/receiver/) uses lossless techniques only**, like Protocol Buffers for logs. Structure is stored once per event type as a template, with only changing values shipped per event; all fields and values remain intact. Compact events contain template references instead of full repetitive structure.

    **Why they go to S3, not Datadog:** Datadog has no plugin to expand 10x's template-plus-values wire form, so compact events cannot be queried natively in Datadog. They go to S3 (and to Splunk/Elasticsearch, which can expand them). Compaction reduces S3 storage, not the Datadog bill.

    **To cut the Datadog bill in-platform, down-tier patterns to Flex Logs instead.** tier_down routes low-value patterns straight to Datadog's Flex Logs index, where they stay queryable in Datadog at a lower cost, no S3 round-trip.

    **To get S3 data back into Datadog:** [Retriever](https://doc.log10x.com/apps/retriever/) reads compact events from S3, expands them back to standard format, and streams regulated events to Datadog on-demand.

    **Note:** if you want to filter or drop logs (lossy), that's [Receiver](https://doc.log10x.com/apps/receiver/) Filter mode, explicit policies you control.

??? tenx-costopt "Can I see what's driving my Datadog costs, inside Datadog"

    **Yes.** 10x publishes [cost insight metrics](https://doc.log10x.com/run/output/metric/datadog/){target="_blank"} directly to Datadog's time-series API: volume and bytes per event type, tagged by [symbol identity](https://doc.log10x.com/run/initialize/message/){target="_blank"}. Which log patterns drive the most volume shows up on native Datadog dashboards.

    This includes regulation visibility: when [Receiver](https://doc.log10x.com/apps/receiver/) caps noisy events, the metrics show what passed vs. what was filtered, so cost control is monitored without leaving Datadog.

??? tenx-costopt "What types of logs typically see the best optimization"

    **High-volume repetitive logs see the best results:**

    - Health checks and heartbeats
    - Debug statements
    - Access logs
    - Container orchestration events
    - Verbose application traces

    On Datadog, the lever for these is tier_down (route them to the cheaper Flex Logs index, still queryable) or offload to your own S3. In many environments these patterns make up a modeled 60-80% of ingestion volume.

??? tenx-costopt "What happens to logs regulated by Receiver"

    [Receiver](https://doc.log10x.com/apps/receiver/) identifies low-priority logs based on your configured budget and severity thresholds. You control the destination for regulated logs:

    - **Offload to [S3 storage](https://doc.log10x.com/run/input/objectStorage/){target="_blank"}** at $0.023/GB. Non-destructive: the data stays reachable, and [Retriever](https://doc.log10x.com/apps/retriever/) fetches it back on-demand.
    - **Stream to a secondary destination** for further analysis. Non-destructive.
    - **Drop** (Filter mode). This is a lossy choice you opt into: dropped events are gone, so use it only for noise you are willing to lose.

    The non-destructive levers (offload, stream) keep the data; drop is the one option that loses it.

??? tenx-costopt "Can I query offloaded logs after they're filtered"

    **Yes.** [Retriever](https://doc.log10x.com/apps/retriever/) provides on-demand fetch of the events offloaded to your own S3.

    When you need specific data back for investigation or compliance, stream specific time ranges and log types directly into Datadog, paying ingestion costs only when you actually need the data.

### :material-speedometer: Performance & Migration

??? tenx-perf "Is there any query overhead inside Datadog"

    **No in-Datadog expansion overhead.** Events that reach Datadog, the hot path to the Standard index and the down-tiered slice in the Flex Logs index, are standard log format. There is nothing for Datadog to expand at query time, so queries run exactly as they would on any Datadog logs.

    Compacted events live in S3, not Datadog. When you need them back, the [Retriever](https://doc.log10x.com/apps/retriever/) expands them on the S3 read path and streams the result to Datadog, the expansion happens in the Retriever, not inside a Datadog query.

    Flex Logs queries are slower than Standard-index queries (that tradeoff is part of the Flex tier itself, set by the compute size you pick), but that is Datadog's tiering, not a 10x expansion cost.

??? tenx-perf "How do I migrate from non-optimized to 10x optimization"

    **Safe migration procedure:**

    **Phase 1: Test (Week 1)**

    ```
    1. Deploy the Receiver with Filter mode + tier_down on 1-2 test nodes
       (route low-value patterns to a Datadog Flex Logs index)
    2. Route subset of traffic (5-10%) to 10x
    3. Remainder bypasses 10x (standard Datadog flow)
    4. Monitor:
       - Check Datadog for missing logs
       - Verify the down-tiered slice lands in the Flex index
       - Verify Standard-index volume drops on the down-tiered patterns
       - Look for errors in sidecar logs
       (Use Compact mode only on the S3 offload path, not for Datadog.)
    ```

    **Phase 2: Gradual rollout (Weeks 2-3)**

    ```
    Increase traffic percentage:
    - 10% -> 25% -> 50% -> 100%

    At each stage:
    - Monitor sidecar CPU/memory
    - Verify Datadog shows consistent log volume across Standard + Flex
    - Check that the down-tier and offload split stays consistent
    ```

    **Phase 3: Stabilization (Week 4+)**

    ```
    Once at 100%:
    - Monitor weekly for consistency
    - Set baseline metrics (GB/day moved to Flex, GB/day offloaded, bill reduction)
    - Plan next step (add S3 offload for patterns to remove from Datadog entirely)
    ```

    **Rollback procedure (if needed):**

    ```bash
    # If issues arise, disable 10x immediately:
    # Remove the Receiver sidecar from your forwarder pod spec
    # and roll out the change (or revert the Helm values)

    # Datadog Agent automatically routes to original sources
    # No logs are lost (sidecar doesn't modify original files)
    # Timeline: < 2 minutes to full bypass
    ```

    **What happens to logs already optimized?**

    If you optimize logs for 2 weeks then disable 10x:
    - Already-optimized logs remain in Datadog (still queryable)
    - New logs flow at full volume (standard ingestion)
    - No data loss, just temporary cost spike until old logs age out

    **Testing before production:**

    Install the 10x MCP and ask it to run a cost POC on your logs. [`log10x_poc_from_local`](https://doc.log10x.com/apps/mcp/tools/poc/poc-from-local/){target="_blank"} runs the engine on your own machine and returns the sample composition by pod, the top patterns, and a list-rate cost projection. Nothing is uploaded. Use it to estimate savings before deploying.

??? tenx-perf "Can I optimize only specific services or log sources"

    **Yes.** You can apply 10x selectively:

    **Option 1: Sidecar to specific pods**

    Add the Receiver as a sidecar only to the forwarder pods whose log sources you want optimized; leave the rest untouched.

    ```yaml
    apiVersion: v1
    kind: Pod
    spec:
      containers:
      - name: app
        image: myapp:1.4.2      # pin your own image too
      - name: receiver
        image: log10x/pipeline-10x:1.1.69
    ```

    **Option 2: Different optimization for different services**

    ```
    Frontend logs (static HTML, JS) → Low repetition → Skip optimization
    Backend logs (microservices) → High repetition → Apply optimization
    Database logs (verbose) → Very high repetition → Aggressive optimization
    ```

    **Option 3: Route by log pattern**

    ```
    If using Fluent Bit or Fluentd:
    - Match app.frontend → Standard routing (no optimization)
    - Match app.backend → Through 10x sidecar
    - Match app.database → Through 10x sidecar
    ```

    **Cost impact per service:**

    Once you deploy, use Datadog metrics to measure savings by service:

    ```
    Datadog Dashboard:
    | Logs → Summary → Group by service
    | Look for: Ingested bytes/day per service
    | Reduction varies by service; measure your own with a local cost POC first
    ```