---
icon: material/currency-usd
title: "Optimization"
---

How the monthly number holds Standard indexing and what sits outside it, why compact events go to S3 rather than Datadog, the cost metrics 10x publishes into Datadog, what happens above a pattern's cap and to services with no rule, what a moved event costs, query overhead, and how to migrate.

### :material-currency-usd: Optimization

??? tenx-costopt "Can 10x hold Standard indexing at a set monthly number"

    **Yes, for the Standard indexing charge.** Four rules define it:

    - Standard indexing for the whole account stays at the monthly number the team sets.
    - Growth above it lands in a Flex Logs index or in the team's own S3, chosen per service. S3 takes those lines off the Datadog bill and out of Datadog search; Flex keeps search, and above 2 billion events a month Datadog prices Flex compute on request, so the Flex charge grows as volume grows. Flex is the default.
    - Each ERROR, CRITICAL or FATAL statement sends up to 50 lines over its limit to Standard in each 4-minute window on each node, released in four slices across the window. The install writes 50; the team can change it.
    - Services the team names are left alone: their lines still count toward the number, so the other services give way. A service can instead get its own monthly number, metered from Datadog's usage metric under its `service` tag (the container name unless mapped to another), and the rest of the account is held to the number less theirs.

    **How the number holds.** A [scheduled run](https://doc.log10x.com/apps/mcp/tools/install/setup-recurring/), hourly by default, reads Datadog's own usage metric month to date, `datadog.estimated_usage.logs.ingested_events`, for the indexes the install names as Standard (the metric counts Flex events too), and sets the rate for the rest of the month from what is left. That metric also counts lines that never pass the Receiver, such as direct Logs API submissions, so they are inside the number: the run subtracts their Standard count from the number and gives the Receiver's services the rest, and its report warns when they alone near the number.

    Each run writes every service's limits, a cap per statement and a total per service, divided across the nodes each service runs on, and one limit, at the level the known services share, for services it has not seen yet. As a service's volume passes its total, its largest statements reach their caps first and small ones keep passing; once what it has passed reaches the total, every non-error line past the first five of its statement in each window takes the service's action. If the number is spent before the month ends, every non-error line past the first five of its statement in each window moves for the rest of the month, and the report says so.

    The run is a Kubernetes CronJob in the cluster by default; a team that cannot run jobs there runs it as a GitHub Actions workflow that commits the limits directly to a [branch the Receiver pulls](https://doc.log10x.com/config/github/).

    **Between runs.** Each service stays bounded on each node. A new service is held at the shared level from its first line until the next run gives it its own limits, up to an hour later. A node added between runs holds its own share until the next run divides the number again. The month can run over by what a new service or an added node passes before the next run, and the next run takes it back from the rest of the month.

    **Seeing it in Datadog.** The Plan and Usage page and the **Log Management - Estimated Usage** dashboard show indexed events per index; the Standard index levels off at the hourly rate the number allows while the Flex index, or the S3 bucket, takes the growth. The same usage metric, grouped by index and service, shows which service carried it.

    **Measured.** On a Datadog org fed by the OpenTelemetry demo, Standard indexing held to the number month to date in Datadog's own usage metric while log volume grew 2.6 to 2.9x, with Flex taking the growth. A service that first appeared during the test was held from its first window, a service with its own number stayed inside it, and a storming error statement sent its limit plus 50 lines to Standard in each window. The run and the commands that repeat it are in [benchmarks/datadog-ceiling](https://github.com/log-10x/benchmarks/tree/main/datadog-ceiling){target="_blank"}.

    **Outside the number:**

    - Ingestion, $0.10 per GB, on every line that reaches Datadog, Flex included. Offloaded lines never reach Datadog's intake.
    - Flex storage, and Flex compute above Starter.
    - The lines that always pass: the first five of every statement and up to 50 over-limit lines of each error statement in each window on each node, released across the window. They still reach Standard once the number is spent, so a burst of errors on many nodes reaches Standard on each. Once a service is past its total, it passes at most 50 more of the first-five lines and 500 more of the error lines in a window on a node ([regulator defaults](https://doc.log10x.com/run/receive/rate/#service-totals){target="_blank"}).

    **Built with Datadog tools alone.** Five pieces hold the number through a statement that spikes on an external change, a few that creep each month and the statements coding agents add each week: an [Observability Pipelines](https://docs.datadoghq.com/observability_pipelines/){target="_blank"} Worker that processes every log, beside the sources or in a central pool, with a regex that turns each message into a signature and a random number per line for sampling, priced per GB on every log; a [log-based metric](https://docs.datadoghq.com/logs/log_configuration/logs_to_metrics/){target="_blank"} per signature, billed as custom metrics; a Flex index listed first whose [index filter](https://docs.datadoghq.com/logs/log_configuration/indexes/){target="_blank"} names the demoted signatures and the random attribute, since an index filter cannot sample and an exclusion filter cannot route; a [Workflow](https://docs.datadoghq.com/service_management/workflows/){target="_blank"} that reads the usage metric hourly, projects month end and rewrites that filter through the Logs Indexes API; and every [log monitor](https://docs.datadoghq.com/monitors/types/log/){target="_blank"} that counts a sampled statement rebuilt as a metric monitor, since log monitors read indexed logs only and Datadog runs no log monitor on Flex.

    An event ID written into the team's own code replaces the first piece for that code, covers no third-party service and no statement an agent adds before anyone assigns it an ID, and leaves the other four standing.

    The Receiver and the hourly run carry the five: the pattern comes from a library compiled from the code, the same on every node; cap, sample and route are decided once on the forwarder, before ingestion; the hourly run is part of the install; and each error statement keeps up to 50 over-limit lines in Standard in every window on every node, released across the window. The pieces, with the Datadog doc behind each, are on the [Comparisons](comparisons.md) page.

??? tenx-costopt "Can Receiver compact output go directly to Datadog"

    **No. Compact events are lossless and S3-bound.** [Receiver compact mode](https://doc.log10x.com/apps/receiver/compact/) stores each event's structure once as a template and ships the changing values per event, with every field and value intact; a compact event carries a template reference instead of the repeated structure.

    **Why S3 and not Datadog:** Datadog has no plugin that expands the template-plus-values form, so compact events cannot be searched in Datadog. They go to S3, and to Splunk and self-hosted Elasticsearch or OpenSearch, which expand them at search time. Compaction reduces S3 storage, not the Datadog bill.

    **The in-Datadog lever is tier_down:** the excess over a pattern's cap goes to a Flex Logs index, where it stays searchable at Flex rates, with no S3 round trip.

    **To bring S3 data back into Datadog:** the [Retriever](https://doc.log10x.com/apps/retriever/) reads compact events from S3, expands them, and streams them to Datadog on demand.

    Sample and drop are the lossy actions; like the others, each is set per service.

??? tenx-costopt "Can I see what's driving my Datadog costs, inside Datadog"

    **Yes.** 10x publishes per-pattern volume and byte counters into Datadog by either of two paths: the engine's own [Datadog metric output](https://doc.log10x.com/run/output/metric/datadog/){target="_blank"}, which posts to the metrics API and takes `DD_API_KEY` and `DD_APP_KEY`, or the [Datadog Agent check](https://doc.log10x.com/install/datadog/){target="_blank"}, which scrapes the engine's Prometheus endpoint and needs no credential on the engine. Each series is tagged by [pattern](https://doc.log10x.com/run/initialize/message/){target="_blank"}, so native Datadog dashboards show which patterns drive the Standard count.

    The series carry `routeState`, so what passed, what went to Flex, what went to S3 under offload and what was dropped are separate series on the same dashboard.

??? tenx-costopt "What types of logs see the most optimization"

    **High-volume repetitive logs:**

    - Health checks and heartbeats
    - Debug statements
    - Access logs
    - Container orchestration events
    - Verbose application traces

    On Datadog the lever for these is tier_down, which routes the excess to a Flex Logs index where it stays searchable, or offload to S3.

??? tenx-costopt "What happens to events above a pattern's cap"

    The [Receiver](https://doc.log10x.com/apps/receiver/) names each line after the log statement that most likely wrote it; that name is the pattern. A line that matches no statement is named by the words on it that also appear in the code and container images the [symbol library](https://doc.log10x.com/compile/link/#symbol-library) was compiled from, or by its shape when no word on it does. A service with a rule of its own has one cap and one action, and each of its patterns is held to that cap; the service's total, set from the monthly number, bounds the sum. Below the cap every event passes. Above it the excess takes the service's action, except that up to 50 lines over the cap of each ERROR, CRITICAL or FATAL statement in each window on each node pass, released across the window. The first five events of a pattern in each window always pass. Caps apply from a container's first line on each node. The allowance is enforced continuously, so a burst cannot spend a day's allowance at once; the settings are on the [rate regulator](https://doc.log10x.com/run/receive/rate/) page.

    The actions:

    - **tier_down** to a Flex Logs index: still searchable in Datadog, seen by log-based metrics and Live Tail, pays ingestion.
    - **Offload** to [S3](https://doc.log10x.com/run/input/objectStorage/){target="_blank"} at S3 storage rates, $0.023 per GB-month on S3 Standard: never ingested by Datadog, and the [Retriever](https://doc.log10x.com/apps/retriever/) fetches it back on demand.
    - **Drop:** gone. Use it only on noise the team is willing to lose.

    tier_down and offload keep every line; sample keeps a share and drop keeps none.

    **Services with no rule.** With a monthly number, the scheduled run holds them at the one limit it writes for services it has not seen, at the level the known services share.

    Without one, the MCP's POC report gives a rule to each service it measured, and every other service takes the fleet default, which [`log10x_advise_install`](https://doc.log10x.com/apps/mcp/tools/install/advise-install/) writes into the Receiver on a Datadog install: a limit per statement on each node, which the POC report states in lines a day, and one route for the lines above it. The limit is twice the largest statement the plan keeps in Standard, measured per node on the POC capture. The lines above it go to Flex, or to S3 on an account without Flex. The Flex index has the filter `@routeState:tier_down` and is ordered ahead of the catch-all index; the install plan prints it as Terraform. The first five lines of every statement in each 4-minute window, up to 1,800 a day, always reach Standard on each node. Without a limit from a POC capture the install writes no fleet default, and a service with no rule keeps every statement in Standard. The engine settings are [`rateReceiverAbsoluteCap`](https://doc.log10x.com/run/receive/rate/#ratereceiverabsolutecap) for the limit and [`rateReceiverDefaultAction`](https://doc.log10x.com/run/receive/rate/#ratereceiverdefaultaction) for the route.

    On the OpenTelemetry demo's own seeded failures, with the Receiver installed from the values the MCP writes, the fleet default held a Kafka duplicate-order flood in Postgres, a service the POC gave no rule, and held nothing outside a failure's causal path; the services with rules were held by their own limits. The run, its limits and the one command that repeats it are in [benchmarks/datadog-containment](https://github.com/log-10x/benchmarks/tree/main/datadog-containment){target="_blank"}.

    **What an error monitor sees.** The limits hold error statements too. A Datadog log monitor on error lines sees, in Standard, every ERROR, CRITICAL and FATAL line under the limit, the always-pass lines, and up to 50 lines over the limit of each error statement in each window on each node, released across the window. A log monitor that alerts above zero on an error statement fires in every window in which the statement appears, and one with a higher threshold counts up to the limit plus 50 lines of that statement in each window on each node. Under tier_down the rest go to Flex, where Datadog runs no log monitors. Log-based metrics count them, because Datadog computes those at ingestion, so a metric monitor on a log-based metric sees every line. Under offload the rest go to S3, out of reach of Datadog's log monitors and log-based metrics, and the Retriever fetches them back on demand.

    The over-limit error count, 50 as installed, is one Receiver setting, the same for every service; above its cap, a statement below ERROR passes only its first five lines in each window on each node.

    **Before and after a move.** Before a rule moves statements off Standard, the MCP reads the account's log monitors and log-based metrics, read-only, and lists the ones whose query reads each rule's service or statement text, including the error monitors of the services the fleet default covers ([`check_dependencies`](https://doc.log10x.com/apps/mcp/tools/install/configure-engine/)).

    After the install, the weekly [change report](https://doc.log10x.com/apps/mcp/tools/costs/whats-new/#datadog-change-report) lists the statements new in each service, each with its lines a day, its estimated Standard indexing cost a month at list price and the rule that held it, if one did, and proposes a rule per service in the same lines-a-day form. The same tool renders two Datadog monitor definitions for the account owner to apply: one fires when a statement's route turns tier_down or offload, the other when a statement's volume rises above its weekly baseline.

    **The team.** A pod label named as the team label ([`k8sTeamLabel`](https://doc.log10x.com/run/initialize/k8s/#k8steamlabel), for example `team`) puts the owning team on the Receiver's Datadog metric series as `tenx_team`, and the change report shows the team when the series carry it.

??? tenx-costopt "Can offloaded events be queried later"

    **Yes.** The [Retriever](https://doc.log10x.com/apps/retriever/) fetches events offloaded to S3 on demand.

    For an investigation or a compliance request, stream a time range and a set of patterns into Datadog and pay ingestion only on what comes back.

### :material-key-variant: Routing Key

??? tenx-costopt "What does 10x add to Datadog's own routing"

    Datadog's [index filters](https://docs.datadoghq.com/logs/log_configuration/indexes/){target="_blank"} route logs by query, and Flex is a setting on an index. What 10x adds is the number the routing holds to, the key the rules run on, the window they run in, and where the excess goes.

    **The attributes.** Each forwarded event carries `routeState`, where the line went, and Datadog-side rules key on it: the Flex index filter `@routeState:tier_down`, and any index filter or monitor the team writes on a route. The scheduled run that holds the monthly number writes every cap afresh each time, so routes follow the patterns the Receiver names now.

    Each event also carries the pattern hash, `tenx_hash`, for reports and lookups: cost by pattern in Log Explorer and on dashboards, the change report, and fetching an offloaded pattern back. The pattern hash is present from the first line with no discovery step, and the same on every pod under the same engine, library and configuration, whatever that pod has seen and whenever it started; a library update can rename some patterns. Once a pattern is known, an index filter can also select it without the attribute, using plain terms and quoted phrases on the message; only the `*:` full-text syntax is barred from index filters ([search syntax](https://docs.datadoghq.com/logs/explorer/search_syntax/){target="_blank"}).

    A template miner at the edge, such as Drain3, names a line by what it has seen: two Drain3 instances that each see half of one stream give a line seen 2 to 10 times different names 39.78% of the time ([pattern-identity benchmark](https://github.com/log-10x/benchmarks/tree/main/pattern-identity){target="_blank"}). The [compiler](https://doc.log10x.com/compile/deploy/) runs in CI or on a schedule, so new log statements, including ones a coding agent wrote, are named as the code ships.

    Without a monthly number, a [scheduled check](https://doc.log10x.com/apps/mcp/tools/install/setup-recurring/) (`tenx-recur`) costs each pattern and sends routing changes as pull requests to the [policy repo](https://doc.log10x.com/apps/mcp/tools/install/set-gitops-repo/).

    **The window.** The cap is enforced continuously against each pattern's allowance, so a burst cannot spend a day's allowance at once, and the scheduled run holds the month to the number. Observability Pipelines Quota measures over a fixed 24-hour window that resets at midnight UTC, and an index daily quota stops indexing every log in the index once reached, until it resets.

    **The excess.** Over the cap, the excess goes to a Flex index inside Datadog, where it stays searchable, or to S3, where the Retriever fetches it back.

    **What a moved event costs.** The cost at Datadog list, computed from the released engine's counts on a public capture, is in [benchmarks/datadog-flex-accounting](https://github.com/log-10x/benchmarks/tree/main/datadog-flex-accounting){target="_blank"}. Standard indexing avoided: $1.70 per million events at 15-day retention and $2.50 at 30-day, billed annually. Flex added, at the rate of the tier the volume falls in: Flex Starter at $0.60 per million stored a month, for at least 3 months, with compute bundled for volumes below 2 billion events a month and 10 billion stored ([Datadog pricing](https://www.datadoghq.com/pricing/?product=log-management){target="_blank"}); above that, storage at $0.05 per million stored a month plus a compute tier Datadog quotes on request. On that capture every target falls in Starter, and the move saves against 30-day Standard retention and costs more than it saves against 15-day. Ingestion, $0.10 per GB, is paid once on every event that reaches Datadog and is unchanged by a Flex move. The README carries the full table.

### :material-speedometer: Performance & Migration

??? tenx-perf "Is there any query overhead inside Datadog"

    **None inside Datadog.** Events that reach Datadog, in the Standard index and in the Flex Logs index, are standard log events. There is nothing for Datadog to expand at query time, so queries run as they do on any Datadog logs.

    Compact events live in S3, not Datadog. The [Retriever](https://doc.log10x.com/apps/retriever/) expands them on the S3 read path and streams the result to Datadog; the expansion happens in the Retriever, not inside a Datadog query.

    Flex Logs queries run slower than Standard-index queries, at a speed Datadog sets by the Flex compute tier.

??? tenx-perf "How do I migrate from non-optimized to 10x optimization"

    **Phase 1: Test (week 1)**

    ```
    1. Deploy the Receiver with tier_down on 1 to 2 test nodes, with the Flex
       index in front of Standard
    2. Route a subset of traffic (5 to 10%) through 10x
    3. The remainder bypasses 10x (standard Datadog flow)
    4. Monitor:
       - Check Datadog for missing logs
       - Verify the marked slice lands in the Flex index
       - Verify the Standard count drops on the capped patterns
       - Look for errors in sidecar logs
       (Compact applies to the S3 offload path only, not to Datadog.)
    ```

    **Phase 2: Gradual rollout (weeks 2 to 3)**

    ```
    Increase the traffic share:
    - 10% -> 25% -> 50% -> 100%

    At each stage:
    - Monitor sidecar CPU and memory
    - Verify Datadog shows consistent log volume across Standard and Flex
    - Check that the split between Flex and offload stays consistent
    ```

    **Phase 3: Stabilization (week 4 on)**

    ```
    Once at 100%:
    - Monitor weekly for consistency
    - Record the baseline: events per day in Standard, in Flex, and offloaded
    - Plan the next step (add offload for patterns to take out of Datadog)
    ```

    **Rollback:**

    ```bash
    # Remove the Receiver sidecar from the forwarder pod spec and roll out the
    # change, or revert the Helm values or the GitOps pull request.
    # On the Datadog Agent path, point logs_config back at the original folder.
    # The sidecar is fail-open: once it stops, logs ship at full volume.
    # The sidecar leaves the original files unchanged.
    ```

    **Logs already moved:**

    After two weeks on 10x and then a rollback:
    - Events already in Flex stay in Flex and searchable until they age out
    - New logs flow at full volume into Standard
    - Rollback leaves every event already in Flex or S3 in place; the Standard count returns to the full-volume rate

    **Testing before production:**

    Install the 10x MCP and ask it to run a cost POC on the logs. [`log10x_poc_from_local`](https://doc.log10x.com/apps/mcp/tools/poc/poc-from-local/){target="_blank"} runs the engine on the local machine and returns the sample composition by pod, the top patterns, and the volume a plan would move out of the Standard index. Nothing is uploaded. Run it before deploying.

??? tenx-perf "Can I optimize only specific services or log sources"

    **Yes.** 10x applies selectively:

    **Option 1: Sidecar to specific pods**

    Add the Receiver as a sidecar only to the forwarder pods whose log sources it should cap; leave the rest untouched.

    ```yaml
    apiVersion: v1
    kind: Pod
    spec:
      containers:
      - name: app
        image: myapp:1.4.2      # pin the application image too
      - name: receiver
        image: log10x/pipeline-10x:<pinned-version>
    ```

    **Option 2: A different action per service**

    ```
    Frontend logs (static HTML, JS) → Low repetition → No cap
    Backend logs (microservices) → High repetition → Cap, tier_down
    Database logs (verbose) → Very high repetition → Cap, offload
    ```

    **Option 3: Route by forwarder tag**

    ```
    With Fluent Bit or Fluentd:
    - Match app.frontend → Standard routing (no 10x)
    - Match app.backend → Through the 10x sidecar
    - Match app.database → Through the 10x sidecar
    ```

    **Cost per service:**

    Once deployed, read the move per service in Datadog:

    ```
    Datadog Dashboard:
    | Logs → Summary → Group by service
    | Look for: indexed events per day per service, Standard against Flex
    | The share moved varies by service; a local cost POC states it first
    ```