---
icon: material/cogs
title: "Compiler"
---

The [Compiler](https://doc.log10x.com/compile/) extracts symbol vocabulary from source code and container images to enable the 10x Engine's [JIT runtime](https://doc.log10x.com/engine/design/#optimization-model) to recognize log event structure.

<div class="grid cards" markdown>

- [:material-shield-outline: **Security**](#security-access)
- [:material-help-circle-outline: **When to Use**](#when-to-use)
- [:material-check-circle-outline: **Coverage**](#coverage)
- [:material-wrench-outline: **Operations**](#operations)

</div>

### :material-shield-outline: Security & Access

??? tenx-security "What does the compiler do"

    The compiler creates a **vocabulary** for the runtime to recognize your logs. At runtime, the 10x Engine [tokenizes](https://doc.log10x.com/run/transform/structure/) incoming events and matches tokens against the symbol library. When symbols match, the engine infers event structure and creates cached templates ([TenXTemplates](https://doc.log10x.com/run/template/)). Subsequent events with the same structure reuse the cached template, fast, accurate, and memory-efficient.

    **Without a symbol library:** The runtime still processes events using JIT inference, creating templates on-demand as events arrive.

    **With a symbol library:** The runtime recognizes patterns upfront, builds more accurate templates, and improves [message extraction](https://doc.log10x.com/run/initialize/message/) for cost tracking and metrics, the core value of the engine.

??? tenx-security "What information does the compiler generate"

    The compiler extracts **symbol values**, tokens that appear in log statements, such as string literals, enum values, and format patterns. Extracted string constants, from log statement formats (e.g., `"error in %s"`), [binary executables](https://doc.log10x.com/compile/scanner/executable/) (via `strings`), or [text/JSON/YAML files](https://doc.log10x.com/compile/scanner/text/), are stored as **64-bit hashes** in the symbol library. The original plain text is never written to the output. The runtime engine matches incoming events against these hashes, not against your source strings.

    Class and method names are stored alongside hashes to identify the source of each log statement, comparable to what appears in a stack trace.

    The compiler does **not** extract application logic, control flow, full source file contents, or environment variables. It does not read `.env` files or runtime state.

    Output is a [symbol library](https://doc.log10x.com/compile/link/#symbol-library) file (`.10x.tar`) containing hash mappings and their source locations. You can inspect the contents before deploying (`tar tf library.10x.tar`).

??? tenx-security "What permissions does the compiler need, and how do I scope access"

    The compiler needs **read-only** access to the sources you configure:

    | Source | Credential | Scope |
    |--------|-----------|-------|
    | [GitHub repos](https://doc.log10x.com/compile/pull/github/) | `GH_TOKEN` | [Fine-grained PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#fine-grained-personal-access-tokens){target="_blank"} with `Contents: Read-only` scoped to target repos (recommended). Classic PAT with `repo` scope also supported. Public repos need no token |
    | [Docker images](https://doc.log10x.com/compile/pull/docker/) | `DOCKER_USERNAME` + `DOCKER_TOKEN` | Pull access to target registries |
    | [Helm charts](https://doc.log10x.com/compile/pull/helm/) | Same as GitHub + Docker | Resolves chart image references, then pulls each image |
    | [Artifactory](https://doc.log10x.com/compile/pull/artifactory/) | `ARTIFACTORY_TOKEN` | Read access to target repositories |

    You control exactly which repos and images the compiler accesses via your [configuration](https://doc.log10x.com/compile/test/#config-files). The compiler only pulls sources you explicitly list, it does not discover or scan anything beyond what you configure.

    All processing runs inside your infrastructure (CI/CD pipeline or k8s CronJob). No source code or symbol data is sent to Log10x or any external service.

??? tenx-security "Does any data leave my network"

    All source code and symbol output stays inside your infrastructure. No source code, symbol data, or log content is transmitted to Log10x or any external service.

    The only outbound calls are:

    - Pulling the 10x container image from `ghcr.io` (can be mirrored to a private registry for air-gapped environments)
    - Engine telemetry, only on a licensed deployment with a service endpoint configured: lightweight Prometheus metrics over TLS 1.3, enumerated in the [telemetry schema](https://doc.log10x.com/security/telemetry/). Payloads contain no log content, no source code, and no PII. A run with no license takes the built-in [evaluation license](https://doc.log10x.com/manage/license/) and makes no outbound call at all

    Input (source code, container images) is pulled from your own repositories. Output (symbol library files) is written to your own storage or pushed to your own GitHub repo via [GitOps](https://doc.log10x.com/engine/gitops/).

??? tenx-security "Can I run the compiler in an air-gapped environment with no egress"

    Yes. The two outbound calls (container image pull + engine telemetry) can both be internalized:

    - **Container image**, Mirror the 10x image from `ghcr.io` to your private registry. Point the CronJob or CI/CD step at your internal registry instead.
    - **Engine telemetry**, Set `TENX_AIRGAPPED=true` (the `airgapped` bootstrap option). The engine verifies the license JWT locally against its embedded public key and makes no outbound call to the 10x gateway at all, no DNS and no TCP. Usage is then reconciled with your account contact instead; [sales@log10x.com](mailto:sales@log10x.com) sets that up. Airgapped mode is honored for every license type, including `demo` and `limited`.

    With both in place, the compiler runs with zero egress. All source access is to your own internal repos and registries.

### :material-help-circle-outline: When to Use

??? tenx-info "Do I need to run the compiler, or do the built-in defaults cover my stack"

    The 10x runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering 131 open-source repositories and 14 public container images across Java, Python, Node.js, Go, C++, Rust, Ruby, .NET, and Scala, including Kubernetes, OpenTelemetry, Spring Boot, Django, Express, nginx, PostgreSQL, Redis, Kafka, and many more.

    **Start with the defaults.** Run a local cost POC on your own log files to see how much the built-in library covers. The runtime reports which events matched known symbols and which did not.

    Run the compiler if:

    - Your applications generate custom logging (lots of proprietary log statements in your code)
    - You use third-party frameworks not in the [default library](https://doc.log10x.com/compile/pull/#default-symbols)
    - You want symbol coverage for your specific application log statements (improves template accuracy for custom events)

??? tenx-info "What happens to log events that don't have matching symbols"

    The runtime engine still processes them. At runtime, the engine [tokenizes](https://doc.log10x.com/run/transform/structure/) each event and matches tokens against the symbol library, then hashes the symbol+delimiter sequence to upsert into the template map.

    **With symbol coverage:** Tokens are classified as known symbols, producing longer, coherent symbol sequences. This enables accurate [message extraction](https://doc.log10x.com/run/initialize/message/) that captures the intended "essence" of the event, essential for cost attribution and metrics.

    **Without symbol coverage:** Tokens are classified as [variables](https://doc.log10x.com/run/transform/structure/#variables), producing shorter, fragmented symbol sequences. [Message extraction](https://doc.log10x.com/run/initialize/message/) still works, but makes best effort based on available symbols. The runtime still builds templates and processes events normally, but message precision is reduced. Events still function with ~80% efficiency from structure inference alone, no failure, just less accurate classification.

### :material-check-circle-outline: Coverage

??? tenx-coverage "My stack uses common open-source frameworks, do I need to run the compiler at all"

    The 10x runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering 131 open-source repositories and 14 public container images. If your stack is covered by the defaults, you don't need to run the compiler.

    Run the compiler to **add custom symbols** for your proprietary application code on top of the defaults. This improves template accuracy for your specific log formats. The compiler output supplements the default library, not replaces it.

    **How to decide:** Run a local cost POC to verify default coverage on your own log files. If default coverage is sufficient, you're done. If you want to optimize logs from custom frameworks or proprietary services, run the compiler.

??? tenx-coverage "I have logs from services where I have no source code, vendor agents, third-party images, AWS managed services. Can the compiler cover these"

    Source code is not required. The compiler handles closed-source workloads via three approaches:

    1. **[Docker image scanning](https://doc.log10x.com/compile/pull/docker/)**, Pull any Docker image from any registry (public or private), including closed-source vendor agents. The compiler exports and scans the image filesystem for symbol content. Images are only re-pulled when their SHA256 changes, making incremental runs efficient. Works for `nginx:latest`, Datadog agent images, Splunk UF, or any vendor-provided image.

    2. **[Executable scanning](https://doc.log10x.com/compile/scanner/executable/)**, Runs the OS `strings` utility on compiled binaries (`.so`, `.dll`, `.dynlib`) to extract symbol values. Runs inside your infrastructure; output is only 64-bit hashes (no binary content leaves your network).

    3. **JIT fallback**, For truly opaque sources (AWS-managed service logs, SaaS tools with no Docker image), the JIT runtime handles them automatically at runtime. The engine creates a template on the first event seen, giving ~80% efficiency without any compiler setup needed.

??? tenx-coverage "Do you support custom and proprietary log formats, or just the default library"

    **Both.** The default library covers common frameworks, but the compiler lets you add custom symbols for proprietary logging patterns. Custom formats work out of the box with 80% coverage from the built-in library.

    **How it works:** The runtime [tokenizes](https://doc.log10x.com/run/transform/structure/) incoming events and matches tokens against the symbol library. With 80% built-in coverage, most generic tokens (class names, log levels, timestamps, method names) already match. For logs from your proprietary services, the runtime still processes them, but template inference is less precise.

    **When to optimize:** If you want accurate [message extraction](https://doc.log10x.com/run/initialize/message/) for cost attribution and metrics, capturing the intended "essence" of each custom event type, run the compiler on your custom code. This generates additional symbols, so the runtime recognizes your proprietary patterns and builds more accurate templates. The compiler is iterative: initial runs take 10-30 minutes to scan your code. Subsequent runs only process changed files, completing in seconds, perfect for CI/CD pipelines. You point it at your repositories once, then it automatically supplements the default symbol library on each deploy.

    See [real-world before/after examples](https://doc.log10x.com/apps/receiver/){target="_blank"} showing how symbol coverage improves lossless volume reduction on production logs.

??? tenx-coverage "My custom format isn't in the library, does it still work"

    Yes. The engine creates a template for every event type it encounters, there are no "unrecognized" events. The [symbol library](https://doc.log10x.com/run/symbol/) provides vocabulary that determines how much of each event is classified as structure (low-cardinality, stored in the template) vs variable data (high-cardinality, shipped per event). The [default library](https://doc.log10x.com/compile/pull/#default-symbols) covers 145 pinned sources and delivers strong reduction on most workloads.

    Running the compiler on custom application code enriches the vocabulary further, producing richer templates, more accurate [message extraction](https://doc.log10x.com/run/initialize/message/), and higher reduction ratios. Without it, the format still works using the default vocabulary, with no configuration or manual regex rules.

### :material-wrench-outline: Operations

??? tenx-operations "How often do I need to run the compiler, and what is the maintenance overhead"

    Minimal. Initial runs take a few minutes; incremental runs on unchanged code complete in seconds. Deploy as a [k8s CronJob](https://doc.log10x.com/compile/deploy/) (every 30 minutes) or trigger from CI/CD on commits.

    How it stays lightweight:

    - **Checksums**, Unchanged source files are skipped automatically. Only modified code is re-scanned, so incremental runs are fast.
    - **Docker images**, Images are only re-pulled when their SHA256 hash changes since the last run. Routine re-runs on an unchanged codebase are near-instantaneous.
    - **Distribution**, Edge and cloud apps pull updated symbol libraries via [GitOps](https://doc.log10x.com/engine/gitops/) at startup and poll for changes periodically. Between runs, events are processed using the previous symbol library version, no data is lost.

    The compiler runs as a Docker container, processes changed files, and exits. No long-running daemon, no runtime agent, no application code changes.

??? tenx-operations "How do symbols relate to templates, and how do I troubleshoot recognition issues"

    **Symbols** are vocabulary, tokens extracted from your source code by the AOT compiler. **Templates** ([TenXTemplates](https://doc.log10x.com/run/template/)) are cached templates created by the runtime engine when it encounters events that match a known symbol pattern.

    The chain: symbol library → runtime engine loads symbols → incoming event matches symbol pattern → engine assigns a cached TenXTemplate → subsequent events with the same structure reuse the template.

    To troubleshoot:

    - Extract and inspect [symbol library contents](https://doc.log10x.com/compile/link/#file-structure) (`tar tf library.10x.tar`), verify your target symbols were captured
    - Run the engine locally on test logs and enable [debug output](https://doc.log10x.com/run/transform/symbol/#debug) to see symbol matching and template assignment in detail
    - Check the engine [log file](https://doc.log10x.com/manage/logging/#log-file-location) for runtime template assignment messages

??? tenx-operations "How does the AOT compiler work"

    :material-code-braces: The [compiler](https://doc.log10x.com/compile/) builds symbol vocabulary from your deployment artifacts in your CI/CD pipeline:

    - **[Helm charts](https://doc.log10x.com/compile/pull/helm/)**, resolves all image references, pulls and scans each one
    - **[Docker images](https://doc.log10x.com/compile/pull/docker/)**, pulls and scans any container image directly
    - **[GitHub repos](https://doc.log10x.com/compile/pull/github/)**, extracts symbols from source code and compiled binaries
    - **[Artifactory](https://doc.log10x.com/compile/pull/artifactory/)**, pulls from artifact repositories

    The runtime ships with a [default symbol library](https://doc.log10x.com/compile/pull/#default-symbols) covering 131 open-source repositories and 14 public container images (Spring Boot, Django, Express, Kafka, Kubernetes, and more). Most environments work out of the box, custom compilation is only needed for proprietary logging frameworks or application-specific log formats.

    The compiler commits custom symbol libraries to GitHub as part of its pipeline. Edge and cloud apps pull them automatically via the [`@github` launch macro](https://doc.log10x.com/config/github/) at startup and poll for changes at a configurable interval. See [compiler workflow](https://doc.log10x.com/compile/#workflow) for the full CI/CD → Pull → Scan → Link → Push → Distribute pipeline.

    The compiler runs in your infrastructure, source code and binaries never leave your network. The compiler itself is free to run with no credits or metered usage.