---
icon: material/transit-connection-variant
title: "Streams"
description: "Regulate logs from platforms that cannot host a process: stream them to a hub and run the engine centrally behind a collector."
---

Some platforms give an application nowhere to run a second process. Azure Functions on the Consumption plan, App Service, and most managed PaaS runtimes have no sidecar, no extension, and no host to install onto. The engine still applies, but it moves off the edge: the platform streams its logs to a hub, and one central engine consumes that hub.

This is the same receiver pipeline that runs as a [sidecar](https://doc.log10x.com/engine/launcher/sidecar/) or a [Lambda extension](https://doc.log10x.com/engine/launcher/extension/), fed by a different input.

## :material-hexagon-multiple-outline: Architecture Flow

<div style="text-align: center;">

```mermaid
graph LR
    A["Managed app"] -->|"Diagnostic<br/>settings"| B["Event Hub"]
    B -->|"Consume"| C["OTel Collector"]
    C -->|"OTLP<br/>127.0.0.1:4317"| D["10x Engine"]
    D -->|"Regulated<br/>127.0.0.1:24225"| C
    C -->|"routeState<br/>routing"| E["Destination"]

    classDef src fill:#ff9900,stroke:#cc7a00,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef tenx fill:#059669,stroke:#047857,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef forwarder fill:#3b82f6,stroke:#1d4ed8,color:#ffffff,stroke-width:2px,rx:8,ry:8
    classDef destination fill:#6b7280,stroke:#4b5563,color:#ffffff,stroke-width:2px,rx:8,ry:8

    class A src
    class B src
    class C forwarder
    class D tenx
    class E destination
```

</div>

**1. The platform streams** its logs to the hub through its own diagnostic pipeline. No application change.

**2. The collector consumes** the hub and hands each record to the engine over loopback.

**3. The engine returns** the record carrying `tenx_hash` and `routeState`, exactly as in every other topology.

**4. A routing connector fans out** on `routeState`: the destination slice ships through the existing exporter, the offload slice lands in object storage, drops end there.

## :material-rocket-launch: Deployment

??? tenx-bootstrap "Step 1: Stream the platform's logs to a hub"

    On Azure, create an Event Hub namespace and hub, then add a [diagnostic setting](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/diagnostic-settings){target="\_blank"} on each resource with **Stream to an event hub** selected and the log categories you want regulated.

    ``` { .console .copy }
    az eventhubs namespace create -g <rg> -n <namespace> -l <region>
    az eventhubs eventhub create -g <rg> --namespace-name <namespace> -n logs
    az monitor diagnostic-settings create \
      --name tenx-stream \
      --resource <resource-id> \
      --event-hub-rule <auth-rule-id> \
      --event-hub logs \
      --logs '[{"categoryGroup":"allLogs","enabled":true}]'
    ```

    The same shape applies elsewhere: any managed platform that can publish its logs to a stream (Event Hub, Kinesis, Pub/Sub, Kafka) can feed this topology.

??? tenx-forwarderinputs "Step 2: Collector reads the hub, pairs with the engine"

    The receiver key is `azure_event_hub`, and `format: azure` unwraps the Azure Monitor envelope into individual records. The engine pairing below is the same loopback pattern used by every other topology.

    ```yaml
    receivers:
      azure_event_hub:
        connection: ${env:EVENTHUB_CONNECTION_STRING}
        group: $Default
        # 'azure' parses the Azure Monitor resource-log envelope from
        # diagnostic settings. For application logs written straight to the
        # hub use 'raw' -- the azure unmarshaler rejects plain text
        # (measured: "invalid character 'I' looking for beginning of value").
        format: azure
        # Checkpoints partition offsets so a restart resumes rather than
        # replaying. Without it, every restart re-reads the retention window
        # (measured: 400 sent, 2,800 delivered after restarts).
        storage: file_storage

      # Return path from the engine (loopback, same pod)
      otlp/tenx:
        protocols:
          grpc:
            endpoint: 127.0.0.1:24225

    processors:
      # Only with format: raw -- the payload arrives as a BYTES body, which
      # the engine cannot pattern (every event collapses to one identity).
      # Decode restores text; String() is not the fix, it yields base64
      # (both measured).
      transform/decode:
        log_statements:
          - context: log
            statements:
              - set(log.body, Decode(log.body, "utf-8"))

    extensions:
      file_storage:
        directory: /var/lib/otelcol/storage

    exporters:
      # Hand-off to the engine (loopback, same pod)
      otlp/tenx:
        endpoint: 127.0.0.1:4317
        tls:
          insecure: true

    connectors:
      routing/tenx:
        default_pipelines: [logs/tenx-destination]
        table:
          # context: log is required -- routeState is a LOG attribute
          - context: log
            condition: attributes["routeState"] == "offload"
            pipelines: [logs/tenx-offload]
          - context: log
            condition: attributes["routeState"] == "drop"
            pipelines: [logs/tenx-drop]
    ```

??? tenx-config "Step 3: Run the engine beside the collector"

    The engine and the collector run as peer containers in one pod, talking over loopback. Any container runtime works; verified on Azure Container Apps with this exact pairing (two containers, one app, localhost between them), and AKS, GKE, ECS, or a single VM are the same shape.

    ```
    tenx @run/input/forwarder/otel-collector @apps/receiver

    TENX_LICENSE_KEY=<your license JWT>
    outputOffload=true
    symbolMessageHashField=tenx_hash
    log10xMetricsEnabled=false
    ```

    On Kubernetes this is the [Receiver](https://doc.log10x.com/apps/receiver/deploy/) install with the collector as the paired forwarder. Nothing about the engine changes; only its input does.

??? tenx-monitoring "Step 4: Size the consumer"

    Unlike edge topologies, this one is centralized, so throughput is a capacity decision rather than a per-node one.

    | Consideration | Guidance |
    |---|---|
    | Partitions | The hub's partition count caps consumer parallelism. Scale replicas up to that number, not beyond |
    | Checkpoints | Without a storage extension a restart replays the retention window |
    | Egress | Logs leave the source region twice unless the consumer runs in the same region |
    | Ordering | Per-partition only, which pattern identity does not depend on |

??? tenx-config "Step 5: Auto-tune from a policy repo (optional)"

    The engine pulls a git repo and hot-reloads per-pattern mutes on every push, so anything that rewrites that repo on a schedule closes the tuning loop. Two schedulers are supported by `setup_recurring`: a GitHub Actions workflow (`github_actions`), or — keeping the whole loop inside the subscription — an Azure Container Apps scheduled Job (`container_apps_job`) that clones the repo, recomputes, and pushes.

    ```
    tenx @run/input/forwarder/otel-collector @apps/receiver @run/receive/rate @gitops

    # Pull lane -- engine 1.1.69 or later (GH_DEST)
    GH_ENABLED=true
    GH_TOKEN=<fine-grained PAT, contents:read on the policy repo>
    GH_REPO=<org>/<policy-repo>
    GH_BRANCH=main
    GH_SYNC_INTERVAL=30s
    GH_DEST=/tmp/policy
    rateReceiverLookupFile=/tmp/policy/test/mutes.csv
    rateReceiverFieldNames=message_pattern

    # <policy-repo>/test/mutes.csv -- <pattern>,<rate>:<untilEpochSec>:<reason>
    # 4102444800 is 2100-01-01, a placeholder; a past epoch mutes nothing, silently.
    pattern,disposition
    noisy_heartbeat_ok,0:4102444800:liveness spam OPS-1234
    ```

    `GH_DEST` mirrors the pulled repo to a stable path that `rateReceiverLookupFile` points into. The path must be writable by the engine user — the shipped container runs as uid 1000, so a root-level path like `/policy` fails the launch; `/tmp/policy` works everywhere. The mute format and severity floor are documented at the [rate receiver](https://doc.log10x.com/run/receive/rate/). The same loop on Lambda uses S3 instead of git, in the [extension](https://doc.log10x.com/engine/launcher/extension/) Step 5.

    An Azure Files share in place of the git repo does not work, and the failure is silent — measured on Container Apps: a REST upload changes the file without waking the engine's reload poll (17 minutes blind), and a second SMB client gets a sharing violation from the reader's own mount, so a scheduled Job cannot write the file either. Deliver policy through the git loop above.

## :material-scale-balance: When to use this pattern

Use it where the platform runs no process of yours. Where one can run, a
[sidecar](https://doc.log10x.com/engine/launcher/sidecar/) or an
[extension](https://doc.log10x.com/engine/launcher/extension/) regulates at the
source, so volume never leaves the node and the platform never bills to
transport it.

Streaming regulates after the platform has emitted and transported the logs,
which still removes the larger cost: what the destination stores, indexes and
queries.