---
icon: simple/prometheus
---

The 10x Engine publishes its metrics as Prometheus series: per-pattern volume from the [aggregators](/run/aggregate/), and a small set of health series about the pipeline itself. They land in the backend the pipeline's [metric output](/run/output/metric/) points at, and that backend's own API reads them. This page names the series and the labels, and shows the Prometheus HTTP API calls that read them.

!!! tip "Used by the MCP"
    The [MCP cost tools](/apps/mcp/) run these queries against the metrics backend configured for the environment.

Use these series to build cost analyses, dashboards, and automation per pattern.

## :material-api: Endpoints

Every Prometheus-compatible backend (Prometheus, Mimir, Cortex, Amazon Managed Prometheus, Grafana Cloud, Google Managed Prometheus) serves the [Prometheus HTTP API](https://prometheus.io/docs/prometheus/latest/querying/api/){target="_blank"}. The endpoints below are the ones the MCP uses. Authenticate the way that backend requires.

| Endpoint | Methods | Description |
|----------|---------|-------------|
| [/query](https://prometheus.io/docs/prometheus/latest/querying/api/#instant-queries){target="_blank"} | POST/GET | Instant queries |
| [/query_range](https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries){target="_blank"} | POST/GET | Range queries |
| [/query_exemplars](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-exemplars){target="_blank"} | POST/GET | Query exemplars |
| [/series](https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers){target="_blank"} | POST/GET | Find series by label |
| [/labels](https://prometheus.io/docs/prometheus/latest/querying/api/#getting-label-names){target="_blank"} | POST/GET | Get label names |
| [/label/{name}/values](https://prometheus.io/docs/prometheus/latest/querying/api/#querying-label-values){target="_blank"} | GET | Get label values |

??? tenx-api "Example: Query Pipeline Health"
	```bash
	curl -s "$PROMETHEUS_URL/api/v1/query?query=tenx_pipeline_up"
	```

	Response:
	```json
	{
	  "status": "success",
	  "data": {
	    "resultType": "vector",
	    "result": [
	      {
	        "metric": {
	          "__name__": "tenx_pipeline_up",
	          "tenx_app": "receiver",
	          "tenx_env": "edge",
	          "tenx_host_name": "edge-node-1",
	          "tenx_pipeline_uuid": "a1b2c3d4-...",
	          "tenx_reported_name": "my-10x-pipeline"
	        },
	        "value": [1704109200, "1"]
	      }
	    ]
	  }
	}
	```

Datadog, Elastic and CloudWatch receive the same series through their own [metric outputs](/run/output/metric/), under the naming each of those pages describes.

## :material-test-tube: Demo Queries

An engine installed with an anonymous demo license writes to the log10x hosted backend at `prometheus.log10x.com`. The same license reads back what it wrote, without an API key, via the `/api/v1/demo/*` mirror of the query endpoints. Authenticate with the demo license JWT in an `Authorization: Bearer` header:

| Endpoint | Methods |
|----------|---------|
| `/api/v1/demo/query` | POST/GET |
| `/api/v1/demo/query_range` | POST/GET |
| `/api/v1/demo/query_exemplars` | POST/GET |
| `/api/v1/demo/series` | POST/GET |
| `/api/v1/demo/labels` | POST/GET |
| `/api/v1/demo/label/{name}/values` | GET |

Series read through the demo endpoints carry one more label, `TENX_Tenant`: the environment ID the hosted gateway stamps on each series as it is written.

!!! note "Demo window"
    Demo queries are bounded to the **last 3 hours of data** and are rate limited. A request whose `start`, `end`, or `time` falls outside the 3-hour window, or whose PromQL range selector (e.g. `[24h]`) exceeds it, returns `400` with an explanation. For full history, point the engine's metric output at a backend of your own.

??? tenx-api "Example: Query demo data with a demo license"
	```bash
	# Mint a demo license (response includes { "license": "<jwt>", ... })
	LICENSE=$(curl -s -X POST "https://api.log10x.com/api/v1/license/demo" | jq -r .license)

	# Read back what that license wrote (last 3 hours only)
	curl -s -H "Authorization: Bearer $LICENSE" \
	  "https://prometheus.log10x.com/api/v1/demo/query?query=tenx_pipeline_up"
	```

## :material-chart-line: Available Metrics

### Pipeline Health

| Metric | Type | Description |
|--------|------|-------------|
| `tenx_pipeline_up` | Gauge | Pipeline running status (1 = up) |
| `tenx_pipeline_bootstrap_time_seconds` | Gauge | Pipeline startup time |
| `tenx_pipeline_runtime_seconds` | Gauge | Total pipeline runtime |
| `tenx_pipeline_input_time_seconds` | Gauge | Time spent processing input |
| `tenx_pipeline_invocation_time_seconds` | Gauge | Function invocation time |
| `tenx_launch_failure_total` | Counter | A pipeline failed to launch |

### Event Counters

| Metric | Type | Description |
|--------|------|-------------|
| `totalEvents_total` | Counter | Lines read from the input, counting every line of a multi-line group |
| `totalBytes_total` | Counter | UTF-8 bytes read from the input |
| `tokenized_total` | Counter | Input entries matched to a template |
| `nonTokenized_total` | Counter | Input entries no template matched |
| `eventGroups_total` | Counter | Template-matched entries, each a group of one or more lines |
| `singleEvents_total` | Counter | Unmatched entries, one line each |

### Volume Metrics

| Metric | Type | Description |
|--------|------|-------------|
| `all_events_summaryVolume_total` | Counter | Total event volume |
| `all_events_summaryBytes_total` | Counter | Total event bytes |
| `emitted_events_summaryVolume_total` | Counter | Emitted event volume |
| `emitted_events_summaryBytes_total` | Counter | Emitted event bytes |
| `emitted_events_optimized_size_total` | Counter | Optimized output size |
| `indexed_events_summaryVolume_total` | Counter | Indexed event volume |
| `indexed_events_summaryBytes_total` | Counter | Indexed event bytes |
| `streamed_events_summaryVolume_total` | Counter | Streamed event volume |
| `streamed_events_summaryBytes_total` | Counter | Streamed event bytes |

## :material-tag-outline: Available Labels

Labels allow filtering and grouping metrics by dimension. They come in three groups.

**Pipeline labels** ride on every pipeline health series and event counter.

| Label | Description |
|-------|-------------|
| `tenx_reported_name` | Pipeline name from the config |
| `tenx_pipeline_uuid` | Unique pipeline instance ID |
| `tenx_host_name` | Host running the pipeline |
| `tenx_parent_uuid` | Parent pipeline ID, present only when one is set |

**Runtime attributes** ride on every series, including the volume metrics. They are set with `runtimeAttributes` in the app config, and the engine prefixes each name with `tenx_`. A deployment can add its own.

| Label | Description |
|-------|-------------|
| `tenx_app` | Application: `receiver`, `reporter`, `retriever`, `compile` |
| `tenx_env` | Where the app runs: `edge` for the Receiver and the Reporter, `cloud` for the Retriever and the compiler, `local` for the MCP |
| `tenx_sub_app` | Retriever sub-application: `index`, `query`, `stream`, `sub-query` |
| `tenx_fwd_input` | Log forwarder input source, for example `cloudwatch` |

**Enrichment labels** ride on the volume metrics. The [initialize](/run/initialize/) modules extract them from each event.

| Label | Description |
|-------|-------------|
| `message_pattern` | Matched message template |
| `tenx_hash` | Hash of the matched template, stable across hosts and restarts |
| `severity_level` | Log severity level |
| `http_code` | HTTP response code |
| `http_message` | HTTP response message |
| `k8s_namespace` | Kubernetes namespace |
| `k8s_container` | Kubernetes container name |
| `k8s_pod` | Kubernetes pod name. Off by default; set `podNameField` in the k8s module to enable it |
| `tenx_user_service` | User-defined service label |
| `tenx_user_process` | User-defined process label. Enabled together with `k8s_pod` |
| `index_app` | Target index application (Retriever) |
| `index_file` | Target index file (Retriever) |

## :material-code-braces: Query Examples

The examples read a Prometheus-compatible backend at `$PROMETHEUS_URL`. Add the authentication that backend requires.

??? tenx-api "List All Metric Names"
	```bash
	curl -s "$PROMETHEUS_URL/api/v1/label/__name__/values" | jq
	```

??? tenx-api "Query Event Volume by Application"
	```bash
	curl -s "$PROMETHEUS_URL/api/v1/query?query=sum(totalEvents_total)by(tenx_app)" | jq
	```

??? tenx-api "Query Range (Last Hour)"
	```bash
	curl -s "$PROMETHEUS_URL/api/v1/query_range?\
	query=rate(totalEvents_total[5m])&\
	start=$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)&\
	end=$(date -u +%Y-%m-%dT%H:%M:%SZ)&\
	step=60s" | jq
	```

??? tenx-api "Get Series Metadata"
	```bash
	curl -s "$PROMETHEUS_URL/api/v1/series?match[]=tenx_pipeline_up" | jq
	```