# Endpoint metrics | Modal Docs

- **URL:** https://modal.com/docs/guide/endpoint-metrics
- **Summary:** Every endpoint reports live inference metrics so you can see how it’s performing under real traffic — latency, throughput, and how many requests are in flight. Open an endpoint from the Endpoints tab and go to the Activity view to see them.

Copy page

Endpoint metrics
================

Every endpoint reports live inference metrics so you can see how it’s performing under real traffic — latency, throughput, and how many requests are in flight. Open an endpoint from the **Endpoints** tab and go to the **Activity** view to see them.

There are two types of metrics available:

*   **Inference metrics** — LLM engine-specific metrics designed to give you more performance observability.
*   **Server metrics** — the standard Modal container health metrics.

What the metrics mean 

**Latency** (reported as p50 / p95 / p99):

*   **Time to first token (TTFT)** — how long after a request arrives before the first output token streams back. The number users feel first.
*   **Inter-token latency (ITL)** — average gap between successive output tokens. Drives perceived “typing speed.”
*   **End-to-end latency (E2E)** — total time to complete a request.

**Throughput:**

*   **Requests per second (QPS)** — request arrival rate.
*   **Token throughput** — tokens/second, split into prefill (processing the prompt, with a separate line for cache-hit tokens) and decode (generating output).

**Request load:**

*   **Request activity** — the rate of requests arriving at and completing on the endpoint over time.
*   **Running** — requests currently being processed.
*   **Queued** — requests waiting for a free slot. Sustained queueing means the fleet is saturated and scaling up.

**Speculative decoding** (only for recipes that use it) — the average number of draft tokens accepted per step; higher means speculation is paying off.

Caveats 

*   **Metrics need traffic.** Latency and throughput are computed over recent rolling windows; an idle or scaled-to-zero endpoint shows no current data.
*   **Cold starts skew early numbers.** The first requests after a scale-up include model load time. Look at steady-state windows when evaluating performance.
*   **Percentiles need volume.** p95/p99 are only meaningful once enough requests have accumulated in the window.
*   Endpoint metrics are available in the dashboard. To get repeatable performance numbers under a controlled load, [run a benchmark](https://modal.com/docs/guide/endpoint-benchmarks)
    .

[Endpoint metrics](https://modal.com/docs/guide/endpoint-metrics#endpoint-metrics)
[What the metrics mean](https://modal.com/docs/guide/endpoint-metrics#what-the-metrics-mean)
[Caveats](https://modal.com/docs/guide/endpoint-metrics#caveats)