LLM Inference Metrics Explained: TTFT, TPOT, ITL and Goodput
LLM inference performance is not one speed number. TTFT describes the wait for the first output, TPOT and ITL describe how output continues, and E2EL describes the whole request. TPS and RPS measure processing capacity; SLO and goodput tell you whether that capacity meets a stated service goal.
Start with the symptom: a slow start suggests checking TTFT; pauses after the start suggest checking output gaps; an overloaded service needs throughput and SLO checks together. These are directions for investigation, not proof of a root cause.
Documentation checked October 18, 2026. All numerical examples and both diagrams are hypothetical. The short Python arithmetic was actually executed; no LLM, paid API, GPU or inference-engine benchmark was run.
1. TTFT — how long until the answer starts?
Definition: Time to First Token is elapsed time from a defined request start to the first generated output token received at the measurement point. Its unit is milliseconds or seconds; lower means a shorter initial wait under comparable conditions.
A token is a model’s processing unit, not necessarily a word or character. Prefill processes the input; decode generates output. Client-observed TTFT can also include queueing and transport, so it is not simply GPU prefill time. Initial empty messages and HTTP headers are not generated text. NVIDIA NIM’s metric definitions.
0 ms
500 ms · TTFT = 500 ms
1,300 ms · E2EL = 1,300 ms
After-first output span: 800 ms. Six output tokens mean five intervening gaps.
Use: compare initial responsiveness at the same input size and load. Caution: receiving a first chunk of text is only a proxy when you do not have actual token events. Keep the measurement boundary explicit.
2. TPOT — how fast does the answer continue on average?
Definition: Time Per Output Token describes average elapsed generation time per output token after the first. Here the unit is ms/token, and the calculation assumes one delivered token per event, N greater than one, and a last-token completion boundary.
(1,300 − 500) ÷ (6 − 1) = 160 ms/token
Subtracting one matters: six tokens create five gaps. With zero or one output token, this interval-based formula is undefined. A shorter average interval means faster continuation, but it does not guarantee that every individual gap is short.
Use: compare continuation speed without hiding the initial wait inside the average. Caution: do not assume every tool’s “time per token” has the same numerator or denominator. Some metric names overlap, so read the tool’s definition before comparing results.
3. ITL — where are the pauses hidden?
Definition: in this guide, individual Inter-Token Latency values are successive token-arrival gaps. Their unit is milliseconds. Our six-token example arrives at 500, 600, 700, 1,000, 1,100 and 1,300 ms, giving gaps of 100, 100, 300, 100 and 200 ms.
Gap 1: 100 ms
Gap 2: 100 ms
Gap 3: 300 ms
Gap 4: 100 ms
Gap 5: 200 ms
Average: 160 ms. Longest gap: 300 ms.
Important naming difference: NIM/AIPerf also uses ITL for an average interval and treats it as TPOT. Therefore “ITL” on a dashboard does not by itself tell you whether you are seeing an individual gap, a request average or an aggregated distribution. See the tool-specific ITL formula.
Use: inspect uneven streaming instead of relying only on a mean. A chunk is a delivery bundle and may contain several tokens; inter-chunk latency measures gaps between bundles. Network chunks and SSE events are not automatically tokens. Client gap measurements cannot, by themselves, identify server compute or memory as the cause.
4. E2EL — when is the complete result available?
Definition: End-to-End Latency measures elapsed time from request start to the stated completion boundary. Here that boundary is receipt of the last output token, giving 1,300 ms. Lower is better for the same task and comparable output length.
Under our interval assumptions, E2EL = TTFT + TPOT × (N − 1), or 500 + 160 × 5 = 1,300 ms. This is an accounting identity for this example, not a universal substitute for measuring completion.
Use: judge when a generated report, summary or code answer is ready. Caution: an application may continue waiting for a completion signal or perform extra work after the final text. Name that endpoint separately. NIM’s documented AIPerf definition excludes a trailing done/empty signal. End-to-end latency boundary.
5. TPS — how much token work gets through?
Definition: Tokens Per Second is a rate whose scope must be stated. It may mean total server output throughput or the output rate of one request. Both use tok/s, but answer different questions.
For an invented closed benchmark run with 9,000 output tokens over 12 seconds, aggregate output throughput is 750 tok/s. For our one six-token request, including its initial wait gives 6 ÷ 1.3 ≈ 4.62 tok/s. Excluding that wait and using five post-first intervals gives 5 ÷ 0.8 = 6.25 tok/s. State which calculation you mean.
Use: evaluate capacity, then inspect request latency alongside it. Caution: output-only throughput is not input-plus-output throughput. A higher aggregate TPS does not establish a faster individual experience. Record the benchmark window and request inclusion rules. NVIDIA’s benchmarking discussion.
6. RPS — how many requests finish?
Definition: Requests Per Second here means successful completed requests divided by the declared measurement duration. Our invented run completes 120 requests in 12 seconds: 10 req/s. This is not the same as the rate of newly arriving requests.
Concurrency is the number of requests active at once. Batching processes work from multiple requests together. These are load and scheduling conditions, not synonyms for RPS. If arrivals exceed sustainable completion capacity, outstanding work can grow. Load-control distinctions.
Use: size a service for a defined request mix. Caution: ten short answers and ten long answers can have the same RPS but very different token work. Report failed requests, timeouts, warmup and sequence lengths rather than comparing isolated headline rates.
7. SLO — what counts as an acceptable response?
Definition: a Service Level Objective states the performance target. For this lesson, a request passes only if TTFT ≤ 800 ms and TPOT ≤ 200 ms/token. These are invented teaching thresholds, not recommended targets for every application.
P50 is the median. P95 and P99 summarize the high-latency tail: roughly 95% or 99% of observations are at or below the corresponding value, subject to the estimator. Specify the metric, sample count and percentile method. A P95 of request-average TPOT does not describe the distribution of every individual token gap.
Use: make the tolerated wait explicit instead of saying “fast enough.” Caution: a mean can hide slow users, and a small sample gives unstable tail estimates. Latency objectives also do not prove answer correctness. Track quality separately. vLLM’s benchmark CLI exposes percentile reporting and request-level goodput objectives. vLLM benchmark options.
8. Goodput — how much useful capacity meets that target?
Definition: in this guide, goodput is successful completed requests meeting all stated latency objectives divided by the same run duration. Its unit is req/s, not a percentage. The throughput-with-latency-constraints perspective is central to the DistServe research paper.
72 requests meet both objectives / 12 s = 6 req/s goodput
72 / 120 = 60% attainment among completed requests.
Use: compare throughput at an acceptable latency, not maximum throughput alone. Caution: qualifying criteria, error treatment, workload and time window must match. These counts are declared examples, not measured service capacity. vLLM supports per-request TTFT, TPOT and E2EL constraints for its goodput calculation; always verify the installed version. Goodput configuration.
9. Reproduce the arithmetic, then choose the right next test
The following complete example was executed locally with Python 3.12.14 on Windows. It only calculates the invented timestamps and counters above. It makes no network request, sleeps for no simulated delay and measures no real inference system.
Python · synthetic arithmetic only
# Synthetic arithmetic only: no network, model or GPU benchmark.
arrivals_ms = [500, 600, 700, 1000, 1100, 1300]
gaps_ms = [b - a for a, b in zip(arrivals_ms, arrivals_ms[1:])]
ttft_ms = arrivals_ms[0]
e2el_ms = arrivals_ms[-1]
tpot_ms = (e2el_ms - ttft_ms) / (len(arrivals_ms) - 1)
assert gaps_ms == [100, 100, 300, 100, 200]
assert tpot_ms == 160
print(f"TTFT={ttft_ms} ms")
print(f"TPOT={tpot_ms:.0f} ms/token")
print(f"Individual gaps={gaps_ms} ms")
print(f"E2EL={e2el_ms} ms")
print(f"Request output rate={len(arrivals_ms)/(e2el_ms/1000):.2f} tok/s")
print(f"System output throughput={9000/12:.0f} tok/s")
print(f"Completed-request rate={120/12:.0f} req/s")
print(f"SLO-qualified goodput={72/12:.0f} req/s")
print(f"SLO attainment={72/120:.0%}")
Observed local output · not a model benchmark
TTFT=500 ms
TPOT=160 ms/token
Individual gaps=[100, 100, 300, 100, 200] ms
E2EL=1300 ms
Request output rate=4.62 tok/s
System output throughput=750 tok/s
Completed-request rate=10 req/s
SLO-qualified goodput=6 req/s
SLO attainment=60%
For an actual streaming client, continue with our runnable C# TTFT and E2EL timing lab. That lab uses a simulated SSE server over real local HTTP; it is not a paid-provider or GPU result. It answers the instrumentation question, while this guide explains the metrics.
| Question | Start with |
|---|---|
| Why does the answer start late? | TTFT and queue/input/cache conditions |
| Why does it pause after starting? | Individual token or chunk gaps, with scope verified |
| When is the whole answer usable? | E2EL and its endpoint |
| How much can the service handle? | TPS, completed RPS and matching workload |
| How much meets the promised latency? | SLO attainment and goodput, plus errors |
Takeaway: choose one user question, a measurement boundary and a workload before comparing numbers. Keep first response, continuation, total completion and system capacity separate. A big tok/s value is only informative when you know whose tokens, over which window, at what latency and quality.
Appendix. Benchmark conditions and primary sources
Before a real comparison, record model and tokenizer versions, input sequence length (ISL), output sequence length (OSL), language, hardware, inference engine and precision. Also record cache state, concurrency or arrival process, warmup, request errors and measurement location. Client and server timestamps describe different boundaries; elapsed durations should use an appropriate monotonic clock.
- NVIDIA NIM Benchmarking 2.0.0: metrics — tool-specific TTFT, average ITL, endpoint and throughput definitions.
- NVIDIA: LLM inference benchmarking fundamentals — rate scope, workload and load-control context.
- vLLM bench serve reference — percentile and goodput options; latest documentation checked October 18, not a locally installed/tested vLLM version.
- DistServe, OSDI 2024 — serving under TTFT/TPOT latency constraints; no paper benchmark result is presented as our measurement.
Both diagrams and all example values are original educational material. Only local arithmetic was executed. No model quality, tokenization behavior, provider performance, investment return or search-traffic improvement was measured or promised.
댓글
댓글 쓰기
안녕하세요 :)