Calculate LLM p50, p95 and SLO Goodput from Request Logs in Python
To calculate LLM latency percentiles and SLO Goodput from request logs, keep two views: compute p50 and p95 for successful responses, then count requests that meet every declared latency threshold over a stated time window. Keep errors and timeouts in the all-attempt attainment denominator—not in a made-up zero-latency bucket.
A dashboard can show a reasonable median while half the requests fail your definition of a useful response. This small Python exercise makes that difference visible, without a GPU, API key or paid LLM call.
Start here: if p95 differs between tools, first compare their percentile method and included requests. Save the two files below and run python analyze_logs.py requests.jsonl. These are synthetic logs; the analyzer really ran, but no model generated these timings.
Checked October 23, 2026. Executed on Windows with CPython 3.14.3, using only the standard library. Thresholds and the 60-second window are teaching choices, not recommended production targets.
New to the vocabulary? Start with the LLM inference metrics guide. For collecting streaming observations rather than analyzing saved logs, see the C# TTFT and E2EL measurement exercise.
1. The terms, in one minute
- TTFT: time from sending a request to the first nonempty model content, in milliseconds (ms).
- p50 / p95: positions in a latency distribution. This exercise uses R7 linear interpolation and successful requests only.
- E2EL: time from sending the request to its final content, in ms; it is not the time of a later empty completion marker.
- SLO Goodput: successful requests meeting both our TTFT and E2EL thresholds, divided by observation seconds; unit: requests/s.
- Attainment: that same good-request count divided by all attempted requests, including errors and timeouts; unit: fraction or percent.
NVIDIA’s metric definitions distinguish first content, final content and aggregate throughput. A chunk may contain multiple tokens; this exercise never treats HTTP/SSE chunks as individual tokens or derives token-level ITL from them.
2. Create a fully resolved, anonymous log
Save this as requests.jsonl: one JSON object per line. The six attempts were invented for this tutorial. No prompts, account identifiers or customer data are included.
{"id":"r1","status":"ok","start_ms":0,"end_ms":1000,"ttft_ms":100}
{"id":"r2","status":"ok","start_ms":10000,"end_ms":11200,"ttft_ms":200}
{"id":"r3","status":"ok","start_ms":20000,"end_ms":21800,"ttft_ms":300}
{"id":"r4","status":"ok","start_ms":30000,"end_ms":35000,"ttft_ms":900}
{"id":"r5","status":"error","start_ms":40000,"end_ms":40400,"ttft_ms":null}
{"id":"r6","status":"timeout","start_ms":50000,"end_ms":56000,"ttft_ms":null}All timestamps are elapsed milliseconds from the start of a declared [0, 60,000 ms] observation window. Each attempt starts before its end, and every outcome is known inside it. For an ok row, end_ms marks final content. For an error or timeout, it marks the terminal failure or deadline—not a successful response.
For example, r3 has E2EL = 21,800 − 20,000 = 1,800 ms. Its TTFT is 300 ms. Our teaching SLO accepts a request only when it succeeds, TTFT ≤ 500 ms, and E2EL ≤ 2,000 ms. Equality passes.
6 attempts in a declared 60-second window
Keep every outcome; reject invalid records.
↓
4 successful responses
Use their measured TTFT and E2EL for p50 and p95.
3 meet both thresholds; 1 is too slow.
↓
1 error + 1 timeout stay visible
Goodput = 3 / 60 = 0.05 requests/s.
Attainment = 3 / 6 = 50%.
3. Declare how p50 and p95 are calculated
Sort n latency values in ascending order, indexed from zero. For percentile p, R7 uses h = (n − 1) × p / 100, then interpolates between the values at floor(h) and ceil(h). NIST’s percentile reference explains that software can use different estimators, particularly noticeable with small samples. This is a declared choice, not the only valid method.
The successful E2EL values are 1,000, 1,200, 1,800 and 5,000 ms. The p50 lies halfway between 1,200 and 1,800: 1,500 ms. For p95, h = 2.85, so the result is 1,800 + 0.85 × (5,000 − 1,800) = 4,520 ms. It is an interpolated value, not an actual request.
Only four successful samples support those numbers. They demonstrate the arithmetic; they cannot establish production tail reliability, confidence bounds or a maximum sustainable capacity.
4. Run the standard-library Python analyzer
Save this complete script as analyze_logs.py in the same folder. It stops on duplicate IDs, unknown outcomes, missing successful TTFT, nonfinite or impossible timings, and attempts outside the fully resolved window. It does not silently drop invalid rows to improve the result.
import json
import math
import sys
def percentile(values, p):
"""R7 linear interpolation; caller supplies validated finite values."""
if not values:
return None
x = sorted(values)
h = (len(x) - 1) * p / 100
lo, hi = math.floor(h), math.ceil(h)
return x[lo] + (h - lo) * (x[hi] - x[lo])
def analyze(rows, window_ms, ttft_slo_ms, e2el_slo_ms):
for value in (window_ms, ttft_slo_ms, e2el_slo_ms):
if type(value) not in (int, float) or not math.isfinite(value) or value < 0:
raise ValueError('window/SLO must be finite nonnegative numbers')
if window_ms == 0:
raise ValueError('window must be positive')
seen, ttfts, e2els = set(), [], []
good = errors = timeouts = 0
for r in rows:
if not isinstance(r, dict):
raise ValueError('each row must be an object')
rid = r.get('id')
if not isinstance(rid, str) or not rid or rid in seen:
raise ValueError('id must be unique and nonempty')
seen.add(rid)
status = r.get('status')
if status not in ('ok', 'error', 'timeout'):
raise ValueError('unresolved/unknown status')
start, end = r.get('start_ms'), r.get('end_ms')
for value in (start, end):
if type(value) not in (int, float) or not math.isfinite(value):
raise ValueError('timestamps must be finite numbers')
if not 0 <= start < window_ms or not start <= end <= window_ms:
raise ValueError('attempt not fully resolved inside this window')
first = r.get('ttft_ms')
if first is not None:
if type(first) not in (int, float) or not math.isfinite(first) or not 0 <= first <= end - start:
raise ValueError('invalid first-token time')
if status == 'ok':
if first is None:
raise ValueError('successful content response needs TTFT')
elapsed = end - start
ttfts.append(first)
e2els.append(elapsed)
good += first <= ttft_slo_ms and elapsed <= e2el_slo_ms
else:
errors += status == 'error'
timeouts += status == 'timeout'
return {
'window_seconds': window_ms / 1000,
'percentile_method': 'R7 linear interpolation; successful requests only',
'attempts': len(rows), 'successful': len(e2els),
'errors': errors, 'timeouts': timeouts, 'good': good,
'attainment': good / len(rows) if rows else None,
'goodput_req_s': good / (window_ms / 1000),
'successful_ttft_ms': {f'p{p}': percentile(ttfts, p) for p in (50, 95)},
'successful_e2el_ms': {f'p{p}': percentile(e2els, p) for p in (50, 95)},
}
if __name__ == '__main__':
with open(sys.argv[1], encoding='utf-8') as f:
rows = [json.loads(line) for line in f if line.strip()]
# This tutorial's fixed, fully resolved 60-second observation window.
print(json.dumps(analyze(rows, 60000, 500, 2000), indent=2, allow_nan=False))Open a terminal in that folder and run:
python --version
python analyze_logs.py requests.jsonlNo packages need installing. The script reads only the named local file and prints JSON; it sends no network requests. Use a trusted, bounded-size log export: this teaching script loads the file into memory and is not a hardened production ingestion service.
5. Read the actual execution result
The following output came from executing the script above against the six synthetic lines on CPython 3.14.3:
{
"window_seconds": 60.0,
"percentile_method": "R7 linear interpolation; successful requests only",
"attempts": 6,
"successful": 4,
"errors": 1,
"timeouts": 1,
"good": 3,
"attainment": 0.5,
"goodput_req_s": 0.05,
"successful_ttft_ms": {
"p50": 250.0,
"p95": 810.0
},
"successful_e2el_ms": {
"p50": 1500.0,
"p95": 4520.0
}
}| Metric | Result |
|---|---|
| E2EL p50 | 1,500 ms |
| E2EL p95 | 4,520 ms |
| SLO pass | 50% |
| Goodput | 0.05 req/s |
The median successful completion fits under 2 seconds, yet only half of all attempts qualify. Dividing 3 by 4 would answer a different question—“what fraction of successes meet the SLO?”—and hide the error and timeout. Dividing 3 by 60 seconds is a rate, not a success percentage.
The AIPerf Goodput tutorial similarly distinguishes good requests per second from the fraction meeting all configured thresholds. This script is independent of AIPerf: it does not run that tool or claim identical accounting.
6. What was tested—and what was not
The accompanying local test suite ran 8 unittest methods successfully. It checks R7 arithmetic, empty/singleton samples, the all-attempt denominator, threshold equality, empty logs, invalid windows or thresholds, malformed request records, and duplicate IDs. Empty latency samples return null, not 0 ms; an empty log has undefined attainment and zero counted Goodput for the declared window.
Executed: local Python parsing, validation and arithmetic on synthetic fixtures. Not executed: an LLM, GPU workload, paid endpoint, real streaming benchmark, AIPerf or load generator. The test result validates this calculation example, not the speed or capacity of any deployed model.
7. Before using real request logs
- Match the measurement boundaries. Use one client-side clock for request submission, first content and final content. Keep tokenizer, model, input/output lengths and sampling settings with the run metadata. Do not mix server compute time with end-user elapsed time.
- Choose the cohort and window explicitly. Here, the denominator is a fixed 60 seconds including idle time. NVIDIA’s referenced batch throughput uses first request to last response; that is a different interval. Do not expect the same rate from different boundaries.
- Resolve boundary-crossing attempts deliberately. This example rejects attempts starting before or ending after the window. Do not simply filter those rows away. For real traffic, define an admission cohort with a declared drain policy, or a fixed completion window with outstanding/censored counts, and report its limitations. A timeout is not a completed response whose latency you know.
- Keep unsuccessful outcomes. Define warmup, retries, cancellations and invalid-record handling before comparison. A retry is another attempt unless you explicitly compute a separate logical-request measure. Never remove slow failures after seeing the chart.
- Validate tail conclusions with enough representative data. Report sample counts, duration and workload mix. Do not average p95 values from separate runs and call that the combined p95; pool comparable raw samples or use a suitable mergeable histogram with declared error.
If software claims a large efficiency gain, this kind of accounting is one part of the evidence—not proof by itself. Compare equivalent useful output, latency targets and failure handling before connecting benchmark throughput to hardware needs. See the Flapping Airplanes GPU-and-memory evidence checklist for that broader distinction.
Summary: publish the denominator beside the percentile
A useful result states the workload, n, percentile estimator, success/failure policy, observation window, latency thresholds, Goodput and all-attempt attainment together. “p95 = 4,520 ms” alone cannot tell you whether the service is acceptable.
For this tiny synthetic fixture: 6 attempts, 4 successes, 3 jointly good requests, R7 E2EL p50/p95 of 1,500/4,520 ms, 50% attainment and 0.05 good requests/s over 60 seconds. Those are reproducible arithmetic results—not a capacity recommendation.
Appendix. Why no tokens/s or ITL?
The schema has no output-token counts or token timestamps, so the script cannot honestly produce token throughput or token-level ITL. It also does not assess answer correctness. In a real application, “good” may require a separate task-quality or validity check; add that declared condition rather than assuming a fast HTTP success is useful.
Sources and reproducibility
- NIST/SEMATECH handbook: percentiles and estimator differences—R7 choice and interpolation.
- NVIDIA NIM Benchmarking metrics, visible documentation version 2.0.0—latency/content and batch interval definitions.
- AIPerf: benchmark Goodput, visible documentation version 0.13.0—joint thresholds, rates and fractions.
- Python 3.14 JSON documentation and math documentation—standard-library parsing, finite-value checks and interpolation helpers.
Sources checked October 23, 2026. Original synthetic data, Python example and schematic; no third-party figure reproduced. Python version actually executed: 3.14.3 on Windows. No API keys, package installation, model downloads or paid calls required.
댓글
댓글 쓰기
안녕하세요 :)