Measure LLM Streaming TTFT and E2EL in C#: A Beginner’s Hands-On Guide
Start a monotonic timer just before sending the request, record the first non-empty generated-text event, and keep the last-text timestamp separate from the completion signal. This guide shows the distinction in a runnable C# streaming lab.
C# code over real loopback HTTP, using a simulated SSE generator. No actual LLM was called. No model, provider, GPU, token count or paid inference performance was measured.
Environment checked: 2026-10-16 · Windows build 10.0.26200 x64 · .NET SDK 10.0.400 · .NET runtime 10.0.11. Other operating systems and .NET versions were not executed for this article.
1. Definitions first: what will we measure?
- Token: a model’s processing unit. It is not necessarily a word, character, HTTP chunk or SSE event.
- TTFT — Time to First Token: wait from a defined request start to the first generated output token at the measurement point. This text client uses the first non-empty generated-text event as a proxy, not a token-generation timestamp.
- E2EL — End-to-End Latency: total latency for a request. In our successful-text result,
e2elLastTextMsends at the last non-empty text event. An incomplete request has no successful E2EL result. - Application-completion latency: wait to a documented success signal. Our
completionMsends at the simulator’s[DONE]event. Do not silently compare it with last-token benchmark latency. - SSE — Server-Sent Events: UTF-8 text events sent over HTTP. A blank line finishes an event; multiple
data:lines are joined with newlines. - Delta / metadata / chunk: a delta is an incremental text update; metadata describes the response; a transport chunk is a delivery unit. None is guaranteed to equal one token.
2. Follow one request on the timeline
The server can send headers immediately and still make you wait for text. Timing headers would miss that wait. The first metadata frame can also arrive before any generated text.
Figure 1 · One measured local request
REAL C# EXECUTION · SIMULATED GENERATOR · NOT AN LLM BENCHMARK
Start the monotonic clock immediately before GetAsync.
Connection ready. Metadata and an empty delta are not generated text.
“Hello” arrives. This is our client-side TTFT proxy.
Final text is received. Successful last-text E2EL ends here.
Application completion is observed later. It is a different endpoint.
Last-text E2EL = last text − start
Completion latency = done event − start
The last text and the final success signal can arrive at different times. NVIDIA’s GenAI-Perf discussion explicitly excludes a trailing done/empty response from its end-to-end token latency. Our output therefore exposes both endpoints, rather than calling every stop time the same E2EL. See NVIDIA’s explanation of metric boundaries.
Figure 2 · What does this client actually measure?
HTTP request → response headers → SSE event parsing.
Artificial delays → JSON text events → [DONE]. Real local HTTP; no model, GPU or paid API.
First text, last text, successful completion and failure status.
Network / gateway → queue → prefill → decode → streamed output.
A client timer alone cannot separate these internal stages. Server instrumentation is needed for that attribution.
3. Run the C# example locally
Purpose: learn the measurement code without buying inference credits or configuring credentials. The mock emits {"delta":"…"} events and a final [DONE]. This is a deliberately narrow teaching protocol, not a provider-compatible LLM client.
- Install a .NET 10 SDK if it is not already available. Confirm the SDK with
dotnet --version. - Create a console project using the commands below.
- Replace the generated
Program.cswith the complete program in the Appendix. There are no third-party package dependencies. - Run
normal, then try the failure scenarios. Each run starts and stops its own loopback server; it is not left running in the background.
dotnet --version
dotnet new console -n StreamingTimingLab -f net10.0
# Replace StreamingTimingLab/Program.cs with the full code below.
dotnet run --project StreamingTimingLab -c Release -- normalThe key client setting: use ResponseHeadersRead so the HTTP operation returns before the entire body has been buffered. Pass an overall cancellation token into body reads as well: the usual HttpClient timeout does not automatically cover the later body-reading phase in this mode. Microsoft documents this timeout boundary.
// Excerpt only. The Appendix contains the complete runnable program.
using var response = await http.GetAsync(
url, HttpCompletionOption.ResponseHeadersRead, cancel.Token);
// After parsing a non-empty text delta:
result.FirstTextMs ??= receivedAt;
result.LastTextMs = receivedAt;
// On the simulator's successful [DONE] event:
result.CompletionMs = NowMs();
result.Status = "completed";Why Stopwatch? It measures elapsed time using the platform’s timer rather than subtracting two wall-clock display times. UTC is recorded separately for the audit trail. Microsoft Stopwatch reference.
4. Read the actual execution results
These are real C# executions of an artificial streaming workload. The mock deliberately waits 350 ms before its first text, adds later delays, splits a UTF-8 character across writes, and sends completion after the final text. Runtime, scheduling, local transport and parser overhead explain why the observations are not exactly the requested delays.
| Run | TTFT proxy (ms) | Last-text E2EL (ms) |
|---|---|---|
| 1 | 382.680 | 627.875 |
| 2 | 387.058 | 630.852 |
| 3 | 390.237 | 635.672 |
In run 1, headers were observed at 30.131 ms, the first text at 382.680 ms, the final text at 627.875 ms, and completion at 734.832 ms. The trailing completion interval was 106.957 ms. It is observable overhead in this mock, not evidence about a real model.
One recorded stdout object from the final compiled program:
{
"measurementMode": "local_mock_no_llm",
"actualLlmMeasured": false,
"scenario": "normal",
"startedAtUtc": "2026-10-16T02:33:18.4012743+00:00",
"status": "completed",
"headersMs": 30.131,
"firstTextMs": 382.68,
"lastTextMs": 627.875,
"e2elLastTextMs": 627.875,
"completionMs": 734.832,
"textEvents": 3,
"text": "Hello, 世界!"
}All three runs assembled Hello, 世界! from three text events. That is an event count, not a measured token count. Each observation came from a fresh program invocation, without a warm-up phase. Three samples are useful for checking the example—not for a reliable p95, a provider ranking, or a GPU-capacity claim.
5. Check failures before trusting a number
The local behavior suite passed 8 of 8 tests. Failure scenarios intentionally return process exit code 2; their successful E2EL field remains null, not zero.
dotnet run --project StreamingTimingLab -c Release -- empty
dotnet run --project StreamingTimingLab -c Release -- eof
dotnet run --project StreamingTimingLab -c Release -- timeout
dotnet run --project StreamingTimingLab -c Release -- error
dotnet run --project StreamingTimingLab -c Release -- malformed
dotnet run --project StreamingTimingLab -c Release -- http-error
dotnet run --project StreamingTimingLab -c Release -- unterminated- normal: metadata and an empty delta do not start TTFT; multiline data and split UTF-8 preserve the output.
- empty: a valid completion with no generated text has
firstTextMs = nullande2elLastTextMs = null, but a measured completion time. - eof: partial text followed by a closed connection is
incomplete. A partial first-text time may still exist; successful E2EL does not. - timeout: the client enforces a 250 ms deadline while the mock stalls the body. No first text or successful completion is reported.
- error / malformed / http-error: a stream error, invalid JSON or HTTP 503 is recorded as failure—not a fast successful response.
- unterminated: a done line without the following blank line is not a complete SSE event. The client must not silently claim success.
This sample limits event and accumulated text size. It does not implement automatic retries, reconnection, tool-call aggregation, provider-specific event types, or production load generation. A malformed schema outside its narrow protocol may require additional validation before reuse.
6. What changes when you measure a real LLM?
No approved target endpoint, model or inference budget was used for this draft. Provider, model, prompt length, output-token count, GPU, server-stage durations, real ITL and real-model E2EL are unmeasured.
- Adapt the request and parser. A real service may require POST, different payloads, approved authentication and a different completion convention. Do not point this custom GET/delta demo at an arbitrary API and call it compatible.
- Choose the timestamp contract before testing. Define request start, which event carries generated text, what successful completion means, and whether UI rendering or tool execution is in scope.
- Fix comparable conditions. Record model/version, prompt and output-token lengths, reasoning settings, concurrency, request rate, cache state, region and failures. Keep client and server measurements separate.
- Use server instrumentation for attribution. A slow first-text event alone cannot tell you whether network, queueing or prefill caused the delay.
- Report token metrics only with valid token accounting. Multiple tokens can arrive in one text event. Event gaps do not establish token-by-token ITL; this program does not calculate TPOT, TPS or token-level ITL.
For serving-side context, vLLM documents distinct request and streaming-output metrics. Check the exact deployed version and collector definitions before comparing them with this client. No vLLM server was run here.
For the infrastructure question behind latency changes, read why prefix caching saves compute without removing the memory trade-off. That related English analysis is not a benchmark result from this lab.
7. Quick comparison: do not mix these clocks
- headersMs: headers observed by the client; not the first generated token and not a directly instrumented first-byte timestamp.
- firstTextMs: request-to-first non-empty text event; a clearly labelled client TTFT proxy.
- lastTextMs: last text seen so far, even if the request later fails.
- e2elLastTextMs: the last-text latency exposed only after successful completion; null for failed or empty-text responses.
- completionMs: latency to this protocol’s success signal; may be later than the final text.
Next check: first reproduce the normal and failure cases locally. Only then adapt one approved real service and collect enough comparable requests. Keep failures in the report; do not replace missing measurements with zero.
Appendix. Full runnable Program.cs
Copy the entire program into the .NET 10 console project. The first half measures the stream; the second half is the deliberately delayed local simulator. The sample makes no external network request.
Expand the complete C# program
using System.Diagnostics;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Text.Json;
// Local teaching simulator only: no model, API key, GPU or external request.
string scenario = args.Length == 0 ? "normal" : args[0];
string[] allowed = ["normal", "empty", "eof", "timeout",
"error", "malformed", "http-error", "unterminated"];
if (!allowed.Contains(scenario))
throw new ArgumentException("Unknown scenario: " + scenario);
var listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start();
int port = ((IPEndPoint)listener.LocalEndpoint).Port;
Task server = ServeMockAsync(listener, scenario);
Measurement result;
try
{
// The short deadline deliberately tests a stalled body in timeout mode.
TimeSpan deadline = TimeSpan.FromMilliseconds(scenario == "timeout" ? 250 : 5000);
result = await MeasureAsync(new Uri($"http://127.0.0.1:{port}/stream"), deadline);
}
finally
{
listener.Stop();
await server;
}
Console.WriteLine(JsonSerializer.Serialize(new
{
measurementMode = "local_mock_no_llm",
actualLlmMeasured = false,
scenario,
startedAtUtc = result.StartedAtUtc,
status = result.Status,
headersMs = result.HeadersMs,
firstTextMs = result.FirstTextMs, // Client TTFT proxy, not a token timestamp.
lastTextMs = result.LastTextMs,
e2elLastTextMs = result.Status == "completed" ? result.LastTextMs : null,
completionMs = result.CompletionMs, // Application completion; includes the done trailer.
textEvents = result.TextEvents, // Event count, NEVER token count.
text = result.Text
}));
Environment.ExitCode = result.Status == "completed" ? 0 : 2;
static async Task<Measurement> MeasureAsync(Uri url, TimeSpan deadline)
{
var result = new Measurement { StartedAtUtc = DateTimeOffset.UtcNow };
using var cancel = new CancellationTokenSource(deadline);
using var http = new HttpClient(new SocketsHttpHandler { UseProxy = false });
var clock = Stopwatch.StartNew();
double NowMs() => Math.Round(clock.Elapsed.TotalMilliseconds, 3);
try
{
using var response = await http.GetAsync(
url, HttpCompletionOption.ResponseHeadersRead, cancel.Token);
result.HeadersMs = NowMs();
if (!response.IsSuccessStatusCode)
{
result.Status = "http_error";
return result;
}
if (response.Content.Headers.ContentType?.MediaType != "text/event-stream")
{
result.Status = "invalid_content_type";
return result;
}
await using var body = await response.Content.ReadAsStreamAsync(cancel.Token);
// StreamReader preserves UTF-8 characters split across transport reads.
using var reader = new StreamReader(body, Encoding.UTF8);
var data = new StringBuilder();
var output = new StringBuilder();
while (await reader.ReadLineAsync(cancel.Token) is { } line)
{
if (line.Length != 0)
{
// Comments, event names and other fields are not generated text.
if (line.StartsWith("data:") || line == "data")
{
string value = line == "data" ? "" : line[5..];
if (value.StartsWith(' ')) value = value[1..];
data.Append(value).Append('\n');
if (data.Length > 1_048_576) throw new IOException("Event too large.");
}
continue;
}
// A blank line dispatches one SSE event, not one model token.
if (data.Length == 0) continue;
string payload = data.ToString(0, data.Length - 1);
data.Clear();
if (payload.Length == 0) continue;
if (payload == "[DONE]") // A convention in THIS simulator, not the SSE standard.
{
result.CompletionMs = NowMs();
result.Status = "completed";
break;
}
using JsonDocument json = JsonDocument.Parse(payload);
if (json.RootElement.TryGetProperty("error", out _))
{
result.Status = "stream_error";
break;
}
if (json.RootElement.TryGetProperty("delta", out var delta)
&& delta.ValueKind == JsonValueKind.String)
{
string text = delta.GetString()!;
if (text.Length == 0) continue;
double receivedAt = NowMs();
result.FirstTextMs ??= receivedAt;
result.LastTextMs = receivedAt;
result.TextEvents++;
output.Append(text);
if (output.Length > 1_048_576) throw new IOException("Output too large.");
result.Text = output.ToString();
}
}
// EOF is not proof of successful application completion.
return result;
}
catch (OperationCanceledException) { result.Status = "timeout"; }
catch (JsonException) { result.Status = "invalid_json"; }
catch (HttpRequestException) { result.Status = "network_error"; }
catch (IOException) { result.Status = "network_error"; }
return result;
}
static async Task ServeMockAsync(TcpListener listener, string scenario)
{
try
{
using TcpClient peer = await listener.AcceptTcpClientAsync();
await using NetworkStream stream = peer.GetStream();
using var request = new StreamReader(stream, Encoding.UTF8, leaveOpen: true);
while (await request.ReadLineAsync() is { Length: > 0 }) { }
async Task Send(string text)
{
await stream.WriteAsync(Encoding.UTF8.GetBytes(text));
await stream.FlushAsync();
}
if (scenario == "http-error")
{
await Send("HTTP/1.1 503 Unavailable\r\nContent-Length: 0\r\nConnection: close\r\n\r\n");
return;
}
await Send("HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nCache-Control: no-cache\r\nConnection: close\r\n\r\n");
await Send(": heartbeat\n\ndata: {\"role\":\"assistant\"}\n\ndata: {\"delta\":\"\"}\n\n");
if (scenario == "timeout") { await Task.Delay(1000); return; }
if (scenario == "empty")
{
await Task.Delay(100);
await Send("data: [DONE]\n\n");
return;
}
await Task.Delay(350);
if (scenario == "malformed") { await Send("data: {broken-json}\n\n"); return; }
if (scenario == "error") { await Send("data: {\"error\":\"simulated\"}\n\n"); return; }
// Valid multi-line SSE data: the joined JSON contains a newline.
await Send("data: {\"delta\":\ndata: \"Hello\"}\n\n");
if (scenario == "eof") return;
await Task.Delay(100);
await Send("data: {\"delta\":\", \"}\n\n");
await Task.Delay(100);
byte[] tail = Encoding.UTF8.GetBytes("data: {\"delta\":\"世界!\"}\n\n");
int split = Array.IndexOf(tail, (byte)0xE4) + 1;
await stream.WriteAsync(tail.AsMemory(0, split));
await stream.FlushAsync();
await Task.Delay(20);
await stream.WriteAsync(tail.AsMemory(split));
await stream.FlushAsync();
await Task.Delay(100);
if (scenario == "unterminated")
{
await Send("data: [DONE]\n"); // No blank line: not a complete SSE event.
return;
}
await Send("data: [DONE]\n\n");
await Task.Delay(150); // Connection closes later; client need not wait for it.
}
catch (IOException) { /* Client may cancel/dispose the local connection. */ }
catch (SocketException) { /* Local listener can be stopped during cleanup. */ }
}
sealed class Measurement
{
public DateTimeOffset StartedAtUtc { get; init; }
public string Status { get; set; } = "incomplete";
public double? HeadersMs { get; set; }
public double? FirstTextMs { get; set; }
public double? LastTextMs { get; set; }
public double? CompletionMs { get; set; }
public int TextEvents { get; set; }
public string Text { get; set; } = "";
}
Primary sources and verification
- NVIDIA: LLM Inference Benchmarking—metric boundaries and trailing completion signals
- Microsoft: HttpCompletionOption and body-read timeout scope
- Microsoft: Stopwatch elapsed-time measurement
- WHATWG HTML Standard: UTF-8 and SSE event interpretation
- vLLM: metric design and request/streaming measurement distinctions
Documentation checked 2026-10-16. Code and original diagrams by Deoks. Local execution, not real-model benchmarking; no third-party figures were copied.
댓글
댓글 쓰기
안녕하세요 :)