When More GPU FLOPS Do Not Mean Faster LLM Decoding

More GPU FLOP/s does not necessarily make LLM decoding faster. If a decode workload is limited by moving data rather than doing arithmetic, raising its compute ceiling alone leaves the main constraint unchanged. Batch size, memory traffic and the actual execution path determine whether extra compute can help.

Start with one question: what limits the next decode step at the batch and context length you actually serve? A peak-compute comparison cannot answer that by itself.

Evidence checked October 22, 2026, Korea time. This is a documentation-based explanation with executed synthetic arithmetic—not a GPU, CUDA, inference-engine or paid-LLM benchmark. No real accelerator specifications or measured token rates are used in the worked example.

Contents · five terms before the comparison

1. FLOPs and FLOP/s: work performed versus a rate of doing work.
2. Decode and batch: generating continuations and grouping work in one engine step.
3. Arithmetic intensity and Roofline: work per byte and a conditional performance ceiling.
4. Memory bandwidth: bytes transferred per second, not bytes of storage.
5. ITL and throughput: output timing versus total completed token work.

1. FLOPs describe work; FLOP/s describes a rate

FLOPs is a count of floating-point operations. FLOP/s is operations per second; TFLOP/s means 10¹² operations per second. Product pages often abbreviate the rate as “FLOPS,” which is why the headline uses that familiar spelling.

Memory bandwidth measures bytes per second. Capacity answers whether data fits; bandwidth answers how quickly it can move. A larger memory allocation is not automatically a faster data path.

NVIDIA’s GPU performance guide distinguishes arithmetic, memory and latency limitations. Its arithmetic-intensity estimate assumes enough parallel work to use the machine. A tiny or poorly parallelized kernel can fall short of both headline ceilings.

Before comparing rates, align the numeric format and execution conditions. Do not treat different precision modes or a sparsity-qualified peak as interchangeable measurements of one unchanged workload.

2. Why decoding is not the same workload as prefill

Prefill processes the prompt; decode incrementally generates its continuation. A KV cache stores attention keys and values from earlier tokens. A decode batch groups work processed in an engine step; it is not necessarily every request waiting at the service.

The JAX scaling book’s inference analysis explains how weight reuse across a larger token batch raises the arithmetic intensity of linear operations. Ordinary autoregressive decoding has a sequential dependency within each sequence, so batching usually means working across multiple sequences.

That creates a trade-off: better weight reuse can increase total throughput, while memory for separate KV states and latency requirements limit the usable batch. Attention traffic also matters, particularly as context grows. “Decode is memory-bound” is therefore a useful hypothesis for some regimes—not a verdict on every kernel, model or batch size.

This article does not repeat the KV-reuse or speculative-decoding mechanisms. Its narrower question is how to compare the arithmetic and data-movement ceilings of a declared workload.

3. Roofline is a ceiling, not a token-speed prediction

Arithmetic intensity, I, is useful floating-point work divided by bytes transferred at a specified memory boundary. For an HBM-level estimate, use HBM traffic—not an unrelated cache or interconnect bandwidth.

Performance ceiling = min(P, W × I)

Ideal time floor = max(F / P, Q / W)

P: compute rate in FLOP/s · W: bandwidth in bytes/s · F: operation count · Q: bytes transferred · I = F/Q. These expressions assume an idealized overlap of arithmetic and transfer, with all other work omitted.

NVIDIA’s Nsight Compute Roofline documentation combines compute, bandwidth and intensity. The ridge point is I = P/W. Below it, the bandwidth ceiling is lower; above it, the compute ceiling is lower. The memory level and compute instruction path must match the analysis.

Use this to decide what to investigate. Do not label a service “bandwidth-bound” simply because its achieved performance sits far below a peak line: low utilization, dependencies and uncounted work can invalidate that shortcut.

4. A worked example: twice the compute, unchanged time floor

Every number below is invented. Assume one device with enough capacity, a synthetic workload containing 7 billion two-byte weights, and one multiply plus one addition per weight per token. All weights are read once per batch step. Ignore attention, KV traffic, activations, launches, scheduling and communication.

At batch 1, that is F = 14 billion operations and Q = 14 billion bytes, or 14 GB in decimal units. Give the imaginary device P = 500 TFLOP/s and W = 2 TB/s. Then I = 1 FLOP/byte and the ridge point is 250 FLOPs/byte.

A · Baseline

14 GB ÷ 2 TB/s = 7 ms transfer
14 billion FLOPs ÷ 500 TFLOP/s = 0.028 ms arithmetic

Ideal floor: max(7, 0.028) = 7 ms

↓ Compare one changed input at a time

B · Double compute only

Transfer stays 7 ms.
Arithmetic falls to 0.014 ms.
Ideal floor stays 7 ms.

C · Double bandwidth only

Transfer falls to 3.5 ms.
Arithmetic stays 0.028 ms.
Ideal floor becomes 3.5 ms.

Original diagram; synthetic arithmetic, not observed hardware performance. A lower ideal floor is not a guaranteed application speedup. All assumptions and values also appear in the text.

Now keep the baseline device and reuse the weights across larger batches. Under this deliberately weight-only model, Q remains 14 GB and transfer time remains 7 ms while F grows with batch size. All table times are milliseconds:

Batch	Math (ms)	Floor (ms)
1	0.028	7
64	1.792	7
512	14.336	14.336

Batch 512 crosses the toy ridge point, making compute the larger term. This is not a recommendation to batch 512 real users: the omitted KV state, attention, activation traffic and latency constraints could make that configuration impractical.

Calculation check: Node.js 24.14.1 on Windows, October 22, 2026; 12 arithmetic assertions executed. No GPU or LLM was run. The calculation validates the displayed numbers, not the realism of the simplified workload.

5. Check the service, then the kernel

Inter-token latency (ITL) describes spacing between output tokens; system throughput describes output across requests. NVIDIA’s NIM benchmarking metrics separates these from time to first token and total request latency. An improved total token rate does not by itself establish a smoother response for one person.

For the definitions and timing boundaries, follow the same-language guide LLM inference metrics: TTFT, TPOT, ITL and goodput. Here the next task is a controlled comparison, not another metric glossary.

1. Write down the workload. Keep the model revision, precision, input/output lengths, engine configuration and accelerator count comparable.
2. Separate load from batch. Sweep concurrency, record the batch actually scheduled and note waiting requests. NVIDIA’s benchmark parameters guide explains why concurrent clients and engine batch size can differ.
3. Keep timing distributions. Compare ITL and request-latency percentiles alongside throughput, under a stated service target. Do not hide a worse tail behind a better average.
4. Find the expensive stage. NVIDIA’s compute triage guide recommends timeline analysis when kernel priority is unclear, then targeted profiling. Change one limiting factor and remeasure absolute duration.

These are proposed checks, not tests performed for this article. A profiler’s collection overhead or changed clock/cache settings also deserves a note when comparing timings.

6. What this changes in AI-infrastructure analysis

Author’s interpretation: compare a system’s balance of resources against the workload it can economically serve, rather than reading peak compute as finished inference capacity. HBM capacity, HBM bandwidth and inter-device communication answer different design questions; one cannot substitute for the other in the calculation.

That does not mean “memory always wins” or that compute demand has peaked. A different batch, attention design, precision or execution path can move the bottleneck. Nor does a technical bottleneck prove any supplier’s sales, margin or share-price outlook.

For a separate capacity-and-platform question, see Vera Rubin NVL72 memory demand: HBM4 and SOCAMM2 scenarios. Rack memory arithmetic and measured decoding performance are complementary evidence—not interchangeable proof. This is technical and industry analysis, not a buy-or-sell recommendation.

Summary · which number answers which question?

Number	Useful question	Not enough to prove
Peak FLOP/s	What is the stated compute ceiling?	Actual decode speed
HBM bytes/s	How fast could data cross this boundary?	That the workload saturates it
Arithmetic intensity	Which ideal ceiling is lower?	The real end-to-end bottleneck
Measured ITL + throughput	What did this deployment deliver?	Performance at another load or configuration

The next check: state the memory boundary, calculate both ideal times, then measure the actual workload. Faster arithmetic matters when arithmetic is on the critical path; the headline alone cannot tell you whether it is.

Appendix · assumptions, limits and primary sources

GB and TB here are decimal, 10⁹ and 10¹² bytes. FLOP counts use two operations per multiply-add. The toy example assumes resident weights and complete ideal overlap; it omits all other traffic and overhead. Multi-device sharding, MoE routing, offload and speculative decoding require additional accounting. No particular CUDA, GPU or inference-engine version was installed or benchmarked for this post.

The NVIDIA performance-background guide contains older hardware examples; this article uses its general reasoning, not those devices as current product recommendations. NIM documentation identified itself as version 2.0.0 when checked. The JAX chapter is dated February 4, 2025; its TPU examples are not presented as GPU measurements.

• NVIDIA, GPU Performance Background — Understanding Performance.
• JAX scaling book, All About Transformer Inference.
• NVIDIA Nsight Compute, Roofline charts.
• NVIDIA NIM Benchmarking, Metrics.
• NVIDIA NIM Benchmarking, Parameters and Best Practices.
• NVIDIA Nsight Compute, Compute Triage Guide.

Primary sources checked October 22, 2026 KST. Explanations, synthetic examples and the diagram are independently written. No source illustrations, screenshots or font files were copied.