When More GPU FLOPS Do Not Mean Faster LLM Decoding
More GPU FLOP/s does not necessarily make LLM decoding faster. If a decode workload is limited by moving data rather than doing arithmetic, raising its compute ceiling alone leaves the main constraint unchanged. Batch size, memory traffic and the actual execution path determine whether extra compute can help. Start with one question: what limits the next decode step at the batch and context length you actually serve? A peak-compute comparison cannot answer that by itself. Evidence checked October 22, 2026, Korea time. This is a documentation-based explanation with executed synthetic arithmetic—not a GPU, CUDA, inference-engine or paid-LLM benchmark. No real accelerator specifications or measured token rates are used in the worked example. Contents · five terms before the comparison FLOPs and FLOP/s : work performed versus a rate of doing work. Decode and batch : generating continuations and grouping work in one engine step. Arithmetic intensity and Roofline : work per byte and a c...