Why CUDA OOM Happens Even When GPU Memory Looks Free
CUDA out of memory does not always contradict a dashboard showing spare memory. The dashboard, the framework’s allocator and the failing request can describe different scopes or moments. First match the device and timestamp, then distinguish live tensors, allocator-held memory and the next allocation—not just the headline free-memory number.
Scope checked October 20, 2026, Korea time: PyTorch 2.14 CUDA documentation, principally its native caching allocator; NVIDIA’s device-memory documentation; and a versioned vLLM 0.20.0 implementation example. These are documentation-based explanations, not a CUDA, PyTorch or LLM benchmark. The numerical examples below are synthetic arithmetic, checked with Node.js on Windows.
1. “Free” belongs to a particular memory view
Allocated memory is the current device memory occupied by tensors reported by PyTorch’s memory_allocated(). Reserved memory is the memory managed by its caching allocator, reported by memory_reserved(). Both APIs return bytes for a selected device; neither is a whole-server memory total. See the allocated-memory reference and reserved-memory reference.
In the native-allocator picture, allocated tensor memory sits inside the allocator’s reserved pool. Do not add the two readings. A lower allocated reading can coexist with a larger reserved pool; that alone is not evidence of a leak.
Device-level free memory describes another scope. NVIDIA’s nvidia-smi documentation describes frame-buffer memory and process/context reporting. Driver accounting and reporting support matter: process GPU-memory usage is unavailable under Windows WDDM. A process metric, a whole-device metric and a reading from another GPU are not interchangeable.
Imagine the next operation asks for 12 GiB. The 8 GiB device-free reading is smaller, but the 16 GiB pool difference does not settle the outcome either: what can be reused or returned matters. The correct conclusion from these totals is “inspect the allocation state”, not “12 GiB will certainly fit” or “fragmentation is proven.”
2. A reserved–allocated gap is a clue, not a diagnosis
Allocator fragmentation is a mismatch between the available allocation pieces and the request that must be satisfied. It is not established merely by subtracting two counters. PyTorch’s memory-snapshot documentation distinguishes tensor-allocated blocks, blocks waiting for another stream to finish, and inactive blocks. It also explains that cache release can return entirely inactive segments.
That is why “reserved minus allocated” should not be labeled “immediately reclaimable memory.” Nor should it be translated into “this many additional requests will fit.” The allocator’s state and the size of the next request are missing from that subtraction.
There is another visibility limit: the PyTorch profiler does not see allocations made outside its allocator, such as some NCCL communication allocations. A clean-looking framework trace is therefore not proof that nothing else occupies the device. Diagnose the relevant layer before blaming the application—or buying a larger GPU.
3. An idle snapshot is not a peak-memory budget
Peak memory is the largest use within a defined observation interval. Headroom is capacity deliberately left for variation and uncertainty. The practical question is whether a specified workload fits at its demanding moments, not whether a quiet screenshot looks comfortable.
For a serving example, KV cache stores attention keys and values used as a model continues generation. vLLM 0.20.0’s GPU-worker implementation derives a cache budget after non-cache memory profiling and applicable CUDA-graph memory estimation. A CUDA graph is a captured sequence of GPU work for replay. This versioned example is not a claim that every engine or configuration budgets memory identically.
KV budget = device capacity − model weights − other peak use − safety margin
80 − 48 − 20 − 4 = 8 GiB for KV data
Here “other peak use” is an assumed combined allowance for activations, workspace, runtime/communication and any graph-related use; it excludes weights, KV data and the separate margin. The categories must not overlap. This is a planning toy, not the earlier allocator snapshot or vLLM’s exact formula.
- At an assumed 0.5 GiB of KV per active request, 8 GiB holds at most 16 such requests in the arithmetic.
- At 1 GiB per request, that falls to 8.
- If the assumed other peak use rises from 20 to 28 GiB, the same budget leaves zero for KV data.
The last case is not a forecast and the request counts are not measured serving limits. They ignore such details as scheduling, cache sharing and allocation granularity. Their purpose is narrower: identical advertised capacity can produce different usable workload budgets. Average usage is not enough to select an admission limit.
4. Why empty_cache() is not a universal OOM fix
Cache release returns unused allocator-held memory; it does not remove tensors that are still needed. PyTorch’s CUDA memory-management notes explain that empty_cache() releases unused cached memory for other GPU applications, but does not free memory occupied by tensors. It does not create extra physical memory for PyTorch.
The same notes describe max_split_size_mb as a last-resort option when a failing workload shows substantial inactive split blocks; it applies to the native backend and is ignored with cudaMallocAsync. That is a reason to check the backend and evidence—not a setting to paste into every production server. This article makes no allocator or driver changes.
5. What this means for AI infrastructure capacity
Usable serving capacity is the workload that meets an operating target under stated conditions. It is not just the number of installed memory bytes. A latency SLO is a defined latency objective for requests. As an engineering interpretation, a useful capacity claim should name the model, input/output lengths, concurrent workload, relevant memory peaks, and success/latency criteria.
More HBM can relax a genuine capacity constraint. Better allocation reuse or a different workload can also change what fits. Neither observation translates directly into a forecast for memory shipments, chip orders or a supplier’s profit. Separate the hardware supply question from the software admission policy.
This is the distinction behind our NVL72 HBM4 and SOCAMM2 capacity scenarios: hardware configuration arithmetic is not a measured count of useful simultaneous requests. Once a workload fits, the TTFT, TPOT, ITL and goodput guide explains why token speed alone is not the final service-quality test.
6. Compare the signal before choosing an action
| Observation | What to check next |
|---|---|
| Device looks free, but one request fails | Match device and timestamp; retain the failed allocation size and workload conditions. |
| Reserved is larger than allocated | Inspect allocator state before calling the difference a leak or fully reclaimable space. |
| A small workload fits, a larger one does not | Compare the defined peak and workload shape—not only the idle reading. |
| Framework and device totals disagree | Keep the measurement scopes separate; investigate other allocations and reporting limits. |
Bottom line: start with one device, one failing request and one timestamped set of observations. Keep evidence before changing configuration. Do not kill another tenant’s process, reset the GPU, expand permissions or assume an allocator flag is safe just to make an OOM disappear.
Appendix · methods, limits and primary sources
Method: the arithmetic above was executed and checked with six assertions using Node.js v24.14.1, Windows x64. No GPU was probed; PyTorch/vLLM were not installed or run for this article, and no real LLM request was made. It is not a step-by-step production remediation runbook. Graph, allocator, operating-system and distributed-runtime behavior must be checked against the actual installed versions.
Terminology: GPU is the accelerator device; VRAM refers to its device memory; HBM is a high-bandwidth memory technology used by many AI accelerators. More bytes, higher bandwidth and more successful requests are three different claims. This is technical and industry education, not a recommendation to buy or sell securities.
- PyTorch 2.14 · memory_allocated and memory_reserved — counter scope and units.
- PyTorch 2.14 · Understanding CUDA Memory Usage — segments, block states and allocations outside the profiler.
- PyTorch 2.14 · CUDA memory management — cache release, allocator backends and tuning caveats.
- NVIDIA · nvidia-smi reference — device/context accounting and OS reporting limitations.
- vLLM 0.20.0 · GPU-worker implementation — version-specific cache-budget profiling.
Sources checked October 20, 2026 KST. Numerical capacities, margins and request sizes are original illustrative assumptions, not vendor specifications, measured results or purchase forecasts.
댓글
댓글 쓰기
안녕하세요 :)