PCIe, NVLink and CXL: Why More Memory Is Not Always More GPU Memory

PCIe, NVLink and CXL are not interchangeable ways to add GPU memory. PCIe commonly carries device I/O; NVLink connects supported processors through product-specific paths; CXL adds memory and cache-coherency protocols over a PCIe-derived physical interface. More addressable capacity does not automatically become local HBM, or deliver HBM-like access time.

The useful question is not simply “Which connection is fastest?” It is which processor is accessing which memory, over what route, with what software support? A capacity number without that route can be misleading when planning AI infrastructure.

Quick check: before accepting “more AI memory,” label the requesting processor, the memory’s physical location, the link/protocol, and the supported access mechanism. If any field is missing, do not count the capacity as interchangeable GPU HBM.

Documentation checked October 24, 2026, KST. This is a source-based architecture explanation with original synthetic arithmetic—not a GPU, PCIe, CXL, CUDA or inference benchmark. Product examples are deliberately named and version-scoped.

For the performance context, see why more GPU FLOPS need not speed up LLM decoding. For the separate capacity accounting, see the GPU-side HBM4 versus CPU-side SOCAMM2 explanation.

1. Define the connection before comparing it

• PCIe: PCI Express, a general-purpose high-speed interconnect commonly used for GPUs, network interfaces and NVMe storage.
• NVLink / NVLink Switch: NVIDIA interconnect technology and switching for supported processor configurations; the GPU scale-up fabric is one important use.
• NVLink-C2C: a distinct chip-to-chip connection; Grace Hopper is a concrete coherent CPU–GPU example.
• CXL: Compute Express Link, an interconnect with I/O, caching and memory-access protocols; available features depend on generation and device type.
• Coherence: rules that keep participating cached views of memory consistent. It is not a bandwidth guarantee or a promise that all memory is equally close.
• Capacity / bandwidth / latency: respectively how much fits, how much data moves per second, and how long an operation takes. State GB or GiB, GB/s or GiB/s, and the latency endpoint.

2. PCIe connects devices; the slot does not define memory semantics

A server can use PCIe for several different jobs: communicating with an accelerator, reaching a network adapter, or accessing NVMe storage. That does not make the attached device’s memory or storage a single interchangeable pool.

For a concrete product boundary, NVIDIA’s DGX H100/H200 guide documents a system with PCIe-generation CPU I/O, GPUs, NVSwitch, network cards, SSDs and system DIMMs. It includes a product topology diagram precisely because the installed route matters; the components coexist rather than replacing one another.

A PCIe device can expose memory windows, perform transfers, or support peer access under the right conditions. But “it plugs into PCIe” alone does not establish coherent CPU–GPU shared memory, a direct GPU-to-device route, or the effective bandwidth of an application. A root complex, switch, CPU-socket boundary, permissions and software support can change the usable path.

3. GPU-to-GPU NVLink is a scale-up path, not new HBM chips

NVIDIA’s NVLink documentation describes GPU-to-GPU communication through links and NVLink Switch. Keep the platform generation attached to its claims: Hopper, Blackwell and Vera Rubin are not one identical topology or link configuration.

Peer access and communication can let software work with data held by another supported GPU. They do not physically enlarge the HBM attached to the requesting GPU. Nor does a connected multi-GPU system automatically make an application’s allocations, scheduling or communication costs disappear.

For infrastructure planning, separate local memory capacity, peer-access capability and the framework’s actual placement strategy. A workload may benefit greatly from the fabric and still spend time communicating or moving data between owners. This is why a rack-wide aggregate bandwidth headline is not a per-request memory-speed measurement.

4. NVLink-C2C connects a different pair of endpoints

The GH200 Grace Hopper Superchip combines a Grace CPU and Hopper GPU using NVLink-C2C. NVIDIA documents a coherent CPU–GPU memory model for that product. Its architecture explanation distinguishes CPU-attached LPDDR5X from GPU-attached HBM and describes access across the connection.

That is not a claim that every ordinary PCIe GPU supports the same coherent access. It also is not the same route as GPU A communicating with GPU B through NVLink Switch. Even within a coherent addressable system, CPU-resident and GPU-resident data occupy different physical memory tiers.

Practical reading rule: when a specification says “NVLink,” ask whether it means a GPU scale-up fabric or the CPU–GPU C2C connection, then name the exact product. The shared brand does not erase the endpoint distinction.

5. CXL reuses a physical foundation—not every PCIe capability

CXL is not simply a third unrelated cable competing with PCIe. It uses a PCIe-derived electrical foundation while introducing additional protocol behavior. A host port, device, firmware and software stack must support the required CXL mode; an ordinary PCIe connection is not sufficient evidence.

The foundational CXL 1.1 specification separates these roles:

• CXL.io provides PCIe-based I/O functions, including device discovery and management.
• CXL.cache supports a device coherently caching host memory.
• CXL.mem provides access to device-attached memory.

Those are foundational protocol definitions, not a statement that 1.1 is the newest version. The Consortium’s CXL 3.1 technical Q&A identifies traditional Type-3 memory expanders as CXL.mem targets with CXL.io, and discusses direct peer-to-peer memory access by capable accelerators. Therefore, neither “every GPU can directly use any CXL pool” nor “CXL can never support accelerator-to-memory access” is a safe generalization.

A CPU-visible CXL memory region is not automatically an extension of a GPU’s local HBM. Establish the supported host/device/switch topology, mappings and runtime path before making that claim. More capacity can be valuable for a workload that tolerates the access path; it does not, by itself, demonstrate faster decoding.

6. Four routes that must stay separate

A · Local GPU memory
GPU cores ↔ local memory controller ↔ local HBM
No PCIe, NVLink or CXL hop is inserted into this local-memory arrow.

B · Supported GPU scale-up fabric
GPU A + its HBM ↔ NVLink / switch fabric ↔ GPU B + its HBM
Peer communication is not additional local HBM on GPU A.

C · Grace Hopper product example
CPU + LPDDR5X ↔ NVLink-C2C ↔ GPU + HBM
Coherent CPU–GPU access; different physical memory locations remain.

D · CXL memory-expansion example
CXL-capable CPU host ↔ supported CXL path ↔ Type-3 controller + DRAM
GPU access requires its own supported route; none is assumed here.

Original conceptual route cards, not a copied board diagram or one combined server configuration. Arrows identify endpoints and access paths, not equal speed, latency or universal compatibility. Sources: NVIDIA product/architecture documentation and the CXL Consortium references linked above.

Connection	First question
PCIe I/O	Which device and actual route?
GPU NVLink	Which peer GPUs and fabric?
NVLink-C2C	Which coherent processor pair?
CXL memory	Who can access the region, and how?

7. A transfer budget shows why capacity is not enough

Consider an entirely invented path that must move 256 MiB before a consumer can proceed. Assume an already established usable payload rate of 32 GiB/s. These are teaching inputs, not specifications or measurements of PCIe, NVLink, CXL or HBM.

Ideal transfer-only time = (256 / 1,024) GiB ÷ 32 GiB/s = 7.8125 ms. If the workload allows only 2 ms for that serialized movement, it cannot meet that budget under these assumptions—even if the source tier has abundant free capacity.

The corresponding ideal rate required for that deadline is 0.25 GiB ÷ 0.002 s = 125 GiB/s. Queueing, setup and protocol costs are excluded. This simple model assumes the entire transfer lies on the critical path; overlap, caching or transferring less data can change the workload. It does not rank the named interconnects or predict token latency.

8. What this changes for AI infrastructure decisions

Capacity expansion: a larger accessible tier may make a previously impossible working set feasible. Check where weights, KV state and other data actually reside, and whether movement fits the application’s latency target. Do not relabel all accessible bytes as HBM.

Scale-up: more capable GPU communication can support larger parallel jobs, but the useful gain depends on placement and collective traffic. “More connected GPUs” is not a measurement of achieved application throughput.

Commercial interpretation: technical capability, a qualified server configuration, a customer deployment, a component shipment and supplier revenue are separate evidence steps. NVIDIA’s product documentation establishes specific interconnect offerings; a CXL specification establishes protocol capabilities. Neither establishes an order forecast, a memory-supplier win or a stock-price outcome.

This is architecture and procurement context, not a buy/sell recommendation. The growth question is conditional: which memory tier and connection remove a real bottleneck at an acceptable cost and service target?

9. A six-field checklist for the next memory claim

1. Requester: CPU, local GPU, peer GPU or another accelerator?
2. Physical owner: local HBM, CPU DRAM, expansion-device DRAM or storage?
3. Route: exact ports, switches, sockets and protocol generation—not just a brand name.
4. Semantics: explicit copy, mapped access, cache coherence or storage I/O? Do not infer one from another.
5. Software support: which OS, driver, runtime and application placement policy actually use the route?
6. Measurement scope: per-lane/link/GPU/rack, one direction or aggregate, payload or line rate, and what workload/latency target?

If a headline supplies only capacity or aggregate bandwidth, those missing fields are the next questions—not permission to assume interchangeability.

Appendix. Coherence is not locality, and SSDs are not DRAM

Coherence addresses consistency among participating caches. Locality concerns where data physically lives and what route accesses it. A coherent remote or attached memory tier can still have different bandwidth and latency from a local tier.

Likewise, an NVMe SSD on PCIe remains storage with its own access model; a Type-3 DRAM expander is a different device. Software can explicitly stage or tier data between media, but sharing a physical interconnect family does not turn NAND into DRAM or HBM.

Sources, scope and verification

• NVIDIA DGX H100/H200 system guide—named system components and topology reference.
• NVIDIA NVLink / NVLink Switch—GPU scale-up paths and generation-specific scope.
• GH200 product documentation and Grace Hopper architecture explanation—coherent C2C and distinct CPU/GPU memory locations; not a claim about every GPU.
• CXL 1.1, June 2019, terminology and protocol overview—foundational definitions only. Specification copyright: © 2019-2020 COMPUTE EXPRESS LINK CONSORTIUM, INC. ALL RIGHTS RESERVED. No specification diagram reproduced.
• CXL Consortium, CXL 3.1 webinar Q&A, March 19, 2024—device roles and direct peer-access qualification.

Checked October 24, 2026, KST. Original transfer-budget arithmetic executed locally with Node.js 24.14.1 on Windows; no actual server topology inspection, transfer benchmark, GPU/CXL hardware test or paid API call. No product throughput measurements or shipment forecasts are implied. The current Consortium site discusses CXL 4.0; its existence does not establish support in an installed system.