All-Reduce vs. All-to-All: Why AI Network Traffic Patterns Differ
All-Reduce combines corresponding values and gives the combined result to every participant. All-to-All redistributes different chunks to different participants without doing that reduction. They solve different problems. A network that handles one well can still struggle with the other because message sizes, destinations, synchronization and the physical routes differ.
Before comparing AI cluster bandwidth numbers, identify the collective, group size, payload size, data type and rank placement. Keep the workload fixed. A faster All-Reduce score is not evidence of faster expert routing, and fewer bytes in an example do not make All-to-All a replacement for All-Reduce.
Evidence checked October 26, 2026, Korea time: NCCL 2.32.3 documentation, PyTorch 2.14 documentation and the public Megatron Core implementation. The diagrams and numbers are original synthetic examples. Fourteen local Node.js assertions checked their arithmetic and data layouts; no distributed, GPU, NCCL, network or LLM benchmark was executed.
Contents · six terms before the comparison
- Rank: one participant in a communication group. This example maps one rank to one GPU on each of four hosts; a rank number is not inherently a physical GPU ordinal.
- Collective: a group communication operation with a defined result and participation rules.
- Chunk: a piece of a buffer assigned to a destination; not an LLM token by definition.
- Data parallelism: replicas process different data and synchronize training information such as gradients.
- Expert parallelism: different ranks hold different experts in a mixture-of-experts model.
- Topology: how devices and links are connected; it determines which logical transfers share physical paths.
1. All-Reduce: combine values, then make the result available everywhere
In a sum All-Reduce, corresponding elements from each rank are added. Each rank receives the same resulting vector. NCCL also supports reductions such as minimum and maximum; the mathematical operation is part of the definition.
R0: [1, 2]
R1: [10, 20]
R2: [100, 200]
R3: [1000, 2000]
R0: [1111, 2222]
R1: [1111, 2222]
R2: [1111, 2222]
R3: [1111, 2222]
The NCCL collective definitions separate All-Reduce from Reduce, whose result is held at a selected root. They also explain that Reduce followed by Broadcast has the same collective outcome as All-Reduce. That does not mean an implementation must use that particular schedule.
Ring and tree algorithms can implement the same mathematical operation with different communication schedules. Real floating-point sums can also depend on accumulation order, so the exact integer illustration is not a promise of bitwise reproducibility across implementations.
2. All-to-All: route each chunk to its designated rank
For the equal-chunk All-to-All in the NCCL documentation, each rank starts with one chunk for each destination. Destination Rj receives the j-th chunk from every source. The operation exchanges data; it does not sum those chunks into one shared vector.
R0: A0, A1, A2, A3
R1: B0, B1, B2, B3
R2: C0, C1, C2, C3
R3: D0, D1, D2, D3
R0: A0, B0, C0, D0
R1: A1, B1, C1, D1
R2: A2, B2, C2, D2
R3: A3, B3, C3, D3
This resembles rearranging rows into destination columns. It differs from All-Gather, where all participants receive the collection of everybody's input. The word “all” does not mean that every rank receives the same data in every collective.
Nor does All-to-All require every pair to transmit at the same instant. The logical exchange and the implementation's schedule are separate. Real routed workloads can have unequal destination sizes, packing, padding and metadata; they are not automatically the balanced four-by-four example above.
3. Why gradient synchronization and expert routing create different traffic
Data-parallel training: replicas need to coordinate their gradients. PyTorch DistributedDataParallel documents gradient synchronization and gradient All-Reduce. The communication group and tensor sizes matter; the operation does not mean every worker sends all its weights to a central machine for every step.
Expert-parallel MoE: a router assigns token activations to experts held by different ranks. Activations must reach the selected experts, and expert results must be returned and combined. That creates a dispatch-and-return problem. It is not the same operation as summing corresponding gradient entries across replicas.
The public Megatron Core token dispatcher includes an All-to-All-based path with an expert-parallel dispatch exchange and a later combine exchange. It also includes an All-Gather-based dispatcher. MoE therefore does not imply that every engine or configuration must use exactly the same All-to-All implementation. The observed source is a moving main branch, not a benchmark or a fixed release claim.
The DeepSpeed MoE tutorial similarly defines an expert-parallel group and shows how its ranks distribute the experts. Expert count, expert-parallel group size and the selected routing policy describe different aspects of the model.
For an All-to-All-based expert path, the communication burden depends on activation width, precision, token assignments, how many selected experts are remote, placement and return traffic. Skewed assignments can make some ranks receive more work than others. Transport performance and expert compute balance both matter; one global average can hide the rank that finishes last.
Moving activations to resident experts is also different from offloading expert weights between memory tiers. The earlier MoE memory and expert-offloading analysis covers that separate capacity-and-timing question. This article focuses on communication semantics and traffic accounting.
4. A four-rank traffic example: count what actually crosses a rank boundary
Assume four ranks on four different hosts. For each operation, every rank starts with a total input buffer of 64 MiB, where 1 MiB is 1,048,576 bytes. Count outbound payload at its source once. Exclude protocol headers, retries, padding, metadata, extra MoE exchanges and any replication beyond the stated algorithm.
Illustrative ring All-Reduce: a Reduce-Scatter phase followed by an All-Gather phase gives an outbound payload factor of 2 × (n − 1) ÷ n. With n = 4, that is 1.5 × 64 MiB = 96 MiB sent per rank. This is the stated ring accounting model, not an observed NIC counter.
Uniform one-way All-to-All exchange: divide 64 MiB into four 16 MiB chunks. Keep one chunk locally and send the other three to the other ranks. That gives 48 MiB sent per rank. The factor is (n − 1) ÷ n = 0.75 for this uniform model.
| Measure | Value |
|---|---|
| Ranks / hosts | 4 / 4 |
| Input per rank | 64 MiB |
| Equal chunk | 16 MiB |
| Ring sends per rank | 96 MiB |
| Exchange sends per rank | 48 MiB |
| Local exchange chunk | 16 MiB |
Summing source-side sends across the four ranks gives 384 MiB for the ring example and 192 MiB for the single exchange. Do not add receiver copies again and call that the same measure. These are logical directed payload counts; they do not count every switch hop or the physical link bytes along a multi-hop route.
The two operations have different outputs and cannot replace one another. The smaller byte count does not prove half the completion time. The MoE example also needs a later return/combine exchange, which the single All-to-All row does not include. Different placement, message sizes, routing imbalance and overlap change the actual job.
The nccl-tests performance notes use operation-dependent factors to derive bus bandwidth from algorithm bandwidth. That derived column is not a direct measurement of every physical link. Compare the collective, ranks, size definition and column definition before comparing two reported GB/s numbers.
5. Topology changes where the pressure lands
A logical rank pair does not uniquely identify a physical route. Two GPUs may communicate within a host, through host interconnects or across a network. In the example above, one rank per host deliberately makes every inter-rank transfer remote; grouping multiple ranks inside a host changes that accounting boundary.
A ring schedule can create relatively regular neighbor exchanges, while a destination exchange can spread traffic across many logical peers. Neither description proves how much traffic each physical uplink carries. Routes can share links, and limited capacity across a partition of the fabric can constrain traffic even when individual NIC ports have high advertised rates.
Both collectives can be affected by congestion and slow participants. With routed experts, look at per-destination counts and expert compute time as well as network averages. With gradient synchronization, look at message sizes, group membership, timing relative to compute and the final completion time. The operation name identifies the required result; it does not identify the bottleneck by itself.
6. What an AI infrastructure comparison should report
- The workload: training gradient synchronization, expert dispatch, expert return or another communication stage; name the actual collective and reduction operation where relevant.
- The group: ranks, GPUs per host and how data, tensor and expert parallel groups map to the machines. A job may use several types of collective.
- The payload: total buffer size versus per-peer chunk size, data type, actual routed counts, padding and whether one or both MoE exchanges are included.
- The measurement: completion time and a clearly defined bandwidth column; include repeated-run variation and the slow ranks rather than only a peak average.
- The outcome: end-to-end training or serving behavior with compute and communication together. Do not translate a collective GB/s score directly into tokens per second, model quality or a supplier's revenue.
For a socket-level diagnostic, the related iperf3 bandwidth tutorial explains why TCP payload rate is a different measurement. A communication microbenchmark and an LLM serving test answer different questions too.
The practical infrastructure conclusion is to procure and tune against the traffic your workload produces. NIC rate alone cannot show whether GPUs will wait for a collective, whether expert assignments will be balanced or whether a different fabric pays for itself. Those decisions need measurements on the intended placement and workload.
Appendix · execution, limitations and sources
Actually executed: Node.js v24.14.1 on Windows; 14 local assertions checked integer elementwise sums, replicated outputs, the four-by-four chunk transpose, label conservation and the payload arithmetic. Not executed: distributed processes, NCCL, CUDA, RDMA, NIC tests, model training, inference or paid LLM calls. No hardware benchmark, system setting, authentication or cloud resource was changed.
The four-rank inputs, labels and 64 MiB assumption are educational choices. No real model, GPU, interconnect or vendor performance number is assigned to them. The examples do not determine memory orders, GPU shipments, a stock-price outlook or an investment recommendation.
- NVIDIA NCCL 2.32.3: Collective Operations — All-Reduce, Reduce, All-Gather, Reduce-Scatter and equal-chunk All-to-All semantics.
- PyTorch 2.14: DistributedDataParallel — replica gradient synchronization and gradient All-Reduce.
- NVIDIA Megatron-LM: MoE token dispatcher implementation — All-to-All dispatch/combine path and an alternative All-Gather path; moving source checked October 26, 2026.
- DeepSpeed: Mixture of Experts — expert-parallel group and expert distribution.
- NVIDIA nccl-tests: Performance — algorithm and operation-adjusted bus bandwidth; moving source checked October 26, 2026.
댓글
댓글 쓰기
안녕하세요 :)