Grace Blackwell needs a new scheduler: 27% more usable GPUs from the same racks

Not all high-speed interconnects are created equal, and using the wrong one can be quite costly: running the same distributed training job over InfiniBand rather than NVLink can degrade performance by 43%.Historically, NVLink was limited to communication within a single node, meaning that multi-node AI workloads were forced to communicate over InfiniBand or a cloud equivalent (EFA on AWS, GPUDirect-TCPX on GCP). NVIDIA’s NVL72 systems remove this limitation by connecting all 72 GPUs in a rack with a unified NVLink fabric, so GPUs on different nodes can exchange data with an order of magnitude more bandwidth.However, maximizing the benefits of NVL72 systems is highly dependent on workload placement: a multi-node job spread across racks is still bottlenecked by InfiniBand. Preserving rack-locality gets harder as hardware failures take nodes offline and jobs finishing at different times scatter free capacity across racks.This blog covers the communication bottleneck in large-scale training and inference, the NVL72 rack-scale fabric, and slice sizing for resilience to GPU failures, as well as how the SkyPilot Platform keeps multi-node jobs rack-local through topology-aware placement, in-place recovery from hardware failures, and automatic defragmentation of shared clusters.Frontier training jobs run across thousands of GPUs. Doing so requires answering two questions:How do you shard your model across GPUs to ensure that it can actually run?Given your compute budget, how many copies of the sharded model can you run to maximize training throughput?The first question is typically answered with a combination of:Tensor Parallelism (TP): splitting a layer’s weight matrices across GPUsExpert Parallelism (EP): placing a mixture-of-experts (MoE) layer’s experts on different GPUsPipeline Parallelism (PP): splitting a model layer-wise across GPUsThe answer to the second question sets the degree of Data Parallelism (DP): the number of copies of the sharded model that train in parallel, each on a different slice of the training data.Of these parallelism strategies, tensor parallelism’s all-reduce and expert parallelism’s all-to-all communication patterns sit on the critical path during training, making them the most important to keep on NVLink (§9.1.2).Tensor ParallelismTensor parallelism splits the weight matrices inside each layer across GPUs, so every GPU only stores and computes a slice of each layer.Transformer layers are made up of two blocks: self-attention and a multi-layer perceptron (MLP). At its core, each block is two matrix multiplications (GEMMs) in a row. Megatron-LM (§3) showed that by partitioning a block’s first GEMM column-wise and its second GEMM row-wise, the GPUs only need to synchronize via all-reduce once per block, after the second GEMM, instead of after each one.Figure 1. Tensor parallelism across two GPUs. A is split column-wise, so each GPU computes its own slice of y without communicating. B is split row-wise to match, so each GPU produces a partial z, and one all-reduce sums the partial results.Even with this optimization, tensor parallelism still requires two all-reduces per transformer layer during the forward pass. These all-reduce steps sit on the critical path during training because they cannot be fully overlapped with computation. As a result, tensor parallelism is typically kept on NVLink: Hugging Face measured a 43% drop in training throughput when it moved off NVLink onto InfiniBand.Expert ParallelismIn a mixture-of-experts (MoE) layer, the MLP block is replaced by many smaller MLPs, called experts, and a router that sends each token to a small subset of experts. Expert parallelism splits MoE layers such that these small, independent experts are placed on different GPUs.Under expert parallelism, each MoE layer requires two all-to-all communication steps during the forward pass, dispatch and combine. A router selects the top-k experts for each token and dispatches tokens to the GPUs that have those experts in memory. Each GPU then computes its local experts on the tokens it received via a single grouped matrix multiplication. Finally, the expert outputs are sent back to each token's original GPU and combined.Figure 2. Expert parallelism distributes experts across GPUs. The all-to-all communication dispatches tokens to their assigned experts and combines results. (Source: NVIDIA, Figure 4.)As models add more MoE layers and route each token to more experts, the volume of all-to-all traffic grows, introducing a communication bottleneck.Training an MoE model with 64-way expert parallelism spreads each MoE layer’s experts across 64 GPUs.

On standard 8-GPU servers, that moves the all-to-all traffic off high-bandwidth NVLink and through much narrower InfiniBand interconnects where unoptimized communication can consume up to 60% of training time (§1.2).

On NVL72 systems, all 64 GPUs stay within a single NVLink domain, effectively eliminating the communication bottleneck (§9.2.2).The parallelism techniques discussed above are not unique to training. Beyond enabling inference engines to serve models that are too large for a single GPU, model parallelism can also improve throughput. Splitting a model's weights across more GPUs frees up memory for the KV cache, so the engine can process more concurrent requests or support larger context windows. In some cases, the freed memory grows faster than the number of GPUs: in one vLLM benchmark, splitting a model tensor-parallel across two GPUs increased the amount of KV cache blocks by 13.9x and delivered 3.9x more throughput.Figure 3. GPU memory breakdown for Llama-3.1-70B on one vs. two H100s. (Data: vLLM.)Expert parallelism follows the same logic. With a low degree of expert parallelism, each GPU holds many experts, leaving less room for the KV cache. WideEP, which refers to expert-parallel configurations spanning multiple nodes, significantly reduces the number of experts per GPU. The trade-off is network traffic: every all-to-all now spans more GPUs. On standard 8-GPU servers, an expert-parallel group can span at most eight GPUs before it has to communicate over InfiniBand, where per-GPU bandwidth drops 18x. NVL72 removes this limit by keeping all 72 GPUs in a rack on NVLink. When serving DeepSeek-R1 at interactive speeds, GB200 NVL72 with 32-way expert parallelism delivers up to 4.4x higher throughput per GPU than 8-GPU B200 servers, which are capped at 8-way.NVL72 extends NVLink from a single node to the entire rack, so the whole rack acts like a gigantic accelerator with shared memory.Figure 4. The NVLink hierarchy within an NVL72 rack. (Source: NVIDIA, Figure 1.)Every GPU on the NVL72 rack has 18 NVLink ports each delivering 100 GB/s of bidirectional bandwidth, which sums up to a total of 1.8 TB/s for each GPU. Each NVLink port on a given GPU maps to one of 18 NVSwitch chips that power the rack's fabric backplane. Because each NVSwitch chip accepts 72 NVLink lines, one from each of the GPUs, the aggregate internal bandwidth within the rack is 130 TB/s.Figure 5. Each GPU connects one NVLink port to each of the rack's 18 NVSwitch chips. (Source: NADDOD.)This network topology ensures that the fabric is fully non-blocking, meaning that any GPU in the rack can transmit data to any other GPU with uniform latency, making it ideal for large-scale distributed AI workloads.Figure 6. The NVSwitch fabric is fully non-blocking. Any GPU can communicate with any other GPU in the rack at full bandwidth with the same latency. (Source: NVIDIA, Figure 1)With NVL72 systems, it is tempting to allocate entire 18-node racks to workloads to maximize cluster utilization. In practice, teams actually get more usable capacity out of their cluster by leaving a couple of spare nodes in each rack.GPUs are unreliable. On its gigawatt-scale supercluster, Meta measured a mean time between interruptions (MTBI) of roughly 490 days per 4-GPU node after putting significant engineering effort into improving reliability. NVL72 clusters make the 72-GPU rack the atomic unit of compute, which increases the fault blast radius by 18x compared to the 4-GPU nodes that make up the rack. The MTBI for the rack is given by:Based on the formula above, a rack would be interrupted roughly every 490 / 18 ≈ 27 days.

If jobs are allocated entire 18-node racks, each of those interruptions makes the entire rack unschedulable until the failed node is repaired.

On a 128-rack cluster, this means that a rack would become unusable for full-rack jobs roughly every 5 hours.Reserving two nodes per rack as spares changes this.

Usable capacity of a 128-rack cluster for different slice sizes as node unavailability increases. Each slice must fit within a single rack. (Adapted from NVIDIA, Figure 6.)At SkyPilot, we manage tens of thousands of our customers’ GPUs, and on production NVL72 clusters we have measured unavailability rates of around 2%.

At that rate, NVIDIA’s model shows that 16-node slices provide 27% more usable capacity than 18-node slices (2,037 vs. 1,602 nodes).