A practical guide to running 8x RTX PRO 6000's

Maximize parallelism across 8x RTX PRO 6000's: high-concurrency inference, model fleets, and 70B fine-tuning.When we got our hands on an 8x NVIDIA RTX PRO 6000 Blackwell system paired with an AMD EPYC 9555 (64 cores, 128 threads, Zen 5), the first question we asked was simple: what can we run with this?With 768 GB of aggregate GDDR7 VRAM on tap, the knee-jerk instinct was to test whether we could shard larger models like Llama-3.1 405B or DeepSeek-R1 671B across all eight cards. But that seems a job more suited to this config's much more expensive sibling in NVIDIA's data center lineup.That realization led us to reframe the architecture.Instead of forcing deep model parallelism over PCIe, it makes more sense to explore what workloads we can handle at a fraction of the price of something like an HGX B200/B300 without sacrificing too much performance.TLDR:Splitting massive 400B+ models across eight PCIe GPUs introduces heavy latency, so that type of workload is out. Let's leave that to the cream of NVIDIA's monstrous server lineup. This platform's strength is high-density parallel execution, where it keeps up with its more powerful counterparts.Isolated serving (TP=1) maximizes compute: Running independent single-GPU instances eliminates inter-card bus traffic, delivering more efficient compute utilization and minimal inter-token latency.Unprecedented KV cache density: Dense mid-sized models (such as Qwen3.8-27B in FP8) leave ~60 GB of usable VRAM per card for KV caching, sustaining up to 1.15 million active tokens per card (~9.2 million tokens node-wide at 4k context).Multi-tenant model fleets: One node can concurrently handle 24 to 32 specialized 8B microservice endpoints (or up to 8 dedicated 32B endpoints) or 8 parallel 1080p video diffusion clips.On-premise 70B+ fine-tuning: Sharding model states across 768 GB VRAM while offloading AdamW optimizer moments to the EPYC 9555's host DDR5 memory enables native 70B fine-tuning without multi-node cloud clusters.1. Interconnect Constraints: PCIe Gen 5 vs. NVLinkInterconnect bandwidth and collective latency determine whether a workload scales efficiently via model parallelism or independent concurrency:InterconnectUnidirectionalBidirectionalRelativeAllReduce latencyPricePCIe Gen 5 x1664 GB/s128 GB/s1x (baseline)12-25 μs$$NVLink 4 (Hopper)450 GB/s900 GB/s~7x1.5-3.0 μs$$$NVLink 5 (Blackwell)900 GB/s1,800 GB/s~14x1.0-2.0 μs$$$$Although going with PCIe 5.0 clearly has limitations, the advantages and capabilities of NVLink you see above also come attached to an abnormally high price tag.Now, this is unavoidable when you're training or serving frontier models that span 100s of gigabytes or even terabytes.