Six GPUs on One x16 Slot: R9700s and RTX 3090s Behind a PCIe Switch

I put six GPUs on the one x16 slot of an ASUS TUF X570-Plus, a consumer board with a Ryzen 9 3900X and 128 GB of RAM. Four are AMD Radeon AI PRO R9700s with 32 GB each, and two are the RTX 3090s from my PCIe splitter post, one ROG Strix and one Founders Edition. They all sit behind a Broadcom PEX88096 PCIe switch card, with ROCm and CUDA side by side on one Debian 13 host.

On the four R9700s, split four ways with tensor parallelism (TP4), Qwen3.8-27B prefills a 32k-token prompt at about 4,700 tokens per second and a 233k-token one at 3,340, and sixteen clients get 601 tokens per second of output between them.

Thankfully, I got all of this hardware before it doubled in price, again.

The PCIe switch and the topology 🔗

The switch card came from eBay and goes in slot 1, the slot wired to the CPU. The board’s other full-length slot hangs off the X570 chipset and runs at x4 at most, and the chipset shares its own link to the CPU with the USB, SATA and everything else on it. Six GPUs behind that would be fighting the rest of the board for four lanes.

In slot 1, the CPU runs one gen4 x16 link to the switch and the GPUs enumerate behind it, so the board doesn’t need a bifurcation setting.

Each GPU gets its own gen4 x8 link that ends at the switch. The R9700s are on a group of four ports on bus 11, and the 3090s on a group of two on bus 0d.

I went with a switch over another riser because a riser can’t get six GPUs out of one slot. Bifurcation splits an x16 slot into at most four x4 links, while the switch gives every card its own x8. It also lets the R9700s talk to each other directly, peer-to-peer (P2P) (bifurcation vs PCIe switches). All four copying at once moved 54.7 GB/s, about twice the roughly 28 GB/s the x16 uplink carries one way, so that traffic never leaves the switch.

BIOS settings 🔗

In the BIOS: CSM off, Above 4G Decoding on, and Resizable BAR off. A BAR is the window the CPU uses to reach a card’s memory, and Resizable BAR lets it cover all of it, so turning it off is an odd setting for a box built around large BARs. With it on, the board stopped at the white VGA light. To turn it back off, I had to unplug a 3090 just to get into the BIOS.

The BIOS can only turn Resizable BAR on for every card or none, and with 128 GB of RAM it can’t fit six 32 GB BARs. Linux sizes them instead, with a small kernel module, gpu_rebar_layout, that runs from the initramfs before any GPU driver loads. It gives just the four R9700s their 32 GB BARs and works around a Linux 7.0 bug that sized the switch’s window too small for them. The module and the bug will have their own post (coming soon).

Kernel command line: BARs, gpu-unlock, BACO and ACS 🔗

iommu=pt
pci=disable_acs_redir=0000:11:00.0;0000:11:08.0;0000:11:10.0;0000:11:18.0
gpu_rebar_layout.ids=1002:7551
gpu_unlock.targets=all
amdgpu.runpm=0

iommu=pt keeps the IOMMU on in passthrough mode, which is how the switch and P2P were tested.

gpu_rebar_layout.ids=1002:7551 tells the module which cards to resize. I haven’t published it, but feel free to reach out if you’d find it useful.

The real fix for the window-sizing bug is a patch series on linux-pci. I tested it on this box and sent a Tested-by to the mailing list, which was an experience in itself. Even with it merged, this board would still need the resize and a rescan before the GPU drivers load.

gpu_unlock.targets=all tells gpu-unlock which cards to unlock, here the two 3090s. It removes the GeForce half-rate cap on tensor math with FP32 accumulation and has to run before the NVIDIA driver binds, so it’s in the initramfs and loads after my module.

amdgpu.runpm=0 keeps the R9700s out of BACO, the low-power state they drop into at idle, after one build of my module lost all four cards to a BACO hang. The cards no longer power down at idle.

ACS (Access Control Services) redirect forces peer traffic between switch ports up to the CPU’s root port. pci=disable_acs_redir= turns it off on the four R9700 ports so the switch routes their traffic locally, at the cost of isolation between those cards (what a switch costs).

If a boot goes wrong, press e at the GRUB menu and delete gpu_rebar_layout.ids= and gpu_unlock.targets= for that boot. Without them both modules do nothing. Every boot also logs two can't assign lines, for a bridge window on 0d:18.0 and for the first 3090’s ROM. They’re harmless: the 3090 doesn’t need its ROM, and every real BAR gets assigned.

The NVIDIA driver 🔗

The 3090s run NVIDIA’s stock 580 open module. The community-patched driver that turns on P2P for GeForce cards needs large BARs, and these 3090s kept their 256 MB ones, so it has nothing to work with here. When I tried it anyway, it hung a card on the first peer copy. The stock driver is also what gpu-unlock is built for. On it, nvidia-smi topo -p2p r shows CNS (chipset not supported) between the 3090s, and copies go through host memory.

The R9700s: radiance at TP4 🔗

The R9700s run radiance 1.4.0 on ROCm 7.2.4 with Qwen3.8-27B in MXFP4, at TP4, with a 262,144-token context and an FP8 KV cache, at a 250 W power cap per card.

In the concurrency test each client sends a 1,024-token prompt and gets 512 tokens back. One client gets 151.1 tokens per second, and sixteen get 601.1 between them, with a median time to first token of 2.18 seconds and a worst case of 3.49. A single request decodes at 180 to 394 tokens per second after the first token, depending on what it’s writing.

Turning P2P off roughly halves the 16-client total, which the P2P post (coming soon) will dig into.

The 3090s: vLLM at TP2 🔗

The 3090s run the same vLLM 0.28 image as in the splitter post, at TP2 with P2P off, multi-token prediction (MTP) at k=3 and the KVarN cache, serving Swift-1.5, a W4A16 quant of Qwen3.8-27B (4-bit weights, 16-bit activations). With gpu-unlock on, a 10.8k-token prompt prefills at about 2,390 tokens per second, and decode runs at 67 tokens per second for one stream and 400 for eight.

gpu-unlock makes vLLM prefill 13 to 20% faster with no clear decode gain, more in the gpu-unlock post (coming soon).

I had help with this one. Claude wrote the module with me, ran the commands and benchmarks on the box over SSH, and helped me draft this post. I did the physical work, read all the words, checked the numbers, and rewrote anything that sounded like a chatbot, so the mistakes are mine.

Sources 🔗

• radiance, the inference engine on the R9700s
• gpu-unlock, the kernel module that removes the GeForce half-rate tensor cap on the 3090s
• Linux kernel-parameters.txt, the pci=disable_acs_redir entry