# Six GPUs on One x16 Slot: R9700s and RTX 3090s Behind a PCIe Switch

> Four Radeon AI PRO R9700s and two RTX 3090s behind a PEX88096 switch on one X570 slot: BIOS, kernel flags, ROCm and CUDA, and Qwen3.8-27B at 601 tok/s.

- URL: https://doug.sh/posts/six-gpus-one-x16-slot/
- Author: Doug Calobrisi
- Date: 2026-11-08
- Tags: llm, ai, self-hosted, gpu, pcie, amd, nvidia, rocm, vllm, qwen, linux


I put six GPUs on the one x16 slot of an ASUS TUF X570-Plus, a consumer board with a Ryzen 9 3900X and
128 GB of RAM. Four are AMD Radeon AI PRO R9700s with 32 GB each, and two are the RTX 3090s from [my PCIe
splitter post](/posts/pcie-splitter-rtx-3090/), one ROG Strix and one Founders Edition. They all sit
behind a Broadcom PEX88096 PCIe switch card, with ROCm and CUDA side by side on one Debian 13 host.

On the four R9700s, split four ways with tensor parallelism (TP4), Qwen3.8-27B prefills a 32k-token prompt at about 4,700 tokens per second and a
233k-token one at 3,340, and sixteen clients get 601 tokens per second of output between them.

![Cartoon: six graphics cards with eyes sit squeezed on one grey sofa with a plaque reading PCIe switch on its arm, and a cable labelled x16 runs from the sofa into a small PC tower beside it. Four blue cards labelled R9700 share an open newspaper reading ROCm, and two bigger blue cards labelled 3090 share one reading CUDA. To the right, a person in an orange hoodie, jeans and slippers holds a tray of six steaming cups of coffee and looks at the sofa with a tired, satisfied smile.](/images/six-gpus/one-couch.png)

| Part | What's in the box |
|---|---|
| Board | ASUS TUF Gaming X570-Plus, BIOS 5044 |
| CPU and RAM | Ryzen 9 3900X, 128 GB |
| Switch | Broadcom PEX88096 card, gen4 x16 to the CPU |
| GPUs | 4x Radeon AI PRO R9700 32 GB, 2x RTX 3090 24 GB, gen4 x8 each |
| OS | Debian 13, kernel 7.0.6-2-pve |
| R9700s | ROCm 7.2.4, radiance 1.4.0, Qwen3.8-27B MXFP4, TP4, P2P on |
| 3090s | NVIDIA 580 open module, vLLM 0.28, Swift-1.5 W4A16, TP2, MTP k=3, KVarN, P2P off |
| Display | GeForce GT 710 in a chipset slot, not one of the six |

Thankfully, I got all of this hardware before it doubled in price, again.

## The PCIe switch and the topology

The switch card came from [eBay](https://www.ebay.com/itm/136917131125) and goes in slot 1, the slot wired to the CPU. The board's
other full-length slot hangs off the X570 chipset and runs at x4 at most,
and the chipset shares its own link to the CPU with the USB, SATA and everything else on it. Six GPUs
behind that would be fighting the rest of the board for four lanes.

In slot 1, the CPU runs one gen4 x16 link to the switch and the GPUs enumerate behind it, so the board
doesn't need a bifurcation setting.

![Photo of the switch card: a blue PCIe card with a full-height bracket and an x16 edge connector. A finned heatsink with a blower fan covers the PEX88096 chip in the middle, and six cable connectors sit along the top and right edges, labelled A2, A1, B2, B1, C2 and C1. There's no power connector on the board, only a small fan header.](/images/six-gpus/switch-card.png)

Each GPU gets its own gen4 x8 link that ends at the switch. The R9700s are on a group of four ports on
bus `11`, and the 3090s on a group of two on bus `0d`.

![Diagram of the topology, headed One x16 slot, six GPUs. A box for the Ryzen 9 3900X holds root port 00:03.1, which sends one thick line labelled gen4 x16 uplink, about 28 GB/s each way, down to a wide box labelled PEX88096 switch. Inside the switch are three port groups. The group labelled bus 11, ACS redirect off has four links down to four R9700 boxes marked 32 GB, at 14:00.0, 17:00.0, 1a:00.0 and 1d:00.0, each tagged 32 GB BAR in blue. The group labelled bus 0d, ACS redirect on has two links down to an RTX 3090 Strix at 0e:00.0 and an RTX 3090 FE at 0f:00.0, each tagged 256 MB BAR in orange. A dashed empty group has four dashed stubs and two thinner ones, labelled unused ports (bus 1f, 25). A blue bracket under the R9700s reads ROCm 7.2.4, radiance 1.4.0, TP4, Qwen3.8-27B MXFP4, P2P on. An orange bracket under the 3090s reads NVIDIA 580 open, vLLM 0.28, TP2, Swift-1.5 W4A16, P2P off. Off to the right, a thin line runs from the CPU to a box labelled X570 chipset, which holds a GT 710 at 05:00.0, display only.](/images/six-gpus/topology.svg)

I went with a switch over another riser because a riser can't get six GPUs out of one slot. Bifurcation
splits an x16 slot into at most four x4 links, while the switch gives every card its own x8. It also lets
the R9700s talk to each other directly, peer-to-peer (P2P) ([bifurcation vs PCIe switches](/posts/pcie-bifurcation-switches-retimers/)). All four copying at once moved
54.7 GB/s, about twice the roughly 28 GB/s the x16 uplink carries one way, so that traffic never leaves the switch.

## BIOS settings

In the BIOS: CSM off, Above 4G Decoding on, and Resizable BAR off. A BAR is the window the CPU uses to
reach a card's memory, and Resizable BAR lets it cover all of it, so turning it off is an odd setting for a
box built around large BARs. With it on, the board stopped at the white VGA light. To turn it
back off, I had to unplug a 3090 just to get into the BIOS.

![Cartoon: inside an open black PC on a wooden desk, a small light labelled VGA glows white, and five blue graphics cards with eyes, four labelled R9700 and one labelled 3090, cling to the case and look out eagerly. The monitor beside it shows a blue BIOS screen reading Re-Size BAR: Disabled, with the mouse pointer on it. A person in an orange hoodie presses a key with one hand, eyes closed in relief with a sweat drop on their forehead, while holding a sixth card, a blue 3090 sulking at being unplugged, under the other arm, its power cable dangling.](/images/six-gpus/unplug-to-reach-bios.png)

The BIOS can only turn Resizable BAR on for every card or none, and with 128 GB of RAM it can't fit six
32 GB BARs. Linux sizes them instead, with a small kernel module, `gpu_rebar_layout`, that runs from the
initramfs before any GPU driver loads. It gives just the four R9700s their 32 GB BARs and works around a
Linux 7.0 bug that sized the switch's window too small for them. The module and the bug will have
their own post (coming soon).

## Kernel command line: BARs, gpu-unlock, BACO and ACS

```
iommu=pt
pci=disable_acs_redir=0000:11:00.0;0000:11:08.0;0000:11:10.0;0000:11:18.0
gpu_rebar_layout.ids=1002:7551
gpu_unlock.targets=all
amdgpu.runpm=0
```

`iommu=pt` keeps the IOMMU on in passthrough mode, which is how the switch and P2P were tested.

`gpu_rebar_layout.ids=1002:7551` tells the module which cards to resize. I haven't published it, but feel
free to [reach out](/about/) if you'd find it useful.

The real fix for the window-sizing bug is a patch series on linux-pci. I tested it on this box and sent a Tested-by to the
mailing list, which was an experience in itself.
Even with it merged, this board would still need the resize and a rescan before
the GPU drivers load.

`gpu_unlock.targets=all` tells [gpu-unlock](https://github.com/Zagitta/gpu-unlock) which cards to unlock,
here the two 3090s. It removes the GeForce half-rate cap on tensor math with FP32 accumulation and has to
run before the NVIDIA driver binds, so it's in the initramfs and loads after my module.

`amdgpu.runpm=0` keeps the R9700s out of BACO, the low-power state they drop into at idle, after one build of my module lost
all four cards to a BACO hang.
The cards no longer power down at idle.

ACS (Access Control Services) redirect forces peer traffic between switch ports up to the CPU's root port. `pci=disable_acs_redir=` turns
it off on the four R9700 ports so the switch routes their traffic locally, at the cost of isolation
between those cards ([what a switch
costs](/posts/pcie-bifurcation-switches-retimers/#what-a-pcie-switch-costs-bar-space-the-uplink-acs-and-power)).

If a boot goes wrong, press `e` at the GRUB menu and delete `gpu_rebar_layout.ids=` and
`gpu_unlock.targets=` for that boot. Without them both modules do nothing. Every boot also logs two
`can't assign` lines, for a bridge window on `0d:18.0` and for the first 3090's ROM. They're harmless:
the 3090 doesn't need its ROM, and every real BAR gets assigned.

## The NVIDIA driver

The 3090s run NVIDIA's stock 580 open module. The community-patched driver that turns on P2P for GeForce
cards needs large BARs, and these 3090s kept their 256 MB ones, so it has nothing to work with here. When
I tried it anyway, it hung a card on the first peer copy. The stock driver is also what [gpu-unlock](https://github.com/Zagitta/gpu-unlock) is
built for. On it, `nvidia-smi topo -p2p r` shows `CNS` (chipset not supported) between the 3090s, and copies go through host memory.

## The R9700s: radiance at TP4

The R9700s run [radiance](https://codeberg.org/StillDeadcode/radiance) 1.4.0 on ROCm 7.2.4 with Qwen3.8-27B in MXFP4, at TP4, with a
262,144-token context and an FP8 KV cache, at a 250 W power cap per card.

In the concurrency test each client sends a 1,024-token prompt and gets 512 tokens back. One client gets
151.1 tokens per second, and sixteen get 601.1 between them, with a median time to first token of 2.18
seconds and a worst case of 3.49. A single request decodes at 180 to 394 tokens per second after the
first token, depending on what it's writing.

Turning P2P off roughly halves the 16-client total, which the P2P post (coming soon)
will dig into.

| prompt, one request | prefill tok/s | time to first token |
|---:|---:|---:|
| 2k | 4,562 | 0.44 s |
| 8k | 4,651 | 1.68 s |
| 16k | 4,730 | 3.30 s |
| 32k | 4,674 | 6.67 s |
| 64k | 4,418 | 14.10 s |
| 128k | 3,973 | 31.35 s |
| 233k | 3,340 | 69.92 s |

## The 3090s: vLLM at TP2

The 3090s run the same vLLM 0.28 image as in the splitter post, at TP2 with P2P off, multi-token prediction (MTP) at k=3 and the
KVarN cache, serving Swift-1.5, a W4A16 quant of Qwen3.8-27B (4-bit weights, 16-bit activations). With
gpu-unlock on, a 10.8k-token prompt prefills at about 2,390 tokens per second, and decode runs at 67
tokens per second for one stream and 400 for eight.

gpu-unlock makes vLLM prefill 13 to 20% faster with no clear decode gain, more in the gpu-unlock
post (coming soon).

*I had help with this one. Claude wrote the module with me, ran the commands and benchmarks
on the box over SSH, and helped me draft this post. I did the physical work, read all the words, checked
the numbers, and rewrote anything that sounded like a chatbot, so the mistakes are mine.*

## Sources

- [radiance](https://codeberg.org/StillDeadcode/radiance), the inference engine on the R9700s
- [gpu-unlock](https://github.com/Zagitta/gpu-unlock), the kernel module that removes the GeForce half-rate tensor cap on the 3090s
- Linux [kernel-parameters.txt](https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/kernel-parameters.txt), the `pci=disable_acs_redir` entry