Counting PCI Address Space for Resizable BAR on Six GPUs

· 988 words · 5 minute read

With Resizable BAR on and six GPUs installed, my motherboard stops at the white VGA light on its POST LEDs and stays there. The six are four AMD Radeon AI PRO R9700s with 32 GB each and two RTX 3090s, all behind a PCIe switch card in the one x16 slot, and the build post has the rest of the box. A cheap GeForce GT 710 in a chipset slot drives the monitor.

Resizable BAR worked fine with only the four R9700s. Once I plugged in the 3090s the board hung, and I had to unplug a 3090 just to get back into the BIOS and turn Resizable BAR off. With it off, all six cards boot fine. The BIOS doesn’t say why it hangs, but counting the address space explains it.

Counting the 64-bit PCI address space 🔗

A BAR, short for base address register, is a range of the CPU’s physical address space that maps onto a device’s memory, so the CPU and other cards can reach it with ordinary reads and writes. A GPU exposes 256 MB this way by default.

Resizable BAR lets the BIOS grow that to the card’s whole memory, 32 GB on an R9700, and AMD’s ROCm won’t let the cards copy to each other directly without it. The large BARs, and the switch’s bridge windows around them, all have to fit in one address window, the stretch of 64-bit address space the BIOS sets aside for PCI devices to share. The cards’ small register BARs live below 4 GB and don’t count.

On this board the window starts just after the top of RAM and ends at 512 GB. With 128 GB installed, /proc/iomem shows:

100000000-202f2fffff : System RAM
2030000000-7fffffffff : PCI Bus 0000:00

0x2030000000 is just under 129 GB. A 32 GB BAR has to start on a multiple of 32 GB, so the first one can go at 160 GB.

Each card also has a small second BAR for doorbells, which are registers the driver writes to tell the card there’s new work. The way the BIOS and Linux 7.0 lay it out, that BAR goes right after the big one and spills into the next 32 GB slot, so every card with a 32 GB BAR costs 64 GB of window. The patched kernel later tucked two doorbells just below their big BARs instead, which wastes less, but the BIOS has to boot first, so 64 GB a card is what I budget.

The 3090s have 24 GB, but BAR sizes are powers of two, so their full-size BAR is 32 GB too. With all six at 32 GB, the first five land at 160, 224, 288, 352, and 416 GB. A sixth at 480 GB would fill the window to 512 GB exactly, with its doorbell hanging off the end, so the BIOS would have had room for five.

The board’s address space drawn as a vertical strip, 0 GB at the bottom and 512 GB at the top, under the heading “128 GB of RAM, Resizable BAR on for all six”. A legend reads “32 GB BAR” in blue, “doorbell BAR (drawn wider)” as a thin dark navy band, and “rest of the slot, unused” as a dashed outline. The axis is labelled 0, 64, 128, 160, 224, 288, 352, 416, 480 and 512 GB. A grey block labelled “128 GB RAM” fills the bottom, and a note beside its top edge reads “window starts at 0x2030000000, just under 129 GB”. The empty stretch above it up to 160 GB reads “too small for a 32 GB BAR”. From 160 GB up, five blue blocks labelled “card 1: 32 GB BAR” to “card 5: 32 GB BAR” each sit on a 32 GB boundary with a navy doorbell band on top and a dashed box labelled “unused” filling the rest of the next 32 GB, and a bracket beside them reads “five cards fit, 160 to 480 GB: each 32 GB BAR starts on a 32 GB boundary, and its doorbell spills into the next one: 64 GB of window a card”. The sixth blue block, “card 6: 32 GB BAR”, fills 480 to 512 GB exactly, and its orange doorbell band sits above the dark 512 GB line, labelled in orange “sixth card’s doorbell lands past 512 GB, the top of the window”. A footer reads “Window from /proc/iomem with 128 GB installed. Card positions are the arithmetic: the BIOS hung at the white VGA light instead of laying them out.”

Keeping 128 GB of RAM and giving up P2P on the 3090s 🔗

Less RAM should have fixed it, because the window starts right after RAM, but I’ve only ever booted with 128 GB.

RAM First 32 GB boundary Six cards with 32 GB BARs
128 GB, installed 160 GB five fit, the sixth’s doorbell crosses 512 GB
96 GB 128 GB fit, ending around 480 GB
64 GB 96 GB fit, ending around 448 GB

I’d bought the 128 GB on purpose, and memory is stupid expensive right now, so the 3090s kept their 256 MB BARs and copy to each other through system memory. A 1 GB copy that way ran at 13.0 GB/s, close to the 14.0 GB/s the R9700s get directly over the same eight-lane links. Turning P2P off on the R9700s, though, roughly halved what they serve to 16 clients, which the P2P post (coming soon) will dig into.

The BIOS turns Resizable BAR on for every card or none, so I left it off and resized just the R9700s from Linux, which ran into a Linux 7.0 bug. The post on fitting four 32 GB BARs behind the switch covers the bug and the kernel module I wrote to work around it.

Six 32 GB BARs on the patched kernel 🔗

While testing the patched kernel, I also tried giving all six cards 32 GB BARs. It still failed. The switch’s root port asked for a 352 GB window. The first place that can start is 160 GB, and 160 + 352 is 512 GB, the very top of the window. The GT 710’s 160 MB sits in the last gigabyte of that range:

0x2800000000 + 0x5800000000 = 0x8000000000    160 GB + 352 GB = 512 GB
GT 710 at 0x7fe8000000

A horizontal strip of address space from 128 to 512 GB under the heading “Patched kernel, all six asking for 32 GB BARs”. A small grey block labelled “RAM” sits at the left end. One long blue bar runs from 160 GB, marked “0x2800000000”, to 512 GB, marked “0x8000000000”, split by a thin white line at 416 GB into “four R9700s: 256 GB, 64 GB each” and “two 3090s: 96 GB”. A note under it reads “RAM ends just under 129 GB; the 352 GB request starts at the first 32 GB boundary after it and runs to the top of the window”. Dashed lines open the last gigabyte into a zoomed strip labelled “the last gigabyte, zoomed”, from 511 GB to 512 GB, all blue except a small orange block about two thirds of the way along, labelled in orange “GT 710, 160 MB at 0x7fe8000000”. A footer reads “Sizes from the patched kernel’s log, not placements.”

The 352 GB is conservative. The log sizes the R9700 side at 256 GB, a full 64 GB block each, which leaves 96 GB for the two 3090s. That’s less than two 64 GB blocks, and the log doesn’t show how it splits. The patch’s changelog says the sizing “has to play it safe” for windows like these, so it doesn’t count on packing doorbells into the gaps.

Without the GT 710, the request would fit exactly, with nothing to spare. I haven’t tried it.

Cartoon: five big blue graphics cards sit squeezed into a row of grey cinema seats, four labelled R9700 and one labelled 3090. The last seat, marked 512 GB, is taken by a small grey graphics card labelled GT 710, in glasses and a cardigan, smiling with a little monitor on its lap. A sixth blue card labelled 3090 stands beside the row holding its ticket and gives it a flat stare, while a person in an orange hoodie behind it covers their face with one hand.

Where the box ended up: 128 GB and four large BARs 🔗

The box runs Linux 7.0 with the module. The four R9700s have their 32 GB BARs, and the 3090s have their default BARs and go through system memory when they need to talk.

I had help with this one. Claude worked through the address arithmetic from the logs with me and helped me draft this post. I did the physical work, read all the words, checked the numbers, and rewrote anything that sounded like a chatbot, so the mistakes are mine.

Sources 🔗

linux pcie gpu amd nvidia llm self-hosted

More in the Multi-GPU builds series

Related posts