GPU P2P On vs Off: Tensor Parallel LLM Serving on Four R9700s

· 807 words · 4 minute read

My four AMD Radeon AI PRO R9700s live in the box with six GPUs in one x16 slot, where they serve Qwen3.8-27B split across all four with tensor parallelism. Peer-to-peer (P2P) lets the cards read and write each other’s memory without a trip through the CPU. On the R9700s it needs a 32 GB BAR on each card, the window the CPU uses to reach the card’s memory. Getting those behind the switch took a kernel module, and the BARs post covers it.

A 1 GB copy between two R9700s ran at 14.0 GB/s with P2P on. To see what P2P does for serving, I ran the same benchmark three times: P2P on, P2P off, and P2P on again to check the first run.

With P2P off, prefill dropped 45 to 55%, decode about 30%, and throughput with 16 clients about 53%. Median time to first token under load more than tripled.

Cartoon: a relay race on a running track with two lanes. In the top lane, labelled P2P on, blue graphics cards run and hand batons straight to the next card. In the bottom lane, labelled P2P off, sweating orange cards queue at toll booths labelled host, each with a robot attendant, and wait for the gate to lift before passing the baton. A person in an orange hoodie watches from the start line with a checkered flag.

Tensor parallelism and all-reduce 🔗

Tensor parallelism splits each layer’s weights across the cards, so every card computes part of every layer for every token. Before the next layer can start, every card needs the sum of those partial results. That’s an all-reduce: each card sends its piece and gets the total back. In Megatron-LM’s layout there are two per layer, one after attention and one after the MLP.

With P2P on, the R9700s swap those partial sums directly through the switch. With P2P off, the sums go up to pinned host memory and back.

Routing sketch: two side-by-side diagrams of the same hardware, a box labelled CPU and host RAM at the top, an x16 uplink down to a box labelled PCIe switch, and four x8 links from the switch to four cards labelled R9700. The left panel, headed “P2P on” with the subtitle “partial sums turn around in the switch”, draws the four x8 links in blue joined by a blue bar inside the switch, and greys out the uplink, labelled “x16 uplink, the sums stay off it”. Under the cards: “a peer transfer crosses two x8 links: card, switch, card”. The right panel, headed “P2P off” with the subtitle “partial sums go through host memory”, draws the four x8 links in orange, merging inside the switch into a thick orange uplink labelled “x16 uplink, shared” and “≈28 GB/s each way” that runs up to an orange box labelled “CPU, pinned host memory”. Under the cards: “a host-staged transfer goes up its x8 link and the shared uplink, then back down”. The footer reads “Peer transfers stay in the switch because ACS redirect is off on the R9700 ports. Four concurrent ring copies moved 54.7 GB/s inside the switch.”

Running radiance with P2P on and off 🔗

The engine is radiance 1.4.0 serving Qwen3.8-27B in MXFP4 at tensor parallel 4 (TP4), with each card capped at 250 W. These flags stayed the same across the three runs, apart from the last line.

--max-num-seqs 16
--max-model-len 262144
--kv-cache-dtype fp8
--tp-wire wht6
--tp-wire-min-kb 128
--p2p off    # the middle run only

With --p2p off, radiance’s startup log says all-reduces go through host memory. Its graph debug output also shows host all-reduce kernels replacing the peer ones, so turning P2P off changes both the code and the route.

Results: P2P off vs on 🔗

Measurement P2P on (a) P2P off P2P on (b)
Prefill tok/s, 2k prompt 4,562 2,069 4,579
Prefill tok/s, 32k prompt 4,674 2,142 4,690
Prefill tok/s, 128k prompt 3,973 2,001 3,969
Prefill tok/s, 234k prompt (233,525 tokens) 3,340 1,834 3,337
Decode tok/s, 2k context 293.3 207.5 304.2
Decode tok/s, 234k context 253.0 167.9 228.9
Total output tok/s, 1 client 151.1 100.5 154.3
Total output tok/s, 16 clients 601.1 285.7 608.1
Median time to first token, 16 clients 2.18 s 7.93 s 2.38 s
Longest time to first token, 16 clients 3.49 s 10.06 s 3.48 s
GSM8K accuracy, 250 questions 98.8% 99.2% 98.8%
Output tokens per joule, 4 cards, 16 clients 0.61 0.39 0.62
Average power, 4 cards, 16 clients 985 W 739 W 988 W

Horizontal bar chart headed “P2P off, as a share of P2P on”, subtitled “tokens per second with P2P off, divided by the mean of the two P2P-on runs”. Orange bars run on an axis from 0% to 100%, with a dashed line at 100% labelled “P2P on (mean of two runs) = 100%”. Prefill, 2k prompt: 45%. Prefill, 32k prompt: 46%. Prefill, 128k prompt: 50%. Prefill, 234k prompt: 55%. Decode, 2k context: 69%. Decode, 234k context: 70%. 1 client, total tok/s: 66%. 16 clients, total tok/s: 47%. The footer reads “radiance 1.4.0, Qwen3.8-27B MXFP4, TP4, 250 W per card. P2P off ran once. The 234k prompt is 233,525 tokens.”

The chart shows what was left, so the 2k prompt at 45% is a 55% drop. Short prompts lost the most prefill: 55% at 2k against 45% at 234k. The 233,525-token prompt waited 127.31 seconds for its first token with P2P off and 69.92 with it on. GSM8K, 250 grade-school math problems, came out one question apart and took 80% longer.

The two P2P-on runs were within 4% of each other on throughput, except long-context decode at about 10%, and median time to first token was 9% apart.

With P2P off the four cards drew 739 W instead of about 985 W, but throughput fell further, so GPU energy per output token rose about 59%. That counts only the power the four cards report.

Copy test vs all-reduce 🔗

The copy test moved one big buffer between one pair of cards. An all-reduce on four cards has every card sending and receiving at once.

With all four R9700s copying around a ring at once, the switch moved 54.7 GB/s. The host path carries about 28 GB/s each way over the x16 uplink to the CPU. All four cards share it, so they get about half that between them.

Checking P2P on your build 🔗

  1. Benchmark your serving workload with P2P on and off, in addition to a copy test.
  2. Run P2P on twice, with short and long prompts, and one client and many.
  3. Check whether card-to-card traffic stays on a switch or goes through the CPU.
  4. Check that your driver supports P2P on your cards before you buy hardware for it.

I had help with this one. Claude ran the benchmarks on the box over SSH and helped me draft this post. I did the physical work, read all the words, checked the numbers, and rewrote anything that sounded like a chatbot, so the mistakes are mine.

Sources 🔗

llm self-hosted gpu pcie amd rocm qwen linux

More in the Multi-GPU builds series

Related posts