# GPU P2P On vs Off: Tensor Parallel LLM Serving on Four R9700s

> With GPU peer-to-peer (P2P) off, four Radeon AI PRO R9700s at tensor parallel 4 lost 45 to 55% of prefill speed, 30% of decode and 53% of 16-client throughput.

- URL: https://doug.sh/posts/gpu-p2p-tensor-parallel/
- Author: Doug Calobrisi
- Date: 2026-11-11
- Tags: llm, self-hosted, gpu, pcie, amd, rocm, qwen, linux


My four AMD Radeon AI PRO R9700s live in [the box with six GPUs in one x16
slot](/posts/six-gpus-one-x16-slot/), where they serve Qwen3.8-27B split across all four with tensor
parallelism. Peer-to-peer (P2P) lets the cards read and write each other's memory without a trip through
the CPU. On the R9700s it needs a 32 GB BAR on each card, the window the CPU uses to reach the card's
memory. Getting those behind the switch took [a kernel
module](https://github.com/dougcalobrisi/gpu-rebar-layout), and [the BARs
post](/posts/gpu-bars-behind-a-pcie-switch/) covers it.

A 1 GB copy between two R9700s ran at 14.0 GB/s with P2P on. To see what P2P does for serving, I ran
the same benchmark three times: P2P on, P2P off, and P2P on again to check the first run.

With P2P off, prefill dropped 45 to 55%, decode about 30%, and throughput with 16 clients about 53%.
Median time to first token under load more than tripled.

![Cartoon: a relay race on a running track with two lanes. In the top lane, labelled P2P on, blue graphics cards run and hand batons straight to the next card. In the bottom lane, labelled P2P off, sweating orange cards queue at toll booths labelled host, each with a robot attendant, and wait for the gate to lift before passing the baton. A person in an orange hoodie watches from the start line with a checkered flag.](/images/gpu-p2p/p2p-relay.png)

## Tensor parallelism and all-reduce

Tensor parallelism splits each layer's weights across the cards, so every card computes part of every
layer for every token. Before the next layer can start, every card needs the sum of those partial
results. That's an all-reduce: each card sends its piece and gets the total back. In Megatron-LM's layout
there are two per layer, one after attention and one after the MLP.

With P2P on, the R9700s swap those partial sums directly through the switch. With P2P off, the sums go up
to pinned host memory and back.

![Routing sketch: two side-by-side diagrams of the same hardware, a box labelled CPU and host RAM at the top, an x16 uplink down to a box labelled PCIe switch, and four x8 links from the switch to four cards labelled R9700. The left panel, headed "P2P on" with the subtitle "partial sums turn around in the switch", draws the four x8 links in blue joined by a blue bar inside the switch, and greys out the uplink, labelled "x16 uplink, the sums stay off it". Under the cards: "a peer transfer crosses two x8 links: card, switch, card". The right panel, headed "P2P off" with the subtitle "partial sums go through host memory", draws the four x8 links in orange, merging inside the switch into a thick orange uplink labelled "x16 uplink, shared" and "≈28 GB/s each way" that runs up to an orange box labelled "CPU, pinned host memory". Under the cards: "a host-staged transfer goes up its x8 link and the shared uplink, then back down". The footer reads "Peer transfers stay in the switch because ACS redirect is off on the R9700 ports. Four concurrent ring copies moved 54.7 GB/s inside the switch."](/images/gpu-p2p/all-reduce-paths.svg)

## Running radiance with P2P on and off

The engine is [radiance](https://codeberg.org/StillDeadcode/radiance) 1.4.0 serving Qwen3.8-27B in MXFP4
at tensor parallel 4 (TP4), with each card capped at 250 W. These flags stayed the same across the three
runs, apart from the last line.

```
--max-num-seqs 16
--max-model-len 262144
--kv-cache-dtype fp8
--tp-wire wht6
--tp-wire-min-kb 128
--p2p off    # the middle run only
```

With `--p2p off`, radiance's startup log says all-reduces go through host memory. Its graph debug output
also shows host all-reduce kernels replacing the peer ones, so turning P2P off changes both the code
and the route.

## Results: P2P off vs on

| Measurement | P2P on (a) | P2P off | P2P on (b) |
|---|---:|---:|---:|
| Prefill tok/s, 2k prompt | 4,562 | 2,069 | 4,579 |
| Prefill tok/s, 32k prompt | 4,674 | 2,142 | 4,690 |
| Prefill tok/s, 128k prompt | 3,973 | 2,001 | 3,969 |
| Prefill tok/s, 234k prompt (233,525 tokens) | 3,340 | 1,834 | 3,337 |
| Decode tok/s, 2k context | 293.3 | 207.5 | 304.2 |
| Decode tok/s, 234k context | 253.0 | 167.9 | 228.9 |
| Total output tok/s, 1 client | 151.1 | 100.5 | 154.3 |
| Total output tok/s, 16 clients | 601.1 | 285.7 | 608.1 |
| Median time to first token, 16 clients | 2.18 s | 7.93 s | 2.38 s |
| Longest time to first token, 16 clients | 3.49 s | 10.06 s | 3.48 s |
| GSM8K accuracy, 250 questions | 98.8% | 99.2% | 98.8% |
| Output tokens per joule, 4 cards, 16 clients | 0.61 | 0.39 | 0.62 |
| Average power, 4 cards, 16 clients | 985 W | 739 W | 988 W |

![Horizontal bar chart headed "P2P off, as a share of P2P on", subtitled "tokens per second with P2P off, divided by the mean of the two P2P-on runs". Orange bars run on an axis from 0% to 100%, with a dashed line at 100% labelled "P2P on (mean of two runs) = 100%". Prefill, 2k prompt: 45%. Prefill, 32k prompt: 46%. Prefill, 128k prompt: 50%. Prefill, 234k prompt: 55%. Decode, 2k context: 69%. Decode, 234k context: 70%. 1 client, total tok/s: 66%. 16 clients, total tok/s: 47%. The footer reads "radiance 1.4.0, Qwen3.8-27B MXFP4, TP4, 250 W per card. P2P off ran once. The 234k prompt is 233,525 tokens."](/images/gpu-p2p/p2p-off-results.svg)

The chart shows what was left, so the 2k prompt at 45% is a 55% drop. Short prompts lost the most
prefill: 55% at 2k against 45% at 234k. The 233,525-token prompt waited 127.31 seconds for its first
token with P2P off and 69.92 with it on. GSM8K, 250 grade-school math problems, came out one question
apart and took 80% longer.

The two P2P-on runs were within 4% of each other on throughput, except long-context decode at about 10%,
and median time to first token was 9% apart.

With P2P off the four cards drew 739 W instead of about 985 W, but throughput fell further, so GPU energy
per output token rose about 59%. That counts only the power the four cards report.

## Copy test vs all-reduce

The copy test moved one big buffer between one pair of cards. An all-reduce on four cards has every card
sending and receiving at once.

With all four R9700s copying around a ring at once, the switch moved 54.7 GB/s. The host path carries
about 28 GB/s each way over the x16 uplink to the CPU. All four cards share it, so they get about half
that between them.

## Checking P2P on your build

1. Benchmark your serving workload with P2P on and off, in addition to a copy test.
2. Run P2P on twice, with short and long prompts, and one client and many.
3. Check whether card-to-card traffic stays on a switch or goes through the CPU.
4. Check that your driver supports P2P on your cards before you buy hardware for it.

*I had help with this one. Claude ran the benchmarks on the box over SSH and helped me draft this post. I
did the physical work, read all the words, checked the numbers, and rewrote anything that sounded like a
chatbot, so the mistakes are mine.*

## Sources

- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053), two all-reduces per transformer layer, after attention and after the MLP
- [radiance](https://codeberg.org/StillDeadcode/radiance), the inference engine on the R9700s