Distributed inference
When a model doesn’t fit on one GPU, or you want it to run faster on several, mistral.rs can spread it out in three ways:
| Mode | How it splits the model | Use it when |
|---|---|---|
| NCCL tensor parallelism | Every layer is split across all GPUs, and they work on each token together. | You have similar NVIDIA GPUs, on one machine or several. This is the default, high-throughput path. |
| Layer mapping | Each GPU holds a different range of whole layers, and each token passes from one GPU to the next. | Your GPUs differ in size or speed, the model can’t be split evenly across your GPU count, or you don’t have NCCL. |
| Ring backend | Like tensor parallelism, but between machines over plain TCP instead of NCCL. | You want to split a model across machines that can’t use NCCL, such as Macs. |
With a CUDA build and more than one visible GPU, mistral.rs picks a mode for you:
- If the binary includes NCCL, it uses tensor parallelism. Linux CUDA prebuilt binaries include NCCL.
- Otherwise, it uses layer mapping and gives each GPU a share of the layers that fits its free memory.
The startup log says which layout it chose. Multi-node NCCL and the ring backend are never picked automatically; you turn them on with the environment variables below.
Single machine, multiple GPUs
Section titled “Single machine, multiple GPUs”Build requirements
Section titled “Build requirements”The Linux CUDA prebuilt binaries from the installer already include NCCL. Installer source builds include it when NCCL is installed on the machine; pass --no-nccl to leave it out.
To build by hand with NCCL:
cargo install mistralrs-cli --features "cuda nccl flash-attn cudnn"Drop nccl from the list if NCCL isn’t installed.
Select GPUs
Section titled “Select GPUs”Set CUDA_VISIBLE_DEVICES to choose which GPUs mistral.rs can see:
CUDA_VISIBLE_DEVICES=0,1 mistralrs serve -m Qwen/Qwen3-32B --quant 4mistral.rs renumbers the visible GPUs from 0. With CUDA_VISIBLE_DEVICES=2,3, refer to them as 0 and 1 in --device-layers.
Tensor parallelism needs the model’s attention heads to split evenly across the GPUs. The KV heads must also split evenly, or, when there are fewer KV heads than GPUs, the GPU count must be a multiple of the KV head count. If that doesn’t work for all visible GPUs, mistral.rs uses the largest GPU count that does and logs the size it chose. For example, Qwen3-32B has 64 attention heads and 8 KV heads, so on 3 GPUs it runs tensor-parallel on 2, and on 6 GPUs it runs on 4. If no count works, the model loads without tensor parallelism. Use CUDA_VISIBLE_DEVICES to control exactly which GPUs take part.
Layer mapping (CUDA P2P)
Section titled “Layer mapping (CUDA P2P)”By default, layer mapping gives each GPU a contiguous range of layers, in GPU order, sized to that GPU’s free memory. It budgets weights, working memory, and the KV cache together; see memory budget.
It doesn’t account for GPU speed or how the GPUs are connected (PCIe or NVLink, same or different CPU socket). When those differ, set the layer counts yourself, for example putting fewer layers on a smaller, slower, or busier GPU:
-n (or --device-layers) takes GPU:LAYERS pairs separated by semicolons:
# Qwen3-32B has 64 layers: even splitmistralrs serve -n "0:32;1:32" -m Qwen/Qwen3-32B --quant 4
# Uneven splitmistralrs serve -n "0:44;1:20" -m Qwen/Qwen3-32B --quant 4num_device_layers takes a list of "GPU:LAYERS" strings:
from mistralrs import Runner, Which
runner = Runner( which=Which.Plain(model_id="Qwen/Qwen3-32B"), num_device_layers=["0:32", "1:32"],)On a binary with NCCL, tensor parallelism takes over and --device-layers is ignored. Set MISTRALRS_NO_NCCL=1 to use layer mapping instead:
MISTRALRS_NO_NCCL=1 mistralrs serve -n "0:44;1:20" -m Qwen/Qwen3-32B --quant 4Data moves between GPUs only where one GPU’s layers end and the next GPU’s begin. It goes straight from GPU to GPU when the driver allows it; otherwise it’s copied through CPU memory, which is slower, and the startup log shows a warning.
If the model doesn’t fit across your GPUs, automatic placement puts the remaining layers on the CPU, which is much slower and turns paged attention off. If you’ve set --kv-cache or --gpu-memory-utilization, loading fails instead.
For per-layer or per-tensor placement, see the topology guide.
Multi-node NCCL
Section titled “Multi-node NCCL”Multi-node NCCL runs tensor parallelism across several machines, each with one or more NVIDIA GPUs. Every node needs:
- The same CUDA and NCCL build, model, dtype, quantization, and flags.
- The same number of visible GPUs. The total GPU count (GPUs per node times nodes) must split the model’s heads evenly, as described above. Unlike on a single machine, a count that doesn’t work fails at startup.
- No
MISTRALRS_NO_NCCLorRING_CONFIGset.
Configure each node with these environment variables:
| Variable | Set on | Value |
|---|---|---|
MISTRALRS_MN_GLOBAL_WORLD_SIZE |
Every node | Total GPUs across all nodes. Setting it turns on multi-node mode. |
MISTRALRS_MN_LOCAL_WORLD_SIZE |
Every node | GPUs on this node. |
MISTRALRS_MN_HEAD_NUM_WORKERS |
Head node | Number of worker nodes. |
MISTRALRS_MN_HEAD_PORT |
Head node | Port the workers connect to. |
MISTRALRS_MN_WORKER_SERVER_ADDR |
Worker nodes | Head node’s address and port. |
MISTRALRS_MN_WORKER_ID |
Worker nodes | This worker’s number, starting at 0. |
Two nodes with four GPUs each:
Head node:
CUDA_VISIBLE_DEVICES=0,1,2,3 \MISTRALRS_MN_GLOBAL_WORLD_SIZE=8 \MISTRALRS_MN_LOCAL_WORLD_SIZE=4 \MISTRALRS_MN_HEAD_NUM_WORKERS=1 \MISTRALRS_MN_HEAD_PORT=9000 \mistralrs serve -m Qwen/Qwen3-32B --quant 4Worker node:
CUDA_VISIBLE_DEVICES=0,1,2,3 \MISTRALRS_MN_GLOBAL_WORLD_SIZE=8 \MISTRALRS_MN_LOCAL_WORLD_SIZE=4 \MISTRALRS_MN_WORKER_SERVER_ADDR=10.0.0.1:9000 \MISTRALRS_MN_WORKER_ID=0 \mistralrs serve -m Qwen/Qwen3-32B --quant 4Send requests to the head node. The head’s port (9000 here) must be reachable from every worker.
Ring backend
Section titled “Ring backend”The ring backend connects machines in a loop over TCP, with each machine running one part of the model. Build with the ring feature on every machine:
# NVIDIA GPUscargo install mistralrs-cli --features "cuda flash-attn ring"
# Apple Siliconcargo install mistralrs-cli --features "metal ring"If the binary also includes nccl, set MISTRALRS_NO_NCCL=1 when launching so the ring backend is used.
Give each machine its own JSON config and point RING_CONFIG at it. For two machines at 10.0.0.1 and 10.0.0.2:
{ "master_ip": "10.0.0.1", "master_port": 9000, "port": 9001, "right_ip": "10.0.0.2", "right_port": 9002, "rank": 0, "world_size": 2}{ "master_ip": "10.0.0.1", "master_port": 9000, "port": 9002, "right_ip": "10.0.0.1", "right_port": 9001, "rank": 1, "world_size": 2}RING_CONFIG=ring.json mistralrs serve -m Qwen/Qwen3-32B --quant 4| Field | Meaning |
|---|---|
rank |
This machine’s position in the ring, from 0 to world_size - 1. Rank 0 is the master: send requests to it. |
world_size |
Number of machines. Must be a power of 2 (2, 4, 8, and so on). |
master_ip, master_port |
Where the other ranks reach rank 0. Required on every rank except 0. |
port |
Port this machine listens on for the previous rank. |
right_ip, right_port |
Address and port of the next rank. The last rank points back to rank 0. |
Each rank waits only 10 seconds to reach the next one, so start all machines at about the same time.
See also
Section titled “See also”- Topology: per-layer and per-tensor placement on one machine.
- Environment variables: every distributed setting in one table.