Python SDK getting started
The Python SDK loads the model in-process and wraps the same Rust engine that backs the mistralrs binary.
from mistralrs import Runner, Which, ChatCompletionRequest
runner = Runner( which=Which.Plain(model_id="Qwen/Qwen3-4B"), in_situ_quant="4",)
response = runner.send_chat_completion_request( ChatCompletionRequest( model="default", messages=[ {"role": "user", "content": "In one sentence, what is Rust known for?"} ], max_tokens=256, ))
print(response.choices[0].message.content)The first run downloads the weights into the Hugging Face cache.
Installing
Section titled “Installing”You need Python 3.10 or newer. One package covers CPU on Linux (x86_64 and aarch64) and Windows, and Metal on Apple Silicon Macs; pip picks the right wheel for your platform:
pip install mistralrsNVIDIA GPU (CUDA)
Section titled “NVIDIA GPU (CUDA)”CUDA wheels are published on GitHub releases instead of PyPI, because pip can’t pick a wheel by GPU. Choose the wheel suffix from the CUDA version nvidia-smi reports and your GPU’s compute capability ({cc}, such as 89 for an RTX 4090):
| Driver reports | Wheel suffix |
|---|---|
CUDA 13.3+ on Hopper / sm90 |
+cuda133.sm90 |
CUDA 13.2+ on Ampere/Ada / sm80, sm86, sm89 |
+cuda132.sm{cc} |
CUDA 13.2+ on Blackwell / sm100, sm120, sm121 |
+cuda132.sm{cc} |
CUDA 13.1+ on Blackwell / sm100, sm120, sm121 |
+cuda131.sm{cc} |
| CUDA 13.0+ | +cuda130.sm{cc} |
CUDA 12.9+ on GB10 / sm121 |
+cuda129.sm121 |
| CUDA 12.8+ | +cuda128.sm{cc} |
Install with --find-links pointed at the release. Replace 0.9.5 / v0.9.5 with the release you want:
pip install "mistralrs==0.9.5+cuda128.sm89" \ --find-links https://github.com/EricLBuehler/mistral.rs/releases/expanded_assets/v0.9.5Look up your GPU’s compute capability in hardware support. CUDA wheels bundle the CUDA runtime, so you don’t need a CUDA toolkit installed. They don’t include cuTile, the optional NVIDIA backend that speeds up MoE and quantized models and that NVFP4 models require. If you need it, use the CLI from the installer instead.
Every wheel exposes the same from mistralrs import ... API.
The pieces
Section titled “The pieces”Runner owns the loaded model. Creating one loads the weights, so create it once and reuse it for every request in your process.
Which tells Runner what kind of model to load:
| Variant | Use for |
|---|---|
Which.Plain(model_id=...) |
Text models |
Which.MultimodalPlain(model_id=...) |
Models that take images, audio, or video |
Which.GGUF(...) |
GGUF checkpoints |
Which.Embedding(model_id=...) |
Embedding models |
Which.Lora(model_id=..., adapters=[...]) |
A base model with LoRA adapters |
in_situ_quant quantizes the weights while loading, like the CLI’s --isq. Pass "8" for about half the memory of the original weights or "4" for about a quarter; "2", "3", "5", and "6" also work. Leave it out to load at full precision. See quantization types for named formats.
Embeddings, speech, image generation, and multimodal requests are covered in the Python reference. Full example: plain.
Memory and device placement
Section titled “Memory and device placement”By default, mistral.rs places the model on your devices automatically and gives the KV cache the memory left over. On CUDA and Metal it uses paged attention, so all running requests share one cache pool.
max_seqs on Runner sets how many requests run at once. The default is 32.
To size the cache yourself, pass memory_goal to Runner:
memory_goal |
What happens |
|---|---|
| Unset (default) | The cache gets whatever memory is left after loading the model. |
"4GiB" |
The cache gets exactly 4 GiB. |
"65536tokens" |
The cache holds 65,536 tokens in total. |
0.9 |
Everything mistral.rs allocates stays under 90% of GPU memory. |
The cache is shared, so with "65536tokens" and 8 requests running, each has room for about 8,192 tokens. Setting memory_goal on CPU, or with no_paged_attn=True, is an error. See memory budget for the details and layer mapping to place layers by hand.
Streaming tokens
Section titled “Streaming tokens”Set stream=True to get an iterator of chunks instead of a single response. Using the runner from above:
stream = runner.send_chat_completion_request( ChatCompletionRequest( model="default", messages=[{"role": "user", "content": "Write me a haiku about ownership."}], max_tokens=128, stream=True, ))
for chunk in stream: delta = chunk.choices[0].delta.content if delta: print(delta, end="", flush=True)print()Each chunk has the OpenAI streaming shape, with new text in choices[0].delta.content. That field can be None, for example on the last chunk, which carries finish_reason, so check it before printing.
Full example: streaming. For async iteration, FastAPI, and handling errors mid-stream, see streaming from Python.
Multi-turn chat
Section titled “Multi-turn chat”Each request stands alone. To continue a conversation, send the earlier messages again, including the model’s replies:
messages = [{"role": "user", "content": "Name a Rust web framework."}]response = runner.send_chat_completion_request( ChatCompletionRequest(model="default", messages=messages))
messages.append({"role": "assistant", "content": response.choices[0].message.content})messages.append({"role": "user", "content": "Show a hello-world server in it."})response = runner.send_chat_completion_request( ChatCompletionRequest(model="default", messages=messages))Agent requests can keep history for you instead; see sessions.