Skip to content

Python SDK getting started

The Python SDK loads the model in-process and wraps the same Rust engine that backs the mistralrs binary.

from mistralrs import Runner, Which, ChatCompletionRequest
runner = Runner(
which=Which.Plain(model_id="Qwen/Qwen3-4B"),
in_situ_quant="4",
)
response = runner.send_chat_completion_request(
ChatCompletionRequest(
model="default",
messages=[
{"role": "user", "content": "In one sentence, what is Rust known for?"}
],
max_tokens=256,
)
)
print(response.choices[0].message.content)

The first run downloads the weights into the Hugging Face cache.

You need Python 3.10 or newer. One package covers CPU on Linux (x86_64 and aarch64) and Windows, and Metal on Apple Silicon Macs; pip picks the right wheel for your platform:

Terminal window
pip install mistralrs

CUDA wheels are published on GitHub releases instead of PyPI, because pip can’t pick a wheel by GPU. Choose the wheel suffix from the CUDA version nvidia-smi reports and your GPU’s compute capability ({cc}, such as 89 for an RTX 4090):

Driver reports Wheel suffix
CUDA 13.3+ on Hopper / sm90 +cuda133.sm90
CUDA 13.2+ on Ampere/Ada / sm80, sm86, sm89 +cuda132.sm{cc}
CUDA 13.2+ on Blackwell / sm100, sm120, sm121 +cuda132.sm{cc}
CUDA 13.1+ on Blackwell / sm100, sm120, sm121 +cuda131.sm{cc}
CUDA 13.0+ +cuda130.sm{cc}
CUDA 12.9+ on GB10 / sm121 +cuda129.sm121
CUDA 12.8+ +cuda128.sm{cc}

Install with --find-links pointed at the release. Replace 0.9.5 / v0.9.5 with the release you want:

Terminal window
pip install "mistralrs==0.9.5+cuda128.sm89" \
--find-links https://github.com/EricLBuehler/mistral.rs/releases/expanded_assets/v0.9.5

Look up your GPU’s compute capability in hardware support. CUDA wheels bundle the CUDA runtime, so you don’t need a CUDA toolkit installed. They don’t include cuTile, the optional NVIDIA backend that speeds up MoE and quantized models and that NVFP4 models require. If you need it, use the CLI from the installer instead.

Every wheel exposes the same from mistralrs import ... API.

Runner owns the loaded model. Creating one loads the weights, so create it once and reuse it for every request in your process.

Which tells Runner what kind of model to load:

Variant Use for
Which.Plain(model_id=...) Text models
Which.MultimodalPlain(model_id=...) Models that take images, audio, or video
Which.GGUF(...) GGUF checkpoints
Which.Embedding(model_id=...) Embedding models
Which.Lora(model_id=..., adapters=[...]) A base model with LoRA adapters

in_situ_quant quantizes the weights while loading, like the CLI’s --isq. Pass "8" for about half the memory of the original weights or "4" for about a quarter; "2", "3", "5", and "6" also work. Leave it out to load at full precision. See quantization types for named formats.

Embeddings, speech, image generation, and multimodal requests are covered in the Python reference. Full example: plain.

By default, mistral.rs places the model on your devices automatically and gives the KV cache the memory left over. On CUDA and Metal it uses paged attention, so all running requests share one cache pool.

max_seqs on Runner sets how many requests run at once. The default is 32.

To size the cache yourself, pass memory_goal to Runner:

memory_goal What happens
Unset (default) The cache gets whatever memory is left after loading the model.
"4GiB" The cache gets exactly 4 GiB.
"65536tokens" The cache holds 65,536 tokens in total.
0.9 Everything mistral.rs allocates stays under 90% of GPU memory.

The cache is shared, so with "65536tokens" and 8 requests running, each has room for about 8,192 tokens. Setting memory_goal on CPU, or with no_paged_attn=True, is an error. See memory budget for the details and layer mapping to place layers by hand.

Set stream=True to get an iterator of chunks instead of a single response. Using the runner from above:

stream = runner.send_chat_completion_request(
ChatCompletionRequest(
model="default",
messages=[{"role": "user", "content": "Write me a haiku about ownership."}],
max_tokens=128,
stream=True,
)
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
print()

Each chunk has the OpenAI streaming shape, with new text in choices[0].delta.content. That field can be None, for example on the last chunk, which carries finish_reason, so check it before printing.

Full example: streaming. For async iteration, FastAPI, and handling errors mid-stream, see streaming from Python.

Each request stands alone. To continue a conversation, send the earlier messages again, including the model’s replies:

messages = [{"role": "user", "content": "Name a Rust web framework."}]
response = runner.send_chat_completion_request(
ChatCompletionRequest(model="default", messages=messages)
)
messages.append({"role": "assistant", "content": response.choices[0].message.content})
messages.append({"role": "user", "content": "Show a hello-world server in it."})
response = runner.send_chat_completion_request(
ChatCompletionRequest(model="default", messages=messages)
)

Agent requests can keep history for you instead; see sessions.