Model family notes
Most checkpoints need only mistralrs run -m <model-id>. This page collects the families that need an extra flag, have a setting worth knowing, or behave differently from the defaults. Choose a model helps you pick a model for a task, the supported models reference lists every architecture, and block-diffusion models like DiffusionGemma have their own page.
Reasoning controls
Section titled “Reasoning controls”Reasoning models accept a reasoning_effort setting:
| Value | Effect |
|---|---|
off (or none) |
Turns thinking off. |
low, medium, high, xhigh |
Turns thinking on and passes the tier to the model’s chat template. |
| Not set | Thinking is on, with no tier, except in requests that ask for speech output, where it’s off. |
Set it per request with "reasoning_effort": "low", or for a mistralrs run session with --reasoning-effort low. The older enable_thinking boolean still works. If you send both and they disagree, such as enable_thinking: true with reasoning_effort: "off", the request fails.
What a tier changes is up to the chat template. A template that only reads enable_thinking treats every tier as plain “on”; Muse Glimmer’s maps the tiers to its own reasoning strengths.
Thinking models (Qwen3, SmolLM3)
Section titled “Thinking models (Qwen3, SmolLM3)”Qwen3 and SmolLM3 think by default. To turn it off:
mistralrs run -m Qwen/Qwen3-4B --thinking false--thinking and --reasoning-effort apply to both interactive and one-shot (-i) runs. You can also end a message with /no_think or /think to switch for that turn:
How many rs are in blueberry? /no_thinkcurl http://localhost:1234/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [{"role": "user", "content": "How many rs are in blueberry?"}], "enable_thinking": false }'runner.send_chat_completion_request( ChatCompletionRequest( model="default", messages=[{"role": "user", "content": "How many rs are in blueberry?"}], enable_thinking=False, ))Qwen3 also publishes FP8 checkpoints. Pass the FP8 model ID and the weights load as published, without --quant. You can store the KV cache in FP8 as well, which halves its memory compared to BF16:
mistralrs serve -m Qwen/Qwen3.8-27B-FP8 --pa-cache-type f8e4m3An FP8 KV cache needs CUDA with compute capability 8.0 or newer, or Metal. See FP8 KV cache.
RecursiveSI/flint is a text-only MoE model with optional reasoning, built-in tool calling, and a 262,144-token context window.
mistralrs serve -m RecursiveSI/flint --quant 4mistralrs run -m RecursiveSI/flint-int4-gguf -f flint-int4-q4_1.ggufIt works with --quant, MoQE, UQFF, and GGUF files in a supported storage format such as Q4_1 or a K-quant. NVFP4 GGUF files and X-LoRA don’t load.
Flint decides on each turn whether to reason, so enable_thinking and reasoning_effort have no effect. Its reasoning comes back in reasoning_content. When you send the conversation back, keep that field on every assistant message, including ones with tool calls.
Tool calls come back as standard tool_calls, streaming or not. Send the tool definitions with every request. See the Flint tool calling example.
The checkpoint sets its own sampling defaults: temperature 1.0, top-p 0.95, and top-k 20. For hard problems, the model authors recommend an output budget of at least 32,768 tokens, since reasoning counts toward it. A finish_reason of "length" means the model ran out of budget, possibly before it answered.
MoE models and MoQE
Section titled “MoE models and MoQE”MoQE quantizes only the routed experts of a mixture-of-experts (MoE) model, which hold most of its weights, and leaves the rest at its original precision:
mistralrs run -m Qwen/Qwen3-30B-A3B --isq 4 --isq-organization moqeIt works with DeepSeek V2/V3, GLM-4.7, GLM-4.7-Flash, Phi 3.5 MoE, Qwen3 MoE, Qwen3-VL MoE, Qwen3.5/Qwen3.6 MoE, LFM2/LFM2.5 MoE, and Flint. Use it with --isq or when you build a UQFF with mistralrs quantize. A GGUF or UQFF that --quant picks loads exactly as it was published, so MoQE doesn’t change it. In Python, pass organization=IsqOrganization.MoQE to Which.Plain(...) or Which.MultimodalPlain(...).
The router that picks experts is sensitive to small numerical changes, so outputs shift slightly between quantization levels.
MLA models (DeepSeek V2/V3, GLM-4.7-Flash)
Section titled “MLA models (DeepSeek V2/V3, GLM-4.7-Flash)”DeepSeek V2, DeepSeek V3 (including the full R1, which uses the V3 architecture), and GLM-4.7-Flash use multi-head latent attention. On Linux CUDA builds with paged attention, mistral.rs caches a compressed latent for these models instead of full keys and values, so the same KV cache memory holds many more tokens. On Metal, on CPU, or with paged attention off, they use a regular-size cache.
The FP8 KV cache (--pa-cache-type f8e4m3) doesn’t work with the compressed cache. See paged attention for sizing the cache.
GPT-OSS
Section titled “GPT-OSS”GPT-OSS ships its expert weights already quantized to MXFP4, so load it without a quantization flag:
mistralrs run -m openai/gpt-oss-20bIf you quantize it further with --isq, only the attention layers and the output head change; the experts stay in MXFP4.
Qwen3 Next
Section titled “Qwen3 Next”Qwen3-Next and Qwen3-Coder-Next mix linear-attention layers with standard attention. In the standard config, one layer in four is standard attention. Only those layers keep a KV cache that grows with the context; the others keep a fixed-size state per request. So the KV cache grows at about a quarter of the rate it would if every layer were standard attention.
Qwen3.5 and Qwen3.6
Section titled “Qwen3.5 and Qwen3.6”Dense and MoE checkpoints of both are supported, and Qwen3.6 loads through the same code as Qwen3.5. To go past the native context length, turn on YaRN scaling with a config override and raise the serving limit. This extends Qwen3.5-27B from its native 262,144 tokens:
mistralrs serve -m Qwen/Qwen3.5-27B \ --hf-overrides '{"text_config":{"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}}' \ --max-model-len 1010000factor: 4.0 stretches 262,144 tokens to about 1M, and --max-model-len 1010000 lets one request use that much. The override merges into the checkpoint’s config, so its other RoPE settings stay as they are. You need both parts, since --max-model-len alone doesn’t turn on scaling. Static YaRN can lower quality on short inputs, so turn it on only when you need more than 262,144 tokens.
Qwen3.8-Flash-Next
Section titled “Qwen3.8-Flash-Next”Qwen3.8-Flash-Next is a large hybrid MoE model. The checkpoint is about 360 GB, and 102 GB of that is a BF16 hashed embedding table (PLE). Quantize it to fit, and on CUDA build with the cuda,flash-attn,cutile features:
mistralrs serve -m Qwen/Qwen3.8-Flash-Next --quant 4mistralrs run -m unsloth/Qwen3.8-Flash-Next-GGUF --quant 4- Embedding table. On CUDA, the PLE table is quantized separately at startup: to 8-bit if it fits in GPU memory, otherwise to 4-bit (about 29 GB). If it still doesn’t fit, it stays in system RAM, which makes prompt processing slower. On CPU and Metal, the table is always read on the host.
- Paged attention works only on CUDA, so pass
--paged-attn offon other devices. The FP8 KV cache and tensor parallelism don’t work with this model. - MTP. Add
--mtpto use the model’s built-in speculative decoding, from safetensors or a UQFF that includes the MTP weights. It picks a draft depth of 2, 3, 4, or 6 tokens as it goes;--mtp-n-predict Nfixes it atN. MTP needs extra memory and recomputes cached prompt prefixes, so repeated prompts take longer to prefill. GGUF files, external MTP models, and DFlash drafters can’t do speculative decoding with this model. - GGUF supports text, images, and video, and
--quantpicks the projector along with the model file.
LFM2 and LFM2.5
Section titled “LFM2 and LFM2.5”Dense, MoE, and VL checkpoints are supported, and the VL models take images. Tool calling works out of the box: mistral.rs detects the Liquid tool-call format from the checkpoint’s chat template.
LFM3 and LFM3-VL
Section titled “LFM3 and LFM3-VL”LiquidAI/LFM3-1.7B is a text model, and LiquidAI/LFM3-VL-2B adds images and video. Both run on CPU, Metal, and CUDA, and load from safetensors, UQFF, or GGUF, with or without quantization:
mistralrs run -m LiquidAI/LFM3-1.7B --cpu --isq q4kmistralrs run -m LiquidAI/LFM3-VL-2B --isq q4k --image photo.jpg -i "Describe this image."mistralrs run -m LiquidAI/LFM3-VL-2B --isq q4k --video clip.mp4 -i "Describe what happens in order."mistralrs run -m LiquidAI/LFM3-VL-2B-GGUF --quant 4 --image photo.jpg -i "Describe this image."- Video is sampled at 2 frames per second, with 4 to 768 frames per clip. Animated GIFs work as is; other formats need FFmpeg on your PATH.
--isqquantizes the language model and the projector. The vision encoder stays at full precision.- The GGUF needs only the model file and its projector; no original config is required.
Examples: Python text, images, and video; HTTP text, images, and video.
LFM3-Audio and LFM3-Omni
Section titled “LFM3-Audio and LFM3-Omni”Both models take audio in and can answer with text, speech, or both:
| Model | Accepts | Returns |
|---|---|---|
LiquidAI/LFM3-Audio-2B |
Text and audio | Text, speech, or both |
LiquidAI/LFM3-Omni-2B |
Text, images, video, and audio together | Text, speech, or both |
mistralrs serve -m LiquidAI/LFM3-Audio-2B --isq q8_0mistralrs serve -m LiquidAI/LFM3-Omni-2B-GGUF --quant 4mistralrs run -m LiquidAI/LFM3-Omni-2B --isq q4k --audio question.wav --image photo.jpg \ -i "Answer the spoken question about this picture."- Audio input is mixed down to mono and resampled to 16 kHz.
- Omni samples video at 2 frames per second, with 4 to 384 frames. It doesn’t extract a video’s soundtrack, so send that separately as audio.
--isqquantizes the language model, plus Omni’s vision projector. The audio encoder, audio projector, speech decoder, and codec stay at full precision, and so does Omni’s vision encoder.- The GGUF projector holds the audio and speech parts, so the model file and projector are all you need.
Speech models covers requesting speech output, the audio endpoints, speech sampling, and limits. Examples: Audio and Omni over HTTP, and Audio and Omni in Python.
d1 and d1-Omni
Section titled “d1 and d1-Omni”LiquidAI/d1-2B and LiquidAI/d1-Omni-2B are decision models. Instead of chatting, they answer named questions about some text or JSON state and return probabilities, extracted values, or boxes. Send questions to POST /v1/systemone, or call send_decision_request in the Python and Rust SDKs.
| Model | Accepts |
|---|---|
| d1-2B | Text, images, and video |
| d1-Omni-2B | Text, images, video, and audio, including spoken questions |
Both load from safetensors, ISQ, UQFF, or GGUF:
mistralrs serve -m LiquidAI/d1-Omni-2Bmistralrs serve -m LiquidAI/d1-Omni-2B-GGUF --quant 8mistralrs serve -m LiquidAI/d1-Omni-2B-GGUF --quant 8 --tok-model-id LiquidAI/d1-Omni-2BThe GGUF on its own answers noul, single-selection choice, and score questions. To also get text, number, and date extraction, multiple choice, estimates, and boxes, add --tok-model-id with the original model, or with a local directory holding its config.json and calibration.json. For the d1-2B GGUF, pass --tok-model-id LiquidAI/d1-2B. Start with Use decision models.
Gemma repositories are gated. Accept the license on the model’s Hugging Face page, then run hf auth login or set HF_TOKEN so mistral.rs can download it.
Gemma 4 accepts image, audio, and video parts in one message. By default it uses constrained decoding to keep tool calls in its native format; see tool calling.
Muse Glimmer
Section titled “Muse Glimmer”mistralrs run -m meta-models/Muse-Glimmer-30BMuse Glimmer takes text, images, and video, but not audio. The model card says it was trained mainly for images, so expect weaker results on video. Tool calls work through the normal APIs, including required, named, and parallel calls. reasoning_effort sets its reasoning strength to low, medium, high, or xhigh, and the template uses high if you leave it out.
The GGUF needs --tok-model-id meta-models/Muse-Glimmer-30B and handles images only; see GGUF compatibility.
MatFormer
Section titled “MatFormer”Gemma 3n (google/gemma-3n-E4B-it) packs several model sizes into one checkpoint. Load a smaller slice when you need to save memory and time, at some cost in quality:
mistralrs run -m google/gemma-3n-E4B-it \ --matformer-config-path matformer_configs/gemma3n.csv \ --matformer-slice-name "Config for E2.49B (block-level)"The bundled matformer_configs/gemma3n.csv lists the slices: Main model (the full E4B), Config for official E2B Model, and sizes in between, from Config for E1.96B (layer-level) up to Config for E3.79B (block-level). Without a slice name, the full model loads.
| Where | Settings |
|---|---|
CLI (run, serve, bench) |
--matformer-config-path, --matformer-slice-name |
| TOML config | matformer_config_path, matformer_slice_name |
Python Which selectors |
matformer_config_path, matformer_slice_name |
| Rust model builders | with_matformer_config_path, with_matformer_slice_name |
Mistral Small tool calling
Section titled “Mistral Small tool calling”Mistral Small 3 checkpoints can call tools, but some repositories don’t ship a template that supports it. Use the bundled one:
mistralrs serve -m mistralai/Mistral-Small-3.2-24B-Instruct-2506 --quant 4 \ --jinja-explicit chat_templates/mistral_small_tool_call.jinjaLLaVA chat templates
Section titled “LLaVA chat templates”Mistral-based LLaVA checkpoints work with their default template. Vicuna-based ones need the Vicuna template:
mistralrs run -m llava-hf/llava-v1.6-vicuna-7b-hf \ -c chat_templates/vicuna.json --image photo.jpg -i "Describe this image."Multimodal device mapping
Section titled “Multimodal device mapping”When you split a multimodal model across GPUs, only the text model’s layers are spread out. The vision and audio encoders load whole on the first GPU, so it needs room for them on top of its share of layers. Automatic device mapping accounts for this. If you assign layers yourself with -n, give the first GPU fewer of them, for example -n "0:10;1:30".
- Phi 3.5 Vision works best with one image per request; multiple images are resized together.
- Phi 4 Multimodal accepts images and audio in the same message.
- Phi 3.5 MoE has 16 experts and routes each token to 2 of them. Quantize the experts with MoQE.
Reporting issues
Section titled “Reporting issues”Open a GitHub issue with the model ID, the command or request you ran, your backend (CUDA, Metal, or CPU), and the error.