Skip to content

Model family notes

Most checkpoints need only mistralrs run -m <model-id>. This page collects the families that need an extra flag, have a setting worth knowing, or behave differently from the defaults. Choose a model helps you pick a model for a task, the supported models reference lists every architecture, and block-diffusion models like DiffusionGemma have their own page.

Reasoning models accept a reasoning_effort setting:

Value Effect
off (or none) Turns thinking off.
low, medium, high, xhigh Turns thinking on and passes the tier to the model’s chat template.
Not set Thinking is on, with no tier, except in requests that ask for speech output, where it’s off.

Set it per request with "reasoning_effort": "low", or for a mistralrs run session with --reasoning-effort low. The older enable_thinking boolean still works. If you send both and they disagree, such as enable_thinking: true with reasoning_effort: "off", the request fails.

What a tier changes is up to the chat template. A template that only reads enable_thinking treats every tier as plain “on”; Muse Glimmer’s maps the tiers to its own reasoning strengths.

Qwen3 and SmolLM3 think by default. To turn it off:

Terminal window
mistralrs run -m Qwen/Qwen3-4B --thinking false

--thinking and --reasoning-effort apply to both interactive and one-shot (-i) runs. You can also end a message with /no_think or /think to switch for that turn:

How many rs are in blueberry? /no_think

Qwen3 also publishes FP8 checkpoints. Pass the FP8 model ID and the weights load as published, without --quant. You can store the KV cache in FP8 as well, which halves its memory compared to BF16:

Terminal window
mistralrs serve -m Qwen/Qwen3.8-27B-FP8 --pa-cache-type f8e4m3

An FP8 KV cache needs CUDA with compute capability 8.0 or newer, or Metal. See FP8 KV cache.

RecursiveSI/flint is a text-only MoE model with optional reasoning, built-in tool calling, and a 262,144-token context window.

Terminal window
mistralrs serve -m RecursiveSI/flint --quant 4
mistralrs run -m RecursiveSI/flint-int4-gguf -f flint-int4-q4_1.gguf

It works with --quant, MoQE, UQFF, and GGUF files in a supported storage format such as Q4_1 or a K-quant. NVFP4 GGUF files and X-LoRA don’t load.

Flint decides on each turn whether to reason, so enable_thinking and reasoning_effort have no effect. Its reasoning comes back in reasoning_content. When you send the conversation back, keep that field on every assistant message, including ones with tool calls.

Tool calls come back as standard tool_calls, streaming or not. Send the tool definitions with every request. See the Flint tool calling example.

The checkpoint sets its own sampling defaults: temperature 1.0, top-p 0.95, and top-k 20. For hard problems, the model authors recommend an output budget of at least 32,768 tokens, since reasoning counts toward it. A finish_reason of "length" means the model ran out of budget, possibly before it answered.

MoQE quantizes only the routed experts of a mixture-of-experts (MoE) model, which hold most of its weights, and leaves the rest at its original precision:

Terminal window
mistralrs run -m Qwen/Qwen3-30B-A3B --isq 4 --isq-organization moqe

It works with DeepSeek V2/V3, GLM-4.7, GLM-4.7-Flash, Phi 3.5 MoE, Qwen3 MoE, Qwen3-VL MoE, Qwen3.5/Qwen3.6 MoE, LFM2/LFM2.5 MoE, and Flint. Use it with --isq or when you build a UQFF with mistralrs quantize. A GGUF or UQFF that --quant picks loads exactly as it was published, so MoQE doesn’t change it. In Python, pass organization=IsqOrganization.MoQE to Which.Plain(...) or Which.MultimodalPlain(...).

The router that picks experts is sensitive to small numerical changes, so outputs shift slightly between quantization levels.

MLA models (DeepSeek V2/V3, GLM-4.7-Flash)

Section titled “MLA models (DeepSeek V2/V3, GLM-4.7-Flash)”

DeepSeek V2, DeepSeek V3 (including the full R1, which uses the V3 architecture), and GLM-4.7-Flash use multi-head latent attention. On Linux CUDA builds with paged attention, mistral.rs caches a compressed latent for these models instead of full keys and values, so the same KV cache memory holds many more tokens. On Metal, on CPU, or with paged attention off, they use a regular-size cache.

The FP8 KV cache (--pa-cache-type f8e4m3) doesn’t work with the compressed cache. See paged attention for sizing the cache.

GPT-OSS ships its expert weights already quantized to MXFP4, so load it without a quantization flag:

Terminal window
mistralrs run -m openai/gpt-oss-20b

If you quantize it further with --isq, only the attention layers and the output head change; the experts stay in MXFP4.

Qwen3-Next and Qwen3-Coder-Next mix linear-attention layers with standard attention. In the standard config, one layer in four is standard attention. Only those layers keep a KV cache that grows with the context; the others keep a fixed-size state per request. So the KV cache grows at about a quarter of the rate it would if every layer were standard attention.

Dense and MoE checkpoints of both are supported, and Qwen3.6 loads through the same code as Qwen3.5. To go past the native context length, turn on YaRN scaling with a config override and raise the serving limit. This extends Qwen3.5-27B from its native 262,144 tokens:

Terminal window
mistralrs serve -m Qwen/Qwen3.5-27B \
--hf-overrides '{"text_config":{"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}}' \
--max-model-len 1010000

factor: 4.0 stretches 262,144 tokens to about 1M, and --max-model-len 1010000 lets one request use that much. The override merges into the checkpoint’s config, so its other RoPE settings stay as they are. You need both parts, since --max-model-len alone doesn’t turn on scaling. Static YaRN can lower quality on short inputs, so turn it on only when you need more than 262,144 tokens.

Qwen3.8-Flash-Next is a large hybrid MoE model. The checkpoint is about 360 GB, and 102 GB of that is a BF16 hashed embedding table (PLE). Quantize it to fit, and on CUDA build with the cuda,flash-attn,cutile features:

Terminal window
mistralrs serve -m Qwen/Qwen3.8-Flash-Next --quant 4
mistralrs run -m unsloth/Qwen3.8-Flash-Next-GGUF --quant 4
  • Embedding table. On CUDA, the PLE table is quantized separately at startup: to 8-bit if it fits in GPU memory, otherwise to 4-bit (about 29 GB). If it still doesn’t fit, it stays in system RAM, which makes prompt processing slower. On CPU and Metal, the table is always read on the host.
  • Paged attention works only on CUDA, so pass --paged-attn off on other devices. The FP8 KV cache and tensor parallelism don’t work with this model.
  • MTP. Add --mtp to use the model’s built-in speculative decoding, from safetensors or a UQFF that includes the MTP weights. It picks a draft depth of 2, 3, 4, or 6 tokens as it goes; --mtp-n-predict N fixes it at N. MTP needs extra memory and recomputes cached prompt prefixes, so repeated prompts take longer to prefill. GGUF files, external MTP models, and DFlash drafters can’t do speculative decoding with this model.
  • GGUF supports text, images, and video, and --quant picks the projector along with the model file.

Dense, MoE, and VL checkpoints are supported, and the VL models take images. Tool calling works out of the box: mistral.rs detects the Liquid tool-call format from the checkpoint’s chat template.

LiquidAI/LFM3-1.7B is a text model, and LiquidAI/LFM3-VL-2B adds images and video. Both run on CPU, Metal, and CUDA, and load from safetensors, UQFF, or GGUF, with or without quantization:

Terminal window
mistralrs run -m LiquidAI/LFM3-1.7B --cpu --isq q4k
mistralrs run -m LiquidAI/LFM3-VL-2B --isq q4k --image photo.jpg -i "Describe this image."
mistralrs run -m LiquidAI/LFM3-VL-2B --isq q4k --video clip.mp4 -i "Describe what happens in order."
mistralrs run -m LiquidAI/LFM3-VL-2B-GGUF --quant 4 --image photo.jpg -i "Describe this image."
  • Video is sampled at 2 frames per second, with 4 to 768 frames per clip. Animated GIFs work as is; other formats need FFmpeg on your PATH.
  • --isq quantizes the language model and the projector. The vision encoder stays at full precision.
  • The GGUF needs only the model file and its projector; no original config is required.

Examples: Python text, images, and video; HTTP text, images, and video.

Both models take audio in and can answer with text, speech, or both:

Model Accepts Returns
LiquidAI/LFM3-Audio-2B Text and audio Text, speech, or both
LiquidAI/LFM3-Omni-2B Text, images, video, and audio together Text, speech, or both
Terminal window
mistralrs serve -m LiquidAI/LFM3-Audio-2B --isq q8_0
mistralrs serve -m LiquidAI/LFM3-Omni-2B-GGUF --quant 4
mistralrs run -m LiquidAI/LFM3-Omni-2B --isq q4k --audio question.wav --image photo.jpg \
-i "Answer the spoken question about this picture."
  • Audio input is mixed down to mono and resampled to 16 kHz.
  • Omni samples video at 2 frames per second, with 4 to 384 frames. It doesn’t extract a video’s soundtrack, so send that separately as audio.
  • --isq quantizes the language model, plus Omni’s vision projector. The audio encoder, audio projector, speech decoder, and codec stay at full precision, and so does Omni’s vision encoder.
  • The GGUF projector holds the audio and speech parts, so the model file and projector are all you need.

Speech models covers requesting speech output, the audio endpoints, speech sampling, and limits. Examples: Audio and Omni over HTTP, and Audio and Omni in Python.

LiquidAI/d1-2B and LiquidAI/d1-Omni-2B are decision models. Instead of chatting, they answer named questions about some text or JSON state and return probabilities, extracted values, or boxes. Send questions to POST /v1/systemone, or call send_decision_request in the Python and Rust SDKs.

Model Accepts
d1-2B Text, images, and video
d1-Omni-2B Text, images, video, and audio, including spoken questions

Both load from safetensors, ISQ, UQFF, or GGUF:

Terminal window
mistralrs serve -m LiquidAI/d1-Omni-2B
mistralrs serve -m LiquidAI/d1-Omni-2B-GGUF --quant 8
mistralrs serve -m LiquidAI/d1-Omni-2B-GGUF --quant 8 --tok-model-id LiquidAI/d1-Omni-2B

The GGUF on its own answers noul, single-selection choice, and score questions. To also get text, number, and date extraction, multiple choice, estimates, and boxes, add --tok-model-id with the original model, or with a local directory holding its config.json and calibration.json. For the d1-2B GGUF, pass --tok-model-id LiquidAI/d1-2B. Start with Use decision models.

Gemma repositories are gated. Accept the license on the model’s Hugging Face page, then run hf auth login or set HF_TOKEN so mistral.rs can download it.

Gemma 4 accepts image, audio, and video parts in one message. By default it uses constrained decoding to keep tool calls in its native format; see tool calling.

Terminal window
mistralrs run -m meta-models/Muse-Glimmer-30B

Muse Glimmer takes text, images, and video, but not audio. The model card says it was trained mainly for images, so expect weaker results on video. Tool calls work through the normal APIs, including required, named, and parallel calls. reasoning_effort sets its reasoning strength to low, medium, high, or xhigh, and the template uses high if you leave it out.

The GGUF needs --tok-model-id meta-models/Muse-Glimmer-30B and handles images only; see GGUF compatibility.

Gemma 3n (google/gemma-3n-E4B-it) packs several model sizes into one checkpoint. Load a smaller slice when you need to save memory and time, at some cost in quality:

Terminal window
mistralrs run -m google/gemma-3n-E4B-it \
--matformer-config-path matformer_configs/gemma3n.csv \
--matformer-slice-name "Config for E2.49B (block-level)"

The bundled matformer_configs/gemma3n.csv lists the slices: Main model (the full E4B), Config for official E2B Model, and sizes in between, from Config for E1.96B (layer-level) up to Config for E3.79B (block-level). Without a slice name, the full model loads.

Where Settings
CLI (run, serve, bench) --matformer-config-path, --matformer-slice-name
TOML config matformer_config_path, matformer_slice_name
Python Which selectors matformer_config_path, matformer_slice_name
Rust model builders with_matformer_config_path, with_matformer_slice_name

Mistral Small 3 checkpoints can call tools, but some repositories don’t ship a template that supports it. Use the bundled one:

Terminal window
mistralrs serve -m mistralai/Mistral-Small-3.2-24B-Instruct-2506 --quant 4 \
--jinja-explicit chat_templates/mistral_small_tool_call.jinja

Mistral-based LLaVA checkpoints work with their default template. Vicuna-based ones need the Vicuna template:

Terminal window
mistralrs run -m llava-hf/llava-v1.6-vicuna-7b-hf \
-c chat_templates/vicuna.json --image photo.jpg -i "Describe this image."

When you split a multimodal model across GPUs, only the text model’s layers are spread out. The vision and audio encoders load whole on the first GPU, so it needs room for them on top of its share of layers. Automatic device mapping accounts for this. If you assign layers yourself with -n, give the first GPU fewer of them, for example -n "0:10;1:30".

  • Phi 3.5 Vision works best with one image per request; multiple images are resized together.
  • Phi 4 Multimodal accepts images and audio in the same message.
  • Phi 3.5 MoE has 16 experts and routes each token to 2 of them. Quantize the experts with MoQE.

Open a GitHub issue with the model ID, the command or request you ran, your backend (CUDA, Metal, or CPU), and the error.