Run GGUF models
Quick start
Section titled “Quick start”# A local filemistralrs run -f /path/to/model.gguf
# The 4-bit file from a Hugging Face repository, as a chat or a servermistralrs run -m unsloth/Qwen3.5-4B-GGUF --quant 4mistralrs serve -m unsloth/Qwen3.5-4B-GGUF --quant 4The .gguf extension tells mistral.rs the format, so you don’t need --format gguf. Once loaded, a GGUF model works like the same model in any other format, including tool calling, decision requests, and LFM3 speech. GGUF supports a different set of models than safetensors, so check GGUF compatibility before you download.
Select a GGUF file
Section titled “Select a GGUF file”| Goal | Command |
|---|---|
| Exact local file | mistralrs run -f /path/model-Q4_K_M.gguf |
| Exact file from a Hub repo | mistralrs run -m owner/repo -f model-Q4_K_M.gguf |
| A bit width from a Hub repo | mistralrs run -m owner/repo --quant 4 |
| A bit width from a local directory | mistralrs run -m /path/to/gguf-dir --quant 4 |
The same model flags work with run, serve, and bench. With a local -f file and no -m, mistral.rs treats the file’s directory as the model directory, so you don’t need -m ..
--quant picks one of the files already in the repository without changing its weights. For each bit width, it looks for these first:
--quant |
Preferred file |
|---|---|
2 |
Q2_K |
3 |
Q3_K_M |
4 |
Q4_K_M |
5 |
Q5_K_M |
6 |
Q6_K |
8 |
Q8_0 |
If that file isn’t there, it takes another of the same width, such as Q4_K_S or Q4_0 for 4. To pick a variant yourself, give its name from the filename, like --quant q8_0, --quant q5_k_s, or --quant ud-q4_k_xl. Use -f when a repository holds several different models rather than sizes of one. --quant auto doesn’t select GGUF files, and IQ variants (IQ4_XS, IQ3_M, and so on) don’t load.
Split files take a semicolon-separated list:
mistralrs run -m owner/repo \ -f 'model-00001-of-00002.gguf;model-00002-of-00002.gguf'Without -m, local split files must all be in the same directory.
Model assets and projectors
Section titled “Model assets and projectors”A GGUF file usually carries its own tokenizer and chat template, so a text model needs nothing else. Multimodal models can need two more kinds of files:
- Projector. Multimodal models keep their vision or audio encoder in a second GGUF file, usually named
mmproj-*.gguf. From a Hub repository, mistral.rs picks one for you, preferring an unquantized projector. With a local-ffile, put the projector in the same directory and it’s found automatically, as long as there’s only one. - Original model files. Some multimodal models also need the original model’s
config.jsonor processor files. mistral.rs gets them from the repository you loaded, or from the original model named in the GGUF’s metadata.
When the automatic choice is wrong or impossible, name the files yourself:
| Option | Use it when |
|---|---|
--mmproj <file.gguf> |
You want a specific projector, or it isn’t next to your local file. Separate several projector files with ;. |
--tok-model-id <id-or-path> |
The GGUF doesn’t say which model it came from, or you’re offline. Pass the original Hub ID or a local directory with its config, tokenizer, and processor files. |
-t <tokenizer.json> |
You want to use a specific tokenizer file. |
-c <template> |
You want a different chat template. |
Multimodal GGUF
Section titled “Multimodal GGUF”Send images, audio, or video with the same flags and request formats as any other multimodal model:
mistralrs run -m unsloth/gemma-4-E4B-it-GGUF --quant 4 \ --image photo.jpg -i "Describe this image."For a fully local model, put the projector next to the main GGUF, and point --tok-model-id at the original model’s files if the model needs them:
mistralrs run -f /models/Qwen3.5-4B-Q4_K_M.gguf --tok-model-id /models/Qwen3.5-4BMedia support varies: Qwen3.5 GGUF takes images and video, Gemma 4 also takes audio, and Muse Glimmer takes images only. The compatibility table covers every family. See multimodal input for request formats and the Gemma 4 GGUF server example for a complete request.
Python and Rust
Section titled “Python and Rust”The SDKs take exact filenames instead of a bit width. Use the filenames from the repository you’re loading, and leave out the projector for a text-only model.
from mistralrs import Runner, Which
runner = Runner(which=Which.GGUF( quantized_model_id="unsloth/gemma-4-E4B-it-GGUF", quantized_filename="gemma-4-E4B-it-Q4_K_M.gguf", mmproj_filename="mmproj-BF16.gguf",))use mistralrs::GgufModelBuilder;
let model = GgufModelBuilder::new( "unsloth/gemma-4-E4B-it-GGUF", vec!["gemma-4-E4B-it-Q4_K_M.gguf"],).with_mmproj_files(vec!["mmproj-BF16.gguf"]).build().await?;GGUF with dynamic LoRA
Section titled “GGUF with dynamic LoRA”You can load LoRA adapters on top of a GGUF model and choose one per request:
mistralrs serve -m Qwen/Qwen2.5-0.5B-Instruct-GGUF --quant 4 \ --lora philosophy=closestfriend/brie-qwen2.5-0.5bA request picks an adapter by its alias (philosophy here), and requests without one use the base model. On a multimodal model, adapters apply to the language model only.
Some architectures, including llama, mistral3, deepseek2, and glm4, store weights in GGUF in a layout that adapters don’t match, so mistral.rs refuses to load the adapter. Load the original safetensors checkpoint with your adapter instead. The GGUF reference has the full list, and LoRA adapters covers choosing adapters per request.
Requantize with ISQ
Section titled “Requantize with ISQ”--quant loads a published file as is. To convert a GGUF to a different quantization while it loads, pass the exact file with -f and use --isq:
| Option | Result |
|---|---|
--quant 4 |
Loads the repo’s 4-bit file unchanged. |
--isq q4k |
Converts the weights in the -f file to Q4K as they load. |
mistralrs run -f /path/model-BF16.gguf --isq q4kStart from a BF16 or 8-bit file when you can, because requantizing an already-small file stacks the quality loss. K-quant conversions can use importance data from --imatrix <file> or --calibration-file <file>; see quantize a model.
Run offline
Section titled “Run offline”Set HF_HUB_OFFLINE=1 and mistral.rs uses only local files and the Hugging Face cache:
HF_HUB_OFFLINE=1 mistralrs run -f /models/model-Q4_K_M.ggufWhat you need on disk depends on the model:
| Model | Files you need |
|---|---|
| Text models | The GGUF file. |
| LFM3-VL, LFM3-Audio, LFM3-Omni | The GGUF file and its projector. |
| d1, d1-Omni | The GGUF file and its projector. Without the original config, d1 answers only some question types. |
| Other multimodal models | The GGUF file, its projector, and usually the original model’s config and processor files. |
Put the projector next to the main GGUF. Put the original model’s files in the Hugging Face cache, or in a local directory you pass with --tok-model-id:
HF_HUB_OFFLINE=1 mistralrs run -f /models/model.gguf \ --tok-model-id /models/original-assetsSee running offline for where the cache lives.
Troubleshooting
Section titled “Troubleshooting”| Symptom | What to do |
|---|---|
No file matches --quant |
The error lists the variants in the repo. Pick one of those widths or names, or name an exact file with -f. |
| An IQ file doesn’t load | IQ formats aren’t supported. Choose a Q or K variant from the supported storage formats. |
| Unknown or unsupported architecture | Check GGUF compatibility and use a GGUF of a supported family. |
| No projector found, or the choice is ambiguous | Pass the exact projector file with --mmproj. Some architectures load only as multimodal models and always need one; see multimodal families. |
| Missing config or processor files | Pass the original model ID or a local directory with --tok-model-id. |
| Unsupported tokenizer | Supply the original tokenizer.json with -t, or use --tok-model-id. |