Skip to content

Run GGUF models

Terminal window
# A local file
mistralrs run -f /path/to/model.gguf
# The 4-bit file from a Hugging Face repository, as a chat or a server
mistralrs run -m unsloth/Qwen3.5-4B-GGUF --quant 4
mistralrs serve -m unsloth/Qwen3.5-4B-GGUF --quant 4

The .gguf extension tells mistral.rs the format, so you don’t need --format gguf. Once loaded, a GGUF model works like the same model in any other format, including tool calling, decision requests, and LFM3 speech. GGUF supports a different set of models than safetensors, so check GGUF compatibility before you download.

Goal Command
Exact local file mistralrs run -f /path/model-Q4_K_M.gguf
Exact file from a Hub repo mistralrs run -m owner/repo -f model-Q4_K_M.gguf
A bit width from a Hub repo mistralrs run -m owner/repo --quant 4
A bit width from a local directory mistralrs run -m /path/to/gguf-dir --quant 4

The same model flags work with run, serve, and bench. With a local -f file and no -m, mistral.rs treats the file’s directory as the model directory, so you don’t need -m ..

--quant picks one of the files already in the repository without changing its weights. For each bit width, it looks for these first:

--quant Preferred file
2 Q2_K
3 Q3_K_M
4 Q4_K_M
5 Q5_K_M
6 Q6_K
8 Q8_0

If that file isn’t there, it takes another of the same width, such as Q4_K_S or Q4_0 for 4. To pick a variant yourself, give its name from the filename, like --quant q8_0, --quant q5_k_s, or --quant ud-q4_k_xl. Use -f when a repository holds several different models rather than sizes of one. --quant auto doesn’t select GGUF files, and IQ variants (IQ4_XS, IQ3_M, and so on) don’t load.

Split files take a semicolon-separated list:

Terminal window
mistralrs run -m owner/repo \
-f 'model-00001-of-00002.gguf;model-00002-of-00002.gguf'

Without -m, local split files must all be in the same directory.

A GGUF file usually carries its own tokenizer and chat template, so a text model needs nothing else. Multimodal models can need two more kinds of files:

  • Projector. Multimodal models keep their vision or audio encoder in a second GGUF file, usually named mmproj-*.gguf. From a Hub repository, mistral.rs picks one for you, preferring an unquantized projector. With a local -f file, put the projector in the same directory and it’s found automatically, as long as there’s only one.
  • Original model files. Some multimodal models also need the original model’s config.json or processor files. mistral.rs gets them from the repository you loaded, or from the original model named in the GGUF’s metadata.

When the automatic choice is wrong or impossible, name the files yourself:

Option Use it when
--mmproj <file.gguf> You want a specific projector, or it isn’t next to your local file. Separate several projector files with ;.
--tok-model-id <id-or-path> The GGUF doesn’t say which model it came from, or you’re offline. Pass the original Hub ID or a local directory with its config, tokenizer, and processor files.
-t <tokenizer.json> You want to use a specific tokenizer file.
-c <template> You want a different chat template.

Send images, audio, or video with the same flags and request formats as any other multimodal model:

Terminal window
mistralrs run -m unsloth/gemma-4-E4B-it-GGUF --quant 4 \
--image photo.jpg -i "Describe this image."

For a fully local model, put the projector next to the main GGUF, and point --tok-model-id at the original model’s files if the model needs them:

Terminal window
mistralrs run -f /models/Qwen3.5-4B-Q4_K_M.gguf --tok-model-id /models/Qwen3.5-4B

Media support varies: Qwen3.5 GGUF takes images and video, Gemma 4 also takes audio, and Muse Glimmer takes images only. The compatibility table covers every family. See multimodal input for request formats and the Gemma 4 GGUF server example for a complete request.

The SDKs take exact filenames instead of a bit width. Use the filenames from the repository you’re loading, and leave out the projector for a text-only model.

from mistralrs import Runner, Which
runner = Runner(which=Which.GGUF(
quantized_model_id="unsloth/gemma-4-E4B-it-GGUF",
quantized_filename="gemma-4-E4B-it-Q4_K_M.gguf",
mmproj_filename="mmproj-BF16.gguf",
))

Complete Python example.

You can load LoRA adapters on top of a GGUF model and choose one per request:

Terminal window
mistralrs serve -m Qwen/Qwen2.5-0.5B-Instruct-GGUF --quant 4 \
--lora philosophy=closestfriend/brie-qwen2.5-0.5b

A request picks an adapter by its alias (philosophy here), and requests without one use the base model. On a multimodal model, adapters apply to the language model only.

Some architectures, including llama, mistral3, deepseek2, and glm4, store weights in GGUF in a layout that adapters don’t match, so mistral.rs refuses to load the adapter. Load the original safetensors checkpoint with your adapter instead. The GGUF reference has the full list, and LoRA adapters covers choosing adapters per request.

--quant loads a published file as is. To convert a GGUF to a different quantization while it loads, pass the exact file with -f and use --isq:

Option Result
--quant 4 Loads the repo’s 4-bit file unchanged.
--isq q4k Converts the weights in the -f file to Q4K as they load.
Terminal window
mistralrs run -f /path/model-BF16.gguf --isq q4k

Start from a BF16 or 8-bit file when you can, because requantizing an already-small file stacks the quality loss. K-quant conversions can use importance data from --imatrix <file> or --calibration-file <file>; see quantize a model.

Set HF_HUB_OFFLINE=1 and mistral.rs uses only local files and the Hugging Face cache:

Terminal window
HF_HUB_OFFLINE=1 mistralrs run -f /models/model-Q4_K_M.gguf

What you need on disk depends on the model:

Model Files you need
Text models The GGUF file.
LFM3-VL, LFM3-Audio, LFM3-Omni The GGUF file and its projector.
d1, d1-Omni The GGUF file and its projector. Without the original config, d1 answers only some question types.
Other multimodal models The GGUF file, its projector, and usually the original model’s config and processor files.

Put the projector next to the main GGUF. Put the original model’s files in the Hugging Face cache, or in a local directory you pass with --tok-model-id:

Terminal window
HF_HUB_OFFLINE=1 mistralrs run -f /models/model.gguf \
--tok-model-id /models/original-assets

See running offline for where the cache lives.

Symptom What to do
No file matches --quant The error lists the variants in the repo. Pick one of those widths or names, or name an exact file with -f.
An IQ file doesn’t load IQ formats aren’t supported. Choose a Q or K variant from the supported storage formats.
Unknown or unsupported architecture Check GGUF compatibility and use a GGUF of a supported family.
No projector found, or the choice is ambiguous Pass the exact projector file with --mmproj. Some architectures load only as multimodal models and always need one; see multimodal families.
Missing config or processor files Pass the original model ID or a local directory with --tok-model-id.
Unsupported tokenizer Supply the original tokenizer.json with -t, or use --tok-model-id.