Use pre-quantized UQFF models
UQFF is mistral.rs’s file format for a model that’s already been quantized. Loading one skips the quantization step, and you download only the quantized weights instead of the full-precision checkpoint. Prebuilt UQFF models are published on Hugging Face under mistralrs-community, named <model>-UQFF.
You often don’t need to name a UQFF yourself: --quant 4 loads a matching prebuilt UQFF automatically when one exists. See how mistral.rs chooses what to load.
Load a UQFF model
Section titled “Load a UQFF model”mistralrs run -m mistralrs-community/gemma-4-E4B-it-UQFF --from-uqff 4-m is the UQFF repo, or the local directory that mistralrs quantize wrote. It holds the config and tokenizer along with the quantized weights. --from-uqff picks which file to load:
| Value | Example | What it loads |
|---|---|---|
| A bit width | 4 |
The file at that width, preferring AFQ on Metal and GGML formats (q4k, q8_0) elsewhere |
| A format name | q4k, afq8 |
The file in that format |
| A file name | q4k-0.uqff |
That file |
Large models are split into several files (q4k-0.uqff, q4k-1.uqff, and so on). Name the first one, or use a bit width or format name, and mistral.rs loads the rest.
from_uqff on Which takes a file name or a list of file names:
from mistralrs import Runner, Which
runner = Runner( which=Which.Plain( model_id="<repo>", from_uqff="q4k-0.uqff", ),)UqffTextModelBuilder takes the repo and the first file:
use mistralrs::UqffTextModelBuilder;
let model = UqffTextModelBuilder::new("<repo>", vec!["q4k-0.uqff".into()]) .build() .await?;For vision and audio models, use UqffMultimodalModelBuilder.
UQFF models work with tensor parallelism; each GPU loads only its share of the weights. Full examples: Rust, multimodal.
If loading fails with a UQFF version error, the file was made either by a mistral.rs release older than 0.8.4 or by a newer release than yours. Regenerate it with mistralrs quantize, or upgrade mistral.rs.
Make your own UQFF
Section titled “Make your own UQFF”Quantize the model once into a directory:
mistralrs quantize -m google/gemma-4-E4B-it --isq 4 -o gemma-4-E4B-it-UQFF/A bit width writes both formats at that width, so --isq 4 produces afq4-0.uqff for Metal and q4k-0.uqff for everything else. To make several variants in one run, list them: --isq 4,8 or --isq q4k --isq q8_0. Files are split into parts of up to about 10 GiB.
The directory also gets everything needed to run the model: config, tokenizer, the unquantized leftovers (residual.safetensors), a uqff_report.json describing the variants, and a README model card. quantize asks for the base model and the Hugging Face repo you’ll upload to, to fill in the README. Pass --uqff-base-model and --uqff-repo-id to skip the questions, or --no-readme to skip the README.
Run it:
mistralrs run -m gemma-4-E4B-it-UQFF/ --from-uqff 4To publish it, check the files with mistralrs uqff verify -m gemma-4-E4B-it-UQFF/, then upload the directory with the hf upload command that quantize prints at the end.
To write a single variant to a single file instead of a directory, give -o a .uqff path, like -o gemma-q4k.uqff. The config, tokenizer, and other files are written next to it.
Control the quality
Section titled “Control the quality”- imatrix: for K-quants, pass
--imatrix <file>(llama.cpp.imatrixfiles work) or--calibration-file <text-file>toquantize. See imatrix. - Keep some layers at higher precision: pass
--topology <file>to pin layers to a different format, such as keepinglm_headatq8_0in an otherwise Q4K file. The pins apply to every variant you write. See topology. - Embeddings and output head: these are always stored at higher precision than the rest. A
q4kfile keeps them at Q6K, anafq4file at AFQ6, and aq6kfile at Q8_0. The full table is in the reference.uqff_report.jsonrecords the format each tensor actually got.
Quantize from a GGUF
Section titled “Quantize from a GGUF”You can also make a UQFF from a GGUF file. Name a local file with -f; -m is optional then:
mistralrs quantize -f /path/model-BF16.gguf --isq q4k -o model-UQFF/For a GGUF repo, --quant picks which GGUF file to read and --isq picks the UQFF format to write:
mistralrs quantize -m <gguf-repo> --quant 8 --isq q4k -o model-UQFF/Start from the highest-precision GGUF available, such as BF16 or Q8_0, because quantizing an already quantized file compounds the loss. --quant auto doesn’t work here. For a multimodal GGUF repo, mistral.rs picks the projector file when there’s only one sensible choice; pass --mmproj <file> to choose it yourself, and --tok-model-id <repo> to take the config and tokenizer from another repo.
From the SDKs
Section titled “From the SDKs”To save a UQFF while loading a model in Python, set write_uqff="<path>.uqff" on Which together with in_situ_quant. In Rust, call .write_uqff(path) on the builder along with .with_isq(...). The model keeps running after the file is written.
All quantize flags are in the CLI reference.
Inspect UQFF files
Section titled “Inspect UQFF files”These commands read only metadata, so you can point them at a Hugging Face repo without downloading the weights:
| Command | What it does |
|---|---|
mistralrs uqff verify -m <repo-or-dir> |
Checks that the files are complete and consistent. Run it before publishing. |
mistralrs uqff inspect -m <repo-or-dir> --quant afq3 |
Opens an interactive browser of the tensors in one variant. |
mistralrs uqff report -m <repo-or-dir> |
Lists the variants, their files, and the format of each layer. Add --json for machine-readable output. |
UQFF files made by quantize come with a uqff_report.json. To add one to files that don’t have it, run report with --write:
mistralrs uqff report -m gemma4_26b_a4b/ \ --write \ --base-model google/gemma-4-26B-A4B-it \ --repo-id mistralrs-community/gemma-4-26B-A4B-itThe file layout is described in the UQFF format reference.