Skip to content

Use pre-quantized UQFF models

UQFF is mistral.rs’s file format for a model that’s already been quantized. Loading one skips the quantization step, and you download only the quantized weights instead of the full-precision checkpoint. Prebuilt UQFF models are published on Hugging Face under mistralrs-community, named <model>-UQFF.

You often don’t need to name a UQFF yourself: --quant 4 loads a matching prebuilt UQFF automatically when one exists. See how mistral.rs chooses what to load.

Terminal window
mistralrs run -m mistralrs-community/gemma-4-E4B-it-UQFF --from-uqff 4

-m is the UQFF repo, or the local directory that mistralrs quantize wrote. It holds the config and tokenizer along with the quantized weights. --from-uqff picks which file to load:

Value Example What it loads
A bit width 4 The file at that width, preferring AFQ on Metal and GGML formats (q4k, q8_0) elsewhere
A format name q4k, afq8 The file in that format
A file name q4k-0.uqff That file

Large models are split into several files (q4k-0.uqff, q4k-1.uqff, and so on). Name the first one, or use a bit width or format name, and mistral.rs loads the rest.

UQFF models work with tensor parallelism; each GPU loads only its share of the weights. Full examples: Rust, multimodal.

If loading fails with a UQFF version error, the file was made either by a mistral.rs release older than 0.8.4 or by a newer release than yours. Regenerate it with mistralrs quantize, or upgrade mistral.rs.

Quantize the model once into a directory:

Terminal window
mistralrs quantize -m google/gemma-4-E4B-it --isq 4 -o gemma-4-E4B-it-UQFF/

A bit width writes both formats at that width, so --isq 4 produces afq4-0.uqff for Metal and q4k-0.uqff for everything else. To make several variants in one run, list them: --isq 4,8 or --isq q4k --isq q8_0. Files are split into parts of up to about 10 GiB.

The directory also gets everything needed to run the model: config, tokenizer, the unquantized leftovers (residual.safetensors), a uqff_report.json describing the variants, and a README model card. quantize asks for the base model and the Hugging Face repo you’ll upload to, to fill in the README. Pass --uqff-base-model and --uqff-repo-id to skip the questions, or --no-readme to skip the README.

Run it:

Terminal window
mistralrs run -m gemma-4-E4B-it-UQFF/ --from-uqff 4

To publish it, check the files with mistralrs uqff verify -m gemma-4-E4B-it-UQFF/, then upload the directory with the hf upload command that quantize prints at the end.

To write a single variant to a single file instead of a directory, give -o a .uqff path, like -o gemma-q4k.uqff. The config, tokenizer, and other files are written next to it.

  • imatrix: for K-quants, pass --imatrix <file> (llama.cpp .imatrix files work) or --calibration-file <text-file> to quantize. See imatrix.
  • Keep some layers at higher precision: pass --topology <file> to pin layers to a different format, such as keeping lm_head at q8_0 in an otherwise Q4K file. The pins apply to every variant you write. See topology.
  • Embeddings and output head: these are always stored at higher precision than the rest. A q4k file keeps them at Q6K, an afq4 file at AFQ6, and a q6k file at Q8_0. The full table is in the reference. uqff_report.json records the format each tensor actually got.

You can also make a UQFF from a GGUF file. Name a local file with -f; -m is optional then:

Terminal window
mistralrs quantize -f /path/model-BF16.gguf --isq q4k -o model-UQFF/

For a GGUF repo, --quant picks which GGUF file to read and --isq picks the UQFF format to write:

Terminal window
mistralrs quantize -m <gguf-repo> --quant 8 --isq q4k -o model-UQFF/

Start from the highest-precision GGUF available, such as BF16 or Q8_0, because quantizing an already quantized file compounds the loss. --quant auto doesn’t work here. For a multimodal GGUF repo, mistral.rs picks the projector file when there’s only one sensible choice; pass --mmproj <file> to choose it yourself, and --tok-model-id <repo> to take the config and tokenizer from another repo.

To save a UQFF while loading a model in Python, set write_uqff="<path>.uqff" on Which together with in_situ_quant. In Rust, call .write_uqff(path) on the builder along with .with_isq(...). The model keeps running after the file is written.

All quantize flags are in the CLI reference.

These commands read only metadata, so you can point them at a Hugging Face repo without downloading the weights:

Command What it does
mistralrs uqff verify -m <repo-or-dir> Checks that the files are complete and consistent. Run it before publishing.
mistralrs uqff inspect -m <repo-or-dir> --quant afq3 Opens an interactive browser of the tensors in one variant.
mistralrs uqff report -m <repo-or-dir> Lists the variants, their files, and the format of each layer. Add --json for machine-readable output.

UQFF files made by quantize come with a uqff_report.json. To add one to files that don’t have it, run report with --write:

Terminal window
mistralrs uqff report -m gemma4_26b_a4b/ \
--write \
--base-model google/gemma-4-26B-A4B-it \
--repo-id mistralrs-community/gemma-4-26B-A4B-it

The file layout is described in the UQFF format reference.