Skip to content

Quantize a model

Quantization stores a model’s weights at lower precision so it needs less memory. A 14B model in BF16 (2 bytes per parameter) needs about 28 GB for its weights; at 4 bits it needs about 7 GB.

To quantize, pass a bit width: 2, 3, 4, 5, 6, or 8. Lower is smaller and loses more precision. You can also pass auto to let mistral.rs pick, or a format name like q4k or fp8.

Terminal window
mistralrs run -m Qwen/Qwen3-4B --quant 4

--quant works the same way with run, serve, and bench, and as the quant key in a TOML config.

On the CLI, --quant uses a copy of the model that’s already quantized when one exists, so you skip the conversion and download less. What happens depends on what -m points to:

-m points to What --quant 4 does
A Hugging Face model with safetensors weights Looks for a prebuilt UQFF in mistralrs-community/<name>-UQFF. For Qwen/Qwen3-4B that’s mistralrs-community/Qwen3-4B-UQFF. If that repo has a 4-bit file, mistral.rs downloads and loads it. If not, it downloads the original weights and quantizes them while loading.
A local safetensors directory Quantizes the weights while loading.
A GGUF repo or a local directory of GGUF files Picks the matching GGUF file. For 4 it tries Q4_K_M, then Q4_K_S, Q4_K_L, Q4_1, and Q4_0. You can also pass the file’s own label, like --quant q5_k_s. IQ files aren’t supported.
A UQFF repo or directory Loads its 4-bit UQFF file. If it has none, loading fails and lists the files it does have.

A bit width matches any format at that width, preferring AFQ on Metal and GGML formats (q4k, q8_0) elsewhere. A format name like q4k matches only that format. The startup log says which file or format was picked.

To skip the UQFF lookup and always quantize the original weights, use --isq instead of --quant. With GGUF, --isq requantizes a file you name with -f:

Terminal window
mistralrs serve -m Qwen/Qwen3-4B --isq 4
mistralrs serve -f /path/model-BF16.gguf --isq q4k

The Python and Rust SDKs always quantize while loading and never look for a prebuilt UQFF. To load one from the SDKs, pass it explicitly as shown in the UQFF guide.

Quantizing while loading is called ISQ (in-situ quantization). mistral.rs quantizes each layer as it reads it, so the full-precision model never has to fit in memory at once, but every load takes longer than loading a prebuilt file. To pay that cost only once, save the result as UQFF.

Footprint scales roughly linearly with bits per weight: a BF16 model uses about half the memory at --quant 8 and a quarter at --quant 4. The KV cache is a separate budget that depends on context length, not on weight quantization. Use nvidia-smi (or equivalent) to measure, and mistralrs tune to estimate before downloading anything.

Start with 8 if it fits and 4 if it doesn’t.

Bits Quality label in tune Use it when
8 Near-lossless The model fits at 8 bits. Output stays closest to the original.
6, 5 Good 8-bit is a little too big.
4 Acceptable You need a large model to fit. This is the most common choice.
3, 2 Degraded Nothing larger fits. Quality drops the most here, so add an imatrix to limit the loss.

Each width maps to a format suited to your hardware. On Metal it’s AFQ (4 is AFQ4). On CUDA and CPU it’s a GGML format (4 is Q4K, 8 is Q8_0). 5 is Q5K everywhere. The full mapping is in the numeric shorthands table.

--quant and --isq also take format names:

Terminal window
mistralrs run -m Qwen/Qwen3-4B --quant q4k # 4-bit K-quant
mistralrs run -m Qwen/Qwen3-4B --quant afq4 # 4-bit AFQ, built for Apple Silicon
mistralrs run -m Qwen/Qwen3-4B --quant q8_0 # 8-bit block format
Names What they are
q2k, q3k, q4k, q5k, q6k, q8k GGML K-quants. They run on every backend and are the only formats that use an imatrix. q8k isn’t available in CUDA builds.
afq2, afq3, afq4, afq6, afq8 Affine quantization with native Apple Silicon kernels. Also runs on CUDA and CPU.
q4_0, q4_1, q5_0, q5_1, q8_0, q8_1 Older GGML block formats. q8_1 isn’t available in CUDA builds.
fp8 8-bit floating point (E4M3) weights.
mxfp4 4-bit microscaling. CUDA and Metal only.
hqq4, hqq8 Half-quadratic quantization at 4 or 8 bits.
f8q8 8-bit format for CPU only.

If you pick a format your hardware can’t run, loading fails with an error. Hardware details are in the quantization types reference. To give some layers a different format from the rest, use a topology file.

Terminal window
mistralrs run -m Qwen/Qwen3-4B --quant auto

--quant auto checks your hardware and picks the highest bit width that fits, trying 6, 5, 4, and then 3 bits (6, 4, and 3 on Metal). If none of those fit it tries 2 bits, and it fails if nothing fits. It then loads that width the same way an explicit --quant does, prebuilt UQFF first.

auto always quantizes. To run at full precision, leave out --quant. It doesn’t pick GGUF files, so pass a bit width or label for GGUF repos.

mistralrs tune runs the same check and prints the result instead of loading the model. It reads only the model’s config files, so nothing large is downloaded:

Terminal window
mistralrs tune -m Qwen/Qwen3-4B

You get one row per format with its estimated size, share of GPU memory, how much context fits beside it, the quality label from the bit width table, and a status: Recommended, Fits, Hybrid (split across GPU and CPU), or Too Large. A ready-to-run command for the recommended row is printed below the table.

--profile changes which widths tune prefers. It recommends the first one in the list that fits:

--profile CUDA and CPU Metal
quality 8, 6, 5, 4, 3, 2 bits 8, 6, 4, 3 bits
balanced (default, and what --quant auto uses) 6, 5, 4, 3 bits 6, 4, 3 bits
fast 4, 3, 2 bits 4, 3, 2 bits

Other options:

Terminal window
mistralrs tune -m Qwen/Qwen3-4B --isq q4k # prefer one format if it fits
mistralrs tune -m Qwen/Qwen3-4B --json # machine-readable output
mistralrs tune -m Qwen/Qwen3-4B --emit-config qwen.toml # save the recommendation as a config
mistralrs from-config -f qwen.toml # then run it

All flags are in the CLI reference.

An importance matrix (imatrix) records which weights matter most when the model runs on sample text, so the quantizer can keep more precision where it counts. It helps most at low bit widths, and only K-quants (q2k to q6k) use it. On Metal, name a K-quant (--isq q4k), because bit widths pick AFQ there.

Pass one of these together with --isq:

Flag What it does
--imatrix <file> Uses an existing imatrix: a llama.cpp .imatrix file or a mistral.rs .cimatrix file.
--calibration-file <file> Runs a text file through the model at load time to build the imatrix, then quantizes. This repeats on every load.

You can’t use both at once. In the SDKs, set imatrix or calibration_file on Which (Python), or call .with_imatrix() or .with_calibration_file() (Rust). Full examples: Rust, Python.

To build an imatrix from a server’s real traffic instead, see online calibration.

--isq-organization moqe quantizes only the expert layers of a Mixture of Experts model and keeps the shared layers at their original precision. It’s useful when the experts make up most of the model’s parameters:

Terminal window
mistralrs serve -m microsoft/Phi-3.5-MoE-instruct --isq 4 --isq-organization moqe

Without it (--isq-organization default), every layer mistral.rs can quantize is quantized. The setting only matters when mistral.rs does the quantizing, including when you make a UQFF; it doesn’t change a GGUF or UQFF that’s already quantized. In the SDKs, use organization=IsqOrganization.MoQE (Python) or .with_mixture_qexperts_isq() (Rust). Full example.

If a model was quantized at load, you can switch it to another format without restarting the server:

Terminal window
curl -X POST localhost:1234/re_isq \
-H "Content-Type: application/json" \
-d '{"ggml_type": "Q4K"}'

This requantizes the weights already in memory, so moving to a higher bit width doesn’t bring back precision that’s already gone. It returns an error for an unquantized model, or a UQFF or GGUF loaded as is. The SDK equivalents are runner.send_re_isq("Q4K") (Python) and model.re_isq_model(IsqType::Q4K).await? (Rust).

These load as they are, with no conversion:

  • UQFF: picked up by --quant when a match exists, or loaded by name with --from-uqff. See UQFF.
  • GGUF: pick a file from a GGUF repo with --quant, or name one with -f. See GGUF support for compatible models.
  • GPTQ, AWQ, FP8, and NVFP4 checkpoints: detected from the model’s config, so leave out --quant and --isq. See the quantization types reference.