Skip to content

Online calibration

Online calibration improves a quantized model using the requests it’s already serving. While it’s on, mistral.rs records which weights matter most for your traffic, building an importance matrix. When you apply it, mistral.rs requantizes the model with that data and swaps the new weights in without a restart.

The model keeps serving while statistics are collected, though collecting adds some decode overhead. Requests that arrive while the new weights are being applied wait until that finishes.

Only K-quants (q2k to q6k) use the collected data, so serve with one of those.

Start the server with --isq and a K-quant, either from safetensors or from a GGUF file:

Terminal window
mistralrs serve -m <model> --isq q4k
mistralrs serve -f /path/model-BF16.gguf --isq q4k

Use --isq, not --quant: --quant may load a prebuilt UQFF, and calibration only works on models mistral.rs quantized itself. On Metal, name the format: --isq 4 picks AFQ4 there, which doesn’t use the collected data.

Then start collecting, let traffic run, and apply:

Terminal window
# start recording; adds some decode overhead until you apply
curl -X POST localhost:1234/calibration/start
# check progress
curl localhost:1234/calibration/status
# requantize with the collected data and swap the new weights in
curl -X POST localhost:1234/calibration/apply \
-H "Content-Type: application/json" \
-d '{"save_cimatrix": "traffic.cimatrix"}'

save_cimatrix is optional. Over HTTP it must be a bare file name ending in .cimatrix, and the file is written to the server’s working directory.

Status, start, and apply all return the same fields:

Field Meaning
collecting Whether statistics are being recorded
layers Number of layers that can be calibrated
layers_tracking Number of those layers currently recording
total_rows, min_rows, max_rows Token rows seen across all layers, and by the least and most active layer

apply returns the status as it was just before applying, then stops collection. Wait until min_rows is above zero before you apply: any layer that hasn’t seen traffic yet is requantized without importance data.

If you saved a .cimatrix, you can reuse it on later loads with --imatrix traffic.cimatrix and skip collecting again.

  • Which models work: the model must have been quantized by mistral.rs at load with --isq, from safetensors or a GGUF file. start returns an error for unquantized models and for models loaded with --from-uqff.
  • Which formats benefit: K-quants use the importance data. Other q* formats and AFQ accept start and apply but requantize without it, so you gain nothing. HQQ, FP8, F8Q8, and MXFP4 can’t collect at all, and start returns an error.
  • Where the new weights come from: mistral.rs requantizes from the original weights, either the safetensors checkpoint or the GGUF file you loaded. If a layer can’t be reloaded from there, it’s requantized from its current quantized weights instead, which loses more precision; the log reports how many layers did this. Starting from an already quantized GGUF has the same cost, so use BF16 safetensors or a BF16 GGUF when quality matters most.