Online calibration
Online calibration improves a quantized model using the requests it’s already serving. While it’s on, mistral.rs records which weights matter most for your traffic, building an importance matrix. When you apply it, mistral.rs requantizes the model with that data and swaps the new weights in without a restart.
The model keeps serving while statistics are collected, though collecting adds some decode overhead. Requests that arrive while the new weights are being applied wait until that finishes.
Only K-quants (q2k to q6k) use the collected data, so serve with one of those.
Calibrate a running server
Section titled “Calibrate a running server”Start the server with --isq and a K-quant, either from safetensors or from a GGUF file:
mistralrs serve -m <model> --isq q4kmistralrs serve -f /path/model-BF16.gguf --isq q4kUse --isq, not --quant: --quant may load a prebuilt UQFF, and calibration only works on models mistral.rs quantized itself. On Metal, name the format: --isq 4 picks AFQ4 there, which doesn’t use the collected data.
Then start collecting, let traffic run, and apply:
# start recording; adds some decode overhead until you applycurl -X POST localhost:1234/calibration/start
# check progresscurl localhost:1234/calibration/status
# requantize with the collected data and swap the new weights incurl -X POST localhost:1234/calibration/apply \ -H "Content-Type: application/json" \ -d '{"save_cimatrix": "traffic.cimatrix"}'save_cimatrix is optional. Over HTTP it must be a bare file name ending in .cimatrix, and the file is written to the server’s working directory.
The same steps are methods on Model:
model.begin_calibration().await?;// ... serve traffic ...let status = model.calibration_status().await?;model.apply_calibration(Some("traffic.cimatrix".into())).await?;On a multi-model setup, use the _with_model variant of each method. See the full example.
runner.begin_calibration()# ... serve traffic ...status = runner.calibration_status()runner.apply_calibration(save_cimatrix="traffic.cimatrix")Each method takes an optional model_id for multi-model setups. See the full example.
Status, start, and apply all return the same fields:
| Field | Meaning |
|---|---|
collecting |
Whether statistics are being recorded |
layers |
Number of layers that can be calibrated |
layers_tracking |
Number of those layers currently recording |
total_rows, min_rows, max_rows |
Token rows seen across all layers, and by the least and most active layer |
apply returns the status as it was just before applying, then stops collection. Wait until min_rows is above zero before you apply: any layer that hasn’t seen traffic yet is requantized without importance data.
If you saved a .cimatrix, you can reuse it on later loads with --imatrix traffic.cimatrix and skip collecting again.
What to know
Section titled “What to know”- Which models work: the model must have been quantized by mistral.rs at load with
--isq, from safetensors or a GGUF file.startreturns an error for unquantized models and for models loaded with--from-uqff. - Which formats benefit: K-quants use the importance data. Other
q*formats and AFQ accept start and apply but requantize without it, so you gain nothing. HQQ, FP8, F8Q8, and MXFP4 can’t collect at all, andstartreturns an error. - Where the new weights come from: mistral.rs requantizes from the original weights, either the safetensors checkpoint or the GGUF file you loaded. If a layer can’t be reloaded from there, it’s requantized from its current quantized weights instead, which loses more precision; the log reports how many layers did this. Starting from an already quantized GGUF has the same cost, so use BF16 safetensors or a BF16 GGUF when quality matters most.