Skip to content

mistralrs quantize

Quantize a model and save it as UQFF files

mistralrs quantize [OPTIONS] [COMMAND]
Option Default Description
-m, --model-id <MODEL_ID> Hugging Face model ID or local model directory; optional when -f names local files
-t, --tokenizer <TOKENIZER> Path to local tokenizer.json file
--dtype <DTYPE> auto Weight data type: auto, bf16, f16, or f32. auto picks the best one for the device
--format <FORMAT> Input model format: plain (safetensors) or GGUF. Auto-detected from -f when omitted. Possible values: plain, gguf.
-f, --quantized-file <QUANTIZED_FILE> GGUF filename(s), separated by semicolons for multiple files
--mmproj <MMPROJ> GGUF vision or audio projector file(s), separated by semicolons. Auto-selected when there is only one
--tok-model-id <TOK_MODEL_ID> Optional model ID overriding configuration, tokenizer, and processor assets for a GGUF model
--quant <QUANT> Which GGUF file to read from a GGUF repo, by bit width or quant name such as q4k
--isq <IN_SITU_QUANT> Quantization type(s) to write, such as q4k or 4. Comma-separate or repeat the flag for several, e.g. --isq q4k,q8_0
--isq-organization <ISQ_ORGANIZATION> Which layers to quantize: default for all supported layers, or moqe for only the experts of a mixture-of-experts model, leaving the rest unquantized
--imatrix <IMATRIX> Importance matrix (imatrix) file, which tells the quantizer which weights matter most and improves quality at low bit widths
--calibration-file <CALIBRATION_FILE> Text file to compute an importance matrix from, instead of passing --imatrix
--cpu false Force CPU-only execution
-n, --device-layers <DEVICE_LAYERS> Layers per device as ORD:NUM;..., e.g. 0:10;1:20 puts 10 layers on GPU 0 and 20 on GPU 1
--topology <TOPOLOGY> YAML file setting the device and quantization for each layer range
--hf-cache <HF_CACHE> Custom Hugging Face cache directory
-o, --output <OUTPUT_PATH> Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type)
--no-readme false Skip README.md model card generation (generated by default in directory mode)
--uqff-base-model <UQFF_BASE_MODEL> Base model ID for the generated README (skips interactive prompt)
--uqff-repo-id <UQFF_REPO_ID> HF repo ID for the generated README and upload hint (skips interactive prompt)
--max-edge <MAX_EDGE> Maximum edge length for image resizing (aspect ratio preserved)
--max-num-images <MAX_NUM_IMAGES> Maximum number of images per request
--max-image-length <MAX_IMAGE_LENGTH> Largest image side, in pixels, to reserve memory for when placing layers on devices

Auto-detect model type (recommended)

mistralrs quantize auto [OPTIONS] --isq <IN_SITU_QUANT> --output <OUTPUT_PATH>
Option Default Description
-m, --model-id <MODEL_ID> Hugging Face model ID or local model directory; optional when -f names local files
-t, --tokenizer <TOKENIZER> Path to local tokenizer.json file
--dtype <DTYPE> auto Weight data type: auto, bf16, f16, or f32. auto picks the best one for the device
--format <FORMAT> Input model format: plain (safetensors) or GGUF. Auto-detected from -f when omitted. Possible values: plain, gguf.
-f, --quantized-file <QUANTIZED_FILE> GGUF filename(s), separated by semicolons for multiple files
--mmproj <MMPROJ> GGUF vision or audio projector file(s), separated by semicolons. Auto-selected when there is only one
--tok-model-id <TOK_MODEL_ID> Optional model ID overriding configuration, tokenizer, and processor assets for a GGUF model
--quant <QUANT> Which GGUF file to read from a GGUF repo, by bit width or quant name such as q4k
--isq <IN_SITU_QUANT> required Quantization type(s) to write, such as q4k or 4. Comma-separate or repeat the flag for several, e.g. --isq q4k,q8_0
--isq-organization <ISQ_ORGANIZATION> Which layers to quantize: default for all supported layers, or moqe for only the experts of a mixture-of-experts model, leaving the rest unquantized
--imatrix <IMATRIX> Importance matrix (imatrix) file, which tells the quantizer which weights matter most and improves quality at low bit widths
--calibration-file <CALIBRATION_FILE> Text file to compute an importance matrix from, instead of passing --imatrix
--cpu false Force CPU-only execution
-n, --device-layers <DEVICE_LAYERS> Layers per device as ORD:NUM;..., e.g. 0:10;1:20 puts 10 layers on GPU 0 and 20 on GPU 1
--topology <TOPOLOGY> YAML file setting the device and quantization for each layer range
--hf-cache <HF_CACHE> Custom Hugging Face cache directory
-o, --output <OUTPUT_PATH> required Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type). Examples: -o model/model-q4k.uqff or -o output/
--no-readme false Skip README.md model card generation (generated by default in directory mode)
--uqff-base-model <UQFF_BASE_MODEL> Base model ID for the generated README (skips interactive prompt)
--uqff-repo-id <UQFF_REPO_ID> HF repo ID for the generated README and upload hint (skips interactive prompt)
--max-edge <MAX_EDGE> Maximum edge length for image resizing (aspect ratio preserved)
--max-num-images <MAX_NUM_IMAGES> Maximum number of images per request
--max-image-length <MAX_IMAGE_LENGTH> Largest image side, in pixels, to reserve memory for when placing layers on devices

Text generation model with explicit architecture

mistralrs quantize text [OPTIONS] --isq <IN_SITU_QUANT> --output <OUTPUT_PATH>
Option Default Description
-m, --model-id <MODEL_ID> Hugging Face model ID or local model directory; optional when -f names local files
-t, --tokenizer <TOKENIZER> Path to local tokenizer.json file
--dtype <DTYPE> auto Weight data type: auto, bf16, f16, or f32. auto picks the best one for the device
--format <FORMAT> Input model format: plain (safetensors) or GGUF. Auto-detected from -f when omitted. Possible values: plain, gguf.
-f, --quantized-file <QUANTIZED_FILE> GGUF filename(s), separated by semicolons for multiple files
--mmproj <MMPROJ> GGUF vision or audio projector file(s), separated by semicolons. Auto-selected when there is only one
--tok-model-id <TOK_MODEL_ID> Optional model ID overriding configuration, tokenizer, and processor assets for a GGUF model
--quant <QUANT> Which GGUF file to read from a GGUF repo, by bit width or quant name such as q4k
-a, --arch <ARCH> Model architecture (required for text models)
--isq <IN_SITU_QUANT> required Quantization type(s) to write, such as q4k or 4. Comma-separate or repeat the flag for several, e.g. --isq q4k,q8_0
--isq-organization <ISQ_ORGANIZATION> Which layers to quantize: default for all supported layers, or moqe for only the experts of a mixture-of-experts model, leaving the rest unquantized
--imatrix <IMATRIX> Importance matrix (imatrix) file, which tells the quantizer which weights matter most and improves quality at low bit widths
--calibration-file <CALIBRATION_FILE> Text file to compute an importance matrix from, instead of passing --imatrix
--cpu false Force CPU-only execution
-n, --device-layers <DEVICE_LAYERS> Layers per device as ORD:NUM;..., e.g. 0:10;1:20 puts 10 layers on GPU 0 and 20 on GPU 1
--topology <TOPOLOGY> YAML file setting the device and quantization for each layer range
--hf-cache <HF_CACHE> Custom Hugging Face cache directory
-o, --output <OUTPUT_PATH> required Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type). Examples: -o model/model-q4k.uqff or -o output/
--no-readme false Skip README.md model card generation (generated by default in directory mode)
--uqff-base-model <UQFF_BASE_MODEL> Base model ID for the generated README (skips interactive prompt)
--uqff-repo-id <UQFF_REPO_ID> HF repo ID for the generated README and upload hint (skips interactive prompt)

Multimodal model

mistralrs quantize multimodal [OPTIONS] --isq <IN_SITU_QUANT> --output <OUTPUT_PATH>
Option Default Description
-m, --model-id <MODEL_ID> Hugging Face model ID or local model directory; optional when -f names local files
-t, --tokenizer <TOKENIZER> Path to local tokenizer.json file
--dtype <DTYPE> auto Weight data type: auto, bf16, f16, or f32. auto picks the best one for the device
--format <FORMAT> Input model format: plain (safetensors) or GGUF. Auto-detected from -f when omitted. Possible values: plain, gguf.
-f, --quantized-file <QUANTIZED_FILE> GGUF filename(s), separated by semicolons for multiple files
--mmproj <MMPROJ> GGUF vision or audio projector file(s), separated by semicolons. Auto-selected when there is only one
--tok-model-id <TOK_MODEL_ID> Optional model ID overriding configuration, tokenizer, and processor assets for a GGUF model
--quant <QUANT> Which GGUF file to read from a GGUF repo, by bit width or quant name such as q4k
--isq <IN_SITU_QUANT> required Quantization type(s) to write, such as q4k or 4. Comma-separate or repeat the flag for several, e.g. --isq q4k,q8_0
--isq-organization <ISQ_ORGANIZATION> Which layers to quantize: default for all supported layers, or moqe for only the experts of a mixture-of-experts model, leaving the rest unquantized
--imatrix <IMATRIX> Importance matrix (imatrix) file, which tells the quantizer which weights matter most and improves quality at low bit widths
--calibration-file <CALIBRATION_FILE> Text file to compute an importance matrix from, instead of passing --imatrix
--cpu false Force CPU-only execution
-n, --device-layers <DEVICE_LAYERS> Layers per device as ORD:NUM;..., e.g. 0:10;1:20 puts 10 layers on GPU 0 and 20 on GPU 1
--topology <TOPOLOGY> YAML file setting the device and quantization for each layer range
--hf-cache <HF_CACHE> Custom Hugging Face cache directory
-o, --output <OUTPUT_PATH> required Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type). Examples: -o model/model-q4k.uqff or -o output/
--no-readme false Skip README.md model card generation (generated by default in directory mode)
--uqff-base-model <UQFF_BASE_MODEL> Base model ID for the generated README (skips interactive prompt)
--uqff-repo-id <UQFF_REPO_ID> HF repo ID for the generated README and upload hint (skips interactive prompt)
--max-edge <MAX_EDGE> Maximum edge length for image resizing (aspect ratio preserved)
--max-num-images <MAX_NUM_IMAGES> Maximum number of images per request
--max-image-length <MAX_IMAGE_LENGTH> Largest image side, in pixels, to reserve memory for when placing layers on devices

Embedding model

mistralrs quantize embedding [OPTIONS] --model-id <MODEL_ID> --isq <IN_SITU_QUANT> --output <OUTPUT_PATH>
Option Default Description
-m, --model-id <MODEL_ID> required Hugging Face model ID or local model directory
-t, --tokenizer <TOKENIZER> Path to local tokenizer.json file
--dtype <DTYPE> auto Weight data type: auto, bf16, f16, or f32. auto picks the best one for the device
--isq <IN_SITU_QUANT> required Quantization type(s) to write, such as q4k or 4. Comma-separate or repeat the flag for several, e.g. --isq q4k,q8_0
--isq-organization <ISQ_ORGANIZATION> Which layers to quantize: default for all supported layers, or moqe for only the experts of a mixture-of-experts model, leaving the rest unquantized
--imatrix <IMATRIX> Importance matrix (imatrix) file, which tells the quantizer which weights matter most and improves quality at low bit widths
--calibration-file <CALIBRATION_FILE> Text file to compute an importance matrix from, instead of passing --imatrix
--cpu false Force CPU-only execution
-n, --device-layers <DEVICE_LAYERS> Layers per device as ORD:NUM;..., e.g. 0:10;1:20 puts 10 layers on GPU 0 and 20 on GPU 1
--topology <TOPOLOGY> YAML file setting the device and quantization for each layer range
--hf-cache <HF_CACHE> Custom Hugging Face cache directory
-o, --output <OUTPUT_PATH> required Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type). Examples: -o model/model-q4k.uqff or -o output/
--no-readme false Skip README.md model card generation (generated by default in directory mode)
--uqff-base-model <UQFF_BASE_MODEL> Base model ID for the generated README (skips interactive prompt)
--uqff-repo-id <UQFF_REPO_ID> HF repo ID for the generated README and upload hint (skips interactive prompt)