mistralrs quantize
Quantize a model and save it as UQFF files
mistralrs quantize [OPTIONS] [COMMAND]| Option | Default | Description |
|---|---|---|
-m, --model-id <MODEL_ID> |
Hugging Face model ID or local model directory; optional when -f names local files |
|
-t, --tokenizer <TOKENIZER> |
Path to local tokenizer.json file | |
--dtype <DTYPE> |
auto |
Weight data type: auto, bf16, f16, or f32. auto picks the best one for the device |
--format <FORMAT> |
Input model format: plain (safetensors) or GGUF. Auto-detected from -f when omitted. Possible values: plain, gguf. |
|
-f, --quantized-file <QUANTIZED_FILE> |
GGUF filename(s), separated by semicolons for multiple files | |
--mmproj <MMPROJ> |
GGUF vision or audio projector file(s), separated by semicolons. Auto-selected when there is only one | |
--tok-model-id <TOK_MODEL_ID> |
Optional model ID overriding configuration, tokenizer, and processor assets for a GGUF model | |
--quant <QUANT> |
Which GGUF file to read from a GGUF repo, by bit width or quant name such as q4k |
|
--isq <IN_SITU_QUANT> |
Quantization type(s) to write, such as q4k or 4. Comma-separate or repeat the flag for several, e.g. --isq q4k,q8_0 |
|
--isq-organization <ISQ_ORGANIZATION> |
Which layers to quantize: default for all supported layers, or moqe for only the experts of a mixture-of-experts model, leaving the rest unquantized |
|
--imatrix <IMATRIX> |
Importance matrix (imatrix) file, which tells the quantizer which weights matter most and improves quality at low bit widths | |
--calibration-file <CALIBRATION_FILE> |
Text file to compute an importance matrix from, instead of passing --imatrix |
|
--cpu |
false |
Force CPU-only execution |
-n, --device-layers <DEVICE_LAYERS> |
Layers per device as ORD:NUM;..., e.g. 0:10;1:20 puts 10 layers on GPU 0 and 20 on GPU 1 |
|
--topology <TOPOLOGY> |
YAML file setting the device and quantization for each layer range | |
--hf-cache <HF_CACHE> |
Custom Hugging Face cache directory | |
-o, --output <OUTPUT_PATH> |
Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type) |
|
--no-readme |
false |
Skip README.md model card generation (generated by default in directory mode) |
--uqff-base-model <UQFF_BASE_MODEL> |
Base model ID for the generated README (skips interactive prompt) | |
--uqff-repo-id <UQFF_REPO_ID> |
HF repo ID for the generated README and upload hint (skips interactive prompt) | |
--max-edge <MAX_EDGE> |
Maximum edge length for image resizing (aspect ratio preserved) | |
--max-num-images <MAX_NUM_IMAGES> |
Maximum number of images per request | |
--max-image-length <MAX_IMAGE_LENGTH> |
Largest image side, in pixels, to reserve memory for when placing layers on devices |
mistralrs quantize auto
Section titled “mistralrs quantize auto”Auto-detect model type (recommended)
mistralrs quantize auto [OPTIONS] --isq <IN_SITU_QUANT> --output <OUTPUT_PATH>| Option | Default | Description |
|---|---|---|
-m, --model-id <MODEL_ID> |
Hugging Face model ID or local model directory; optional when -f names local files |
|
-t, --tokenizer <TOKENIZER> |
Path to local tokenizer.json file | |
--dtype <DTYPE> |
auto |
Weight data type: auto, bf16, f16, or f32. auto picks the best one for the device |
--format <FORMAT> |
Input model format: plain (safetensors) or GGUF. Auto-detected from -f when omitted. Possible values: plain, gguf. |
|
-f, --quantized-file <QUANTIZED_FILE> |
GGUF filename(s), separated by semicolons for multiple files | |
--mmproj <MMPROJ> |
GGUF vision or audio projector file(s), separated by semicolons. Auto-selected when there is only one | |
--tok-model-id <TOK_MODEL_ID> |
Optional model ID overriding configuration, tokenizer, and processor assets for a GGUF model | |
--quant <QUANT> |
Which GGUF file to read from a GGUF repo, by bit width or quant name such as q4k |
|
--isq <IN_SITU_QUANT> |
required | Quantization type(s) to write, such as q4k or 4. Comma-separate or repeat the flag for several, e.g. --isq q4k,q8_0 |
--isq-organization <ISQ_ORGANIZATION> |
Which layers to quantize: default for all supported layers, or moqe for only the experts of a mixture-of-experts model, leaving the rest unquantized |
|
--imatrix <IMATRIX> |
Importance matrix (imatrix) file, which tells the quantizer which weights matter most and improves quality at low bit widths | |
--calibration-file <CALIBRATION_FILE> |
Text file to compute an importance matrix from, instead of passing --imatrix |
|
--cpu |
false |
Force CPU-only execution |
-n, --device-layers <DEVICE_LAYERS> |
Layers per device as ORD:NUM;..., e.g. 0:10;1:20 puts 10 layers on GPU 0 and 20 on GPU 1 |
|
--topology <TOPOLOGY> |
YAML file setting the device and quantization for each layer range | |
--hf-cache <HF_CACHE> |
Custom Hugging Face cache directory | |
-o, --output <OUTPUT_PATH> |
required | Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type). Examples: -o model/model-q4k.uqff or -o output/ |
--no-readme |
false |
Skip README.md model card generation (generated by default in directory mode) |
--uqff-base-model <UQFF_BASE_MODEL> |
Base model ID for the generated README (skips interactive prompt) | |
--uqff-repo-id <UQFF_REPO_ID> |
HF repo ID for the generated README and upload hint (skips interactive prompt) | |
--max-edge <MAX_EDGE> |
Maximum edge length for image resizing (aspect ratio preserved) | |
--max-num-images <MAX_NUM_IMAGES> |
Maximum number of images per request | |
--max-image-length <MAX_IMAGE_LENGTH> |
Largest image side, in pixels, to reserve memory for when placing layers on devices |
mistralrs quantize text
Section titled “mistralrs quantize text”Text generation model with explicit architecture
mistralrs quantize text [OPTIONS] --isq <IN_SITU_QUANT> --output <OUTPUT_PATH>| Option | Default | Description |
|---|---|---|
-m, --model-id <MODEL_ID> |
Hugging Face model ID or local model directory; optional when -f names local files |
|
-t, --tokenizer <TOKENIZER> |
Path to local tokenizer.json file | |
--dtype <DTYPE> |
auto |
Weight data type: auto, bf16, f16, or f32. auto picks the best one for the device |
--format <FORMAT> |
Input model format: plain (safetensors) or GGUF. Auto-detected from -f when omitted. Possible values: plain, gguf. |
|
-f, --quantized-file <QUANTIZED_FILE> |
GGUF filename(s), separated by semicolons for multiple files | |
--mmproj <MMPROJ> |
GGUF vision or audio projector file(s), separated by semicolons. Auto-selected when there is only one | |
--tok-model-id <TOK_MODEL_ID> |
Optional model ID overriding configuration, tokenizer, and processor assets for a GGUF model | |
--quant <QUANT> |
Which GGUF file to read from a GGUF repo, by bit width or quant name such as q4k |
|
-a, --arch <ARCH> |
Model architecture (required for text models) | |
--isq <IN_SITU_QUANT> |
required | Quantization type(s) to write, such as q4k or 4. Comma-separate or repeat the flag for several, e.g. --isq q4k,q8_0 |
--isq-organization <ISQ_ORGANIZATION> |
Which layers to quantize: default for all supported layers, or moqe for only the experts of a mixture-of-experts model, leaving the rest unquantized |
|
--imatrix <IMATRIX> |
Importance matrix (imatrix) file, which tells the quantizer which weights matter most and improves quality at low bit widths | |
--calibration-file <CALIBRATION_FILE> |
Text file to compute an importance matrix from, instead of passing --imatrix |
|
--cpu |
false |
Force CPU-only execution |
-n, --device-layers <DEVICE_LAYERS> |
Layers per device as ORD:NUM;..., e.g. 0:10;1:20 puts 10 layers on GPU 0 and 20 on GPU 1 |
|
--topology <TOPOLOGY> |
YAML file setting the device and quantization for each layer range | |
--hf-cache <HF_CACHE> |
Custom Hugging Face cache directory | |
-o, --output <OUTPUT_PATH> |
required | Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type). Examples: -o model/model-q4k.uqff or -o output/ |
--no-readme |
false |
Skip README.md model card generation (generated by default in directory mode) |
--uqff-base-model <UQFF_BASE_MODEL> |
Base model ID for the generated README (skips interactive prompt) | |
--uqff-repo-id <UQFF_REPO_ID> |
HF repo ID for the generated README and upload hint (skips interactive prompt) |
mistralrs quantize multimodal
Section titled “mistralrs quantize multimodal”Multimodal model
mistralrs quantize multimodal [OPTIONS] --isq <IN_SITU_QUANT> --output <OUTPUT_PATH>| Option | Default | Description |
|---|---|---|
-m, --model-id <MODEL_ID> |
Hugging Face model ID or local model directory; optional when -f names local files |
|
-t, --tokenizer <TOKENIZER> |
Path to local tokenizer.json file | |
--dtype <DTYPE> |
auto |
Weight data type: auto, bf16, f16, or f32. auto picks the best one for the device |
--format <FORMAT> |
Input model format: plain (safetensors) or GGUF. Auto-detected from -f when omitted. Possible values: plain, gguf. |
|
-f, --quantized-file <QUANTIZED_FILE> |
GGUF filename(s), separated by semicolons for multiple files | |
--mmproj <MMPROJ> |
GGUF vision or audio projector file(s), separated by semicolons. Auto-selected when there is only one | |
--tok-model-id <TOK_MODEL_ID> |
Optional model ID overriding configuration, tokenizer, and processor assets for a GGUF model | |
--quant <QUANT> |
Which GGUF file to read from a GGUF repo, by bit width or quant name such as q4k |
|
--isq <IN_SITU_QUANT> |
required | Quantization type(s) to write, such as q4k or 4. Comma-separate or repeat the flag for several, e.g. --isq q4k,q8_0 |
--isq-organization <ISQ_ORGANIZATION> |
Which layers to quantize: default for all supported layers, or moqe for only the experts of a mixture-of-experts model, leaving the rest unquantized |
|
--imatrix <IMATRIX> |
Importance matrix (imatrix) file, which tells the quantizer which weights matter most and improves quality at low bit widths | |
--calibration-file <CALIBRATION_FILE> |
Text file to compute an importance matrix from, instead of passing --imatrix |
|
--cpu |
false |
Force CPU-only execution |
-n, --device-layers <DEVICE_LAYERS> |
Layers per device as ORD:NUM;..., e.g. 0:10;1:20 puts 10 layers on GPU 0 and 20 on GPU 1 |
|
--topology <TOPOLOGY> |
YAML file setting the device and quantization for each layer range | |
--hf-cache <HF_CACHE> |
Custom Hugging Face cache directory | |
-o, --output <OUTPUT_PATH> |
required | Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type). Examples: -o model/model-q4k.uqff or -o output/ |
--no-readme |
false |
Skip README.md model card generation (generated by default in directory mode) |
--uqff-base-model <UQFF_BASE_MODEL> |
Base model ID for the generated README (skips interactive prompt) | |
--uqff-repo-id <UQFF_REPO_ID> |
HF repo ID for the generated README and upload hint (skips interactive prompt) | |
--max-edge <MAX_EDGE> |
Maximum edge length for image resizing (aspect ratio preserved) | |
--max-num-images <MAX_NUM_IMAGES> |
Maximum number of images per request | |
--max-image-length <MAX_IMAGE_LENGTH> |
Largest image side, in pixels, to reserve memory for when placing layers on devices |
mistralrs quantize embedding
Section titled “mistralrs quantize embedding”Embedding model
mistralrs quantize embedding [OPTIONS] --model-id <MODEL_ID> --isq <IN_SITU_QUANT> --output <OUTPUT_PATH>| Option | Default | Description |
|---|---|---|
-m, --model-id <MODEL_ID> |
required | Hugging Face model ID or local model directory |
-t, --tokenizer <TOKENIZER> |
Path to local tokenizer.json file | |
--dtype <DTYPE> |
auto |
Weight data type: auto, bf16, f16, or f32. auto picks the best one for the device |
--isq <IN_SITU_QUANT> |
required | Quantization type(s) to write, such as q4k or 4. Comma-separate or repeat the flag for several, e.g. --isq q4k,q8_0 |
--isq-organization <ISQ_ORGANIZATION> |
Which layers to quantize: default for all supported layers, or moqe for only the experts of a mixture-of-experts model, leaving the rest unquantized |
|
--imatrix <IMATRIX> |
Importance matrix (imatrix) file, which tells the quantizer which weights matter most and improves quality at low bit widths | |
--calibration-file <CALIBRATION_FILE> |
Text file to compute an importance matrix from, instead of passing --imatrix |
|
--cpu |
false |
Force CPU-only execution |
-n, --device-layers <DEVICE_LAYERS> |
Layers per device as ORD:NUM;..., e.g. 0:10;1:20 puts 10 layers on GPU 0 and 20 on GPU 1 |
|
--topology <TOPOLOGY> |
YAML file setting the device and quantization for each layer range | |
--hf-cache <HF_CACHE> |
Custom Hugging Face cache directory | |
-o, --output <OUTPUT_PATH> |
required | Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type). Examples: -o model/model-q4k.uqff or -o output/ |
--no-readme |
false |
Skip README.md model card generation (generated by default in directory mode) |
--uqff-base-model <UQFF_BASE_MODEL> |
Base model ID for the generated README (skips interactive prompt) | |
--uqff-repo-id <UQFF_REPO_ID> |
HF repo ID for the generated README and upload hint (skips interactive prompt) |