Run models
mistralrs run opens an interactive chat with a model, and mistralrs serve starts an OpenAI-compatible server on port 1234 (change it with -p):
mistralrs run -m Qwen/Qwen3-4Bmistralrs serve -m Qwen/Qwen3-4BThe first time you use a model, mistral.rs downloads it from Hugging Face and caches it. To ask one question and exit, add -i "Your question" to run. In the interactive chat you can type:
| Command | What it does |
|---|---|
/help |
Lists the commands. |
/system <message> |
Adds a system message without running the model. |
/clear |
Starts a new conversation. |
/temperature <0.0-2.0>, /topk <n>, /topp <0.0-1.0> |
Changes sampling for the next replies. |
/exit |
Quits. |
With a multimodal model, put an image, audio, or video path or URL right in your prompt, such as Describe this: photo.jpg.
The same commands work for every kind of model, because mistral.rs reads the checkpoint’s config to tell a chat model from an embedding, speech, or image generation model. Embedding and decision models have no interactive mode, so use serve for those; Choose a model lists the API for each kind. To load models from code, see the Python and Rust examples. Families that need an extra flag or a template fix are in model family notes.
Select a quantization with --quant
Section titled “Select a quantization with --quant”To load a smaller version of the model, add --quant with a bit width:
mistralrs run -m Qwen/Qwen3-4B --quant 4| Value | What you get |
|---|---|
2, 3, 4, 5, 6, 8 |
Lower is smaller. 8 needs about half the memory of the original weights and 4 about a quarter. |
auto |
mistral.rs picks a level that fits your hardware, or keeps full precision if the model already fits. |
A format name, like q4k or afq8 |
That exact format. |
On a GGUF repository, it picks the file with that bit width. Quantize a model explains what each level costs in quality and lists the format names.
Local model directories
Section titled “Local model directories”Pass a directory path to -m to load a model from disk:
mistralrs run -m /path/to/model-dirLay the directory out like a Hugging Face repository: config.json, the tokenizer (tokenizer.json, or Mistral’s tekken.json), and the .safetensors weights. mistral.rs reads it straight from disk. --quant still works and quantizes the weights as they load.
A directory that holds only GGUF files works the same way: -m /path/to/gguf-dir --quant 4 picks the 4-bit file. If the directory has both GGUF and safetensors weights, add --format gguf to load the GGUF files.
Run GGUF models
Section titled “Run GGUF models”Pass a .gguf file with -f, or a GGUF repository with -m and a bit width:
mistralrs run -f /path/to/model.ggufmistralrs serve -m unsloth/Qwen3.5-4B-GGUF --quant 4The extension is enough, so you don’t need --format gguf. Run GGUF models covers picking files, projectors for multimodal models, and SDK loading.
Forcing an architecture
Section titled “Forcing an architecture”mistral.rs picks the model code from the architectures field in config.json. If a text model’s config names an architecture mistral.rs doesn’t recognize, such as a fine-tune that renamed it, tell it which one to use with --arch:
mistralrs run -m /path/to/finetune --arch qwen3Values include llama, mistral, qwen3, qwen3moe, phi3, and gpt_oss, and an unknown value prints the full list. --arch covers text models only. In Python, pass arch=Architecture.Qwen3 to Which.Plain.
Chat template overrides
Section titled “Chat template overrides”If a model’s chat template is missing or produces garbled turns, supply your own:
--chat-template <file>(-c) takes a.jinjafile, or a.jsonfile with achat_templatefield.--jinja-explicit <file.jinja>wins over every other template source, including one the model ships in its processor config.
Fixes for common models are bundled in the repository’s chat_templates/ directory, for example -c chat_templates/vicuna.json. See chat templates for how to tell when a template is wrong and how to write one.
Hugging Face authentication
Section titled “Hugging Face authentication”Public, ungated models download without logging in. For private or gated models, run hf auth login or set HF_TOKEN. mistral.rs reads the shared Hugging Face credentials, so existing saved tokens continue to work. Gated models also require accepting their license on the model’s Hugging Face page.
Use --token-source none to force anonymous access, even when a token is saved or set in the environment. Account management stays in the HF CLI.
Running offline
Section titled “Running offline”mistralrs download fetches the files needed to run a model without loading it or initializing a GPU. Downloads automatically use native Xet for Xet-backed files and HTTP for other Hub files; no Python package is needed.
Set HF_HUB_OFFLINE=1 to stop mistral.rs from contacting Hugging Face. It then loads only from the local cache, and fails right away if a file is missing.
# With network access: download the model filesmistralrs download -m Qwen/Qwen3-4B
# Later, or on an air-gapped machine with the cache copied overHF_HUB_OFFLINE=1 mistralrs serve -m Qwen/Qwen3-4BPass the same model and artifact options to download and run or serve, such as --quant 4 or -f model.gguf. download rejects --quant auto because it probes inference hardware; choose a level yourself or use mistralrs tune first. When a selection uses ISQ, downloading fetches the source weights and quantization happens when loading the model. See the download reference for all options.
For a whole repository, use hf download <repo>. mistralrs cache list shows what’s cached. The cache lives in $HF_HUB_CACHE if set, otherwise $HF_HOME/hub, otherwise $XDG_CACHE_HOME/huggingface/hub, otherwise ~/.cache/huggingface/hub; --hf-cache <dir> overrides all of them.
A local multimodal GGUF may need config or processor files that aren’t stored beside it. Cache those too, or point --tok-model-id at a local directory that has them. See running GGUF offline.