Skip to content

Run models

mistralrs run opens an interactive chat with a model, and mistralrs serve starts an OpenAI-compatible server on port 1234 (change it with -p):

Terminal window
mistralrs run -m Qwen/Qwen3-4B
mistralrs serve -m Qwen/Qwen3-4B

The first time you use a model, mistral.rs downloads it from Hugging Face and caches it. To ask one question and exit, add -i "Your question" to run. In the interactive chat you can type:

Command What it does
/help Lists the commands.
/system <message> Adds a system message without running the model.
/clear Starts a new conversation.
/temperature <0.0-2.0>, /topk <n>, /topp <0.0-1.0> Changes sampling for the next replies.
/exit Quits.

With a multimodal model, put an image, audio, or video path or URL right in your prompt, such as Describe this: photo.jpg.

The same commands work for every kind of model, because mistral.rs reads the checkpoint’s config to tell a chat model from an embedding, speech, or image generation model. Embedding and decision models have no interactive mode, so use serve for those; Choose a model lists the API for each kind. To load models from code, see the Python and Rust examples. Families that need an extra flag or a template fix are in model family notes.

To load a smaller version of the model, add --quant with a bit width:

Terminal window
mistralrs run -m Qwen/Qwen3-4B --quant 4
Value What you get
2, 3, 4, 5, 6, 8 Lower is smaller. 8 needs about half the memory of the original weights and 4 about a quarter.
auto mistral.rs picks a level that fits your hardware, or keeps full precision if the model already fits.
A format name, like q4k or afq8 That exact format.

On a GGUF repository, it picks the file with that bit width. Quantize a model explains what each level costs in quality and lists the format names.

Pass a directory path to -m to load a model from disk:

Terminal window
mistralrs run -m /path/to/model-dir

Lay the directory out like a Hugging Face repository: config.json, the tokenizer (tokenizer.json, or Mistral’s tekken.json), and the .safetensors weights. mistral.rs reads it straight from disk. --quant still works and quantizes the weights as they load.

A directory that holds only GGUF files works the same way: -m /path/to/gguf-dir --quant 4 picks the 4-bit file. If the directory has both GGUF and safetensors weights, add --format gguf to load the GGUF files.

Pass a .gguf file with -f, or a GGUF repository with -m and a bit width:

Terminal window
mistralrs run -f /path/to/model.gguf
mistralrs serve -m unsloth/Qwen3.5-4B-GGUF --quant 4

The extension is enough, so you don’t need --format gguf. Run GGUF models covers picking files, projectors for multimodal models, and SDK loading.

mistral.rs picks the model code from the architectures field in config.json. If a text model’s config names an architecture mistral.rs doesn’t recognize, such as a fine-tune that renamed it, tell it which one to use with --arch:

Terminal window
mistralrs run -m /path/to/finetune --arch qwen3

Values include llama, mistral, qwen3, qwen3moe, phi3, and gpt_oss, and an unknown value prints the full list. --arch covers text models only. In Python, pass arch=Architecture.Qwen3 to Which.Plain.

If a model’s chat template is missing or produces garbled turns, supply your own:

  • --chat-template <file> (-c) takes a .jinja file, or a .json file with a chat_template field.
  • --jinja-explicit <file.jinja> wins over every other template source, including one the model ships in its processor config.

Fixes for common models are bundled in the repository’s chat_templates/ directory, for example -c chat_templates/vicuna.json. See chat templates for how to tell when a template is wrong and how to write one.

Public, ungated models download without logging in. For private or gated models, run hf auth login or set HF_TOKEN. mistral.rs reads the shared Hugging Face credentials, so existing saved tokens continue to work. Gated models also require accepting their license on the model’s Hugging Face page.

Use --token-source none to force anonymous access, even when a token is saved or set in the environment. Account management stays in the HF CLI.

mistralrs download fetches the files needed to run a model without loading it or initializing a GPU. Downloads automatically use native Xet for Xet-backed files and HTTP for other Hub files; no Python package is needed.

Set HF_HUB_OFFLINE=1 to stop mistral.rs from contacting Hugging Face. It then loads only from the local cache, and fails right away if a file is missing.

Terminal window
# With network access: download the model files
mistralrs download -m Qwen/Qwen3-4B
# Later, or on an air-gapped machine with the cache copied over
HF_HUB_OFFLINE=1 mistralrs serve -m Qwen/Qwen3-4B

Pass the same model and artifact options to download and run or serve, such as --quant 4 or -f model.gguf. download rejects --quant auto because it probes inference hardware; choose a level yourself or use mistralrs tune first. When a selection uses ISQ, downloading fetches the source weights and quantization happens when loading the model. See the download reference for all options.

For a whole repository, use hf download <repo>. mistralrs cache list shows what’s cached. The cache lives in $HF_HUB_CACHE if set, otherwise $HF_HOME/hub, otherwise $XDG_CACHE_HOME/huggingface/hub, otherwise ~/.cache/huggingface/hub; --hf-cache <dir> overrides all of them.

A local multimodal GGUF may need config or processor files that aren’t stored beside it. Cache those too, or point --tok-model-id at a local directory that has them. See running GGUF offline.