Serve multiple models from one process
mistralrs serve -m <model> loads one model. To serve several from one process, list them in a TOML file and start it with mistralrs from-config. Each model runs on its own engine, and every request picks a model with its model field.
Starting a multi-model server
Section titled “Starting a multi-model server”Create models.toml:
command = "serve"default_model_id = "Qwen/Qwen3-4B"
[server]host = "0.0.0.0"port = 1234
[[models]]model_id = "Qwen/Qwen3-4B"
[models.quantization]quant = "4"
[[models]]model_id = "google/gemma-4-E4B-it"
[models.quantization]quant = "4"quant takes the same values as the --quant flag: a bit width from 2 to 8 for a smaller version of the model, auto to fit your hardware, or a format name like q4k. Leave it out to load the original weights. See quantize a model.
Start the server:
mistralrs from-config -f models.toml- Each
[[models]]entry loads one model. It takes the same settings as the CLI flags, grouped into tables such as[models.quantization],[models.format], and[models.device]. The TOML reference lists them all. default_model_idmust match one of themodel_ids. Requests that leave outmodel, or send"default", go to that model.- Each entry’s model type is detected the same way as on the command line. If detection gets it wrong, set
kindtotext,multimodal,embedding,speech, ordiffusion. - CPU mode applies to the whole server. Setting
cpu = truein any entry’s[models.device]runs every model on CPU, and settingtrueon one entry andfalseon another is an error.
All models share the GPU, and mistral.rs divides the memory budget between them. See paged attention for how the split works.
Routing a request to a specific model
Section titled “Routing a request to a specific model”A request picks a model by its ID. If it leaves the model out or sends "default", it goes to default_model_id, and fails if you didn’t set one.
Set the model field:
curl http://localhost:1234/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "google/gemma-4-E4B-it", "messages": [{"role": "user", "content": "Hello!"}] }'Point an OpenAI client at the server and set model to the target ID:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-used")
response = client.chat.completions.create( model="google/gemma-4-E4B-it", messages=[{"role": "user", "content": "Hello!"}],)print(response.choices[0].message.content)The in-process Python Runner loads one model, so use the server to host several from Python.
Every request method on Model has a _with_model variant that takes an optional ID; None uses the default model:
let response = model .send_chat_request_with_model(messages, Some("gemma-vision")) .await?;The ID is the model’s model_id, or its alias if you added it with add_model_with_alias. See multi-model from code for loading the models.
Listing loaded models
Section titled “Listing loaded models”curl http://localhost:1234/v1/modelsEach entry has the usual OpenAI fields (id, object, created, owned_by), plus:
| Field | Meaning |
|---|---|
status |
loaded, unloaded, or reloading. |
tools_available |
Whether the model has tools from MCP or tool callbacks. |
mcp_tools_count, mcp_servers_connected |
How many tools and servers it has from MCP (Model Context Protocol). |
Unloading and reloading on demand
Section titled “Unloading and reloading on demand”Unload a model to free its memory. It stays in the model list with status unloaded:
curl http://localhost:1234/v1/models/unload \ -H "Content-Type: application/json" \ -d '{"model_id": "google/gemma-4-E4B-it"}'| Endpoint | What it does |
|---|---|
POST /v1/models/unload |
Frees the model’s memory. |
POST /v1/models/reload |
Loads it back. |
POST /v1/models/status |
Returns whether it’s loaded, unloaded, or reloading. |
All three take {"model_id": "..."}. You don’t have to reload by hand: a request sent to an unloaded model reloads it first, so that request waits for the load to finish. Request and response schemas are in the generated HTTP API reference.
Multi-model from code (Rust)
Section titled “Multi-model from code (Rust)”Load several models in one process with MultiModelBuilder, then route each request with the _with_model methods shown above:
use mistralrs::{IsqType, MultiModelBuilder, TextModelBuilder, MultimodalModelBuilder};
let model = MultiModelBuilder::new() .add_model(TextModelBuilder::new("Qwen/Qwen3-4B").with_isq(IsqType::Q4K)) .add_model_with_alias( "gemma-vision", MultimodalModelBuilder::new("google/gemma-4-E4B-it").with_isq(IsqType::Q4K), ) .with_default_model("Qwen/Qwen3-4B") .build() .await?;