Skip to content

Serve multiple models from one process

mistralrs serve -m <model> loads one model. To serve several from one process, list them in a TOML file and start it with mistralrs from-config. Each model runs on its own engine, and every request picks a model with its model field.

Create models.toml:

command = "serve"
default_model_id = "Qwen/Qwen3-4B"
[server]
host = "0.0.0.0"
port = 1234
[[models]]
model_id = "Qwen/Qwen3-4B"
[models.quantization]
quant = "4"
[[models]]
model_id = "google/gemma-4-E4B-it"
[models.quantization]
quant = "4"

quant takes the same values as the --quant flag: a bit width from 2 to 8 for a smaller version of the model, auto to fit your hardware, or a format name like q4k. Leave it out to load the original weights. See quantize a model.

Start the server:

Terminal window
mistralrs from-config -f models.toml
  • Each [[models]] entry loads one model. It takes the same settings as the CLI flags, grouped into tables such as [models.quantization], [models.format], and [models.device]. The TOML reference lists them all.
  • default_model_id must match one of the model_ids. Requests that leave out model, or send "default", go to that model.
  • Each entry’s model type is detected the same way as on the command line. If detection gets it wrong, set kind to text, multimodal, embedding, speech, or diffusion.
  • CPU mode applies to the whole server. Setting cpu = true in any entry’s [models.device] runs every model on CPU, and setting true on one entry and false on another is an error.

All models share the GPU, and mistral.rs divides the memory budget between them. See paged attention for how the split works.

A request picks a model by its ID. If it leaves the model out or sends "default", it goes to default_model_id, and fails if you didn’t set one.

Set the model field:

Terminal window
curl http://localhost:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-4-E4B-it",
"messages": [{"role": "user", "content": "Hello!"}]
}'
Terminal window
curl http://localhost:1234/v1/models

Each entry has the usual OpenAI fields (id, object, created, owned_by), plus:

Field Meaning
status loaded, unloaded, or reloading.
tools_available Whether the model has tools from MCP or tool callbacks.
mcp_tools_count, mcp_servers_connected How many tools and servers it has from MCP (Model Context Protocol).

Unload a model to free its memory. It stays in the model list with status unloaded:

Terminal window
curl http://localhost:1234/v1/models/unload \
-H "Content-Type: application/json" \
-d '{"model_id": "google/gemma-4-E4B-it"}'
Endpoint What it does
POST /v1/models/unload Frees the model’s memory.
POST /v1/models/reload Loads it back.
POST /v1/models/status Returns whether it’s loaded, unloaded, or reloading.

All three take {"model_id": "..."}. You don’t have to reload by hand: a request sent to an unloaded model reloads it first, so that request waits for the load to finish. Request and response schemas are in the generated HTTP API reference.

Load several models in one process with MultiModelBuilder, then route each request with the _with_model methods shown above:

use mistralrs::{IsqType, MultiModelBuilder, TextModelBuilder, MultimodalModelBuilder};
let model = MultiModelBuilder::new()
.add_model(TextModelBuilder::new("Qwen/Qwen3-4B").with_isq(IsqType::Q4K))
.add_model_with_alias(
"gemma-vision",
MultimodalModelBuilder::new("google/gemma-4-E4B-it").with_isq(IsqType::Q4K),
)
.with_default_model("Qwen/Qwen3-4B")
.build()
.await?;