TOML configuration
mistralrs from-config -f <path> reads a TOML file. The top-level command field selects serve or run. Every key maps to a CLI flag of the same subcommand; the mapping is listed per table below. For per-flag semantics, see the generated CLI reference.
Minimal example
Section titled “Minimal example”command = "serve"
[server]host = "0.0.0.0"port = 1234
[[models]]model_id = "Qwen/Qwen3-4B"
[models.quantization]quant = "4"mistralrs from-config -f this.toml runs the server.
Memory sizing and device placement are automatic. For optional overrides, see paged attention and KV caching.
Top-level fields
Section titled “Top-level fields”| Field | Type | Applies to | Purpose |
|---|---|---|---|
command |
string | both | "serve" or "run". |
default_model_id |
string | serve | Model id treated as the default. Must match one of the [[models]] entries. |
thinking |
bool | run | Turns thinking on or off (alias enable_thinking). If this and reasoning_effort are both omitted, thinking is on. Maps to --thinking. |
reasoning_effort |
string | run | off (or none), low, medium, high, or xhigh. Omit to leave it unset. Maps to --reasoning-effort. |
adapter |
string | run | Alias of the LoRA adapter to start with, from [models.adapter]. Omit to run the base model. See adapters. |
[global] section
Section titled “[global] section”| Field | CLI flag | Default | Purpose |
|---|---|---|---|
seed |
--seed |
not set | Sampling seed. |
log |
-l, --log |
not set | Log file for requests/responses. |
token_source |
--token-source |
cache |
Token source string (literal:<token>, env:<var>, path:<file>, cache, none). |
-v/--verbose has no TOML equivalent; use RUST_LOG instead.
[runtime] section
Section titled “[runtime] section”| Field | CLI flag | Default | Purpose |
|---|---|---|---|
max_seqs |
--max-seqs |
1 for run, 32 for serve |
Max concurrent sequences. An explicit value overrides the command default. |
max_num_batched_tokens |
--max-num-batched-tokens |
4096 | Most tokens processed in one step, across all requests. Must be nonzero. |
max_prefill_chunk_tokens |
--max-prefill-chunk-tokens |
512 | Largest piece of a new prompt read in one step while other requests are generating. Models with recurrent layers, such as Qwen3.5, read every prompt in pieces up to this size. CUDA and Metal only. Must be nonzero. |
max_decode_steps_before_prefill |
--max-decode-steps-before-prefill |
8 | Most generation steps to run before a waiting prompt gets a turn. Must be nonzero. |
no_kv_cache |
--no-kv-cache |
false | Disable KV cache entirely. |
prefix_cache_n |
--prefix-cache-n |
up to 16 | How many prefixes the prefix cache keeps. Unset, mistral.rs may keep fewer on models with recurrent layers to fit memory. 0 turns prefix caching off. See prefix caching. |
chat_template |
-c, --chat-template |
not set | Custom chat template file (.json or .jinja), applied to every model. Per-model chat_template in [[models]] overrides it. |
jinja_explicit |
-j, --jinja-explicit |
not set | Explicit Jinja template override. Per-model jinja_explicit also exists. |
matformer_config_path |
--matformer-config-path |
not set | MatFormer (nested-submodel) slice config (CSV/JSON). |
matformer_slice_name |
--matformer-slice-name |
not set | MatFormer slice to load. Requires matformer_config_path. |
mtp_model |
--mtp-model |
not set | MTP (multi-token prediction) assistant model id or path. |
mtp_n_predict |
--mtp-n-predict |
not set | MTP draft tokens proposed per target step. |
mtp_draft_sampling |
--mtp-draft-sampling |
auto |
MTP draft policy: auto, greedy, or probabilistic. |
mcp_config |
--mcp-config |
not set | MCP (Model Context Protocol) client configuration JSON for outbound servers. Also reads MCP_CONFIG_PATH if unset. |
agent |
--agent (alias --agentic) |
false | Shortcut for enable_search = true + enable_code_execution = true + enable_shell = true. |
enable_search |
--enable-search |
false | Enable the built-in web search tool. |
search_embedding_model |
--search-embedding-model |
not set | Search reranker; embedding-gemma is the only accepted value. Requires enable_search (or agent). |
enable_code_execution |
--enable-code-execution |
false | Enable Python code execution. |
code_exec_python |
--code-exec-python |
python on Windows, python3 elsewhere |
Python interpreter. Requires enable_code_execution (or agent). |
code_exec_timeout |
--code-exec-timeout |
60 | Per-call timeout in seconds. Requires enable_code_execution (or agent). |
code_exec_workdir |
--code-exec-workdir |
per-session temp dir | Code execution working directory. Requires enable_code_execution (or agent). |
enable_shell |
--enable-shell |
false | Enable the built-in shell tool for Responses tools[*].type="shell". |
shell_path |
--shell-path |
/bin/sh on Unix, cmd on Windows |
Shell executable. Requires enable_shell (or agent). |
shell_timeout |
--shell-timeout |
600 | Per-call shell timeout in seconds. Requires enable_shell (or agent). |
shell_workdir |
--shell-workdir |
per-session temp dir | Root directory for per-session shell working directories. Requires enable_shell (or agent). |
skills_dir |
--skills-dir |
system temp dir | Directory for uploaded OpenAI-compatible Skills. Requires enable_shell (or agent). |
agent_permission |
--agent-permission |
auto |
auto, ask, or deny: whether model-requested agent actions run automatically, require approval, or are denied. code_exec_permission / --code-exec-permission are accepted as aliases. |
To tune max_num_batched_tokens, max_prefill_chunk_tokens, and max_decode_steps_before_prefill, see prefill and decode scheduling.
[server] section (serve only)
Section titled “[server] section (serve only)”| Field | CLI flag | Default | Purpose |
|---|---|---|---|
host |
--host |
0.0.0.0 |
Bind address. |
port |
-p, --port |
1234 | TCP port. |
no_ui |
--no-ui |
false | Disable the built-in web UI (mounted at /ui by default). |
mcp_port |
--mcp-port |
not set | Also expose the loaded model as an MCP server on this port (JSON-RPC 2.0 at POST /mcp). See serve over MCP. |
max_tool_rounds |
--max-tool-rounds |
not set | Default cap on agentic tool loop rounds. Per-request values from the HTTP API override it; the safety cap is 256 when unset. |
tool_dispatch_url |
--tool-dispatch-url |
not set | URL to POST tool calls to for server-side execution. Only configurable server-side, never per-request. |
disable_access_log |
--disable-access-log |
false | Disable info-level HTTP access logs. |
access_log_format |
--access-log-format |
text |
Access-log format: text or json. |
access_log_health |
--access-log-health |
false | Include health, metrics, docs, and UI requests in HTTP access logs. |
disable_request_id_header |
--disable-request-id-header |
false | Stop echoing x-request-id on responses. |
disable_metrics |
--disable-metrics |
false | Disable Prometheus HTTP metrics and recorder installation. |
The MCP client configuration (mcp_config) lives under [runtime], not [server]: it applies to run as well as serve.
[paged_attn] section
Section titled “[paged_attn] section”| Field | CLI flag | Default | Purpose |
|---|---|---|---|
mode |
--paged-attn |
auto |
auto (on for CUDA and Metal, off for CPU), on, or off. |
gpu_memory_utilization |
--gpu-memory-utilization |
not set | Cap on the fraction of GPU memory used by weights, working memory, and KV cache together, such as 0.9. Conflicts with kv_cache. |
kv_cache |
--kv-cache |
not set | Fixed KV cache size shared by all requests, such as "4GiB" or "65536tokens". Conflicts with gpu_memory_utilization. |
block_size |
--pa-block-size |
32 on CUDA and Metal | Tokens per block. |
cache_type |
--pa-cache-type |
auto |
KV cache data type: auto, or f8e4m3 for an FP8 cache that uses half the memory of BF16. |
For automatic sizing, omit both gpu_memory_utilization and kv_cache (kv_cache = "auto" isn’t accepted). Setting either one needs paging, so it can’t be combined with mode = "off".
To give the model a fixed cache budget, add this to the minimal example:
[paged_attn]kv_cache = "4GiB"The budget is shared by all requests, and it’s the total across the model’s GPUs. Use max_model_len in [[models]] to limit each request’s context. See memory budgets and context length for help choosing settings.
[sandbox] section
Section titled “[sandbox] section”OS-level isolation for the code-execution subprocess. Mechanics and threat model: sandbox reference.
| Field | CLI flag | Default | Purpose |
|---|---|---|---|
mode |
--sandbox |
auto |
auto (on for Linux/macOS, no-op elsewhere), on (missing isolation is a hard error), or off. |
profile |
--sandbox-profile |
profile-dependent | developer for agent/code/shell tools, otherwise restricted. |
max_memory_mb |
--sb-max-memory-mb |
2048 | Per-session memory cap in MiB. |
max_cpu_secs |
--sb-max-cpu-secs |
600 | Per-session CPU time cap in seconds. When rlimits apply, this is raised before execution to at least the enabled code or shell timeout. |
max_procs |
--sb-max-procs |
64 | Per-session process/thread cap. |
network |
--sandbox-network |
profile-dependent | none, loopback, or full. Defaults to full for developer, loopback for restricted. |
[[models]] array
Section titled “[[models]] array”Each entry defines one loaded model.
| Field | Type | Required | Purpose |
|---|---|---|---|
kind |
enum | no | Defaults to auto. Set to text, multimodal, diffusion, speech, or embedding only to force a loader. |
model_id |
string | yes | Hugging Face id or local path. |
tokenizer |
path | no | Local tokenizer.json. |
arch |
enum | no | Architecture override (text models). |
dtype |
enum | no | auto, f16, bf16, f32. |
hf_overrides |
JSON object | no | Recursively merged Hugging Face config overrides. |
max_model_len |
integer | no | Most tokens (prompt plus output) one request may use. Defaults to the smaller of the model’s context and what the KV cache holds. |
chat_template |
path | no | Chat template override for this model. |
jinja_explicit |
path | no | Jinja override for this model. |
matformer_config_path |
path | no | MatFormer slice config (CSV/JSON). |
matformer_slice_name |
string | no | MatFormer slice to load. |
Each [[models]] entry can carry nested sections whose field shapes mirror the corresponding CLI flags:
| Section | Purpose |
|---|---|
[models.format] |
Weight format and file overrides: format, quantized_file, mmproj, tok_model_id, and GGML gqa. |
[models.adapter] |
LoRA, legacy LoRA, or X-LoRA adapters. See adapters. |
[models.quantization] |
quant (same as --quant), isq, from_uqff, isq_organization, imatrix, calibration_file. See quantize a model. |
[models.device] |
Device placement: cpu, device_layers, topology, hf_cache. cpu must be consistent across every entry. |
[models.multimodal] |
Multimodal load-time caps (image/video/audio limits). |
Adapters
Section titled “Adapters”[models.adapter] takes one of three adapter modes per model:
| Mode | Keys | Use it for |
|---|---|---|
| Dynamic LoRA | enable_lora, lora, lora_max_adapters, lora_max_rank, lora_max_bytes |
Adapters chosen per request, on text models and on the language model of multimodal models, including GGUF. |
| Legacy LoRA | legacy_lora, legacy_lora_order |
One fixed adapter on a GGML model or Phi-3 GGUF. |
| X-LoRA | xlora, xlora_order, tgt_non_granular_index |
An X-LoRA checkpoint on a plain or GGML text model, or Phi-3 GGUF. |
The modes can’t be combined, and each *_order key is required with its adapter key. Multimodal models take only dynamic LoRA, on the language model: vision, audio, and projector adapters aren’t supported. See LoRA adapters for details.
Dynamic LoRA adapters are picked per request. With command = "run", the top-level adapter key picks the one to start with:
command = "run"adapter = "code"
[[models]]model_id = "Qwen/Qwen3-4B"
[models.adapter]enable_lora = truelora = [ { alias = "code", source = "org/code-lora", revision = "refs/pr/7" }, { alias = "math", source = "./math-adapter" },]lora_max_adapters = 16lora_max_rank = 256lora_max_bytes = 8589934592- Each
loraentry needs a unique, nonemptyaliasand asource: a Hugging Face repo or a local directory. revisionsets the Hugging Face revision of a remote adapter, such asrefs/pr/7. It defaults tomain, not the base model’s revision, and is ignored for local directories.enable_lora = trueturns on dynamic LoRA with no adapter preloaded. Anyloraentry turns it on too.lora_max_adapters(default 16),lora_max_rank(default 256), andlora_max_bytes(default 8 GiB) limit loaded adapters. See adapter limits.
GGUF models take the same keys:
command = "serve"
[[models]]model_id = "Qwen/Qwen2.5-0.5B-Instruct-GGUF"
[models.quantization]quant = "4"
[models.adapter]lora = [ { alias = "philosophy", source = "closestfriend/brie-qwen2.5-0.5b" },]Multi-model example
Section titled “Multi-model example”command = "serve"default_model_id = "Qwen/Qwen3-4B"
[server]host = "0.0.0.0"port = 1234
[runtime]enable_search = truesearch_embedding_model = "embedding-gemma"
[[models]]model_id = "Qwen/Qwen3-4B"
[models.quantization]quant = "4"
[[models]]model_id = "google/gemma-4-E4B-it"
[models.quantization]quant = "4"For a multimodal GGUF repo, model_id and quant are enough when the repo has a single compatible projector:
[[models]]model_id = "unsloth/gemma-4-E4B-it-GGUF"
[models.quantization]quant = "4"To choose exact files, set them in [models.format]. mmproj picks the projector, and tok_model_id sets where the config, tokenizer, and processor come from. A .gguf filename is enough to select the GGUF format:
[[models]]model_id = "unsloth/gemma-4-E4B-it-GGUF"
[models.format]quantized_file = "gemma-4-E4B-it-Q4_K_M.gguf"mmproj = "mmproj-BF16.gguf"tok_model_id = "google/gemma-4-E4B-it"Validation
Section titled “Validation”Invalid configs abort startup with a message identifying the problem:
- At least one entry in
[[models]]. default_model_idmatches amodel_idin[[models]].cpuis consistent across all models when set.search_embedding_modelrequiresenable_search = true(oragent = true).code_exec_python,code_exec_timeout, andcode_exec_workdireach requireenable_code_execution = true(oragent = true).
CLI usage notes
Section titled “CLI usage notes”Flag interactions that hold on the command line and as TOML keys:
quant(--quant) sets a bit width such as"4", a format name, or"auto", and loads a prebuilt quantized copy when one exists. See how mistral.rs chooses what to load."auto"always quantizes: it picks the highest bit width that fits, from 6 bits down to 2. Omitquantfor full precision.quantconflicts withisq(--isq) andfrom_uqff(--from-uqff).mistralrs tuneneeds an explicit level and rejects"auto".
calibration_file(--calibration-file) conflicts withimatrix(--imatrix).- Dynamic LoRA, legacy LoRA, and X-LoRA are mutually exclusive. See adapters.
--matformer-slice-namerequires--matformer-config-path.mistralrs run:--image,--video, and--audiorequire-i/--input.mistralrs bench:--prompt-lenand--depthaccept comma-separated values for sweeps.- Each
--prompt-lenvalue produces a prefill measurement at that prompt length. - Each
--depthvalue produces a decode measurement that prefillsdepthtokens and then generates--gen-lentokens. --depthmust be greater than 0 when--gen-lenis greater than 0.
- Each
Server behavior notes
Section titled “Server behavior notes”- Authentication. mistral.rs does not implement authentication. Put a reverse proxy (nginx, Caddy, Traefik) in front for auth and TLS. OpenAI SDK clients send an
Authorization: Bearer ...header because the SDK requires an API key. mistral.rs doesn’t check it, so any value works. - Logging and metrics. Access logs go to the server’s stdout/stderr by default, with the request id, route, status, and latency.
GET /metricsexposes Prometheus metrics by default. See observability. - Payload logging.
-vturns on debug logging and-vvtrace logging. ARUST_LOGfilter such asRUST_LOG=mistralrs_core=debug,tower_http=infooverrides both.-l <path>logs all requests and responses to a file.
CORS and request body limit
Section titled “CORS and request body limit”These have no CLI flag or TOML key. To change the allowed origins or the body limit, build the server yourself with MistralRsServerRouterBuilder from mistralrs-server-core.
| Setting | Default | Builder method |
|---|---|---|
| Allowed origins | Any | with_allowed_origins |
| Request body limit | 50 MiB | with_max_body_limit |
| Allowed methods | GET, POST, PUT, DELETE |
fixed |
| Allowed headers | Content-Type, Authorization, x-api-key, anthropic-version, anthropic-beta, x-request-id, request-id |
fixed |
| Exposed headers | x-request-id, request-id |
fixed |