Serve an OpenAI-compatible API
mistralrs serve puts a local model behind OpenAI-compatible endpoints under /v1. Point any OpenAI SDK or compatible client at http://localhost:1234/v1 as its base URL.
mistralrs serve -m Qwen/Qwen3-4B --quant 4Then send a request:
curl http://localhost:1234/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "default", "messages": [ {"role": "user", "content": "Write a haiku about local inference."} ], "max_tokens": 128 }'With a single -m model, set model to "default" or leave it out. When serving several models, use a model ID exactly as GET /v1/models lists it.
OpenAI Python client
Section titled “OpenAI Python client”from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-used")
response = client.chat.completions.create( model="default", messages=[{"role": "user", "content": "Say hello from mistral.rs."}],)
print(response.choices[0].message.content)If you’ve loaded LoRA adapters, pick one by setting model to its alias, such as model="code", or keep the base model and pass extra_body={"adapter": "code"}.
The client requires an api_key, but the server ignores it, so any non-empty string works; see authentication. Set stream=True to get tokens as they’re generated (full example).
Endpoints
Section titled “Endpoints”| Task | Endpoint | Guide |
|---|---|---|
| Chat, tools, multimodal input | /v1/chat/completions |
Tool calling |
| Responses, background runs, cancellation | /v1/responses |
Compatibility |
| Anthropic clients | /v1/messages |
Messages API |
| Embeddings | /v1/embeddings |
Embeddings |
| Image generation | /v1/images/generations |
Diffusion |
| Speech and recognition | /v1/audio/speech, /v1/audio/transcriptions, /v1/audio/translations |
Speech models |
| Typed decisions | /v1/systemone |
Decision requests |
Each endpoint needs a model of the matching kind loaded, such as an embedding model for /v1/embeddings. If you’re pointing an existing OpenAI client at mistral.rs, check OpenAI compatibility for fields that behave differently. For example, embeddings reject dimensions instead of truncating.
A running server shows an interactive API explorer (Swagger UI) at http://localhost:1234/docs. The generated HTTP reference lists every route, including model management, legacy Completions, files, LoRA adapters, and Skills. HTTP semantics covers routing, streaming, and history.
Tools, structured output, and agentic features
Section titled “Tools, structured output, and agentic features”OpenAI-compatible function tools work on Chat Completions and Responses, including strict: true for JSON-Schema-constrained tool arguments. See tool calling.
response_format with json_schema and the grammar extension constrain output server-side. See structured output.
To have the server run tools itself, such as web search, code execution, and shell, start it with --agent:
mistralrs serve --agent -m Qwen/Qwen3-4BSee agentic runtime for apps for tool progress events, generated files, Skills, and session state.
Configuration
Section titled “Configuration”| Flag | Default | What it does |
|---|---|---|
-p, --port |
1234 |
Port to listen on. |
--host |
0.0.0.0 |
Address to bind. 127.0.0.1 accepts local connections only. |
--no-ui |
UI on | Turns off the web UI at /ui. |
Every flag is in the CLI reference. To keep a deployment in a file, including several models, use a TOML config.
Examples
Section titled “Examples”Runnable server examples cover streaming, Responses, hosted tools, Skills, and speech/transcription.
For Codex and Claude Code setup, see coding agents.