Skip to content

Serve an OpenAI-compatible API

mistralrs serve puts a local model behind OpenAI-compatible endpoints under /v1. Point any OpenAI SDK or compatible client at http://localhost:1234/v1 as its base URL.

Terminal window
mistralrs serve -m Qwen/Qwen3-4B --quant 4

Then send a request:

Terminal window
curl http://localhost:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "default",
"messages": [
{"role": "user", "content": "Write a haiku about local inference."}
],
"max_tokens": 128
}'

With a single -m model, set model to "default" or leave it out. When serving several models, use a model ID exactly as GET /v1/models lists it.

from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-used")
response = client.chat.completions.create(
model="default",
messages=[{"role": "user", "content": "Say hello from mistral.rs."}],
)
print(response.choices[0].message.content)

If you’ve loaded LoRA adapters, pick one by setting model to its alias, such as model="code", or keep the base model and pass extra_body={"adapter": "code"}.

The client requires an api_key, but the server ignores it, so any non-empty string works; see authentication. Set stream=True to get tokens as they’re generated (full example).

Task Endpoint Guide
Chat, tools, multimodal input /v1/chat/completions Tool calling
Responses, background runs, cancellation /v1/responses Compatibility
Anthropic clients /v1/messages Messages API
Embeddings /v1/embeddings Embeddings
Image generation /v1/images/generations Diffusion
Speech and recognition /v1/audio/speech, /v1/audio/transcriptions, /v1/audio/translations Speech models
Typed decisions /v1/systemone Decision requests

Each endpoint needs a model of the matching kind loaded, such as an embedding model for /v1/embeddings. If you’re pointing an existing OpenAI client at mistral.rs, check OpenAI compatibility for fields that behave differently. For example, embeddings reject dimensions instead of truncating.

A running server shows an interactive API explorer (Swagger UI) at http://localhost:1234/docs. The generated HTTP reference lists every route, including model management, legacy Completions, files, LoRA adapters, and Skills. HTTP semantics covers routing, streaming, and history.

Tools, structured output, and agentic features

Section titled “Tools, structured output, and agentic features”

OpenAI-compatible function tools work on Chat Completions and Responses, including strict: true for JSON-Schema-constrained tool arguments. See tool calling.

response_format with json_schema and the grammar extension constrain output server-side. See structured output.

To have the server run tools itself, such as web search, code execution, and shell, start it with --agent:

Terminal window
mistralrs serve --agent -m Qwen/Qwen3-4B

See agentic runtime for apps for tool progress events, generated files, Skills, and session state.

Flag Default What it does
-p, --port 1234 Port to listen on.
--host 0.0.0.0 Address to bind. 127.0.0.1 accepts local connections only.
--no-ui UI on Turns off the web UI at /ui.

Every flag is in the CLI reference. To keep a deployment in a file, including several models, use a TOML config.

Runnable server examples cover streaming, Responses, hosted tools, Skills, and speech/transcription.

For Codex and Claude Code setup, see coding agents.