Skip to content

Tool calling

Tool calling lets the model ask your code to do something, such as look up the weather or query a database, and then use the result in its answer. You describe each tool with a name and a JSON Schema for its arguments. The model decides when to call one, and mistral.rs returns the call as structured JSON. You need a model and chat template that support tool calls.

You pick one of two ways to run tools:

  • Your app runs it. This is the standard OpenAI flow. The response contains tool_calls, your app runs each one, sends the result back as a tool message, and asks the model again. Use it when the tool lives in your app.
  • mistral.rs runs it. mistral.rs runs the tool, feeds the result back to the model, and repeats until the model answers, all inside one request. This repeat is the tool loop, and each tool call in it is one round. The tool can be a built-in one (web search, Python, shell, Skills), a tool from an MCP (Model Context Protocol) server, a callback you register in the Python or Rust SDK, or your own HTTP endpoint. See agentic runtime for what the response contains.

Send tool definitions with the chat request:

{
"model": "default",
"messages": [{"role": "user", "content": "What's the weather in Tokyo?"}],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city.",
"strict": true,
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
"additionalProperties": false
}
}
}]
}

When the model calls the tool, the reply has finish_reason: "tool_calls" and a tool_calls array:

{"role": "assistant", "tool_calls": [{"id": "call_abc123", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Tokyo\"}"}}]}

Run the function, then send the conversation back with the assistant message and a tool message whose tool_call_id matches, along with the same tools:

{
"messages": [
{"role": "user", "content": "What's the weather in Tokyo?"},
{"role": "assistant", "tool_calls": [...]},
{"role": "tool", "tool_call_id": "call_abc123", "content": "{\"temperature\": 18}"}
],
"tools": [...]
}

Full example

tool_choice controls whether the model calls a tool:

tool_choice What the model does
"auto" (default) Decides for itself whether to call a tool.
"none" Never calls a tool.
"required" Must call at least one tool. Rejected if the request has no tools.
{"type": "function", "function": {"name": "get_weather"}} Must call that tool. In the Responses API, use {"type": "function", "name": "get_weather"}.
{"type": "allowed_tools", "mode": "auto" or "required", "tools": [...]} Can only use the listed function tools. With "required", must call one of them.

With "required" or a named tool, the model may still write some text or reasoning first, but the reply always ends with a valid call to the tool.

Built-in tools such as web search, code_interpreter, and shell are turned on by listing them in tools. You can’t name one in tool_choice or allowed_tools; the request is rejected. Use "required" if you need the model to call one.

allowed_tools narrows which functions the model may call without removing the other definitions from tools. That way you can keep one full tool list and allow different subsets per request:

{
"type": "allowed_tools",
"mode": "required",
"tools": [{"type": "function", "name": "get_weather"}]
}

Pass this object as tool_choice over HTTP. In Rust, use ToolChoice::AllowedTools. Complete examples: HTTP, JavaScript, and Rust.

With function.strict: true, mistral.rs forces the generated arguments to match the tool’s parameters schema as it generates them, so you always get valid JSON with the right fields. Without it, arguments are only parsed after generation and can come back malformed or with extra fields. The tighter the schema, the more it constrains: use required, enum, nested object and array schemas, and additionalProperties: false.

Strict mode only shapes the arguments of a call. It doesn’t make the model call a tool (use tool_choice), and it doesn’t shape the final answer (use response_format). A strict tool with no parameters accepts any JSON object.

Built-in tools are already strict: web search, Python code execution, the file tools, and any MCP tool whose server provides an input schema.

Start the server with the built-in tools you want:

Terminal window
# Web search and Python
mistralrs serve --enable-search --enable-code-execution -m <model>
# Search, Python, and shell together
mistralrs serve --agent -m <model>

Each tool has its own page: web search, code execution, and shell. Build an agent puts them together.

For your own tools, the simplest option is to run them as an MCP server and connect mistral.rs to it; see connect to an MCP server. In the SDKs you can also register a function directly: Python uses Runner(tool_callbacks=...) (full example) and Rust uses with_tool_callback(...) or with_tool_callback_and_tool(...) (full example).

max_tool_rounds caps how many tool calls mistral.rs runs in one request. Without a cap, it stops after 256. Once the cap is reached, mistral.rs stops running tools; if the model then asks for another one, you get that reply instead of a final answer. Pick a cap that fits the task: a few rounds for a single lookup, more for a multi-step task.

Set a server-wide default, and override it on any request:

Where Setting
CLI (server default) --max-tool-rounds 10
HTTP request "max_tool_rounds": 10
Python request ChatCompletionRequest(..., max_tool_rounds=10)
Rust request RequestBuilder::new().with_max_tool_rounds(10)

Setting max_tool_rounds on a request also turns the tool loop on for it. MCP tools count toward the same cap.

--tool-dispatch-url sends tool calls to your own HTTP endpoint. Any call that no built-in tool, MCP server, or callback handles is POSTed there, and the reply goes back to the model:

Terminal window
mistralrs serve --tool-dispatch-url http://localhost:7070/tools -m <model>

The request body:

{"name": "search", "arguments": {"query": "mistralrs"}}

Reply with the tool result as a plain string, or as {"content": "..."}. If the call fails, the model gets an error result instead. The URL is set when the server starts; a request can’t set or change it. Full example

Set on the server at startup Set on each request
Which built-in tools exist: --enable-search, --enable-code-execution, --enable-shell, --agent Which tools the model sees: tools, web_search_options, enable_shell
MCP servers: --mcp-config How the model uses them: tool_choice
Dispatch endpoint: --tool-dispatch-url Round cap: max_tool_rounds, overriding --max-tool-rounds
Tool settings: --code-exec-*, --shell-*, --search-embedding-model Which session to continue: session_id
Default permission: --agent-permission A stricter permission: agent_permission. A request can tighten the server setting but not loosen it.

All flags are in the serve reference. Permissions are covered in permissions and approvals.

When mistral.rs runs tools, the response adds an agentic_tool_calls array with one entry per round. When streaming, agentic_tool_call_progress events arrive before and after each tool runs. Agentic runtime shows how to use them, and the HTTP API reference has the full format.