Send images, audio, and video
Vision, audio, and omni models take media alongside your text. The supported models list shows which inputs each model accepts. To attach media to a chat message, make its content a list of parts, text and media mixed, instead of a single string.
mistralrs run -m Qwen/Qwen3-VL-4B-Instruct \ --image photo.jpg -i "What is this?"--image, --audio, and --video take a file path or URL and need a prompt with -i. Repeat a flag to attach several files. In interactive chat, you can also type a file path into your message and mistral.rs attaches the file.
Replace the example URL with a publicly reachable image:
curl http://localhost:1234/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "default", "messages": [{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}, {"type": "text", "text": "Describe this image."} ]}] }'To send a file from the client’s disk, inline it as a base64 data URL as shown below.
from mistralrs import ChatCompletionRequest, Runner, Which
runner = Runner(which=Which.MultimodalPlain(model_id="Qwen/Qwen3-VL-4B-Instruct"))response = runner.send_chat_completion_request(ChatCompletionRequest( model="default", messages=[{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "file:///absolute/path/photo.jpg"}}, {"type": "text", "text": "Describe this image."}, ]}], max_tokens=256,))print(response.choices[0].message.content)In the Python SDK, url can be a plain file path, a file:// URL, an http(s) URL, or a data URL.
use mistralrs::{ModelBuilder, MultimodalMessages, TextMessageRole};
let model = ModelBuilder::new("Qwen/Qwen3-VL-4B-Instruct").build().await?;let messages = MultimodalMessages::new().add_image_message( TextMessageRole::User, "Describe this image.", vec![image::open("photo.jpg")?],);let response = model.send_chat_request(messages).await?;Use add_audio_message, add_video_message, or add_multimodal_message for other inputs. Complete example.
Content parts and URL forms
Section titled “Content parts and URL forms”| Part | Payload |
|---|---|
text |
{"text":"Your question"} |
image_url |
{"image_url":{"url":"..."}} |
video_url |
{"video_url":{"url":"..."}} |
audio_url |
{"audio_url":{"url":"..."}} |
input_audio |
{"input_audio":{"data":"BASE64","format":"wav"}} |
Each part also has a type field, as in the examples above.
Over HTTP, a url can be:
- An
http://orhttps://URL on the public internet. The server refuses URLs that point at localhost or a private network address. - An inline
data:<mime>;base64,...URL.
Local paths and file:// URLs don’t work over HTTP, because the server won’t read its own disk on a client’s behalf. Read the file on the client and send it inline instead. The CLI and the Python and Rust SDKs read local files directly.
Each media file can be up to 64 MiB. The whole request body is capped at 50 MiB, and base64 makes a file about a third larger, so send large files by URL.
Attach a video with --video clip.mp4 on the CLI, or with a content part:
{"type":"video_url","video_url":{"url":"https://example.com/clip.mp4"}}Formats other than GIF need FFmpeg on the server; see video setup. mistral.rs samples frames from the clip using the model’s own frame rate and frame-count limits. LFM3-VL, for example, takes 2 frames per second, between 4 and 768 frames in total.
The soundtrack isn’t used. If the model accepts audio and you want it to hear the clip too, extract the track (for example, ffmpeg -i clip.mp4 -vn clip.wav) and attach it as a separate audio part.
For WAV or MP3, base64-encode the file and send it in an input_audio part. Don’t add a data: prefix:
{"type":"input_audio","input_audio":{"data":"BASE64_FILE_BYTES","format":"wav"}}For FLAC or OGG, or to point at a URL, use an audio_url part with an https:// or data: URL. mistral.rs decodes WAV, MP3, FLAC, and OGG. Convert anything else first, for example with ffmpeg -i clip.m4a clip.wav. In Rust, build an AudioInput with AudioInput::from_bytes and pass it to add_audio_message. See HTTP audio requests for request limits.
This covers sending audio in. To get spoken replies, see Speech models.
In-memory images from Python
Section titled “In-memory images from Python”A data URL works with both the OpenAI client and the native SDK:
import base64from pathlib import Path
payload = base64.b64encode(Path("photo.jpg").read_bytes()).decode("ascii")messages = [{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{payload}"}}, {"type": "text", "text": "Describe this image."},]}]For a PIL image, save it into an io.BytesIO buffer first, then encode the buffer’s bytes and use the matching MIME type, such as image/png.
Mixing modalities in one request
Section titled “Mixing modalities in one request”A message can hold any number of parts, and the model reads them in the order you send them. To compare two images:
{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/before.jpg"}}, {"type": "image_url", "image_url": {"url": "https://example.com/after.jpg"}}, {"type": "text", "text": "What changed between these images?"}]}You can also mix kinds, such as a chart, a spoken question about it, and a text instruction, as long as the model accepts all of them. LFM3-Omni and Gemma 4 take images, audio, and video together. See the Omni example.
Image size and memory
Section titled “Image size and memory”Each model resizes images, samples video frames, and resamples audio into the shape it expects, so you don’t need to prepare media yourself. Three load-time flags on run and serve change how much image work mistral.rs plans for:
| Flag | Default | What it does |
|---|---|---|
--max-edge <pixels> |
Off | Resizes each image so its longest side is this many pixels, keeping the aspect ratio. Smaller values mean fewer image tokens and less memory per image. The Qwen-VL, LFM3, Idefics, MiniCPM-O, and Muse Glimmer models use it; others, such as Gemma and Mistral 3, always use their own image size. |
--max-num-images <n> |
1 | How many images per request mistral.rs sets memory aside for when it loads the model. |
--max-image-length <pixels> |
1024 | The image side length that memory reservation assumes. |
--max-num-images and --max-image-length don’t reject requests. They decide how much working memory is reserved at load time, which changes how much is left for the KV cache and how layers are split across GPUs. If you regularly send several large images per request and run out of memory, raise them to match what you send, for example --max-num-images 4 --max-image-length 2048.