Speech models
Pick a speech model by what you need:
| Model | What it does | HTTP endpoints |
|---|---|---|
LFM3-Audio (LiquidAI/LFM3-Audio-2B) |
Listens and talks: transcription, translation into English, text-to-speech, and spoken chat | Chat Completions, /v1/audio/speech, /v1/audio/transcriptions, /v1/audio/translations |
LFM3-Omni (LiquidAI/LFM3-Omni-2B) |
Everything LFM3-Audio does, plus images and video as input | Same as LFM3-Audio |
Voxtral Realtime (mistralai/Voxtral-Mini-4B-Realtime-2602) |
Transcription only | Chat Completions, /v1/audio/transcriptions |
Dia (nari-labs/Dia-1.6B) |
Text-to-speech, including two-speaker dialogue | /v1/audio/speech |
LFM3: speech input and output
Section titled “LFM3: speech input and output”mistralrs serve -m LiquidAI/LFM3-Audio-2B --isq q8_0--isq q8_0 quantizes the weights to 8 bits as they load, which roughly halves memory. You can drop it to run at full precision. To download a smaller file instead, load the GGUF build with -m LiquidAI/LFM3-Audio-2B-GGUF --quant 4 (about a 1.1 GB model file) or --quant 8 (about 1.8 GB). mistral.rs fetches the matching audio components automatically. The same works for LiquidAI/LFM3-Omni-2B-GGUF. See quantize a model for the other options.
The quickest way to use LFM3 is through the dedicated audio endpoints, which follow OpenAI’s audio API.
Turn text into speech:
curl http://localhost:1234/v1/audio/speech \ -H 'Content-Type: application/json' \ -d '{"model":"default","input":"Hello from mistral.rs.","response_format":"wav"}' \ --output hello.wavTranscribe a recording:
curl http://localhost:1234/v1/audio/transcriptions \Send the same upload to /v1/audio/translations to get an English translation instead of a transcript.
The speech endpoint takes these options:
| Field | Values |
|---|---|
response_format |
mp3 (default), wav, flac, opus, aac, or pcm (raw 16-bit samples with no file header; LFM3 output is mono at 24 kHz). |
speed |
0.25 to 4.0, default 1.0. |
voice |
Omit it or use default. Each checkpoint has one voice, so named OpenAI voices like alloy return an error. |
instructions |
Extra directions for how to say the text. |
stream_format |
audio (default) returns the finished file. sse streams audio events, incrementally only for pcm at speed 1.0. |
mp3, flac, opus, aac, and any speed other than 1.0 need FFmpeg on the server’s PATH. wav and pcm don’t. The endpoint example shows all of these with the OpenAI Python SDK, and OpenAI compatibility lists the transcription options.
Spoken chat
Section titled “Spoken chat”In a chat request, send audio as an input_audio content part. The reply is text unless you ask for speech with modalities:
modalities |
Reply |
|---|---|
omitted or ["text"] |
Text only |
["text", "audio"] |
Text plus speech |
["audio"] |
Speech only |
When you ask for speech, also pass an audio object with a format: wav, mp3, flac, opus, aac, or pcm16.
{ "modalities": ["text", "audio"], "audio": {"format": "wav", "voice": "default"}}The audio comes back base64-encoded in choices[0].message.audio.data, as mono 24 kHz.
LFM3-Audio expects a task-specific system message. Put the matching one first in your chat messages:
| Task | System message |
|---|---|
| Spoken conversation | Respond with interleaved text and audio. |
| Transcription | Perform ASR in English. |
| Text-to-speech | Perform TTS in American English using a female voice. |
The /v1/audio/* endpoints add these for you. Complete chat examples: LFM3-Audio and LFM3-Omni.
To stream speech as it’s generated, set "stream": true and audio.format to pcm16. Decode each delta.audio.data chunk from base64 and append the bytes in the order they arrive. Most chunks hold about a second of audio, and the last one holds whatever is left, so don’t drop it. Chunk and event details are in the HTTP reference.
To continue a spoken conversation, send the model’s earlier replies back as text. Over HTTP, each completed reply includes audio.transcript and an audio.id. Put the transcript in the assistant message’s content, or send {"role": "assistant", "audio": {"id": "..."}} and the server fills the transcript in for you. IDs expire after an hour and can be lost when the server restarts, so store the transcript if you need lasting history. The Python and Rust SDKs don’t support audio IDs, so keep the reply text yourself. Either way, the model sees earlier turns as text, not as audio.
Reply length and speech sampling
Section titled “Reply length and speech sampling”max_tokens counts text tokens, speech frames, and control tokens together, so speech uses it up much faster than text. If the audio stops early with finish_reason: "length", raise max_tokens.
Speech is sampled with its own fixed settings:
| Setting | LFM3-Audio | LFM3-Omni |
|---|---|---|
| Temperature | 1.0 | 0.4 |
| Top-k | 128 | Off |
| Repetition penalty | 1.1, over the last 4,096 frames | 1.2, over the last 64 frames |
GGUF files can carry their own values. The request’s temperature, top_k, and penalty fields only affect the text tokens, so temperature: 0 won’t make the speech come out the same every time.
When a request asks for speech, thinking is off unless you set enable_thinking or reasoning_effort yourself. These requests can’t use tools, JSON schemas or other constrained output, custom logits processors, or logprobs. Adding any of them returns a validation error.
Python and Rust
Section titled “Python and Rust”Load LFM3 with Which.MultimodalPlain(model_id="LiquidAI/LFM3-Audio-2B") in Python or ModelBuilder::new("LiquidAI/LFM3-Audio-2B") in Rust. For plain text-to-speech, call runner.generate_audio(text) or model.generate_speech(text). You get back the raw samples, the sample rate, and the channel count. The Rust example saves them as a WAV file.
For spoken chat or streaming, set modalities and audio on a Python ChatCompletionRequest, or use Rust’s chat speech output builder.
Voxtral Realtime: transcription
Section titled “Voxtral Realtime: transcription”mistralrs serve -m mistralai/Voxtral-Mini-4B-Realtime-2602Send audio to /v1/audio/transcriptions as shown above, or as an audio part in a chat request. Voxtral returns what was said. It’s built for transcription, so don’t expect it to summarize the audio or answer questions about it.
Only this Realtime checkpoint loads. The earlier Voxtral checkpoints (VoxtralForConditionalGeneration) don’t. mistral.rs doesn’t implement OpenAI’s Realtime API, so you can’t stream live microphone audio to it; send a recorded file instead.
In Python and Rust, load Voxtral like any other multimodal model and send audio in a chat message.
Dia: text-to-speech
Section titled “Dia: text-to-speech”mistralrs serve -m nari-labs/Dia-1.6Bcurl http://localhost:1234/v1/audio/speech \ -H 'Content-Type: application/json' \ -d '{"model":"default","input":"[S1] Did you hear that? [S2] I did. (laughs)","response_format":"wav"}' \ --output dialogue.wavMark each speaker with [S1] or [S2], and write sounds like (laughs) or (coughs) inline where you want them. response_format, speed, and voice take the same values as in the LFM3 table above. Dia doesn’t accept instructions and doesn’t stream: stream_format: "sse" returns an error, and the response is always the finished file.
In Python, load Dia with Which.Speech. In Rust, use SpeechModelBuilder. Then call generate_audio or generate_speech as with LFM3. Complete examples: Python and Rust.