Speech models

mistral.rs supports two speech-related model families:

• Voxtral Realtime: multimodal model accepting audio input for speech transcription through /v1/chat/completions.
• Dia: dedicated text-to-speech model served via /v1/audio/speech.

Voxtral Realtime is classified as a multimodal model because audio is an input modality. Dia is a dedicated speech model.

Voxtral Realtime: audio in, text out

mistralrs serve -m mistralai/Voxtral-Mini-4B-Realtime-2602

The auto-loader recognizes the Voxtral Realtime layout from params.json, consolidated.safetensors, and tekken.json. The earlier VoxtralForConditionalGeneration architecture is not supported.

Voxtral Realtime uses the multimodal chat request shape: audio is an input content part and the response is its transcription. The supported checkpoint is intended for transcription; the text part in these examples does not imply support for summarization, speaker analysis, or other audio tasks.

HTTP

curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "audio_url", "audio_url": {"url": "https://raw.githubusercontent.com/google-gemma/cookbook/refs/heads/main/apps/sample-data/journal1.wav"}},
        {"type": "text", "text": "Transcribe this."}
      ]
    }]
  }'

Python

from mistralrs import ChatCompletionRequest, MultimodalArchitecture, Runner, Which

runner = Runner(
    which=Which.MultimodalPlain(
        model_id="mistralai/Voxtral-Mini-4B-Realtime-2602",
        arch=MultimodalArchitecture.Voxtral,
    )
)

response = runner.send_chat_completion_request(
    ChatCompletionRequest(
        model="default",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "audio_url", "audio_url": {"url": "file:///absolute/path/clip.wav"}},
                    {"type": "text", "text": "Transcribe this audio."},
                ],
            }
        ],
        max_tokens=256,
        temperature=0,
    )
)
print(response.choices[0].message.content)

Rust

use mistralrs::{AudioInput, MultimodalMessages, MultimodalModelBuilder, TextMessageRole};

let model = MultimodalModelBuilder::new("mistralai/Voxtral-Mini-4B-Realtime-2602")
    .build()
    .await?;

let audio_bytes = std::fs::read("clip.wav")?;
let audio = AudioInput::from_bytes(&audio_bytes)?;

let messages = MultimodalMessages::new().add_audio_message(
    TextMessageRole::User,
    "Transcribe this audio.",
    vec![audio],
);

let response = model.send_chat_request(messages).await?;
println!("{}", response.choices[0].message.content.as_ref().unwrap());

Dia: text-to-speech

/v1/audio/speech matches OpenAI:

mistralrs serve -m nari-labs/Dia-1.6B

Dia understands dialogue speaker tags such as [S1] and [S2], and nonverbal parentheticals such as (laughs) or (coughs). Use them in the input string when you want dialogue or expressive speech.

HTTP

curl http://localhost:1234/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "input": "[S1] Hello. This is a test of the text-to-speech system.",
    "response_format": "wav"
  }' \
  --output out.wav

• Output: raw audio bytes.
• response_format: only wav and pcm are read; mp3/opus/aac/flac return a validation error.
• Extra OpenAI fields such as voice, speed, and instructions are silently ignored (the request reads only model, input, and response_format).

Python

import struct
import wave
from pathlib import Path

from mistralrs import Runner, SpeechLoaderType, Which

runner = Runner(
    which=Which.Speech(
        model_id="nari-labs/Dia-1.6B",
        arch=SpeechLoaderType.Dia,
    )
)

response = runner.generate_audio("[S1] mistral r s can generate speech locally.")

output_path = Path("out.wav")
pcm_ints = [int(max(-32768, min(32767, sample * 32767))) for sample in response.pcm]
with wave.open(output_path, "wb") as wav:
    wav.setnchannels(response.channels)
    wav.setsampwidth(2)
    wav.setframerate(response.rate)
    wav.writeframes(b"".join(struct.pack("<h", sample) for sample in pcm_ints))

Rust

use mistralrs::{speech_utils, SpeechLoaderType, SpeechModelBuilder};

let model = SpeechModelBuilder::new("nari-labs/Dia-1.6B", SpeechLoaderType::Dia)
    .build()
    .await?;

let (pcm, rate, channels) = model
    .generate_speech("[S1] mistral r s can generate speech locally.")
    .await?;

let mut output = std::fs::File::create("out.wav")?;
speech_utils::write_pcm_as_wav(&mut output, &pcm, rate as u32, channels as u16)?;