Skip to content

Stream chat responses from Rust

stream_chat_request returns a futures::Stream of Response values that you read with .next().await. The minimal loop is in getting started; this page covers errors, tool events, cancellation, and collecting the full reply.

use mistralrs::Response;
use std::io::Write;
let mut stream = model.stream_chat_request(messages).await?;
let mut out = std::io::BufWriter::new(std::io::stdout());
while let Some(item) = stream.next().await {
match item {
Response::Chunk(chunk) => {
if let Some(text) = chunk.choices.first().and_then(|c| c.delta.content.as_ref()) {
out.write_all(text.as_bytes())?;
out.flush()?;
}
}
Response::InternalError(e) | Response::ValidationError(e) => {
eprintln!("stream error: {e}");
break;
}
Response::ModelError(msg, _) => {
eprintln!("stream error: {msg}");
break;
}
_ => {}
}
}

These are the Response variants a chat stream can send:

Variant Meaning
Response::Chunk New output. Text is in choices[0].delta.content.
Response::InternalError, Response::ValidationError The engine or the request failed. Nothing else follows, so stop reading.
Response::ModelError The model failed. It carries the error message and the partial response built so far.
Response::AgenticToolCallProgress, Response::AgenticToolApprovalRequired, Response::File A server-side tool is starting or finished, needs your approval, or produced a file. See the next section.
Response::BlockDenoisingProgress Progress from a diffusion language model.

Each choice’s last chunk carries its finish_reason, and the stream ends after that. The last chunk also carries token counts in usage.

For chat speech output, select ChatAudioFormat::Pcm16. Audio then arrives in Response::Chunk as base64 PCM in choices[].delta.audio.data: decode each chunk and append the bytes in the order they arrive, including the final one.

The loop above prints only the first choice and skips everything else; match the agentic variants too if your app uses tools. Full examples: streaming and error-handling.

When the agentic loop executes a tool mid-stream (web search, code execution, shell, MCP (Model Context Protocol) tools), the stream interleaves progress events with content chunks, in stream order:

use mistralrs::core::AgenticToolCallPhase;
Response::AgenticToolCallProgress { round, tool_name, phase } => {
match phase {
AgenticToolCallPhase::Calling(_) => println!("[round {round}: calling {tool_name}]"),
AgenticToolCallPhase::Complete(_) => println!("[round {round}: completed {tool_name}]"),
}
}

The non-streaming send_chat_request doesn’t surface these events; it returns only the final response.

The stream borrows the Model for its lifetime, so it can move across await points within the same scope but cannot be sent into a detached task on its own. Model does not implement Clone. To stream inside a spawned task, share the model via Arc and create the stream inside the task:

use std::sync::Arc;
let model = Arc::new(model);
let handle = tokio::spawn({
let model = Arc::clone(&model);
async move {
let mut stream = model.stream_chat_request(messages).await?;
while let Some(item) = stream.next().await {
// forward chunks to a channel, websocket, etc.
}
anyhow::Ok(())
}
});

The response channel behind the stream is bounded, so a consumer that stops polling applies backpressure to the engine. To cancel early, drop the stream: the channel closes and the engine stops generating for that request.

To show text as it arrives and also keep the whole reply, append each piece to a String:

use mistralrs::Response;
use std::io::Write;
let mut stream = model.stream_chat_request(messages).await?;
let mut reply = String::new();
while let Some(item) = stream.next().await {
match item {
Response::Chunk(chunk) => {
if let Some(text) = chunk.choices.first().and_then(|c| c.delta.content.as_deref()) {
print!("{text}");
std::io::stdout().flush()?;
reply.push_str(text);
}
}
Response::InternalError(e) | Response::ValidationError(e) => {
eprintln!("stream error: {e}");
break;
}
Response::ModelError(msg, _) => {
eprintln!("model error: {msg}");
break;
}
_ => {}
}
}

When the loop ends, reply holds the whole reply, or everything generated before an error. With n > 1, chunks for different choices arrive interleaved: keep one buffer per choice and pick it with choice.index.