Speculative decoding (MTP)
Speculative decoding makes generation faster by guessing several tokens ahead. A small drafter proposes the next few tokens, and the main (target) model checks all of them in a single pass. The target keeps the tokens it agrees with and throws away the rest, so one step can produce several tokens instead of one. The drafter is either a separate checkpoint (--mtp-model) or a prediction head built into the model (--mtp). mistral.rs calls both MTP, for multi-token prediction.
mistralrs run -m google/gemma-4-E4B-it --quant 8 \ --mtp-model google/gemma-4-E4B-it-assistant--mtp-model takes a Hugging Face ID or a local path. The same flags work with mistralrs serve.
from mistralrs import Runner, Which
runner = Runner( which=Which.MultimodalPlain(model_id="google/gemma-4-E4B-it"), in_situ_quant="8", mtp_model="google/gemma-4-E4B-it-assistant",)Python takes a separate drafter only. To use the built-in Qwen head (--mtp), use the CLI or Rust.
let model = mistralrs::ModelBuilder::new("google/gemma-4-E4B-it") .with_mtp_model("google/gemma-4-E4B-it-assistant", None) .build() .await?;For a built-in head, pass .with_mtp_config(mistralrs::MtpConfig::builtin(None)) instead. The second argument in both is the draft depth; None picks it automatically. These methods work on the text, multimodal, and auto-detecting model builders.
Speculative decoding is set when the model loads, and then every request uses it. There’s no per-request switch. It works with batched requests and constrained decoding, but it needs paged attention, so it doesn’t run on CPU or with --paged-attn off, and it can’t be combined with dynamic LoRA.
How much it helps
Section titled “How much it helps”It helps most when only a few requests run at once. As concurrency rises, the GPU is already busy, and the extra drafting and checking work starts to cost more than it saves.
The v0.9.3 benchmark ran Qwen3.8-27B (FP8) on one GH200 with 512-token outputs, using the DFlash 2 drafter at a fixed depth of 7 drafts. Compared with the same model and no drafter, output throughput was:
| Concurrent requests | Throughput vs. no drafter |
|---|---|
| 1 | 2.2x |
| 8 | 1.8x |
| 16 | 1.6x |
| 32 | 1.3x |
| 64 | 0.8x |
| 128 | 0.6x |
The default automatic depth drafts fewer tokens under heavy load than this fixed-depth run did.
How many drafts the target accepts also depends heavily on the text. That same drafter accepts about 7 of 7 drafts on math and code, and far fewer on thinking-mode reasoning traces. To see the acceptance rate on your own traffic, use the speculative decoding queries in observability.
Supported models
Section titled “Supported models”| Target model | Drafter | Flag |
|---|---|---|
| Gemma 4 | Gemma 4 assistant checkpoint, such as google/gemma-4-E4B-it-assistant |
--mtp-model |
| Qwen3.5, Qwen3.8 | Head built into the checkpoint | --mtp |
| Qwen3.5, Qwen3.8 | DFlash or DFlash 2 checkpoint, such as incoai/Qwen3.8-27B-DFlash2 |
--mtp-model |
| Qwen3.8-Flash-Next | Head built into the checkpoint (the only option) | --mtp |
Draft depth
Section titled “Draft depth”--mtp-n-predict N sets how many tokens are drafted per step. In Python, pass mtp_n_predict=N to Runner; in Rust, pass Some(N). A higher depth gains more per step when drafts are accepted and wastes more work when they aren’t, so lower depths suit text where acceptance is low.
If you don’t set it, mistral.rs picks the depth as it runs, based on how many drafts are being accepted, how long each step takes, and how many requests are running:
| Drafter | Automatic depth |
|---|---|
| Built-in Qwen head | 2, 3, 4, or 6 |
| Gemma 4 assistant | 2, 3, 4, or 6, capped by num_assistant_tokens in the assistant’s generation_config.json (6 if it isn’t set) |
| DFlash, DFlash 2 | Up to 7, or the drafter’s block size minus 1 if that’s smaller. On CUDA, it drafts the full depth while up to 8 requests are running and at most 3 beyond that. |
Qwen3.5 and Qwen3.8
Section titled “Qwen3.5 and Qwen3.8”Qwen3.5 and Qwen3.8 checkpoints include a prediction head (the mtp.* weights). Add --mtp to use it; there’s nothing extra to download:
mistralrs serve -m Qwen/Qwen3.8-27B --quant 4 --mtp- The head adds the KV cache of one attention layer.
- Safetensors and UQFF checkpoints work if they include the
mtp.*weights. GGUF files don’t support the built-in head. - With the built-in head, cached prompt prefixes are recomputed, so repeated prompts can take longer to reach the first token. DFlash drafters don’t have this cost.
- Outputs and log probabilities can differ slightly from a run without
--mtp, even with greedy sampling.
Qwen3.8-Flash-Next takes the same flag:
mistralrs serve -m Qwen/Qwen3.8-Flash-Next --quant 4 --mtpFlash-Next supports only its built-in head, not DFlash or separate assistants. Its paged attention is CUDA-only, so speculative decoding is too. See model-family notes for loading it.
DFlash and DFlash 2
Section titled “DFlash and DFlash 2”DFlash drafters are small models that propose a whole block of tokens in one pass. The target accepts their drafts much more often than the built-in head’s. Pass one to --mtp-model; mistral.rs tells DFlash and DFlash 2 apart from the checkpoint:
mistralrs serve -m Qwen/Qwen3.8-27B --quant 4 \ --mtp-model incoai/Qwen3.8-27B-DFlash2The drafter is quantized to the same type as the target, so --quant applies to both.
--mtp-draft-sampling picks how the drafter chooses its tokens:
| Value | What happens |
|---|---|
auto (default) |
DFlash 2 samples its drafts on CUDA devices that support it. DFlash 1, and other devices, draft greedily. |
greedy |
The drafter always proposes its most likely tokens. |
probabilistic |
The drafter always samples its drafts. This needs DFlash 2 on a supported CUDA device; otherwise loading fails. |
Gemma 4
Section titled “Gemma 4”Use the assistant checkpoint made for your target, such as google/gemma-4-E4B-it-assistant for google/gemma-4-E4B-it. A downloaded copy works too:
mistralrs run -m google/gemma-4-E4B-it --quant 8 \ --mtp-model ./gemma-4-E4B-it-assistantIf the assistant doesn’t match the target, for example a different vocabulary or hidden size, loading fails before any generation starts.