Skip to content

Sampling parameters

Sampling parameters decide how the model picks each next token: always the most likely one, or a random draw from the likely candidates. You set them per request, and the field names are the same over HTTP, in Python, and in Rust.

Any field you leave out comes from the model’s generation_config.json, which can set temperature, top-k, top-p, min-p, and repetition penalty. Anything the model doesn’t set defaults to temperature 1.0 with no filtering and no penalties.

If a model’s generation_config.json sets do_sample: false, its default top-k is 1, so output stays greedy even when you raise the temperature. Send your own top_k (0 turns it off) to sample from such a model.

Goal Settings
Always pick the most likely token (greedy), for tests or extraction temperature: 0
General chat Leave everything unset to get the model’s own defaults. If the model has none, try what interactive mode falls back to: temperature 0.8, top-k 40, top-p 0.95, and min-p 0.05.
More varied or creative output Raise temperature, and keep top-p or min-p on so very unlikely tokens stay out.
Output that loops or repeats itself Add repetition_penalty a little above 1.0, such as 1.1, or turn on DRY.

Temperature controls how random the choice is:

  • 0, or anything below 1e-7: always the most likely token.
  • Below 1: sharper. The model sticks closer to its top choices.
  • 1.0: the model’s own probabilities, unchanged.
  • Above 1: flatter. Unlikely tokens get picked more often, and output gets less coherent the higher you go.

Top-k keeps only the k most likely tokens and samples among them. top_k: 1 is greedy, larger values allow more variety, and 0 or unset turns it off.

Top-p (nucleus sampling) keeps the smallest set of top tokens whose probabilities add up to p. With top_p: 0.9, the model samples from the tokens that cover 90% of the probability. Lower values are stricter. 1.0 turns it off, and so does 0 or anything outside that range.

Min-p drops tokens that are less likely than min_p times the top token’s probability. With min_p: 0.05, a token survives only if it’s at least 5% as likely as the best one. This adapts to the model’s confidence: when one token dominates, few others survive, and when many are close, more do. 0 turns it off, and so does any value of 1 or more.

For each token, mistral.rs applies settings in this order:

  1. Penalties: DRY, then presence, frequency, and repetition penalties, then logit_bias.
  2. Custom logits processors, which you can add only from Rust with RequestBuilder::add_logits_processor.
  3. Temperature. At temperature 0 the most likely token is picked here and the filters below are skipped.
  4. Top-k, then top-p, then min-p. Top-p is measured on what top-k left.

Three separate parameters push the model away from tokens it has already used:

Parameter What it looks at Effect Off
presence_penalty Generated tokens Subtracts a flat amount from every token that has appeared at least once. 0
frequency_penalty Generated tokens Subtracts an amount proportional to how many times each token has appeared. 0
repetition_penalty Prompt and generated tokens Divides positive scores of seen tokens by the value and multiplies negative ones, Hugging Face style. 1.0

Positive presence and frequency penalties discourage repeats, and negative ones encourage them. A repetition_penalty above 1.0 discourages repeats, and below 1.0 encourages them. presence_penalty and frequency_penalty follow OpenAI’s definitions; repetition_penalty is a different setting, not another name for either.

logit_bias maps token IDs to an amount added to their score. A large negative value effectively bans a token, and a positive one makes it more likely.

DRY targets one kind of repetition: the model copying a run of tokens that already appears earlier in the context, including the prompt. When the latest tokens match an earlier span, DRY penalizes the token that would continue the copy, and the penalty grows with the length of the match. It’s off until you set dry_multiplier.

Field Default What it does
dry_multiplier 0 (off) Overall strength. Any value above 0 turns DRY on.
dry_base 1.75 How fast the penalty grows with each extra matching token.
dry_allowed_length 2 Matches shorter than this aren’t penalized.
dry_sequence_breakers ["\n", ":", "\"", "*"] Strings that end a match, so repeated formatting doesn’t build up a penalty.

The penalty is dry_multiplier * dry_base ^ (match length - dry_allowed_length). With dry_multiplier: 0.8 and the other defaults, continuing a 2-token match costs 0.8, a 4-token match costs 2.45, and a 6-token match costs about 7.5.

Generation stops at whichever comes first:

  • the model’s end-of-sequence (EOS) token,
  • one of your stop strings (stop over HTTP, stop_seqs in Python, set_sampler_stop_toks in Rust, which also takes token IDs),
  • the max_tokens limit.

Set ignore_eos: true to keep going after the model emits EOS. Stop strings and max_tokens still apply, so set one of them to end the response. This helps with base (non-instruct) models that stop too early, and when measuring decode speed.

Interactive mode (mistralrs run) has slash commands that last for the rest of the session:

/temperature 0.7 sampling temperature, 0.0 to 2.0; 0 means greedy
/topk 40 top-k, a positive integer
/topp 0.9 top-p, above 0.0 and up to 1.0

Until you change them, interactive mode uses the model’s generation_config.json, or temperature 0.8, top-k 40, top-p 0.95, and min-p 0.05 if the model has none.

To make sampling repeatable, set a seed. Send seed in a Chat Completions, Completions, or Responses request to seed that request alone: the same request with the same seed and settings makes the same random choices again. With n > 1, choice i uses seed + i, so the choices differ from each other but each one is repeatable.

To seed the whole engine instead, pass --seed on the CLI or seed= to the Python Runner.