Observability
mistralrs serve and mistralrs from-config log every request and serve Prometheus metrics at /metrics out of the box. This page shows what you get and how to change it.
Access logs
Section titled “Access logs”Each request writes two lines to the server’s normal log output: request started when it arrives and request completed when it ends. Both carry the request ID, method, route, and model. The completed line adds the status code and duration in milliseconds. For streamed responses it also says how the stream ended (completed, error, or client_disconnected) and, when available, the prompt and completion token counts and prefill and decode speed in tokens per second.
Health checks, /metrics scrapes, docs and web UI requests, and CORS preflight requests aren’t logged unless you ask for them.
| Option | Effect |
|---|---|
--access-log-format json |
Write each line as a JSON object, for log pipelines. The default is text. |
--access-log-health |
Also log health checks, metrics scrapes, and docs and UI requests. |
--disable-access-log |
Stop writing access logs. With -v, the completed line still appears as a debug message. |
-v, -vv, RUST_LOG |
Show debug (-v) or trace (-vv) logs, or set your own filter with RUST_LOG. |
Request IDs
Section titled “Request IDs”Send an x-request-id header and the server uses that ID in its logs and returns it in the response. If you don’t send one, the server generates one and returns it. Log the ID on the client side too, so you can find the server lines for any failed request.
--disable-request-id-header stops the server from adding the header to responses. The ID still appears in the logs.
Prometheus metrics
Section titled “Prometheus metrics”Point Prometheus at the server’s address:
scrape_configs: - job_name: mistralrs static_configs: - targets: ["localhost:1234"]If Prometheus runs on another machine, use an address it can reach instead of localhost. To turn metrics off, pass --disable-metrics; /metrics then returns 503.
These are the metrics to start with. Every name and label is in the metrics reference.
| To find out | Watch | What to look for |
|---|---|---|
| Request rate and errors | http_requests_total, by status |
A rising share of 5xx responses. |
| End-to-end latency | http_request_duration_seconds |
High p95 or p99 latency, by path and model. |
| How long users wait for the first token | mistralrs_time_to_first_token_seconds |
A rising p99. It includes time spent queued, so check the waiting queue below. |
| How smoothly tokens stream | mistralrs_inter_token_latency_seconds |
A high p95 or p99 means streams stall, often while long new prompts are read. See throughput tuning. |
| Whether requests are queueing | mistralrs_sequences_waiting, and mistralrs_sequences_running divided by mistralrs_sequences_capacity |
Running stays at capacity while the waiting count grows. Raise --max-seqs or add servers. |
| Whether the KV cache is too small | mistralrs_kv_cache_blocks_used divided by mistralrs_kv_cache_blocks_total, and mistralrs_paged_preemptions_total |
Usage near 1.0 and a climbing preemption count: requests are being paused and recomputed. See memory budget. |
| Whether prefix caching helps | mistralrs_prefix_cache_hits_total divided by mistralrs_prefix_cache_lookups_total |
A low hit rate even though requests share a system prompt. |
| Tokens per second | rate() of mistralrs_prefill_tokens_processed_total and mistralrs_decode_tokens_processed_total |
Prompt and output throughput, separately. |
| Whether speculative decoding helps | mistralrs_speculative_draft_tokens_accepted_total and mistralrs_speculative_drafts_total |
Accepted tokens per step near 1.0 means drafting isn’t paying off. See speculative decoding. |
The http_* metrics and the two latency histograms are labeled by path and model. The path label is the route pattern, such as /v1/responses/{response_id}, not the raw URL. The other mistralrs_* metrics have no labels: if one server runs several models, they share those numbers. Run one server per model when you need them per model.
Useful PromQL
Section titled “Useful PromQL”Use rate() for counters and histogram_quantile() for latency:
# Fraction of proposed draft tokens that were acceptedrate(mistralrs_speculative_draft_tokens_accepted_total[5m]) / rate(mistralrs_speculative_draft_tokens_proposed_total[5m])
# Mean tokens accepted per verification step (1.0 = speculation is not helping)1 + rate(mistralrs_speculative_draft_tokens_accepted_total[5m]) / rate(mistralrs_speculative_drafts_total[5m])
# P99 time to first token, all modelshistogram_quantile(0.99, sum(rate(mistralrs_time_to_first_token_seconds_bucket[5m])) by (le))
# P95 inter-token latency for one route/modelhistogram_quantile(0.95, sum(rate(mistralrs_inter_token_latency_seconds_bucket{path="/v1/chat/completions"}[5m])) by (le))
# KV cache pressure (sustained 1.0 with a growing waiting queue means preemptions are likely)mistralrs_kv_cache_blocks_used / mistralrs_kv_cache_blocks_total
# Sequence capacity utilization (sustained 1.0 with a growing waiting queue means requests are queueing)mistralrs_sequences_running / mistralrs_sequences_capacityThe same options go under [server] in a TOML config:
[server]port = 1234access_log_format = "json"access_log_health = falsedisable_access_log = falsedisable_request_id_header = falsedisable_metrics = falseFor health and readiness probes, see the production checklist.