Skip to content

Observability

mistralrs serve and mistralrs from-config log every request and serve Prometheus metrics at /metrics out of the box. This page shows what you get and how to change it.

Each request writes two lines to the server’s normal log output: request started when it arrives and request completed when it ends. Both carry the request ID, method, route, and model. The completed line adds the status code and duration in milliseconds. For streamed responses it also says how the stream ended (completed, error, or client_disconnected) and, when available, the prompt and completion token counts and prefill and decode speed in tokens per second.

Health checks, /metrics scrapes, docs and web UI requests, and CORS preflight requests aren’t logged unless you ask for them.

Option Effect
--access-log-format json Write each line as a JSON object, for log pipelines. The default is text.
--access-log-health Also log health checks, metrics scrapes, and docs and UI requests.
--disable-access-log Stop writing access logs. With -v, the completed line still appears as a debug message.
-v, -vv, RUST_LOG Show debug (-v) or trace (-vv) logs, or set your own filter with RUST_LOG.

Send an x-request-id header and the server uses that ID in its logs and returns it in the response. If you don’t send one, the server generates one and returns it. Log the ID on the client side too, so you can find the server lines for any failed request.

--disable-request-id-header stops the server from adding the header to responses. The ID still appears in the logs.

Point Prometheus at the server’s address:

scrape_configs:
- job_name: mistralrs
static_configs:
- targets: ["localhost:1234"]

If Prometheus runs on another machine, use an address it can reach instead of localhost. To turn metrics off, pass --disable-metrics; /metrics then returns 503.

These are the metrics to start with. Every name and label is in the metrics reference.

To find out Watch What to look for
Request rate and errors http_requests_total, by status A rising share of 5xx responses.
End-to-end latency http_request_duration_seconds High p95 or p99 latency, by path and model.
How long users wait for the first token mistralrs_time_to_first_token_seconds A rising p99. It includes time spent queued, so check the waiting queue below.
How smoothly tokens stream mistralrs_inter_token_latency_seconds A high p95 or p99 means streams stall, often while long new prompts are read. See throughput tuning.
Whether requests are queueing mistralrs_sequences_waiting, and mistralrs_sequences_running divided by mistralrs_sequences_capacity Running stays at capacity while the waiting count grows. Raise --max-seqs or add servers.
Whether the KV cache is too small mistralrs_kv_cache_blocks_used divided by mistralrs_kv_cache_blocks_total, and mistralrs_paged_preemptions_total Usage near 1.0 and a climbing preemption count: requests are being paused and recomputed. See memory budget.
Whether prefix caching helps mistralrs_prefix_cache_hits_total divided by mistralrs_prefix_cache_lookups_total A low hit rate even though requests share a system prompt.
Tokens per second rate() of mistralrs_prefill_tokens_processed_total and mistralrs_decode_tokens_processed_total Prompt and output throughput, separately.
Whether speculative decoding helps mistralrs_speculative_draft_tokens_accepted_total and mistralrs_speculative_drafts_total Accepted tokens per step near 1.0 means drafting isn’t paying off. See speculative decoding.

The http_* metrics and the two latency histograms are labeled by path and model. The path label is the route pattern, such as /v1/responses/{response_id}, not the raw URL. The other mistralrs_* metrics have no labels: if one server runs several models, they share those numbers. Run one server per model when you need them per model.

Use rate() for counters and histogram_quantile() for latency:

# Fraction of proposed draft tokens that were accepted
rate(mistralrs_speculative_draft_tokens_accepted_total[5m])
/ rate(mistralrs_speculative_draft_tokens_proposed_total[5m])
# Mean tokens accepted per verification step (1.0 = speculation is not helping)
1 + rate(mistralrs_speculative_draft_tokens_accepted_total[5m])
/ rate(mistralrs_speculative_drafts_total[5m])
# P99 time to first token, all models
histogram_quantile(0.99,
sum(rate(mistralrs_time_to_first_token_seconds_bucket[5m])) by (le))
# P95 inter-token latency for one route/model
histogram_quantile(0.95,
sum(rate(mistralrs_inter_token_latency_seconds_bucket{path="/v1/chat/completions"}[5m])) by (le))
# KV cache pressure (sustained 1.0 with a growing waiting queue means preemptions are likely)
mistralrs_kv_cache_blocks_used / mistralrs_kv_cache_blocks_total
# Sequence capacity utilization (sustained 1.0 with a growing waiting queue means requests are queueing)
mistralrs_sequences_running / mistralrs_sequences_capacity

The same options go under [server] in a TOML config:

[server]
port = 1234
access_log_format = "json"
access_log_health = false
disable_access_log = false
disable_request_id_header = false
disable_metrics = false

For health and readiness probes, see the production checklist.