Skip to content

Use embedding models

An embedding model turns text into a vector of numbers, so that texts with similar meaning get nearby vectors. You use them for semantic search, clustering, deduplication, and retrieval for RAG. mistral.rs serves them through the OpenAI-compatible /v1/embeddings endpoint, so tools that already use that endpoint work unchanged.

Terminal window
mistralrs serve -m google/embeddinggemma-300m
Model Size Vector length
google/embeddinggemma-300m 300M parameters 768
Qwen/Qwen3-Embedding-0.6B 600M parameters 1,024

EmbeddingGemma is gated: accept its license on Hugging Face, then run hf auth login or set HF_TOKEN before the first download.

Terminal window
curl http://localhost:1234/v1/embeddings \
-H 'Content-Type: application/json' \
-d '{
"model": "default",
"input": "task: search result | query: What is graphene?"
}'

Read the vector from data[0].embedding.

To embed several texts in one call, pass a list as input (or use add_prompts in Rust). You get one entry in data per input, in the same order.

Field Values
encoding_format float (default) returns a list of numbers. base64 returns each vector as base64-encoded little-endian f32 bytes, which is smaller to send.
truncate_sequence mistral.rs extension. true cuts inputs longer than the model’s context to fit. The default, false, returns an error instead.
dimensions Not supported. Sending it returns an error rather than silently giving you a different length.

mistral.rs runs the post-processing steps listed in the checkpoint’s modules.json. Both EmbeddingGemma and Qwen3-Embedding end with a normalize step, so their vectors already have length 1, and a plain dot product gives you cosine similarity. For another model, check its modules.json for a Normalize entry. If there isn’t one and your vector store expects unit vectors, divide each vector by its length before storing and querying.

EmbeddingGemma expects a task prefix on every input. mistral.rs doesn’t add it, so include it in the text you send:

Use Prefix
Search query task: search result | query: <query>
Search document title: <title or none> | text: <document>
Question answering task: question answering | query: <question>
Fact verification task: fact checking | query: <claim>
Classification task: classification | query: <text>
Clustering task: clustering | query: <text>
Similarity task: sentence similarity | query: <text>
Code search task: code retrieval | query: <query>

For search, embed queries with the query prefix and documents with the document prefix. Other models use their own formats; check the model card.

Because both models return unit-length vectors, ranking documents against a query is one matrix product:

import numpy as np
query, *docs = runner.send_embedding_request(EmbeddingRequest(input=[
"task: search result | query: What is graphene?",
"title: none | text: Graphene is a single layer of carbon atoms.",
"title: none | text: Apples grow on trees in temperate climates.",
]))
scores = np.array(docs) @ np.array(query)
print(scores) # one score per document; higher means more similar

For a real corpus, embed your documents once, store the vectors in a vector database, and embed each incoming query with the same model and prefix.