Use embedding models
An embedding model turns text into a vector of numbers, so that texts with similar meaning get nearby vectors. You use them for semantic search, clustering, deduplication, and retrieval for RAG. mistral.rs serves them through the OpenAI-compatible /v1/embeddings endpoint, so tools that already use that endpoint work unchanged.
Loading an embedding model
Section titled “Loading an embedding model”mistralrs serve -m google/embeddinggemma-300m| Model | Size | Vector length |
|---|---|---|
google/embeddinggemma-300m |
300M parameters | 768 |
Qwen/Qwen3-Embedding-0.6B |
600M parameters | 1,024 |
EmbeddingGemma is gated: accept its license on Hugging Face, then run hf auth login or set HF_TOKEN before the first download.
Requesting an embedding
Section titled “Requesting an embedding”curl http://localhost:1234/v1/embeddings \ -H 'Content-Type: application/json' \ -d '{ "model": "default", "input": "task: search result | query: What is graphene?" }'Read the vector from data[0].embedding.
from mistralrs import EmbeddingRequest, Runner, Which
runner = Runner(which=Which.Embedding(model_id="google/embeddinggemma-300m"))vectors = runner.send_embedding_request(EmbeddingRequest( input=["task: search result | query: What is graphene?"], truncate_sequence=True,))print(len(vectors[0])) # 768use mistralrs::{EmbeddingModelBuilder, EmbeddingRequest};
let model = EmbeddingModelBuilder::new("google/embeddinggemma-300m").build().await?;let vectors = model.generate_embeddings( EmbeddingRequest::builder().add_prompt("task: search result | query: What is graphene?"),).await?;Batching
Section titled “Batching”To embed several texts in one call, pass a list as input (or use add_prompts in Rust). You get one entry in data per input, in the same order.
Request fields
Section titled “Request fields”| Field | Values |
|---|---|
encoding_format |
float (default) returns a list of numbers. base64 returns each vector as base64-encoded little-endian f32 bytes, which is smaller to send. |
truncate_sequence |
mistral.rs extension. true cuts inputs longer than the model’s context to fit. The default, false, returns an error instead. |
dimensions |
Not supported. Sending it returns an error rather than silently giving you a different length. |
Normalization
Section titled “Normalization”mistral.rs runs the post-processing steps listed in the checkpoint’s modules.json. Both EmbeddingGemma and Qwen3-Embedding end with a normalize step, so their vectors already have length 1, and a plain dot product gives you cosine similarity. For another model, check its modules.json for a Normalize entry. If there isn’t one and your vector store expects unit vectors, divide each vector by its length before storing and querying.
EmbeddingGemma prompts
Section titled “EmbeddingGemma prompts”EmbeddingGemma expects a task prefix on every input. mistral.rs doesn’t add it, so include it in the text you send:
| Use | Prefix |
|---|---|
| Search query | task: search result | query: <query> |
| Search document | title: <title or none> | text: <document> |
| Question answering | task: question answering | query: <question> |
| Fact verification | task: fact checking | query: <claim> |
| Classification | task: classification | query: <text> |
| Clustering | task: clustering | query: <text> |
| Similarity | task: sentence similarity | query: <text> |
| Code search | task: code retrieval | query: <query> |
For search, embed queries with the query prefix and documents with the document prefix. Other models use their own formats; check the model card.
Comparing vectors
Section titled “Comparing vectors”Because both models return unit-length vectors, ranking documents against a query is one matrix product:
import numpy as np
query, *docs = runner.send_embedding_request(EmbeddingRequest(input=[ "task: search result | query: What is graphene?", "title: none | text: Graphene is a single layer of carbon atoms.", "title: none | text: Apples grow on trees in temperate climates.",]))scores = np.array(docs) @ np.array(query)print(scores) # one score per document; higher means more similarFor a real corpus, embed your documents once, store the vectors in a vector database, and embed each incoming query with the same model and prefix.