Mistral Agents API: Production Deployment Guide
Running services on Mistral's Agents API in production: configuration, concurrency, retries, observability, storage, rate limits, security and CI/CD.
Verified against mistralai 3.8.0 (checked 2026-11-19).
The agent loop, built-in tools and conversation storage run on Mistral’s side, so your service is mostly an API client: it needs sensible concurrency, retries, observability, its own record of which conversations belong to whom, and careful handling of secrets.
1. Configuration
Section titled “1. Configuration”# Credentials (one of)MISTRAL_API_KEY=...MISTRAL_SA_TOKEN_PATH=/var/run/secrets/mistral/token # token file the SDK re-reads on every request; wins over MISTRAL_API_KEY
# Your serviceAGENT_ID_SUPPORT=ag_... # created by a deploy script, not at request timeAGENT_VERSION_SUPPORT=prod # version aliasMISTRAL_SERVER=eu # "eu", "us" or "global"MISTRAL_TIMEOUT_MS=60000MISTRAL_MAX_CONCURRENCY=16Load and validate it once at start-up:
import osfrom pydantic import BaseModel
class Settings(BaseModel): agent_id_support: str agent_version_support: str = "prod" mistral_server: str = "eu" mistral_timeout_ms: int = 60_000 mistral_max_concurrency: int = 16
settings = Settings(**{k.lower(): v for k, v in os.environ.items() if k.lower() in Settings.model_fields})Create agents in a deploy step, not per request. A script (or CI job) creates or updates agents, sets a version alias such as prod, and writes the agent IDs into configuration. Requests then start conversations against agent_version="prod", and a rollback is update_version() or moving the alias.
import osfrom mistralai.client import Mistral
client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])
agent = client.beta.agents.update( agent_id=os.environ["AGENT_ID_SUPPORT"], instructions=open("prompts/support.md").read(), version_message=os.environ.get("GIT_SHA", "manual"),)client.beta.agents.create_version_alias(agent_id=agent.id, alias="prod", version=agent.version)print("support agent now at version", agent.version)2. Client, concurrency and retries
Section titled “2. Client, concurrency and retries”Create one client per process and reuse it; it pools HTTP connections. Bound concurrency explicitly.
import asynciofrom mistralai.client import Mistralfrom mistralai.client.utils import BackoffStrategy, RetryConfig
client = Mistral( server=settings.mistral_server, timeout_ms=settings.mistral_timeout_ms, retry_config=RetryConfig( "backoff", BackoffStrategy(initial_interval=500, max_interval=20_000, exponent=2.0, max_elapsed_time=90_000), retry_connection_errors=True, ),)limit = asyncio.Semaphore(settings.mistral_max_concurrency)
async def start_conversation(inputs: str, metadata: dict[str, str]): async with limit: return await client.beta.conversations.start_async( agent_id=settings.agent_id_support, agent_version=settings.agent_version_support, inputs=inputs, metadata=metadata, )The SDK’s retry policy retries connection errors and the status codes it considers transient. Retrying start() can create a second conversation if the first request actually succeeded; for idempotency, record the conversation ID before acting on the reply, and retry append() only after checking the history.
For a custom transport (proxies, custom TLS), pass httpx2 clients; since v3 the SDK no longer uses httpx:
import httpx2from mistralai.client import Mistral
client = Mistral( client=httpx2.Client(timeout=60, limits=httpx2.Limits(max_connections=50)), async_client=httpx2.AsyncClient(timeout=60, limits=httpx2.Limits(max_connections=50)),)Circuit breaker
Section titled “Circuit breaker”When Mistral (or your network) is down, fail fast instead of piling up requests:
import timefrom mistralai.client import errors
class CircuitBreaker: def __init__(self, threshold: int = 5, reset_after: float = 30.0): self.threshold, self.reset_after = threshold, reset_after self.failures, self.opened_at = 0, 0.0
def allow(self) -> bool: return self.failures < self.threshold or time.monotonic() - self.opened_at > self.reset_after
def record(self, ok: bool) -> None: if ok: self.failures = 0 else: self.failures += 1 if self.failures == self.threshold: self.opened_at = time.monotonic()
breaker = CircuitBreaker()
async def guarded_start(inputs: str): if not breaker.allow(): raise RuntimeError("Mistral temporarily unavailable") try: response = await start_conversation(inputs, metadata={}) except errors.MistralError as e: breaker.record(ok=e.status_code < 500 and e.status_code != 429) raise breaker.record(ok=True) return response3. Observability
Section titled “3. Observability”Log one structured record per call. response.usage has token counts, including connector_tokens for built-in tools; outputs shows which tools ran and which agents handled the request.
import jsonimport loggingimport time
log = logging.getLogger("agents")
async def observed_start(inputs: str, user_id: str): started = time.monotonic() response = await start_conversation(inputs, metadata={"user_id": user_id}) log.info(json.dumps({ "event": "conversation.start", "conversation_id": response.conversation_id, "user_id": user_id, "latency_ms": int((time.monotonic() - started) * 1000), "prompt_tokens": response.usage.prompt_tokens, "completion_tokens": response.usage.completion_tokens, "connector_tokens": response.usage.connector_tokens, "tools": [e.name for e in response.outputs if e.type == "tool.execution"], "handoffs": [e.next_agent_name for e in response.outputs if e.type == "agent.handoff"], })) return responseOpenTelemetry
Section titled “OpenTelemetry”The SDK can emit OpenTelemetry spans for its API calls (operation, model, token usage and, unless redacted, the messages and tool calls), but only once you opt in, with configure_telemetry() or the MISTRAL_SDK_TELEMETRY environment variable (dedicated, global or false). A tracer provider on its own is not enough. Install the extra first:
pip install "mistralai[telemetry]==3.8.0"To send the SDK’s spans through your own pipeline, next to your service’s spans, use global mode:
from opentelemetry import tracefrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporterfrom opentelemetry.sdk.trace import TracerProviderfrom opentelemetry.sdk.trace.export import BatchSpanProcessorfrom mistralai.extra.observability import RedactingSpanExporter, configure_telemetry
provider = TracerProvider()provider.add_span_processor(BatchSpanProcessor( RedactingSpanExporter(OTLPSpanExporter(endpoint="http://otel-collector:4318/v1/traces"))))trace.set_tracer_provider(provider)
# SDK spans now use the global provider. redaction=False only silences the SDK's notice that# its own redaction does not apply in this mode; the exporter above does the redacting.configure_telemetry(client, provider="global", redaction=False)
tracer = trace.get_tracer("support-service")with tracer.start_as_current_span("answer_support_question") as span: response = client.beta.conversations.start(agent_id=settings.agent_id_support, inputs="Where is ORD-1234?") span.set_attribute("conversation.id", response.conversation_id)The SDK’s own spans then appear as children of answer_support_question. In global mode (or when you pass your own TracerProvider) the SDK does not redact anything, which is why the exporter above is wrapped in RedactingSpanExporter; its default policy masks secrets and common personal data (emails, card-like numbers, IP addresses) in span attributes.
configure_telemetry(client) with no provider uses dedicated mode instead: the SDK creates its own exporter, redacts by default (redaction= takes a policy, a callback or False) and sends the spans to Mistral’s telemetry endpoint, authenticated with the client’s API key.
Mistral’s traces and evaluations
Section titled “Mistral’s traces and evaluations”Spans sent in dedicated mode are stored by Mistral, and client.beta.observability (beta) works with them: traces and spans search and aggregate them, judges defines LLM judges that score conversations, datasets builds evaluation sets (including from spans or an uploaded file), and evaluations runs evaluation pipelines over them. This is one way to run the evaluation set mentioned under CI/CD before moving a version alias; a plain pytest suite over a fixed set of prompts is another.
What to alert on
Section titled “What to alert on”Alert on: error rate by status code, p95 latency, tokens per conversation (a rising trend usually means a prompt or tool regression), and 429s.
4. Testing
Section titled “4. Testing”Unit-test your code against scripted API responses rather than the live API. The client accepts any httpx2 client, so an httpx2.MockTransport can return canned conversation entries, for example a function.call followed by the final message.output:
import jsonimport httpx2from mistralai.client import Mistralfrom mistralai.client.models import FunctionCallEntry, FunctionResultEntry
# --- code under test (normally imported from your service) ---def get_order_status(order_id: str) -> dict: return {"order_id": order_id, "status": "shipped"}
def answer_order_question(client: Mistral, agent_id: str, question: str, max_turns: int = 5) -> str: response = client.beta.conversations.start(agent_id=agent_id, inputs=question) for _ in range(max_turns): call = response.outputs[-1] if not isinstance(call, FunctionCallEntry): return response.outputs[-1].content args = json.loads(call.arguments) if isinstance(call.arguments, str) else call.arguments response = client.beta.conversations.append( conversation_id=response.conversation_id, inputs=[FunctionResultEntry(tool_call_id=call.tool_call_id, result=json.dumps(get_order_status(**args)))], ) raise RuntimeError("too many tool calls")
# --- test helpers ---NOW = "2026-10-21T00:00:00Z"
def entry(kind: str, **fields) -> dict: return {"object": "entry", "type": kind, "id": f"{kind}-1", "created_at": NOW, **fields}
def scripted_client(turns: list[list[dict]]) -> tuple[Mistral, list[dict]]: """A client whose conversation calls return `turns` in order; also returns the request bodies sent.""" sent: list[dict] = []
def handler(request: httpx2.Request) -> httpx2.Response: sent.append(json.loads(request.content or b"{}")) return httpx2.Response(200, json={ "object": "conversation.response", "conversation_id": "conv_test", "outputs": turns.pop(0), "usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15}, })
return Mistral(api_key="test", client=httpx2.Client(transport=httpx2.MockTransport(handler))), sent
def test_order_question_runs_the_tool() -> None: client, sent = scripted_client([ [entry("function.call", tool_call_id="tc1", name="get_order_status", arguments='{"order_id": "ORD-1234"}')], [entry("message.output", role="assistant", content="ORD-1234 has shipped.")], ]) assert answer_order_question(client, "ag_test", "Where is ORD-1234?") == "ORD-1234 has shipped." assert sent[1]["inputs"][0]["tool_call_id"] == "tc1" assert json.loads(sent[1]["inputs"][0]["result"])["status"] == "shipped"For async code pass async_client=httpx2.AsyncClient(transport=httpx2.MockTransport(handler)) as well. Script handoffs (agent.handoff entries), pending approvals (function.call with "confirmation_status": "pending") and errors (httpx2.Response(429, ...)) the same way. In TypeScript, pass a fetcher to HTTPClient; see TypeScript production.
Keep a small set of live tests as well, run against a staging agent version before you move the prod alias: mocks prove your loop handles the entries, not that the agent’s instructions produce them.
5. Cost
Section titled “5. Cost”- Model tokens.
usage.prompt_tokensandusage.completion_tokensare billed at the agent’s model rate. Routers, classifiers and extractors rarely need more thanmistral-small-latest. - Tool tokens and calls.
usage.connector_tokenscounts the tokens that tool results (search results, retrieved passages, code output) added to the context; they are included intotal_tokens.usage.connectorscounts calls per tool (for example{"web_search": 1}). Built-in tools may carry their own charges on top of tokens; Mistral’s tool documentation states none, so check its pricing page for web search, code execution and image generation before estimating. - History grows. Stored conversations save you resending the transcript, but the model still reads the whole history on every
append(), so prompt tokens rise with each turn. Start a new conversation, orrestart()from an earlier entry, when the topic changes. - Cap the work. Set
completion_args={"max_tokens": ...}on agents, bound function-call loops, and give each agent only the tools it needs (every tool definition is sent on every turn). - Batch offline work. The Batch API (
client.batch.jobs) is priced below synchronous requests and suits bulk classification or extraction with no tools or conversation state. - Track it. Log usage per conversation (section 3) and keep running totals per user (section 6), so a prompt or tool regression shows up as a cost trend before it shows up on the invoice.
6. Storing conversation references
Section titled “6. Storing conversation references”Mistral stores the conversation; you store who it belongs to.
CREATE TABLE agent_conversations ( conversation_id TEXT PRIMARY KEY, user_id TEXT NOT NULL, agent_id TEXT NOT NULL, agent_version TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), last_used_at TIMESTAMPTZ NOT NULL DEFAULT now(), prompt_tokens INTEGER NOT NULL DEFAULT 0, completion_tokens INTEGER NOT NULL DEFAULT 0, status TEXT NOT NULL DEFAULT 'active' -- active | closed | deleted);CREATE INDEX ON agent_conversations (user_id, last_used_at DESC);Check ownership before every append(), so one user cannot continue another’s conversation by guessing an ID. Use conversation metadata for search and filtering (conversations.list(metadata=...)), but keep authorisation in your own database. When a user deletes their data, call client.beta.conversations.delete(conversation_id=...) as well as removing your rows.
7. Rate limiting your users
Section titled “7. Rate limiting your users”Mistral enforces workspace limits; you should enforce per-user limits so one user cannot exhaust them:
import time
class TokenBucket: def __init__(self, rate_per_sec: float, capacity: int): self.rate, self.capacity = rate_per_sec, capacity self.tokens, self.updated = float(capacity), time.monotonic()
def take(self, n: int = 1) -> bool: now = time.monotonic() self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate) self.updated = now if self.tokens >= n: self.tokens -= n return True return False
buckets: dict[str, TokenBucket] = {}
def allow(user_id: str) -> bool: return buckets.setdefault(user_id, TokenBucket(rate_per_sec=0.2, capacity=5)).take()In a multi-instance deployment, keep the buckets in Redis.
8. Performance
Section titled “8. Performance”- Stream with
start_stream_async()for user-facing requests; time to first token matters more than total time. - Choose models per agent. Routers and classifiers rarely need more than
mistral-small-latest. - Keep instructions stable. Put per-request data in
inputs, notinstructions, so an agent’s definition changes only through a new version that you can test, alias and roll back. - Trim tools. Every tool definition costs prompt tokens on every turn.
- Use
store=Falsefor one-shot requests you will never append to.
9. Security
Section titled “9. Security”- Secrets. Keep
MISTRAL_API_KEYin a secret manager and out of images and source control. The SDK can also read a service-account token from the file named byMISTRAL_SA_TOKEN_PATH, re-reading it on every request so rotation needs no restart; Mistral documents that mechanism for its managed workflow deployments, which mount the token for you. A key passed toMistral(api_key=...)outranks the token file, so construct the client without one to use it. If the variable is set but the file is missing or empty, every request fails withServiceAccountTokenError; unset it rather than leaving it dangling. - Data residency. Use
server="eu"orserver="us"to pin the region. - Guardrails. Attach
GuardrailConfigwithModerationLlmv2Configto public-facing agents. - Prompt injection. Web search results, documents and connector outputs are untrusted. Use
requires_confirmationon tools that write or send, and never put credentials in instructions. - Least privilege for connectors. Scope credentials (
consumer_scope="user"where possible) and expose only needed tools withinclude. - Access control. Map your users to conversations in your database (section 6) and check it on every request.
10. Deployment and CI/CD
Section titled “10. Deployment and CI/CD”This is the one deployment set-up for the Python pages in this guide; the FastAPI streaming server and a chat-completions service deploy the same way.
FROM python:3.12-slimRUN useradd --create-home appWORKDIR /appCOPY requirements.txt .RUN pip install --no-cache-dir -r requirements.txt # pins mistralai==3.8.0COPY --chown=app:app . .USER appEXPOSE 8000CMD ["uvicorn", "service:app", "--host", "0.0.0.0", "--port", "8000"]On Kubernetes, the straightforward set-up is the API key from a Secret (secretKeyRef, shown after the manifest). If you hold a Mistral service-account token, you can instead mount it from a Secret as a file and point MISTRAL_SA_TOKEN_PATH at it, as here:
apiVersion: apps/v1kind: Deploymentmetadata: name: mistral-agentspec: replicas: 2 selector: matchLabels: { app: mistral-agent } template: metadata: labels: { app: mistral-agent } spec: containers: - name: app image: ghcr.io/yourorg/mistral-agent:1.0.0 env: - name: MISTRAL_SA_TOKEN_PATH value: /var/run/secrets/mistral/token - name: AGENT_ID_SUPPORT valueFrom: { configMapKeyRef: { name: mistral-config, key: agentIdSupport } } volumeMounts: - { name: mistral-token, mountPath: /var/run/secrets/mistral, readOnly: true } ports: [{ containerPort: 8000 }] readinessProbe: httpGet: { path: /healthz, port: 8000 } volumes: - name: mistral-token secret: secretName: mistral-sa-token # holds the Mistral token under the key "token" items: [{ key: token, path: token }]The token must be a Mistral service-account token, not the pod’s own Kubernetes service-account token (Mistral has no documented way to accept that). client.beta.service_accounts manages service accounts and their roles; it has no call that issues a token, and no public documentation for issuing one outside managed deployments was found, so confirm how your organisation obtains one. Kubernetes updates a mounted Secret in place (unless it is mounted with subPath), and the SDK re-reads the file on each request, so rotating the Secret needs no restart. With an API key instead, drop the volume and set the variable from a Secret:
env: - name: MISTRAL_API_KEY valueFrom: secretKeyRef: { name: mistral-api-key, key: apiKey }In both cases construct the client with Mistral() and no api_key argument, so the SDK picks the credential up itself.
Because the heavy work happens at Mistral, instances are light: scale on concurrent requests and keep MISTRAL_MAX_CONCURRENCY × replicas below your workspace rate limit.
CI runs the mocked tests from section 4, then creates the new agent version and moves the alias:
name: agentson: push: branches: [main]jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: { python-version: "3.12" } - run: pip install -r requirements.txt pytest - run: pytest -q # the mocked tests from section 4 deploy-agents: needs: test runs-on: ubuntu-latest environment: production steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: { python-version: "3.12" } - run: pip install -r requirements.txt - run: python deploy_agents.py # creates a new agent version and moves the "prod" alias env: MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }} AGENT_ID_SUPPORT: ${{ vars.AGENT_ID_SUPPORT }} GIT_SHA: ${{ github.sha }}Run a small evaluation set against the new agent version before moving the alias, and keep the previous version number so rollback is one call. Build and push the container image in the same workflow with your usual Docker actions.