Mistral Streaming Server (FastAPI)
A FastAPI server that streams Mistral agent conversations to browsers as Server-Sent Events.
Verified against mistralai 3.8.0 (checked 2026-11-19).
This server streams a stored agent’s replies as Server-Sent Events, forwarding text deltas, tool activity and the conversation ID so the client can continue the conversation.
mistralai==3.8.0fastapi>=0.115uvicorn[standard]>=0.30import jsonimport osfrom collections.abc import AsyncIterator
from fastapi import FastAPI, HTTPExceptionfrom fastapi.responses import StreamingResponsefrom mistralai.client import Mistral
app = FastAPI()client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])AGENT_ID = os.environ["AGENT_ID"]
def sse(event: str, data: object) -> str: return f"event: {event}\ndata: {json.dumps(data)}\n\n"
@app.get("/stream")async def stream(q: str, conversation_id: str | None = None) -> StreamingResponse: if not q.strip() or len(q) > 4000: raise HTTPException(status_code=400, detail="q must be 1-4000 characters")
async def events() -> AsyncIterator[str]: if conversation_id: source = await client.beta.conversations.append_stream_async(conversation_id=conversation_id, inputs=q) else: source = await client.beta.conversations.start_stream_async(agent_id=AGENT_ID, inputs=q) async with source as stream: async for event in stream: data = event.data if data.type == "conversation.response.started": yield sse("start", {"conversation_id": data.conversation_id}) elif data.type == "message.output.delta" and isinstance(data.content, str): yield sse("delta", data.content) elif data.type == "tool.execution.started": yield sse("tool", {"name": data.name}) elif data.type == "conversation.response.error": yield sse("error", {"message": data.message}) elif data.type == "conversation.response.done": yield sse("done", {"tokens": data.usage.total_tokens})
return StreamingResponse(events(), media_type="text/event-stream", headers={"Cache-Control": "no-cache"})
@app.get("/healthz")async def healthz() -> dict: return {"ok": True}Run it with uvicorn app:app --port 8080 and try curl -N "http://localhost:8080/stream?q=hello". The browser keeps the conversation_id from the start event and sends it back on the next question.
For plain chat completions without an agent, stream client.chat.stream_async(model=..., messages=...) instead and forward event.data.choices[0].delta.content.
Deployment
Section titled “Deployment”Package and deploy it like any small web service; the production guide has a Dockerfile, Kubernetes manifest and CI workflow; change the CMD module to app:app and they fit this app, including its /healthz probe. Two things matter for streaming specifically:
- Disable response buffering on any proxy in front of the service (for NGINX ingress,
nginx.ingress.kubernetes.io/proxy-buffering: "off"), or events arrive in one burst at the end. - Set the proxy’s read timeout above your longest agent run; tool-heavy runs can be quiet for many seconds between events.
Security
Section titled “Security”- Authenticate the endpoint and rate-limit per user; every request costs tokens.
- Check that a
conversation_idbelongs to the caller before appending to it. - Keep
MISTRAL_API_KEYin a secret store, or use a service-account token viaMISTRAL_SA_TOKEN_PATH. - Do not log full prompts or replies if they may contain personal data; log conversation IDs and token counts.