smolagents: Production Deployment Guide
Running smolagents reliably in production — sandboxing, cost and token tracking, telemetry, error handling, deployment, scaling and testing.
Verified against
smolagents1.26.0 (checked 2026-11-19).
Production Readiness Checklist
Section titled “Production Readiness Checklist”Core Functionality
Section titled “Core Functionality”| Area | Check |
|---|---|
| Versions | smolagents and its extras pinned (smolagents[...]==1.26.0); model provider SDKs pinned |
| Execution | CodeAgent uses a remote executor (docker, e2b, modal, blaxel) for any untrusted input; LocalPythonExecutor is not a security boundary |
| Imports | additional_authorized_imports lists only what tasks need; never "*" |
| Limits | max_steps set per agent; executor_kwargs={"timeout_seconds": ...} for the local executor; HTTP timeouts on the model (timeout= on InferenceClientModel, timeout kwarg on LiteLLM/OpenAI models) |
| Output | final_answer_checks validate the answer’s shape; RunResult.state checked for "max_steps_error" |
| Tools | Each tool unit-tested; descriptions reviewed; side-effecting tools idempotent |
| State | One agent instance per concurrent run (agents keep memory) |
| Telemetry | OpenTelemetry/OpenInference or Weave enabled; token usage recorded per run |
A thin production wrapper:
import loggingfrom dataclasses import dataclass
from smolagents import CodeAgent, InferenceClientModel, LogLevel, RunResult
logger = logging.getLogger(__name__)
@dataclassclass AgentOutcome: ok: bool output: object state: str steps: int input_tokens: int output_tokens: int seconds: float
def build_agent() -> CodeAgent: return CodeAgent( tools=[], model=InferenceClientModel(model_id="Qwen/Qwen3.8-27B", timeout=60), # check the Hub for live providers max_steps=10, executor_type="local", # switch to "docker"/"e2b" for untrusted input executor_kwargs={"timeout_seconds": 30}, verbosity_level=LogLevel.OFF, # -1: no console output at all (0 still prints errors) )
def run_task(task: str) -> AgentOutcome: if not task or len(task) > 10_000: raise ValueError("Task must be between 1 and 10,000 characters") agent = build_agent() # fresh agent: no memory shared between requests result: RunResult = agent.run(task, return_full_result=True) usage = result.token_usage outcome = AgentOutcome( ok=result.state == "success", output=result.output, state=result.state, steps=sum("step_number" in s for s in result.steps), # action steps; result.steps also holds the task and plans input_tokens=usage.input_tokens if usage else 0, output_tokens=usage.output_tokens if usage else 0, seconds=result.timing.duration or 0.0, ) logger.info("agent run", extra={"state": outcome.state, "steps": outcome.steps, "tokens_in": outcome.input_tokens, "tokens_out": outcome.output_tokens}) return outcomeInfrastructure Requirements
Section titled “Infrastructure Requirements”- Model provider quota sized for peak concurrency × average steps per run.
- A sandbox backend for
CodeAgent: Docker daemon reachable from the service (forexecutor_type="docker"), or E2B/Modal/Blaxel credentials. - Somewhere to keep conversation state if you support multi-turn sessions (agents’ memory is in-process).
- A telemetry backend (any OTLP collector, Langfuse, Arize Phoenix, Weave).
A start-up health check that exercises the model without running an agent:
from smolagents import ChatMessage, InferenceClientModelfrom smolagents.models import MessageRole
def check_model(model) -> bool: reply = model.generate([ChatMessage(role=MessageRole.USER, content="Reply with OK.")]) return "OK" in (reply.content or "")Operational Requirements
Section titled “Operational Requirements”- Alerting on error rate,
max_steps_errorrate, p95 latency and tokens per run. - A runbook for provider outages (fallback model, degraded mode).
- Log retention rules that respect the personal data in prompts and tool outputs.
Performance Optimisation
Section titled “Performance Optimisation”Model Selection for Performance
Section titled “Model Selection for Performance”Latency is dominated by the number of model calls and the tokens per call. In practice:
- A strong code model often finishes in fewer steps than a cheaper one, so compare cost per completed task, not price per token.
- Use a cheaper model for managed sub-agents with narrow tasks and a stronger one for the manager.
- Keep
planning_intervalunset for short tasks; planning adds a model call.
Caching & Memoisation
Section titled “Caching & Memoisation”Cache at the tool level, where results are deterministic, rather than caching whole agent answers:
from functools import lru_cache
from smolagents import tool
@lru_cache(maxsize=1024)def _fetch_rate(base: str, quote: str) -> float: # Replace with a real API call return 0.92
@tooldef fx_rate(base: str, quote: str) -> float: """Return the exchange rate from one currency to another.
Args: base: ISO code of the currency to convert from, e.g. "USD". quote: ISO code of the currency to convert to, e.g. "EUR". """ return _fetch_rate(base.upper(), quote.upper())For whole-answer caching (FAQ-style traffic), key on the normalised task plus model ID and prompt version, and cache only state == "success" results.
Prompt Optimisation
Section titled “Prompt Optimisation”- Put durable guidance in
instructions=rather than in every task string. - Write tool descriptions that state the return format; vague descriptions cause extra exploratory steps.
- Return compact tool outputs; trim pages and tables before returning them.
max_print_outputs_lengthstops a strayprint(df)flooding the context.
Parallel Execution
Section titled “Parallel Execution”ToolCallingAgent(max_tool_threads=N)runs a step’s tool calls in parallel.- A
CodeAgentcan call several tools in one code block; nothing more to configure. - Across tasks, run separate agent instances in threads or processes:
from concurrent.futures import ThreadPoolExecutor
tasks = ["Summarise doc A", "Summarise doc B", "Summarise doc C"]with ThreadPoolExecutor(max_workers=4) as pool: outcomes = list(pool.map(run_task, tasks)) # run_task builds a fresh agent each timeCost Management
Section titled “Cost Management”Token Usage Tracking
Section titled “Token Usage Tracking”Every ActionStep and PlanningStep carries token_usage, and RunResult.token_usage sums them. Attach prices outside the library:
from smolagents import ActionStep, PlanningStep
PRICE_PER_MTOK = {"input": 0.50, "output": 1.50} # your provider's prices, per million tokens
def run_cost(result) -> float: u = result.token_usage return (u.input_tokens * PRICE_PER_MTOK["input"] + u.output_tokens * PRICE_PER_MTOK["output"]) / 1_000_000
class StepCostLogger: def __init__(self): self.total = 0
def __call__(self, step, agent=None): if step.token_usage: self.total += step.token_usage.total_tokens print(f"{type(step).__name__} {getattr(step, 'step_number', '')}: running total {self.total} tokens")
cost_logger = StepCostLogger()# CodeAgent(..., step_callbacks={ActionStep: cost_logger, PlanningStep: cost_logger})Cost Optimisation Strategies
Section titled “Cost Optimisation Strategies”- Cap steps.
max_stepsis the main cost ceiling; set it per agent from observed distributions. - Stop early on budget. A step callback can call
agent.interrupt()when a token budget is exceeded;run()then raisesAgentError("Agent interrupted.")before the next step. - Trim memory. For vision or browsing agents, clear
observations_imageson old steps in a callback. - Right-size models per role, and re-evaluate when providers release cheaper tiers.
- Avoid re-runs. Validate inputs before starting an agent; a malformed request that fails at step 8 still costs eight steps.
def budget_guard(limit_tokens: int): used = 0 def callback(step, agent=None): nonlocal used if step.token_usage: used += step.token_usage.total_tokens if used > limit_tokens and agent is not None: agent.interrupt() return callbackMonitoring & Observability
Section titled “Monitoring & Observability”Key Metrics to Track
Section titled “Key Metrics to Track”| Metric | Source |
|---|---|
Runs, success rate, max_steps_error rate | RunResult.state |
| Steps per run | action steps in RunResult.steps (entries with a step_number) |
| Tokens per run (in/out) | RunResult.token_usage |
| Latency per run and per step | RunResult.timing, ActionStep.timing |
| Tool error rate | ActionStep.error |
| Final-answer check failures | step errors after final_answer |
Structured Logging
Section titled “Structured Logging”Set verbosity_level=LogLevel.OFF (-1) to silence the Rich console output (0 still prints errors), then log from a step callback:
import jsonimport logging
from smolagents import ActionStep
log = logging.getLogger("agent")
def log_step(step: ActionStep, agent=None): log.info(json.dumps({ "step": step.step_number, "duration_s": step.timing.duration if step.timing else None, "tokens": step.token_usage.total_tokens if step.token_usage else None, "tools": [tc.name for tc in (step.tool_calls or [])], "error": str(step.error) if step.error else None, }))OpenTelemetry tracing
Section titled “OpenTelemetry tracing”The telemetry extra installs OpenInference’s smolagents instrumentation (openinference-instrumentation-smolagents) together with the OpenTelemetry SDK, the OTLP exporters and Arize Phoenix. The instrumentation emits spans for each run (kind AGENT), each step (CHAIN) and each tool call (TOOL), and for each model call (LLM) made through the built-in model classes. A custom Model subclass of your own is not wrapped, so its calls produce no LLM span. Point the exporter at any OTLP backend (Langfuse, Arize Phoenix, Jaeger, Grafana Tempo):
# pip install "smolagents[telemetry]"from openinference.instrumentation.smolagents import SmolagentsInstrumentorfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporterfrom opentelemetry.sdk.trace import TracerProviderfrom opentelemetry.sdk.trace.export import BatchSpanProcessor
provider = TracerProvider()provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")))SmolagentsInstrumentor().instrument(tracer_provider=provider)Instrument once, at start-up, before any agent runs. To check the wiring without a backend, add an InMemorySpanExporter (from opentelemetry.sdk.trace.export.in_memory_span_exporter) behind a SimpleSpanProcessor and read get_finished_spans() after a run: a one-step CodeAgent with one tool produced CodeAgent.run, Step 1 and one span per tool call, including final_answer. Spans carry the full prompts and tool outputs, so apply the same retention and redaction rules as to your logs.
Security Best Practices
Section titled “Security Best Practices”Input Validation
Section titled “Input Validation”- Bound task length and reject binary or obviously malformed input before starting a run.
- Treat everything a tool returns (web pages, emails, files) as untrusted: it can contain prompt-injection text. Keep privileged tools away from agents that read untrusted content, or put a human approval step in front of them.
- Never interpolate secrets into tasks; tools should read credentials from the environment.
Code Execution Sandboxing
Section titled “Code Execution Sandboxing”LocalPythonExecutor restricts imports and dangerous builtins but runs in your process; it is a guard against accidents, not attackers. For untrusted input:
from smolagents import CodeAgent, InferenceClientModel
model = InferenceClientModel(model_id="Qwen/Qwen3.8-27B")agent = CodeAgent( tools=[], model=model, executor_type="e2b", # or "docker", "modal", "blaxel" additional_authorized_imports=["pandas"],)try: agent.run("Load the CSV at https://example.com/data.csv and report the column means.")finally: agent.cleanup()Notes:
- Remote executors do not support
managed_agents; for multi-agent systems, run the whole service inside a locked-down container (no credentials beyond what tools need, egress restricted) instead. - Tools are rebuilt from their source inside the sandbox and run there, so any credential a tool reads must be present in the sandbox. Give it only the credentials that tool needs.
- With
executor_type="docker", the executor builds a Jupyter-kernel image and starts a container from it, and by default rebuilds the image on every start. Supply your own image withexecutor_kwargs={"image_name": ..., "build_new_image": False}in production. - The Docker container gets a port mapping (published on
127.0.0.1:8888by default, with a random auth token for the kernel gateway) and nothing else: normal network access (itpip installs the authorised imports) and no memory, CPU or process limits. Add limits throughcontainer_run_kwargs, which is passed to docker-py’scontainers.run. Restrict egress at the host or network level; do not rely on the executor for it.
agent = CodeAgent( tools=[], model=model, executor_type="docker", executor_kwargs={ "image_name": "agent-sandbox:1", # pre-built, with the authorised imports installed "build_new_image": False, "container_run_kwargs": {"mem_limit": "1g", "nano_cpus": 1_000_000_000, "pids_limit": 256}, },)Rate Limiting & Throttling
Section titled “Rate Limiting & Throttling”Limit at the edge (API gateway or middleware) per user, and inside the service per provider:
import threading
PROVIDER_SLOTS = threading.BoundedSemaphore(8)
def run_task_limited(task: str): with PROVIDER_SLOTS: return run_task(task)Error Handling & Resilience
Section titled “Error Handling & Resilience”Comprehensive Error Handling
Section titled “Comprehensive Error Handling”Errors inside a step (tool exceptions, code errors, parse failures) are fed back to the model, which usually recovers. What reaches your code:
| Situation | How it surfaces |
|---|---|
| Ran out of steps | RunResult.state == "max_steps_error"; output is a best-effort answer |
| Provider/API failure | AgentGenerationError raised from run() |
Interrupted via agent.interrupt() | AgentError raised from run() |
| Your callback raised | The exception propagates |
import time
from smolagents import AgentError, AgentGenerationError
def resilient_run(task: str, attempts: int = 3): delay = 2.0 for attempt in range(1, attempts + 1): try: outcome = run_task(task) if outcome.ok: return outcome if attempt == attempts: return outcome # return the best-effort answer, flagged not ok except AgentGenerationError: if attempt == attempts: raise time.sleep(delay) delay *= 2 except AgentError: raise # interrupted or unrecoverable: do not retry blindlyFor provider failover, use LiteLLMRouterModel with several deployments, or catch AgentGenerationError and rebuild the agent with a fallback model.
Deployment Options
Section titled “Deployment Options”Containerised Deployment
Section titled “Containerised Deployment”FROM python:3.12-slimENV PYTHONUNBUFFERED=1RUN useradd -m -u 1000 appWORKDIR /appCOPY requirements.txt .RUN pip install --no-cache-dir -r requirements.txtCOPY --chown=app:app . .USER appEXPOSE 8000CMD ["uvicorn", "service:app", "--host", "0.0.0.0", "--port", "8000"]requirements.txt:
smolagents[toolkit,litellm,e2b,telemetry]==1.26.0fastapiuvicorn[standard]prometheus-clientAsync applications
Section titled “Async applications”agent.run() is synchronous and blocks for the whole run, so an async server must run it in a worker thread: FastAPI’s run_in_threadpool (below), or anyio.to_thread.run_sync(agent.run, task) in Starlette, as in the official async_agent example. Both use AnyIO’s default thread limiter of 40 threads, which therefore caps concurrent runs per process. Build an agent per request rather than sharing a global one, because an agent keeps its memory between run() calls.
A service with a streaming endpoint
Section titled “A service with a streaming endpoint”/run returns the finished result; /run/stream sends each plan, step and the final answer as server-sent events while the run progresses. Starlette iterates a plain generator in its thread pool, so the synchronous agent.run(..., stream=True) loop does not block the event loop. The in-flight gauge feeds the autoscaler below.
import json
from fastapi import FastAPI, HTTPExceptionfrom fastapi.concurrency import run_in_threadpoolfrom fastapi.responses import StreamingResponsefrom prometheus_client import Gauge, make_asgi_appfrom pydantic import BaseModelfrom smolagents import ActionStep, AgentError, FinalAnswerStep, PlanningStep
app = FastAPI()app.mount("/metrics", make_asgi_app())RUNS_IN_FLIGHT = Gauge("agent_runs_in_flight", "Agent runs currently executing")
class TaskIn(BaseModel): task: str = ""
@app.get("/health")def health(): return {"status": "ok"}
def tracked_run(task: str) -> AgentOutcome: with RUNS_IN_FLIGHT.track_inprogress(): return run_task(task)
@app.post("/run")async def run(body: TaskIn): try: outcome = await run_in_threadpool(tracked_run, body.task) except ValueError as exc: raise HTTPException(status_code=400, detail=str(exc)) return outcome.__dict__
def sse(event: str, data: dict) -> str: return f"event: {event}\ndata: {json.dumps(data, default=str)}\n\n"
@app.post("/run/stream")def run_stream(body: TaskIn): if not body.task or len(body.task) > 10_000: raise HTTPException(status_code=400, detail="Task must be between 1 and 10,000 characters") agent = build_agent()
def events(): with RUNS_IN_FLIGHT.track_inprogress(): try: for event in agent.run(body.task, stream=True): if isinstance(event, PlanningStep): yield sse("plan", {"plan": event.plan}) elif isinstance(event, ActionStep): yield sse("step", { "step": event.step_number, "observations": (event.observations or "")[:2_000], "error": str(event.error) if event.error else None, }) elif isinstance(event, FinalAnswerStep): yield sse("final", {"output": event.output}) except AgentError as exc: # provider failure or interrupt yield sse("error", {"detail": str(exc)})
return StreamingResponse(events(), media_type="text/event-stream")Set stream_outputs=True on the agent and handle ChatMessageStreamDelta events as well if you want to forward model tokens as they arrive.
Kubernetes Deployment
Section titled “Kubernetes Deployment”apiVersion: apps/v1kind: Deploymentmetadata: name: smolagents-apispec: replicas: 3 selector: matchLabels: { app: smolagents-api } template: metadata: labels: { app: smolagents-api } spec: securityContext: { runAsNonRoot: true, runAsUser: 1000 } containers: - name: api image: registry.example.com/smolagents-api:1.0.0 ports: [{ containerPort: 8000 }] envFrom: - secretRef: { name: smolagents-secrets } # HF_TOKEN, E2B_API_KEY, ... resources: requests: { cpu: 250m, memory: 512Mi } limits: { cpu: "1", memory: 1Gi } readinessProbe: httpGet: { path: /health, port: 8000 } livenessProbe: httpGet: { path: /health, port: 8000 } initialDelaySeconds: 20---apiVersion: v1kind: Servicemetadata: name: smolagents-apispec: selector: { app: smolagents-api } ports: [{ port: 80, targetPort: 8000 }]---apiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata: name: smolagents-apispec: scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: smolagents-api } minReplicas: 3 maxReplicas: 15 metrics: - type: Pods pods: metric: { name: agent_runs_in_flight } target: { type: AverageValue, averageValue: "4" } # concurrent runs one pod should carryThe HPA scales on the agent_runs_in_flight gauge from the service above, not CPU: a pod waiting on the model uses little CPU however busy it is. A custom pod metric needs an adapter that serves Prometheus metrics to the Kubernetes metrics API (Prometheus Adapter, or KEDA with its Prometheus scaler). Run one Uvicorn worker per pod so each pod has a single gauge.
Running executor_type="docker" from inside Kubernetes needs access to a Docker daemon, which is rarely appropriate; prefer E2B, Modal or Blaxel there.
Scaling Strategies
Section titled “Scaling Strategies”Horizontal Scaling
Section titled “Horizontal Scaling”Agents are cheap to construct and hold per-run memory, so the simplest scalable design is one agent per request, with no shared mutable state in the process. Scale pods on in-flight runs (as in the HPA above) or latency rather than CPU, since runs mostly wait on the model. For long tasks, move runs to a job queue (Celery, RQ, Arq, a Kubernetes Job) and return a job ID.
Vertical Scaling
Section titled “Vertical Scaling”Only relevant when you host the model yourself (TransformersModel, VLLMModel): size GPU memory for the model plus KV cache at your target concurrency, and prefer vLLM’s continuous batching to many single-request processes.
Testing & Quality Assurance
Section titled “Testing & Quality Assurance”Unit Testing Tools
Section titled “Unit Testing Tools”Tools are plain objects; test them directly:
def test_fx_rate(): assert fx_rate(base="usd", quote="eur") == 0.92 assert fx_rate.name == "fx_rate" assert set(fx_rate.inputs) == {"base", "quote"}Test agent wiring without network calls by subclassing Model with a scripted reply:
from smolagents import ChatMessage, CodeAgent, Modelfrom smolagents.models import MessageRole
class ScriptedModel(Model): def __init__(self, replies): super().__init__(model_id="scripted") self.replies = list(replies)
def generate(self, messages, stop_sequences=None, response_format=None, tools_to_call_from=None, **kwargs): return ChatMessage(role=MessageRole.ASSISTANT, content=self.replies.pop(0))
def test_agent_uses_tool(): model = ScriptedModel([ "Thought: convert.\n<code>\nrate = fx_rate(base='USD', quote='EUR')\nfinal_answer(round(100 * rate, 2))\n</code>", ]) agent = CodeAgent(tools=[fx_rate], model=model, verbosity_level=0) assert agent.run("Convert 100 USD to EUR") == 92.0A ToolCallingAgent expects structured tool calls rather than text, so its scripted replies are ChatMessage objects with tool_calls. The last reply calls the final_answer tool:
from smolagents import ChatMessage, Model, ToolCallingAgentfrom smolagents.models import ChatMessageToolCall, ChatMessageToolCallFunction, MessageRole
def tool_call(name: str, **arguments) -> ChatMessage: return ChatMessage( role=MessageRole.ASSISTANT, content=None, tool_calls=[ChatMessageToolCall( id=f"call_{name}", type="function", function=ChatMessageToolCallFunction(name=name, arguments=arguments), )], )
class ScriptedToolModel(Model): def __init__(self, replies): super().__init__(model_id="scripted") self.replies = list(replies)
def generate(self, messages, stop_sequences=None, response_format=None, tools_to_call_from=None, **kwargs): return self.replies.pop(0)
def test_tool_calling_agent_uses_tool(): model = ScriptedToolModel([ tool_call("fx_rate", base="USD", quote="EUR"), tool_call("final_answer", answer=92.0), ]) agent = ToolCallingAgent(tools=[fx_rate], model=model, verbosity_level=0) assert agent.run("Convert 100 USD to EUR") == 92.0 assert agent.memory.steps[1].tool_calls[0].name == "fx_rate" # steps[0] is the TaskStepAssertions on agent.memory.steps (which tool was called, with what arguments, what the observation was) test your wiring and prompts-as-data without a model.
For quality, keep a small evaluation set of real tasks with expected answers, run it on every prompt, tool or model change, and track success rate, steps and tokens over time.
Observability Integration (Weights & Biases Weave)
Section titled “Observability Integration (Weights & Biases Weave)”Weave Integration
Section titled “Weave Integration”Weave ships a smolagents integration that is enabled by weave.init; agent runs, steps, model and tool calls are then traced automatically.
# pip install weaveimport weavefrom smolagents import CodeAgent, InferenceClientModel
weave.init("smolagents-production")
agent = CodeAgent(tools=[], model=InferenceClientModel(model_id="Qwen/Qwen3.8-27B"))agent.run("What is 2 ** 32?") # appears as a trace in the Weave UI
@weave.op()def answer_question(question: str) -> str: """Wrap your own entry point to group runs and attach inputs/outputs.""" return str(agent.run(question))Weave’s Evaluation API can then score answer_question against a dataset; see the Weave documentation for current usage.