PydanticAI: Model Providers, FallbackModel & Instrumentation
Every supported provider, provider prefixes, FallbackModel with exception and response handlers, FallbackExceptionGroup, InstrumentedModel, InstrumentationSettings versions 2-6, OTel token/cost histograms, gateway routing, and local models.
Verified against pydantic-ai 2.68.0 (checked 2026-11-19) — source modules: pydantic_ai.providers, pydantic_ai.models, pydantic_ai.models.fallback, pydantic_ai.models.instrumented.
A PydanticAI Agent talks to an LLM through a Model backed by a Provider. The quickest way to wire one up is the 'provider:model-name' string; the full way is constructing SpecificModel(..., provider=SpecificProvider(...)) yourself. This page lists every prefix the installed source recognises and how to compose, gateway, or fall back between them.
Minimal runnable example
Section titled “Minimal runnable example”from pydantic_ai import Agent
agent = Agent('openai:gpt-6.1-sol')print(agent.run_sync('Hello!').output)
# Same thing, constructed explicitly (`openai:` means the Responses API):from pydantic_ai.models.openai import OpenAIResponsesModelfrom pydantic_ai.providers.openai import OpenAIProvider
agent = Agent(OpenAIResponsesModel('gpt-6.1-sol', provider=OpenAIProvider(api_key='sk-...')))A string such as 'openai:gpt-6.1-sol' is parsed by infer_provider_class + infer_provider (providers/__init__.py). The first token before : selects the provider; the remainder is the model name, verbatim.
Provider prefixes
Section titled “Provider prefixes”Verified from infer_provider_class in providers/__init__.py:
| Prefix | Model class | Notes |
|---|---|---|
openai | OpenAIResponsesModel | OPENAI_API_KEY. Since 2.0 the bare prefix means the Responses API (it meant Chat Completions in V1). |
openai-chat | OpenAIChatModel | Chat Completions API — use this to keep the V1 behaviour. |
openai-responses | OpenAIResponsesModel | Explicit form of the default. |
azure / azure-responses | Azure OpenAI | AzureProvider(endpoint=..., api_key=...). |
anthropic | AnthropicModel | ANTHROPIC_API_KEY. |
google | GoogleModel (Gemini API) | GOOGLE_API_KEY / GEMINI_API_KEY. Replaces V1’s google-gla:. |
google-cloud | GoogleModel (Vertex AI) | ADC / service-account credentials. Replaces V1’s google-vertex: / vertexai:. |
bedrock | BedrockConverseModel | AWS credentials resolution. Needs the bedrock extra. |
bedrock-mantle | Bedrock Mantle | Needs the bedrock-mantle extra. |
groq | GroqModel | GROQ_API_KEY. Needs the groq extra. |
mistral | MistralModel | MISTRAL_API_KEY. Needs the mistral extra. |
cohere | CohereModel | CO_API_KEY. Needs the cohere extra. |
xai | XaiModel | XAI_API_KEY. Supports XSearchTool. Replaces V1’s grok:. Needs the xai extra. |
deepseek | OpenAI-compatible DeepSeek | |
openrouter | OpenRouterModel | Route to any OpenRouter model. |
vercel | Vercel AI Gateway | |
cerebras | Cerebras | |
moonshotai | Moonshot / Kimi | |
zai | Z.AI | |
fireworks, together, heroku, crusoe, nebius, ovhcloud, alibaba, sambanova, meta, snowflake | OpenAI-compatible | hosted / regional providers |
huggingface | HF Inference Providers | Needs the huggingface extra. |
ollama | OllamaModel (local) | Set OLLAMA_BASE_URL (e.g. http://localhost:11434/v1); no API key. |
vllm, litellm | OpenAI-compatible | Self-hosted server / LiteLLM proxy (vllm: needs VLLM_BASE_URL). |
github, github-copilot | GitHub Models / Copilot | |
sentence-transformers, voyageai | Embeddings only | pydantic_ai.embeddings. |
gateway/<upstream> | Any upstream via Pydantic AI Gateway | e.g. 'gateway/openai:<model>'. |
typesafe, system-one, openai-codex, openai-chatgpt, openai-decisions | provider-specific model classes | Newer integrations; see the docstrings in pydantic_ai.models. |
test | TestModel | No network; Agent('test'). |
Removed in 2.0: the grok:, google-gla:, google-vertex: / vertexai: prefixes and the Outlines integration. A bare model name with no prefix (Agent('gpt-5')) now raises UserError.
Since 2.0 a bare pip install pydantic-ai installs only the OpenAI, Anthropic and Google providers (plus CLI, MCP, evals, web and Logfire); the retries extra is separate. Add the extras you use, e.g. pip install "pydantic-ai[bedrock,groq]".
The KnownModelName literal in pydantic_ai.models lists several hundred model names. An unknown string with a known prefix still works — it’s passed through to the provider.
Explicit provider construction
Section titled “Explicit provider construction”Each provider accepts an api_key, a pre-built SDK client, or env-var fallback. Typical pattern:
from pydantic_ai import Agentfrom pydantic_ai.models.anthropic import AnthropicModelfrom pydantic_ai.providers.anthropic import AnthropicProvider
model = AnthropicModel( 'claude-sonnet-5-5', provider=AnthropicProvider(api_key='...'),)agent = Agent(model)Useful when you need to:
- Configure a custom
httpx.AsyncClient(timeouts, proxies, retries). - Share a single SDK client across many agents.
- Point at a self-hosted OpenAI-compatible endpoint (pass
base_url=toOpenAIProvider).
OpenAI-compatible providers (OpenAIProvider(base_url='http://localhost:8000/v1', api_key='...')) unlock vLLM, LM Studio, oobabooga, or any homegrown server. For a service with no provider of its own, OpenAICompatibleProvider is the better fit: see Any OpenAI-compatible service.
ModelSettings — provider-agnostic knobs
Section titled “ModelSettings — provider-agnostic knobs”pydantic_ai.settings.ModelSettings is a TypedDict; every key is optional. Most keys a model does not support are ignored silently; the few whose absence changes what the request guarantees (timeout, max_retries, stop_sequences, parallel_tool_calls=False, extra_headers, extra_body) emit an UnsupportedSettingWarning. Settings are layered key by key — the model’s own settings, then the agent’s model_settings, then capabilities, then the run’s model_settings= — and the later layer wins.
| Key | Notes |
|---|---|
max_tokens, temperature, top_p, top_k, seed, stop_sequences, presence_penalty, frequency_penalty, logit_bias | The usual sampling controls. |
thinking | ThinkingLevel = bool | Literal['minimal', 'low', 'medium', 'high', 'xhigh', 'max']. True = provider default effort; False disables thinking where the model allows it (on models that cannot switch it off, 2.61 sends the lowest level they accept); unmapped levels round to the nearest the provider has. |
service_tier | ServiceTier = Literal['auto', 'default', 'flex', 'priority']. Provider-specific keys (openai_service_tier, …) take precedence. |
cache | Library-managed prompt caching: True, False, a retention ('5m', '30m', '1h') or CacheConfig(retention=..., messages=False). Provider-specific cache keys win if set. See also CachePoint in the message history guide. |
tool_choice | None/'auto', 'none', 'required', a list of tool names, or ToolOrOutput(function_tools=[...]). |
parallel_tool_calls | False asks the model for one tool call per response. |
timeout | Per-attempt timeout applied by the provider client. |
request_timeout | One deadline, enforced by PydanticAI, covering every SDK retry and the whole streamed response. |
first_event_timeout | Added in 2.64.0. A deadline for a streamed request to produce its first output (a part or a delta, thinking included); provider events that carry no output do not count. It stops applying once the first event arrives, so long responses are not cut off. On expiry the request is cancelled and ModelRequestTimeout is raised with setting == 'first_event_timeout'; FallbackModel and ModelFallback fall back by default. Ignored for non-streamed requests, where request_timeout covers the wait. |
max_retries | Override the provider SDK’s retry count for one request (0 = single attempt). |
extra_headers, extra_body | Escape hatches for provider features without a first-class key. |
from pydantic_ai import Agent, ModelSettings
agent = Agent( 'anthropic:claude-opus-5-5', model_settings=ModelSettings(temperature=0.1, max_tokens=4096, thinking='high', service_tier='priority'),)Provider-specific extensions (OpenAIChatModelSettings, OpenAIResponsesModelSettings, AnthropicModelSettings, GoogleModelSettings, …) subclass it and add prefixed keys (e.g. openai_reasoning_effort, anthropic_thinking). V1’s OpenAIModelSettings is now OpenAIChatModelSettings.
tool_choice and ToolOrOutput
Section titled “tool_choice and ToolOrOutput”A static tool_choice=['search'] would also hide the output tools, so the run could never finish (PydanticAI raises UserError for that, and for a static 'required'). ToolOrOutput restricts the function tools while keeping output available:
from pydantic_ai import Agentfrom pydantic_ai.settings import ToolOrOutput
agent = Agent('openai:gpt-6.1-sol')
@agent.tool_plaindef search_web(query: str) -> str: """Search the web.""" return f'Results for {query!r}'
@agent.tool_plaindef delete_cache() -> str: """Clear the cache.""" return 'cleared'
result = agent.run_sync( 'What is the population of France?', model_settings={'tool_choice': ToolOrOutput(function_tools=['search_web'])}, # delete_cache hidden)To force a particular tool on one step only (say, the first), return the setting from a capability’s get_model_settings callable rather than setting it statically.
Model-level capabilities
Section titled “Model-level capabilities”These capabilities change which model is used or how it is called, without touching your agent code.
| Capability | What it does |
|---|---|
Thinking(effort=True) | Sets thinking portably; provider-specific thinking settings still win. |
Caching(retention=True, messages=True) | Sets cache (library-managed prompt caching). |
SelectModel(selector) | Picks the model before each model-request step from a ModelSelectionContext (deps, prompt, messages, usage, run_step, usage_limits, deadline, run_id, conversation_id, the lower-precedence model). |
ResolveModelId(resolver) | Turns a model-ID string into a Model (custom names, per-tenant credentials); return None to fall through to the normal lookup. Replaces Temporal’s old provider_factory=. |
RaiseContentFilterError() | Raises ContentFilterError (an UnexpectedModelBehavior) when a response finishes with finish_reason='content_filter', instead of returning an empty or partial output. |
from pydantic_ai import Agentfrom pydantic_ai.capabilities import Caching, RaiseContentFilterError, ResolveModelId, SelectModel, Thinkingfrom pydantic_ai.models import ModelResolutionContext, ModelSelectionContextfrom pydantic_ai.models.anthropic import AnthropicModelfrom pydantic_ai.providers.anthropic import AnthropicProvider
def route(ctx: ModelSelectionContext) -> str: # cheap model for the first step, stronger one once tools have produced results return 'anthropic:claude-haiku-5-5' if ctx.run_step == 1 else 'anthropic:claude-sonnet-5-5'
def resolve(ctx: ModelResolutionContext, model_id: str): if model_id == 'tenant-default': return AnthropicModel('claude-sonnet-5-5', provider=AnthropicProvider(api_key='per-tenant-key')) return None # fall through to the standard 'provider:model' lookup
agent = Agent( 'tenant-default', capabilities=[ ResolveModelId(resolve), SelectModel(route), Thinking(effort='low'), Caching('1h'), RaiseContentFilterError(), ],)Capabilities can also be passed per run (agent.run(..., capabilities=[Thinking(effort='xhigh')])) for a one-off change.
ModelFallback — fallback inside an agent
Section titled “ModelFallback — fallback inside an agent”From 2.64.0, pydantic_ai.capabilities.ModelFallback does the same job as FallbackModel as a capability. The chain is the model selected for the step, followed by the models you list, so it composes with the agent’s own model and with SelectModel. The package documentation recommends it over FallbackModel inside an agent; FallbackModel remains the way to get the behaviour from a bare Model outside a run.
from pydantic_ai import Agentfrom pydantic_ai.capabilities import ModelFallbackfrom pydantic_ai.exceptions import ModelAPIError
agent = Agent( 'openai:gpt-6.1-sol', capabilities=[ ModelFallback( 'anthropic:claude-sonnet-5-5', fallback_on=(ModelAPIError,), # the default also includes ContentFilterError; the same forms as FallbackModel sticky=True, ) ],)- The default is
fallback_on=(ModelAPIError, ContentFilterError). Since 2.65.0 a refusal (a response withfinish_reason='content_filter') therefore moves on to the next model; passfallback_on=(ModelAPIError,)to keep the old behaviour. This applies to non-streamed requests only: a streamed refusal is accepted without a warning, because whether the content filter ended the stream is only known once it has been streamed. An empty one then ends the run withContentFilterError. - With no agent model, the first model you list serves the first attempt.
- A model already tried in the step is skipped, so repeating the agent’s own model in the list is harmless.
- Each attempt re-runs
prepare_model_requestfor the model about to serve it.before_model_requestandwrap_model_requestrun once, and the model never sees a retry prompt. sticky=Truekeeps a conversation that fell back on the model that answered, in later steps and in later runs that continue the message history. That keeps the provider’s prompt cache warm and avoids asking the failing model first every step.- Response handlers in
fallback_onapply to non-streamed requests only. A streamed response has already reached the caller by the time it could be judged, so it is accepted with aUserWarning. A stream that fails to open, or errors before its first event, still falls back.
Checked with two FunctionModels, the first raising ModelOverloadedError: the run returns the second model’s answer.
FallbackModel — wrap primaries with backups
Section titled “FallbackModel — wrap primaries with backups”pydantic_ai.models.fallback.FallbackModel (models/fallback.py) accepts a default + one or more fallbacks. When a request fails (or a response predicate fires), it retries the next model in the list until one succeeds or all are exhausted.
from pydantic_ai import Agentfrom pydantic_ai.models.fallback import FallbackModelfrom pydantic_ai.exceptions import ModelAPIError
model = FallbackModel( 'openai:gpt-6.1-sol', # primary 'anthropic:claude-sonnet-5-5', # first fallback 'google:gemini-3.8-flash', # second fallback fallback_on=(ModelAPIError,),)agent = Agent(model)result = agent.run_sync('What is 2 + 2?')print(result.output)Constructor signature
Section titled “Constructor signature”FallbackModel( default_model: Model | KnownModelName | str, *fallback_models: Model | KnownModelName | str, fallback_on: FallbackOn = (ModelAPIError,),)fallback_on is the core parameter. It can be any of:
| Form | Example | Behaviour |
|---|---|---|
| Tuple of exception types | (ModelAPIError,) | Triggers when any listed exception is raised |
| Single exception type | ModelAPIError | Shorthand for a one-element tuple |
| Sync exception handler | def fn(exc) -> bool | True → fallback, False → re-raise |
| Async exception handler | async def fn(exc) -> bool | Same, awaited |
| Sync response handler | def fn(resp: ModelResponse) -> bool | Called after each successful request; True → fallback |
| Async response handler | async def fn(resp: ModelResponse) -> bool | Same, awaited |
| Sequence mixing any of the above | [ModelAPIError, fn1, fn2] | Evaluated in order; first True wins |
Handler type auto-detection: the source (models/fallback.py:_add_handler) inspects the first parameter’s type hint. If it’s annotated as ModelResponse, it’s a response handler; otherwise (including bare def fn(exc) lambdas) it’s an exception handler.
Exception handlers — sync and async
Section titled “Exception handlers — sync and async”from pydantic_ai.models.fallback import FallbackModelfrom pydantic_ai.exceptions import ModelAPIError
# Sync exception handlerdef rate_limit_handler(exc: Exception) -> bool: """Fall back on 429 rate-limit errors only.""" return isinstance(exc, ModelAPIError) and '429' in str(exc)
# Async exception handler — can call your own metrics serviceasync def async_exc_handler(exc: Exception) -> bool: # Could log to observability platform here return isinstance(exc, ModelAPIError)
model = FallbackModel( 'openai:gpt-6.1-sol', 'anthropic:claude-haiku-5-5', fallback_on=async_exc_handler,)Response handlers — fall back on weak or malformed replies
Section titled “Response handlers — fall back on weak or malformed replies”from pydantic_ai.messages import ModelResponsefrom pydantic_ai.models.fallback import FallbackModelfrom pydantic_ai.exceptions import ModelAPIError
def too_short(resp: ModelResponse) -> bool: """Fall back if the model returned almost nothing.""" texts = [p.content for p in resp.parts if p.part_kind == 'text'] total_chars = sum(len(t) for t in texts) return total_chars < 10
async def async_quality_check(resp: ModelResponse) -> bool: """Async response handler — could POST to a scoring endpoint.""" texts = [p.content for p in resp.parts if p.part_kind == 'text'] return not any(texts) # fall back on empty response
model = FallbackModel( 'openai:gpt-6.1-sol', 'anthropic:claude-sonnet-5-5', fallback_on=[too_short, ModelAPIError],)Mixed sequence — all four forms together
Section titled “Mixed sequence — all four forms together”from pydantic_ai.messages import ModelResponsefrom pydantic_ai.models.fallback import FallbackModelfrom pydantic_ai.exceptions import ModelAPIError
def is_rate_limited(exc: Exception) -> bool: return '429' in str(exc)
def is_server_error(exc: Exception) -> bool: return '5' in str(type(exc).__name__)
def empty_reply(resp: ModelResponse) -> bool: return not any(p.part_kind == 'text' for p in resp.parts)
model = FallbackModel( 'openai:gpt-6.1-sol', 'anthropic:claude-sonnet-5-5', 'google:gemini-3.8-flash', fallback_on=[ ModelAPIError, # exception type is_rate_limited, # exception handler is_server_error, # exception handler empty_reply, # response handler (auto-detected from type hint) ],)FallbackExceptionGroup — when all models fail
Section titled “FallbackExceptionGroup — when all models fail”If every model in the chain fails, FallbackModel raises a FallbackExceptionGroup — an ExceptionGroup subclass that bundles all the individual exceptions:
import asynciofrom pydantic_ai import Agentfrom pydantic_ai.models.fallback import FallbackModel, FallbackExceptionGroupfrom pydantic_ai.exceptions import ModelAPIError
model = FallbackModel( 'openai:gpt-6.1-sol', 'anthropic:claude-haiku-5-5', fallback_on=(ModelAPIError,),)agent = Agent(model)
async def main(): try: await agent.run('Hello') except FallbackExceptionGroup as eg: print(f'All {len(eg.exceptions)} models failed:') for exc in eg.exceptions: print(f' {type(exc).__name__}: {exc}')
asyncio.run(main())FallbackExceptionGroup is an ExceptionGroup, so except* ModelAPIError also works (Python 3.11+, which PydanticAI requires) — but a single try cannot mix except and except* clauses; pick one style.
FallbackModel.models — inspecting the chain
Section titled “FallbackModel.models — inspecting the chain”from pydantic_ai.exceptions import ModelAPIErrorfrom pydantic_ai.models.fallback import FallbackModel
model = FallbackModel( 'openai:gpt-6.1-sol', 'anthropic:claude-haiku-5-5', fallback_on=(ModelAPIError,),)print([m.model_name for m in model.models])# ['gpt-6.1-sol', 'claude-haiku-5-5']Gotchas
Section titled “Gotchas”fallback_on=()(empty tuple) raisesUserErrorat construction time with the message “All exceptions will propagate”. Always supply at least one condition.- Usage reflects the winning model only. If the primary fails and the fallback succeeds,
result.usageonly includes the fallback’s token counts. UseInstrumentedModelon each member of the chain to track per-model cost. - Fallbacks are tried serially, not concurrently. For true parallel hedging, run multiple
Agent.run()calls yourself and take the first result. - Response handlers run after every successful request, including intermediate ones that aren’t the final step (tool-calling loops). Keep them cheap.
InstrumentedModel and InstrumentationSettings
Section titled “InstrumentedModel and InstrumentationSettings”InstrumentedModel (models/instrumented.py) wraps any Model to emit OpenTelemetry spans, log events, and metrics around every model request. InstrumentationSettings controls what goes on the wire.
Quick start
Section titled “Quick start”from pydantic_ai import Agentfrom pydantic_ai.models.instrumented import InstrumentedModel, InstrumentationSettings
# Wrap a single agentagent = Agent( InstrumentedModel( 'openai:gpt-6.1-sol', options=InstrumentationSettings(include_content=True), ))
# Or apply globally to every agent in the processAgent.instrument_all(InstrumentationSettings())InstrumentationSettings constructor
Section titled “InstrumentationSettings constructor”InstrumentationSettings( *, tracer_provider: TracerProvider | None = None, # default: global provider meter_provider: MeterProvider | None = None, # default: global provider include_binary_content: bool = True, include_content: bool = True, include_model_request_parameters: bool = True, version: Literal[2, 3, 4, 5, 6] = 5, use_aggregated_usage_attribute_names: bool = True,)| Parameter | Default | Notes |
|---|---|---|
tracer_provider | global | Pass to scope spans to a custom provider |
meter_provider | global | Token + cost histograms |
include_binary_content | True | Whether to encode base64 images/files in spans |
include_content | True | Set False to strip prompts & completions from spans (PII) |
include_model_request_parameters | True | Record the full request parameters (tool definitions etc.) on spans; set False if large tool schemas bloat spans |
version | 5 | Wire format version (see table below) |
use_aggregated_usage_attribute_names | True | Agent-run spans report gen_ai.aggregated_usage.* to avoid double-counting in backends that aggregate child+parent spans. Default flipped to True in 2.0 |
Version 1 and its event_mode= / logger_provider= arguments were removed in 2.0.
Data format versions
Section titled “Data format versions”| Version | Notes |
|---|---|
1 | Legacy event-based spec — removed in 2.0. |
2 | Newer OTel GenAI spec (deprecated since 2.0 — emits PydanticAIDeprecationWarning, as do 3 and 4). Messages in gen_ai.input.messages / gen_ai.output.messages span attributes; system instructions in gen_ai.system_instructions. |
3 | Same as v2 + thinking tokens (e.g. Claude extended thinking, o-series reasoning tokens). |
4 | Same as v3 + GenAI semantic conventions for multimodal content. URL media uses type='uri'; inline binaries use type='blob' with modality. |
5 | Same as v4 + CallDeferred and ApprovalRequired control-flow exceptions no longer set the span to ERROR — spans are left UNSET since deferrals are not failures. Default (since 1.103.0). |
6 | Same as v5, but tool results are emitted in a message with role='tool' instead of role='user'. Opt-in. |
from pydantic_ai.models.instrumented import InstrumentationSettings
# Production — PII scrubbed, version 5 (default)prod_settings = InstrumentationSettings( include_content=False, include_binary_content=False, version=5,)
# Development — full content for debuggingdev_settings = InstrumentationSettings( include_content=True, version=5,)OTel metrics emitted
Section titled “OTel metrics emitted”InstrumentationSettings creates two meters on init (source: models/instrumented.py):
| Metric | Unit | Description |
|---|---|---|
gen_ai.client.token.usage | {token} | Input and output token counts per request (type=input/output) |
operation.cost | {USD} | Monetary cost per request (via genai-prices) |
# Both meters are histogram-type; bucket boundaries follow the OTel GenAI spec.# They are automatically populated on every InstrumentedModel request — no extra code needed.To avoid double-counting on observability backends that sum child+parent spans:
from pydantic_ai.models.instrumented import InstrumentationSettings
# Already the default since 2.0: agent-run spans use# gen_ai.aggregated_usage.input_tokens / output_tokens, model-request spans keep gen_ai.usage.*.# To keep the V1 attribute names on run spans (e.g. for existing dashboards):settings = InstrumentationSettings(use_aggregated_usage_attribute_names=False)include_content=False — PII scrubbing
Section titled “include_content=False — PII scrubbing”from pydantic_ai import Agentfrom pydantic_ai.models.instrumented import InstrumentedModel, InstrumentationSettings
pii_safe = InstrumentationSettings( include_content=False, # no prompts or completions in spans include_binary_content=False, # no base64-encoded images/files)
agent = Agent(InstrumentedModel('openai:gpt-6.1-sol', options=pii_safe))# Spans will contain timing, token counts, and model name — but NO text content.messages_to_otel_messages — see what lands on the span
Section titled “messages_to_otel_messages — see what lands on the span”The V1 messages_to_otel_events() helper went with version 1. To see the GenAI-convention message list that version 2+ writes to gen_ai.input.messages, call messages_to_otel_messages():
from pydantic_ai.models.instrumented import InstrumentationSettingsfrom pydantic_ai.messages import ModelRequest, UserPromptPart
settings = InstrumentationSettings()messages = [ModelRequest(parts=[UserPromptPart(content='Hello')])]print(settings.messages_to_otel_messages(messages))instrument_model helper function
Section titled “instrument_model helper function”instrument_model (models/instrumented.py:instrument_model) is the low-level primitive. It only wraps if the model isn’t already instrumented:
from pydantic_ai.models.openai import OpenAIChatModelfrom pydantic_ai.models.instrumented import instrument_model, InstrumentationSettings
raw_model = OpenAIChatModel('gpt-6.1-sol')model = instrument_model(raw_model, instrument=InstrumentationSettings(version=5))# If raw_model were already an InstrumentedModel, it's returned unchanged.Global instrumentation with Agent.instrument_all
Section titled “Global instrumentation with Agent.instrument_all”import logfirefrom pydantic_ai import Agentfrom pydantic_ai.models.instrumented import InstrumentationSettings
# Logfire configures the global OTel providers — instrument_all picks them uplogfire.configure()Agent.instrument_all(InstrumentationSettings( include_content=True, version=5, use_aggregated_usage_attribute_names=True,))
# Every Agent created after this call will have spans emittedagent1 = Agent('openai:gpt-6.1-sol')agent2 = Agent('anthropic:claude-sonnet-5-5')Per-provider tracing with custom TracerProvider
Section titled “Per-provider tracing with custom TracerProvider”from opentelemetry.sdk.trace import TracerProviderfrom opentelemetry.sdk.trace.export import SimpleSpanProcessorfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporterfrom pydantic_ai.models.instrumented import InstrumentedModel, InstrumentationSettings
# Route only model spans to a dedicated OTLP endpointmodel_provider = TracerProvider()model_provider.add_span_processor( SimpleSpanProcessor(OTLPSpanExporter(endpoint='http://otel-collector:4318/v1/traces')))
settings = InstrumentationSettings(tracer_provider=model_provider, include_content=True)model = InstrumentedModel('openai:gpt-6.1-sol', options=settings)Gateway routing
Section titled “Gateway routing”Prefix a known provider with gateway/ to route it through the Pydantic AI Gateway:
from pydantic_ai import Agent
agent = Agent('gateway/openai:gpt-6.1-sol') # needs PYDANTIC_AI_GATEWAY_API_KEY# => uses the gateway provider, normalising to the upstream OpenAI profilenormalize_gateway_provider (providers/gateway.py) strips the prefix so model-profile lookups still resolve correctly.
Local models
Section titled “Local models”from pydantic_ai import Agent
# Ollama (no key; set OLLAMA_BASE_URL=http://localhost:11434/v1)agent = Agent('ollama:llama3.1')
# Any OpenAI-compatible local serverfrom pydantic_ai.models.openai import OpenAIChatModelfrom pydantic_ai.providers.openai import OpenAIProvider
model = OpenAIChatModel( 'qwen2.5-coder:32b', provider=OpenAIProvider(base_url='http://localhost:8000/v1', api_key='x'),)Any OpenAI-compatible service
Section titled “Any OpenAI-compatible service”OpenAICompatibleProvider (pydantic_ai.providers.openai_compatible) is for a hosted gateway or server that speaks the OpenAI API but is not OpenAI. Unlike OpenAIProvider, it takes a required name, and a vendor/model name such as qwen/qwen3-coder gets that vendor’s model profile.
from pydantic_ai import Agentfrom pydantic_ai.models.openai import OpenAIChatModelfrom pydantic_ai.providers.openai_compatible import OpenAICompatibleProvider
provider = OpenAICompatibleProvider('my-gateway', base_url='https://gateway.example.com/v1', api_key='...')agent = Agent(OpenAIChatModel('qwen/qwen3-coder', provider=provider))The name identifies the service in message history and telemetry. Stateful data such as response and file IDs, and signed or encrypted data such as encrypted reasoning, is only sent back to a provider with a matching name (or canonical_system), so renaming it later stops history recorded under the old name from being replayed that way. Use OpenAIResponsesModel instead of OpenAIChatModel if the service implements the Responses API.
OpenAIProvider injects api_key='api-key-not-set' when you pass base_url without a key, which keeps the OpenAI SDK happy against local servers that don’t require auth.
Patterns
Section titled “Patterns”1. Provider-level concurrency limit
Section titled “1. Provider-level concurrency limit”from pydantic_ai import limit_model_concurrencyfrom pydantic_ai.models.openai import OpenAIChatModel
model = limit_model_concurrency(OpenAIChatModel('gpt-6.1-sol'), 8) # or ConcurrencyLimit(max_running=8, max_queued=100)Enforces a max of 8 concurrent in-flight requests at the model layer.
2. Region-aware fallback
Section titled “2. Region-aware fallback”from pydantic_ai.exceptions import ModelAPIErrorfrom pydantic_ai.models.fallback import FallbackModel
model = FallbackModel( 'bedrock:us.anthropic.claude-sonnet-5-5', 'bedrock:eu.anthropic.claude-sonnet-5-5', fallback_on=(ModelAPIError,),)3. Rate-limit-aware fallback with response sniff
Section titled “3. Rate-limit-aware fallback with response sniff”from pydantic_ai.exceptions import ModelAPIErrorfrom pydantic_ai.messages import ModelResponsefrom pydantic_ai.models.fallback import FallbackModel
def empty_or_short(resp: ModelResponse) -> bool: for p in resp.parts: if getattr(p, 'part_kind', None) == 'text' and len(p.content) >= 20: return False return True
model = FallbackModel('openai:gpt-6.1-sol', 'anthropic:claude-sonnet-5-5', fallback_on=[empty_or_short, ModelAPIError])4. Self-hosted vLLM with a shared httpx client
Section titled “4. Self-hosted vLLM with a shared httpx client”import httpxfrom pydantic_ai.models.openai import OpenAIChatModelfrom pydantic_ai.providers.openai import OpenAIProvider
shared = httpx.AsyncClient(timeout=60, limits=httpx.Limits(max_connections=50))provider = OpenAIProvider(base_url='http://vllm:8000/v1', api_key='x', http_client=shared)model = OpenAIChatModel('meta-llama/Llama-3.1-8B-Instruct', provider=provider)5. Swap model per environment with agent.override
Section titled “5. Swap model per environment with agent.override”import os
from pydantic_ai import Agentfrom pydantic_ai.models.fallback import FallbackModel
agent = Agent('openai:gpt-6.1-sol', defer_model_check=True)env = os.getenv('APP_ENV', 'test')prompt = 'Hello'
if env == 'production': ctx = agent.override(model='openai:gpt-6.1-sol')elif env == 'canary': ctx = agent.override(model=FallbackModel('openai:gpt-6.1-sol', 'anthropic:claude-sonnet-5-5'))else: from pydantic_ai.models.test import TestModel ctx = agent.override(model=TestModel())with ctx: result = agent.run_sync(prompt)6. FallbackModel with async response quality check
Section titled “6. FallbackModel with async response quality check”import asyncioimport httpxfrom pydantic_ai import Agentfrom pydantic_ai.messages import ModelResponsefrom pydantic_ai.models.fallback import FallbackModelfrom pydantic_ai.exceptions import ModelAPIError
async def quality_check(resp: ModelResponse) -> bool: """POST response to an internal scoring endpoint; fall back if score < 0.7.""" texts = [p.content for p in resp.parts if p.part_kind == 'text'] text = ' '.join(texts) try: async with httpx.AsyncClient(timeout=1.0) as client: r = await client.post('http://scorer/score', json={'text': text}) return r.json()['score'] < 0.7 except Exception: return False # don't fall back if scorer is down
model = FallbackModel( 'openai:gpt-6-luna', 'anthropic:claude-haiku-5-5', fallback_on=[quality_check, ModelAPIError],)agent = Agent(model)Reference
Section titled “Reference”infer_provider,infer_provider_class—providers/__init__.pyKnownModelName—models/__init__.py(near the top)FallbackModel—models/fallback.pyFallbackExceptionGroup—models/fallback.pyInstrumentedModel—models/instrumented.pyInstrumentationSettings—models/instrumented.pyinstrument_model—models/instrumented.pyModelSettingsbase —settings.pylimit_model_concurrency—models/concurrency.py