Skip to content
PaPy

PydanticAI: Model Providers, FallbackModel & Instrumentation

Every supported provider, provider prefixes, FallbackModel with exception and response handlers, FallbackExceptionGroup, InstrumentedModel, InstrumentationSettings versions 2-6, OTel token/cost histograms, gateway routing, and local models.

Verified against pydantic-ai 2.68.0 (checked 2026-11-19) — source modules: pydantic_ai.providers, pydantic_ai.models, pydantic_ai.models.fallback, pydantic_ai.models.instrumented.

A PydanticAI Agent talks to an LLM through a Model backed by a Provider. The quickest way to wire one up is the 'provider:model-name' string; the full way is constructing SpecificModel(..., provider=SpecificProvider(...)) yourself. This page lists every prefix the installed source recognises and how to compose, gateway, or fall back between them.

from pydantic_ai import Agent
agent = Agent('openai:gpt-6.1-sol')
print(agent.run_sync('Hello!').output)
# Same thing, constructed explicitly (`openai:` means the Responses API):
from pydantic_ai.models.openai import OpenAIResponsesModel
from pydantic_ai.providers.openai import OpenAIProvider
agent = Agent(OpenAIResponsesModel('gpt-6.1-sol', provider=OpenAIProvider(api_key='sk-...')))

A string such as 'openai:gpt-6.1-sol' is parsed by infer_provider_class + infer_provider (providers/__init__.py). The first token before : selects the provider; the remainder is the model name, verbatim.

Verified from infer_provider_class in providers/__init__.py:

PrefixModel classNotes
openaiOpenAIResponsesModelOPENAI_API_KEY. Since 2.0 the bare prefix means the Responses API (it meant Chat Completions in V1).
openai-chatOpenAIChatModelChat Completions API — use this to keep the V1 behaviour.
openai-responsesOpenAIResponsesModelExplicit form of the default.
azure / azure-responsesAzure OpenAIAzureProvider(endpoint=..., api_key=...).
anthropicAnthropicModelANTHROPIC_API_KEY.
googleGoogleModel (Gemini API)GOOGLE_API_KEY / GEMINI_API_KEY. Replaces V1’s google-gla:.
google-cloudGoogleModel (Vertex AI)ADC / service-account credentials. Replaces V1’s google-vertex: / vertexai:.
bedrockBedrockConverseModelAWS credentials resolution. Needs the bedrock extra.
bedrock-mantleBedrock MantleNeeds the bedrock-mantle extra.
groqGroqModelGROQ_API_KEY. Needs the groq extra.
mistralMistralModelMISTRAL_API_KEY. Needs the mistral extra.
cohereCohereModelCO_API_KEY. Needs the cohere extra.
xaiXaiModelXAI_API_KEY. Supports XSearchTool. Replaces V1’s grok:. Needs the xai extra.
deepseekOpenAI-compatible DeepSeek
openrouterOpenRouterModelRoute to any OpenRouter model.
vercelVercel AI Gateway
cerebrasCerebras
moonshotaiMoonshot / Kimi
zaiZ.AI
fireworks, together, heroku, crusoe, nebius, ovhcloud, alibaba, sambanova, meta, snowflakeOpenAI-compatiblehosted / regional providers
huggingfaceHF Inference ProvidersNeeds the huggingface extra.
ollamaOllamaModel (local)Set OLLAMA_BASE_URL (e.g. http://localhost:11434/v1); no API key.
vllm, litellmOpenAI-compatibleSelf-hosted server / LiteLLM proxy (vllm: needs VLLM_BASE_URL).
github, github-copilotGitHub Models / Copilot
sentence-transformers, voyageaiEmbeddings onlypydantic_ai.embeddings.
gateway/<upstream>Any upstream via Pydantic AI Gatewaye.g. 'gateway/openai:<model>'.
typesafe, system-one, openai-codex, openai-chatgpt, openai-decisionsprovider-specific model classesNewer integrations; see the docstrings in pydantic_ai.models.
testTestModelNo network; Agent('test').

Removed in 2.0: the grok:, google-gla:, google-vertex: / vertexai: prefixes and the Outlines integration. A bare model name with no prefix (Agent('gpt-5')) now raises UserError.

Since 2.0 a bare pip install pydantic-ai installs only the OpenAI, Anthropic and Google providers (plus CLI, MCP, evals, web and Logfire); the retries extra is separate. Add the extras you use, e.g. pip install "pydantic-ai[bedrock,groq]".

The KnownModelName literal in pydantic_ai.models lists several hundred model names. An unknown string with a known prefix still works — it’s passed through to the provider.

Each provider accepts an api_key, a pre-built SDK client, or env-var fallback. Typical pattern:

from pydantic_ai import Agent
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider
model = AnthropicModel(
'claude-sonnet-5-5',
provider=AnthropicProvider(api_key='...'),
)
agent = Agent(model)

Useful when you need to:

  • Configure a custom httpx.AsyncClient (timeouts, proxies, retries).
  • Share a single SDK client across many agents.
  • Point at a self-hosted OpenAI-compatible endpoint (pass base_url= to OpenAIProvider).

OpenAI-compatible providers (OpenAIProvider(base_url='http://localhost:8000/v1', api_key='...')) unlock vLLM, LM Studio, oobabooga, or any homegrown server. For a service with no provider of its own, OpenAICompatibleProvider is the better fit: see Any OpenAI-compatible service.

pydantic_ai.settings.ModelSettings is a TypedDict; every key is optional. Most keys a model does not support are ignored silently; the few whose absence changes what the request guarantees (timeout, max_retries, stop_sequences, parallel_tool_calls=False, extra_headers, extra_body) emit an UnsupportedSettingWarning. Settings are layered key by key — the model’s own settings, then the agent’s model_settings, then capabilities, then the run’s model_settings= — and the later layer wins.

KeyNotes
max_tokens, temperature, top_p, top_k, seed, stop_sequences, presence_penalty, frequency_penalty, logit_biasThe usual sampling controls.
thinkingThinkingLevel = bool | Literal['minimal', 'low', 'medium', 'high', 'xhigh', 'max']. True = provider default effort; False disables thinking where the model allows it (on models that cannot switch it off, 2.61 sends the lowest level they accept); unmapped levels round to the nearest the provider has.
service_tierServiceTier = Literal['auto', 'default', 'flex', 'priority']. Provider-specific keys (openai_service_tier, …) take precedence.
cacheLibrary-managed prompt caching: True, False, a retention ('5m', '30m', '1h') or CacheConfig(retention=..., messages=False). Provider-specific cache keys win if set. See also CachePoint in the message history guide.
tool_choiceNone/'auto', 'none', 'required', a list of tool names, or ToolOrOutput(function_tools=[...]).
parallel_tool_callsFalse asks the model for one tool call per response.
timeoutPer-attempt timeout applied by the provider client.
request_timeoutOne deadline, enforced by PydanticAI, covering every SDK retry and the whole streamed response.
first_event_timeoutAdded in 2.64.0. A deadline for a streamed request to produce its first output (a part or a delta, thinking included); provider events that carry no output do not count. It stops applying once the first event arrives, so long responses are not cut off. On expiry the request is cancelled and ModelRequestTimeout is raised with setting == 'first_event_timeout'; FallbackModel and ModelFallback fall back by default. Ignored for non-streamed requests, where request_timeout covers the wait.
max_retriesOverride the provider SDK’s retry count for one request (0 = single attempt).
extra_headers, extra_bodyEscape hatches for provider features without a first-class key.
from pydantic_ai import Agent, ModelSettings
agent = Agent(
'anthropic:claude-opus-5-5',
model_settings=ModelSettings(temperature=0.1, max_tokens=4096, thinking='high', service_tier='priority'),
)

Provider-specific extensions (OpenAIChatModelSettings, OpenAIResponsesModelSettings, AnthropicModelSettings, GoogleModelSettings, …) subclass it and add prefixed keys (e.g. openai_reasoning_effort, anthropic_thinking). V1’s OpenAIModelSettings is now OpenAIChatModelSettings.

A static tool_choice=['search'] would also hide the output tools, so the run could never finish (PydanticAI raises UserError for that, and for a static 'required'). ToolOrOutput restricts the function tools while keeping output available:

from pydantic_ai import Agent
from pydantic_ai.settings import ToolOrOutput
agent = Agent('openai:gpt-6.1-sol')
@agent.tool_plain
def search_web(query: str) -> str:
"""Search the web."""
return f'Results for {query!r}'
@agent.tool_plain
def delete_cache() -> str:
"""Clear the cache."""
return 'cleared'
result = agent.run_sync(
'What is the population of France?',
model_settings={'tool_choice': ToolOrOutput(function_tools=['search_web'])}, # delete_cache hidden
)

To force a particular tool on one step only (say, the first), return the setting from a capability’s get_model_settings callable rather than setting it statically.

These capabilities change which model is used or how it is called, without touching your agent code.

CapabilityWhat it does
Thinking(effort=True)Sets thinking portably; provider-specific thinking settings still win.
Caching(retention=True, messages=True)Sets cache (library-managed prompt caching).
SelectModel(selector)Picks the model before each model-request step from a ModelSelectionContext (deps, prompt, messages, usage, run_step, usage_limits, deadline, run_id, conversation_id, the lower-precedence model).
ResolveModelId(resolver)Turns a model-ID string into a Model (custom names, per-tenant credentials); return None to fall through to the normal lookup. Replaces Temporal’s old provider_factory=.
RaiseContentFilterError()Raises ContentFilterError (an UnexpectedModelBehavior) when a response finishes with finish_reason='content_filter', instead of returning an empty or partial output.
from pydantic_ai import Agent
from pydantic_ai.capabilities import Caching, RaiseContentFilterError, ResolveModelId, SelectModel, Thinking
from pydantic_ai.models import ModelResolutionContext, ModelSelectionContext
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider
def route(ctx: ModelSelectionContext) -> str:
# cheap model for the first step, stronger one once tools have produced results
return 'anthropic:claude-haiku-5-5' if ctx.run_step == 1 else 'anthropic:claude-sonnet-5-5'
def resolve(ctx: ModelResolutionContext, model_id: str):
if model_id == 'tenant-default':
return AnthropicModel('claude-sonnet-5-5', provider=AnthropicProvider(api_key='per-tenant-key'))
return None # fall through to the standard 'provider:model' lookup
agent = Agent(
'tenant-default',
capabilities=[
ResolveModelId(resolve),
SelectModel(route),
Thinking(effort='low'),
Caching('1h'),
RaiseContentFilterError(),
],
)

Capabilities can also be passed per run (agent.run(..., capabilities=[Thinking(effort='xhigh')])) for a one-off change.

ModelFallback — fallback inside an agent

Section titled “ModelFallback — fallback inside an agent”

From 2.64.0, pydantic_ai.capabilities.ModelFallback does the same job as FallbackModel as a capability. The chain is the model selected for the step, followed by the models you list, so it composes with the agent’s own model and with SelectModel. The package documentation recommends it over FallbackModel inside an agent; FallbackModel remains the way to get the behaviour from a bare Model outside a run.

from pydantic_ai import Agent
from pydantic_ai.capabilities import ModelFallback
from pydantic_ai.exceptions import ModelAPIError
agent = Agent(
'openai:gpt-6.1-sol',
capabilities=[
ModelFallback(
'anthropic:claude-sonnet-5-5',
fallback_on=(ModelAPIError,), # the default also includes ContentFilterError; the same forms as FallbackModel
sticky=True,
)
],
)
  • The default is fallback_on=(ModelAPIError, ContentFilterError). Since 2.65.0 a refusal (a response with finish_reason='content_filter') therefore moves on to the next model; pass fallback_on=(ModelAPIError,) to keep the old behaviour. This applies to non-streamed requests only: a streamed refusal is accepted without a warning, because whether the content filter ended the stream is only known once it has been streamed. An empty one then ends the run with ContentFilterError.
  • With no agent model, the first model you list serves the first attempt.
  • A model already tried in the step is skipped, so repeating the agent’s own model in the list is harmless.
  • Each attempt re-runs prepare_model_request for the model about to serve it. before_model_request and wrap_model_request run once, and the model never sees a retry prompt.
  • sticky=True keeps a conversation that fell back on the model that answered, in later steps and in later runs that continue the message history. That keeps the provider’s prompt cache warm and avoids asking the failing model first every step.
  • Response handlers in fallback_on apply to non-streamed requests only. A streamed response has already reached the caller by the time it could be judged, so it is accepted with a UserWarning. A stream that fails to open, or errors before its first event, still falls back.

Checked with two FunctionModels, the first raising ModelOverloadedError: the run returns the second model’s answer.

FallbackModel — wrap primaries with backups

Section titled “FallbackModel — wrap primaries with backups”

pydantic_ai.models.fallback.FallbackModel (models/fallback.py) accepts a default + one or more fallbacks. When a request fails (or a response predicate fires), it retries the next model in the list until one succeeds or all are exhausted.

from pydantic_ai import Agent
from pydantic_ai.models.fallback import FallbackModel
from pydantic_ai.exceptions import ModelAPIError
model = FallbackModel(
'openai:gpt-6.1-sol', # primary
'anthropic:claude-sonnet-5-5', # first fallback
'google:gemini-3.8-flash', # second fallback
fallback_on=(ModelAPIError,),
)
agent = Agent(model)
result = agent.run_sync('What is 2 + 2?')
print(result.output)
FallbackModel(
default_model: Model | KnownModelName | str,
*fallback_models: Model | KnownModelName | str,
fallback_on: FallbackOn = (ModelAPIError,),
)

fallback_on is the core parameter. It can be any of:

FormExampleBehaviour
Tuple of exception types(ModelAPIError,)Triggers when any listed exception is raised
Single exception typeModelAPIErrorShorthand for a one-element tuple
Sync exception handlerdef fn(exc) -> boolTrue → fallback, False → re-raise
Async exception handlerasync def fn(exc) -> boolSame, awaited
Sync response handlerdef fn(resp: ModelResponse) -> boolCalled after each successful request; True → fallback
Async response handlerasync def fn(resp: ModelResponse) -> boolSame, awaited
Sequence mixing any of the above[ModelAPIError, fn1, fn2]Evaluated in order; first True wins

Handler type auto-detection: the source (models/fallback.py:_add_handler) inspects the first parameter’s type hint. If it’s annotated as ModelResponse, it’s a response handler; otherwise (including bare def fn(exc) lambdas) it’s an exception handler.

from pydantic_ai.models.fallback import FallbackModel
from pydantic_ai.exceptions import ModelAPIError
# Sync exception handler
def rate_limit_handler(exc: Exception) -> bool:
"""Fall back on 429 rate-limit errors only."""
return isinstance(exc, ModelAPIError) and '429' in str(exc)
# Async exception handler — can call your own metrics service
async def async_exc_handler(exc: Exception) -> bool:
# Could log to observability platform here
return isinstance(exc, ModelAPIError)
model = FallbackModel(
'openai:gpt-6.1-sol',
'anthropic:claude-haiku-5-5',
fallback_on=async_exc_handler,
)

Response handlers — fall back on weak or malformed replies

Section titled “Response handlers — fall back on weak or malformed replies”
from pydantic_ai.messages import ModelResponse
from pydantic_ai.models.fallback import FallbackModel
from pydantic_ai.exceptions import ModelAPIError
def too_short(resp: ModelResponse) -> bool:
"""Fall back if the model returned almost nothing."""
texts = [p.content for p in resp.parts if p.part_kind == 'text']
total_chars = sum(len(t) for t in texts)
return total_chars < 10
async def async_quality_check(resp: ModelResponse) -> bool:
"""Async response handler — could POST to a scoring endpoint."""
texts = [p.content for p in resp.parts if p.part_kind == 'text']
return not any(texts) # fall back on empty response
model = FallbackModel(
'openai:gpt-6.1-sol',
'anthropic:claude-sonnet-5-5',
fallback_on=[too_short, ModelAPIError],
)

Mixed sequence — all four forms together

Section titled “Mixed sequence — all four forms together”
from pydantic_ai.messages import ModelResponse
from pydantic_ai.models.fallback import FallbackModel
from pydantic_ai.exceptions import ModelAPIError
def is_rate_limited(exc: Exception) -> bool:
return '429' in str(exc)
def is_server_error(exc: Exception) -> bool:
return '5' in str(type(exc).__name__)
def empty_reply(resp: ModelResponse) -> bool:
return not any(p.part_kind == 'text' for p in resp.parts)
model = FallbackModel(
'openai:gpt-6.1-sol',
'anthropic:claude-sonnet-5-5',
'google:gemini-3.8-flash',
fallback_on=[
ModelAPIError, # exception type
is_rate_limited, # exception handler
is_server_error, # exception handler
empty_reply, # response handler (auto-detected from type hint)
],
)

FallbackExceptionGroup — when all models fail

Section titled “FallbackExceptionGroup — when all models fail”

If every model in the chain fails, FallbackModel raises a FallbackExceptionGroup — an ExceptionGroup subclass that bundles all the individual exceptions:

import asyncio
from pydantic_ai import Agent
from pydantic_ai.models.fallback import FallbackModel, FallbackExceptionGroup
from pydantic_ai.exceptions import ModelAPIError
model = FallbackModel(
'openai:gpt-6.1-sol',
'anthropic:claude-haiku-5-5',
fallback_on=(ModelAPIError,),
)
agent = Agent(model)
async def main():
try:
await agent.run('Hello')
except FallbackExceptionGroup as eg:
print(f'All {len(eg.exceptions)} models failed:')
for exc in eg.exceptions:
print(f' {type(exc).__name__}: {exc}')
asyncio.run(main())

FallbackExceptionGroup is an ExceptionGroup, so except* ModelAPIError also works (Python 3.11+, which PydanticAI requires) — but a single try cannot mix except and except* clauses; pick one style.

FallbackModel.models — inspecting the chain

Section titled “FallbackModel.models — inspecting the chain”
from pydantic_ai.exceptions import ModelAPIError
from pydantic_ai.models.fallback import FallbackModel
model = FallbackModel(
'openai:gpt-6.1-sol',
'anthropic:claude-haiku-5-5',
fallback_on=(ModelAPIError,),
)
print([m.model_name for m in model.models])
# ['gpt-6.1-sol', 'claude-haiku-5-5']
  • fallback_on=() (empty tuple) raises UserError at construction time with the message “All exceptions will propagate”. Always supply at least one condition.
  • Usage reflects the winning model only. If the primary fails and the fallback succeeds, result.usage only includes the fallback’s token counts. Use InstrumentedModel on each member of the chain to track per-model cost.
  • Fallbacks are tried serially, not concurrently. For true parallel hedging, run multiple Agent.run() calls yourself and take the first result.
  • Response handlers run after every successful request, including intermediate ones that aren’t the final step (tool-calling loops). Keep them cheap.

InstrumentedModel and InstrumentationSettings

Section titled “InstrumentedModel and InstrumentationSettings”

InstrumentedModel (models/instrumented.py) wraps any Model to emit OpenTelemetry spans, log events, and metrics around every model request. InstrumentationSettings controls what goes on the wire.

from pydantic_ai import Agent
from pydantic_ai.models.instrumented import InstrumentedModel, InstrumentationSettings
# Wrap a single agent
agent = Agent(
InstrumentedModel(
'openai:gpt-6.1-sol',
options=InstrumentationSettings(include_content=True),
)
)
# Or apply globally to every agent in the process
Agent.instrument_all(InstrumentationSettings())
InstrumentationSettings(
*,
tracer_provider: TracerProvider | None = None, # default: global provider
meter_provider: MeterProvider | None = None, # default: global provider
include_binary_content: bool = True,
include_content: bool = True,
include_model_request_parameters: bool = True,
version: Literal[2, 3, 4, 5, 6] = 5,
use_aggregated_usage_attribute_names: bool = True,
)
ParameterDefaultNotes
tracer_providerglobalPass to scope spans to a custom provider
meter_providerglobalToken + cost histograms
include_binary_contentTrueWhether to encode base64 images/files in spans
include_contentTrueSet False to strip prompts & completions from spans (PII)
include_model_request_parametersTrueRecord the full request parameters (tool definitions etc.) on spans; set False if large tool schemas bloat spans
version5Wire format version (see table below)
use_aggregated_usage_attribute_namesTrueAgent-run spans report gen_ai.aggregated_usage.* to avoid double-counting in backends that aggregate child+parent spans. Default flipped to True in 2.0

Version 1 and its event_mode= / logger_provider= arguments were removed in 2.0.

VersionNotes
1Legacy event-based spec — removed in 2.0.
2Newer OTel GenAI spec (deprecated since 2.0 — emits PydanticAIDeprecationWarning, as do 3 and 4). Messages in gen_ai.input.messages / gen_ai.output.messages span attributes; system instructions in gen_ai.system_instructions.
3Same as v2 + thinking tokens (e.g. Claude extended thinking, o-series reasoning tokens).
4Same as v3 + GenAI semantic conventions for multimodal content. URL media uses type='uri'; inline binaries use type='blob' with modality.
5Same as v4 + CallDeferred and ApprovalRequired control-flow exceptions no longer set the span to ERROR — spans are left UNSET since deferrals are not failures. Default (since 1.103.0).
6Same as v5, but tool results are emitted in a message with role='tool' instead of role='user'. Opt-in.
from pydantic_ai.models.instrumented import InstrumentationSettings
# Production — PII scrubbed, version 5 (default)
prod_settings = InstrumentationSettings(
include_content=False,
include_binary_content=False,
version=5,
)
# Development — full content for debugging
dev_settings = InstrumentationSettings(
include_content=True,
version=5,
)

InstrumentationSettings creates two meters on init (source: models/instrumented.py):

MetricUnitDescription
gen_ai.client.token.usage{token}Input and output token counts per request (type=input/output)
operation.cost{USD}Monetary cost per request (via genai-prices)
# Both meters are histogram-type; bucket boundaries follow the OTel GenAI spec.
# They are automatically populated on every InstrumentedModel request — no extra code needed.

To avoid double-counting on observability backends that sum child+parent spans:

from pydantic_ai.models.instrumented import InstrumentationSettings
# Already the default since 2.0: agent-run spans use
# gen_ai.aggregated_usage.input_tokens / output_tokens, model-request spans keep gen_ai.usage.*.
# To keep the V1 attribute names on run spans (e.g. for existing dashboards):
settings = InstrumentationSettings(use_aggregated_usage_attribute_names=False)
from pydantic_ai import Agent
from pydantic_ai.models.instrumented import InstrumentedModel, InstrumentationSettings
pii_safe = InstrumentationSettings(
include_content=False, # no prompts or completions in spans
include_binary_content=False, # no base64-encoded images/files
)
agent = Agent(InstrumentedModel('openai:gpt-6.1-sol', options=pii_safe))
# Spans will contain timing, token counts, and model name — but NO text content.

messages_to_otel_messages — see what lands on the span

Section titled “messages_to_otel_messages — see what lands on the span”

The V1 messages_to_otel_events() helper went with version 1. To see the GenAI-convention message list that version 2+ writes to gen_ai.input.messages, call messages_to_otel_messages():

from pydantic_ai.models.instrumented import InstrumentationSettings
from pydantic_ai.messages import ModelRequest, UserPromptPart
settings = InstrumentationSettings()
messages = [ModelRequest(parts=[UserPromptPart(content='Hello')])]
print(settings.messages_to_otel_messages(messages))

instrument_model (models/instrumented.py:instrument_model) is the low-level primitive. It only wraps if the model isn’t already instrumented:

from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.models.instrumented import instrument_model, InstrumentationSettings
raw_model = OpenAIChatModel('gpt-6.1-sol')
model = instrument_model(raw_model, instrument=InstrumentationSettings(version=5))
# If raw_model were already an InstrumentedModel, it's returned unchanged.

Global instrumentation with Agent.instrument_all

Section titled “Global instrumentation with Agent.instrument_all”
import logfire
from pydantic_ai import Agent
from pydantic_ai.models.instrumented import InstrumentationSettings
# Logfire configures the global OTel providers — instrument_all picks them up
logfire.configure()
Agent.instrument_all(InstrumentationSettings(
include_content=True,
version=5,
use_aggregated_usage_attribute_names=True,
))
# Every Agent created after this call will have spans emitted
agent1 = Agent('openai:gpt-6.1-sol')
agent2 = Agent('anthropic:claude-sonnet-5-5')

Per-provider tracing with custom TracerProvider

Section titled “Per-provider tracing with custom TracerProvider”
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from pydantic_ai.models.instrumented import InstrumentedModel, InstrumentationSettings
# Route only model spans to a dedicated OTLP endpoint
model_provider = TracerProvider()
model_provider.add_span_processor(
SimpleSpanProcessor(OTLPSpanExporter(endpoint='http://otel-collector:4318/v1/traces'))
)
settings = InstrumentationSettings(tracer_provider=model_provider, include_content=True)
model = InstrumentedModel('openai:gpt-6.1-sol', options=settings)

Prefix a known provider with gateway/ to route it through the Pydantic AI Gateway:

from pydantic_ai import Agent
agent = Agent('gateway/openai:gpt-6.1-sol') # needs PYDANTIC_AI_GATEWAY_API_KEY
# => uses the gateway provider, normalising to the upstream OpenAI profile

normalize_gateway_provider (providers/gateway.py) strips the prefix so model-profile lookups still resolve correctly.

from pydantic_ai import Agent
# Ollama (no key; set OLLAMA_BASE_URL=http://localhost:11434/v1)
agent = Agent('ollama:llama3.1')
# Any OpenAI-compatible local server
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider
model = OpenAIChatModel(
'qwen2.5-coder:32b',
provider=OpenAIProvider(base_url='http://localhost:8000/v1', api_key='x'),
)

OpenAICompatibleProvider (pydantic_ai.providers.openai_compatible) is for a hosted gateway or server that speaks the OpenAI API but is not OpenAI. Unlike OpenAIProvider, it takes a required name, and a vendor/model name such as qwen/qwen3-coder gets that vendor’s model profile.

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai_compatible import OpenAICompatibleProvider
provider = OpenAICompatibleProvider('my-gateway', base_url='https://gateway.example.com/v1', api_key='...')
agent = Agent(OpenAIChatModel('qwen/qwen3-coder', provider=provider))

The name identifies the service in message history and telemetry. Stateful data such as response and file IDs, and signed or encrypted data such as encrypted reasoning, is only sent back to a provider with a matching name (or canonical_system), so renaming it later stops history recorded under the old name from being replayed that way. Use OpenAIResponsesModel instead of OpenAIChatModel if the service implements the Responses API.

OpenAIProvider injects api_key='api-key-not-set' when you pass base_url without a key, which keeps the OpenAI SDK happy against local servers that don’t require auth.

from pydantic_ai import limit_model_concurrency
from pydantic_ai.models.openai import OpenAIChatModel
model = limit_model_concurrency(OpenAIChatModel('gpt-6.1-sol'), 8) # or ConcurrencyLimit(max_running=8, max_queued=100)

Enforces a max of 8 concurrent in-flight requests at the model layer.

from pydantic_ai.exceptions import ModelAPIError
from pydantic_ai.models.fallback import FallbackModel
model = FallbackModel(
'bedrock:us.anthropic.claude-sonnet-5-5',
'bedrock:eu.anthropic.claude-sonnet-5-5',
fallback_on=(ModelAPIError,),
)

3. Rate-limit-aware fallback with response sniff

Section titled “3. Rate-limit-aware fallback with response sniff”
from pydantic_ai.exceptions import ModelAPIError
from pydantic_ai.messages import ModelResponse
from pydantic_ai.models.fallback import FallbackModel
def empty_or_short(resp: ModelResponse) -> bool:
for p in resp.parts:
if getattr(p, 'part_kind', None) == 'text' and len(p.content) >= 20:
return False
return True
model = FallbackModel('openai:gpt-6.1-sol', 'anthropic:claude-sonnet-5-5',
fallback_on=[empty_or_short, ModelAPIError])

4. Self-hosted vLLM with a shared httpx client

Section titled “4. Self-hosted vLLM with a shared httpx client”
import httpx
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider
shared = httpx.AsyncClient(timeout=60, limits=httpx.Limits(max_connections=50))
provider = OpenAIProvider(base_url='http://vllm:8000/v1', api_key='x', http_client=shared)
model = OpenAIChatModel('meta-llama/Llama-3.1-8B-Instruct', provider=provider)

5. Swap model per environment with agent.override

Section titled “5. Swap model per environment with agent.override”
import os
from pydantic_ai import Agent
from pydantic_ai.models.fallback import FallbackModel
agent = Agent('openai:gpt-6.1-sol', defer_model_check=True)
env = os.getenv('APP_ENV', 'test')
prompt = 'Hello'
if env == 'production':
ctx = agent.override(model='openai:gpt-6.1-sol')
elif env == 'canary':
ctx = agent.override(model=FallbackModel('openai:gpt-6.1-sol', 'anthropic:claude-sonnet-5-5'))
else:
from pydantic_ai.models.test import TestModel
ctx = agent.override(model=TestModel())
with ctx:
result = agent.run_sync(prompt)

6. FallbackModel with async response quality check

Section titled “6. FallbackModel with async response quality check”
import asyncio
import httpx
from pydantic_ai import Agent
from pydantic_ai.messages import ModelResponse
from pydantic_ai.models.fallback import FallbackModel
from pydantic_ai.exceptions import ModelAPIError
async def quality_check(resp: ModelResponse) -> bool:
"""POST response to an internal scoring endpoint; fall back if score < 0.7."""
texts = [p.content for p in resp.parts if p.part_kind == 'text']
text = ' '.join(texts)
try:
async with httpx.AsyncClient(timeout=1.0) as client:
r = await client.post('http://scorer/score', json={'text': text})
return r.json()['score'] < 0.7
except Exception:
return False # don't fall back if scorer is down
model = FallbackModel(
'openai:gpt-6-luna',
'anthropic:claude-haiku-5-5',
fallback_on=[quality_check, ModelAPIError],
)
agent = Agent(model)
  • infer_provider, infer_provider_class — providers/__init__.py
  • KnownModelName — models/__init__.py (near the top)
  • FallbackModel — models/fallback.py
  • FallbackExceptionGroup — models/fallback.py
  • InstrumentedModel — models/instrumented.py
  • InstrumentationSettings — models/instrumented.py
  • instrument_model — models/instrumented.py
  • ModelSettings base — settings.py
  • limit_model_concurrency — models/concurrency.py