Skip to content
PaPy

PydanticAI: Concurrency Limiting

ConcurrencyLimiter, ConcurrencyLimit, and AbstractConcurrencyLimiter — cap parallel model requests with optional backpressure and OpenTelemetry observability.

Verified against pydantic-ai 2.68.0 (checked 2026-11-19) — source module: pydantic_ai.concurrency.

ConcurrencyLimiter caps the number of simultaneous operations without any external dependencies. When the cap is reached, additional callers queue and wait. Optionally set max_queued to reject work once the queue depth exceeds a threshold.

There are two places to attach a limit:

WhereWhat it limits
Agent(..., max_concurrency=...)Concurrent runs of that agent (each run / run_sync / run_stream / iter holds a slot for its whole duration).
limit_model_concurrency(model, ...) (returns a ConcurrencyLimitedModel)Concurrent model requests through that model, across every agent that uses it.

Both accept any AnyConcurrencyLimit: an int, a ConcurrencyLimit, a shared ConcurrencyLimiter instance, or None. There is no concurrency_limiter key in ModelSettings — passing one there is silently ignored.

import asyncio
from pydantic_ai import Agent
# At most 3 runs of this agent in flight at once; the rest queue.
agent = Agent('openai:gpt-6.1-sol', max_concurrency=3)
async def main():
prompts = [f'What is {i} squared?' for i in range(10)]
results = await asyncio.gather(*[agent.run(p) for p in prompts])
for r in results:
print(r.output)
asyncio.run(main())
from pydantic_ai.concurrency import ConcurrencyLimiter, ConcurrencyLimit
# Simple: max 5 concurrent requests, unlimited queue
limiter = ConcurrencyLimiter(max_running=5)
# With backpressure: raise ConcurrencyLimitExceeded if > 10 tasks are already waiting
limiter = ConcurrencyLimiter(max_running=5, max_queued=10)
# Named (shows in OTel spans and error messages)
limiter = ConcurrencyLimiter(max_running=5, name='openai-prod')
# From a ConcurrencyLimit config object
config = ConcurrencyLimit(max_running=5, max_queued=10)
limiter = ConcurrencyLimiter.from_limit(config, name='openai-prod')
from pydantic_ai.concurrency import ConcurrencyLimit
# Serialisable config — useful when limits come from env vars or config files
config = ConcurrencyLimit(max_running=10, max_queued=50)
from pydantic_ai.concurrency import ConcurrencyLimiter
limiter = ConcurrencyLimiter(max_running=5, name='my-limiter')
print(limiter.max_running) # 5
print(limiter.running_count) # current concurrent calls
print(limiter.waiting_count) # tasks queued right now
print(limiter.available_count) # free slots
print(limiter.name) # 'my-limiter'

When a task must wait, ConcurrencyLimiter.acquire() creates an OpenTelemetry span named "waiting for <source> concurrency" — it shows up in your trace waterfall automatically.

from pydantic_ai import Agent
from pydantic_ai.concurrency import ConcurrencyLimiter
from pydantic_ai.exceptions import ConcurrencyLimitExceeded
strict_limiter = ConcurrencyLimiter(max_running=2, max_queued=5)
agent = Agent('openai:gpt-6.1-sol', max_concurrency=strict_limiter)
async def handle(prompt: str):
try:
# If 5 tasks are already waiting, this raises immediately
return await agent.run(prompt)
except ConcurrencyLimitExceeded as e:
print(f'Rejected: {e}')
# Handle gracefully — e.g. return a 429 to the caller

Sharing a limit across agents — limit the model, not the agent

Section titled “Sharing a limit across agents — limit the model, not the agent”

Wrap the model once and give the wrapped model to every agent; the total concurrent requests stay bounded however many agents use it:

import asyncio
from pydantic_ai import Agent, limit_model_concurrency
from pydantic_ai.concurrency import ConcurrencyLimiter
shared = ConcurrencyLimiter(max_running=10, name='openai-pool')
model = limit_model_concurrency('openai:gpt-6.1-sol', shared)
summariser = Agent(model, instructions='Summarise the text.')
analyst = Agent(model, instructions='Analyse the summary.')
async def process(text: str) -> str:
summary = await summariser.run(text)
analysis = await analyst.run(summary.output)
return analysis.output
async def main(documents: list[str]):
# At most 10 model requests run at once across both agents
return await asyncio.gather(*[process(doc) for doc in documents])

Passing the same ConcurrencyLimiter instance as max_concurrency= to several agents works too, but then the slot is held for the whole run (including tool execution), not just the model call.

Use separate limiters per provider to stay inside provider-specific rate limits:

from pydantic_ai import Agent, limit_model_concurrency
from pydantic_ai.concurrency import ConcurrencyLimit
openai_agent = Agent(limit_model_concurrency('openai:gpt-6.1-sol', 20))
anthropic_agent = Agent(
limit_model_concurrency('anthropic:claude-opus-5-5', ConcurrencyLimit(max_running=5, max_queued=50))
)

Custom limiter — Redis-backed distributed limiting

Section titled “Custom limiter — Redis-backed distributed limiting”

Subclass AbstractConcurrencyLimiter to integrate with external rate limiters:

import asyncio
from pydantic_ai.concurrency import AbstractConcurrencyLimiter
class RedisLimiter(AbstractConcurrencyLimiter):
"""Distributed concurrency limiting backed by Redis."""
def __init__(self, redis_client, key: str, max_running: int):
self._redis = redis_client
self._key = key
self._max = max_running
self._local_sem = asyncio.Semaphore(max_running)
async def acquire(self, source: str) -> None:
# Local semaphore prevents local over-subscription
await self._local_sem.acquire()
# INCR is atomic, so claim a slot first and give it back if we overshot;
# a separate GET-then-INCR would let two processes take the last slot
while True:
if await self._redis.incr(self._key) <= self._max:
return
await self._redis.decr(self._key)
await asyncio.sleep(0.05)
def release(self) -> None:
self._local_sem.release()
asyncio.create_task(self._redis.decr(self._key))

release() is synchronous in the interface, hence the fire-and-forget task; give the key an expiry (or a periodic reconciliation) so slots held by a crashed process are eventually recovered.

max_concurrency= and limit_model_concurrency() accept all these forms interchangeably:

from pydantic_ai import Agent
from pydantic_ai.concurrency import AnyConcurrencyLimit, ConcurrencyLimit, ConcurrencyLimiter
# AnyConcurrencyLimit = int | ConcurrencyLimit | AbstractConcurrencyLimiter | None
Agent('openai:gpt-6.1-sol', max_concurrency=5) # int shorthand
Agent('openai:gpt-6.1-sol', max_concurrency=ConcurrencyLimit(max_running=5, max_queued=20)) # config object
Agent('openai:gpt-6.1-sol', max_concurrency=ConcurrencyLimiter(max_running=5)) # shareable instance
Agent('openai:gpt-6.1-sol', max_concurrency=None) # no limiting (default)

FastAPI integration — request-level limiting

Section titled “FastAPI integration — request-level limiting”

Use a shared limiter with backpressure to protect your service under load:

from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
from pydantic_ai import Agent
from pydantic_ai.concurrency import ConcurrencyLimiter
from pydantic_ai.exceptions import ConcurrencyLimitExceeded
limiter = ConcurrencyLimiter(max_running=10, max_queued=20, name='api')
agent = Agent('openai:gpt-6.1-sol', max_concurrency=limiter)
@asynccontextmanager
async def lifespan(app: FastAPI):
app.state.limiter = limiter # e.g. for a /health endpoint reporting waiting_count
yield
app = FastAPI(lifespan=lifespan)
@app.post('/ask')
async def ask(prompt: str):
try:
result = await agent.run(prompt)
return {'output': result.output}
except ConcurrencyLimitExceeded:
raise HTTPException(status_code=429, detail='Too many concurrent requests')
import asyncio
from pydantic_ai.concurrency import ConcurrencyLimiter
limiter = ConcurrencyLimiter(max_running=5, name='monitor-demo')
async def status_reporter():
while True:
print(
f'running={limiter.running_count} '
f'waiting={limiter.waiting_count} '
f'available={limiter.available_count}'
)
await asyncio.sleep(1.0)

normalize_to_limiter — utility for library authors

Section titled “normalize_to_limiter — utility for library authors”

normalize_to_limiter converts any AnyConcurrencyLimit value into a concrete AbstractConcurrencyLimiter instance (or None). Use it in custom model wrappers or toolsets that accept user-provided concurrency config:

from pydantic_ai.concurrency import (
normalize_to_limiter,
AnyConcurrencyLimit,
ConcurrencyLimit,
ConcurrencyLimiter,
)
def setup_limiter(config: AnyConcurrencyLimit, name: str) -> None:
limiter = normalize_to_limiter(config, name=name)
if limiter is None:
print('No concurrency limiting configured')
else:
print(f'Limiter: {type(limiter).__name__}, max_running={limiter.max_running}')
setup_limiter(None, 'test') # No limiting
setup_limiter(5, 'test') # Creates ConcurrencyLimiter(5)
setup_limiter(ConcurrencyLimit(5, max_queued=20), 'test') # Creates ConcurrencyLimiter(5, max_queued=20)
setup_limiter(ConcurrencyLimiter(10), 'test') # Returns as-is

get_concurrency_context — async context manager wrapper

Section titled “get_concurrency_context — async context manager wrapper”

get_concurrency_context returns an async context manager that acquires and releases the limiter. Passing None returns a no-op manager. This is the internal primitive used by PydanticAI’s agent graph — useful when building custom runners:

import asyncio
from pydantic_ai.concurrency import (
ConcurrencyLimiter,
get_concurrency_context,
normalize_to_limiter,
AnyConcurrencyLimit,
)
async def run_with_limit(work, limit: AnyConcurrencyLimit = None):
"""Run an async callable under an optional concurrency limit."""
limiter = normalize_to_limiter(limit, name='custom-runner')
async with get_concurrency_context(limiter, source='custom-runner'):
return await work()
async def main():
limiter = ConcurrencyLimiter(max_running=3)
tasks = [
run_with_limit(lambda: asyncio.sleep(0.1), limit=limiter)
for _ in range(10)
]
await asyncio.gather(*tasks)
print('All tasks completed with concurrency ≤ 3')
asyncio.run(main())

When a task must wait (no free slots), ConcurrencyLimiter.acquire() creates an OpenTelemetry span automatically:

  • Span name: "waiting for <limiter-name-or-source> concurrency"
  • Attributes: source, waiting_count, max_running, limiter_name (if set), max_queued (if set)

This appears in your distributed trace waterfall as a latency contributor — you can see how long tasks spend waiting for a slot without any extra instrumentation code.

from pydantic_ai.concurrency import ConcurrencyLimiter
# Name appears in the OTel span and in ConcurrencyLimitExceeded error messages
limiter = ConcurrencyLimiter(max_running=5, name='openai-gpt4o-pool')
SymbolModuleNotes
ConcurrencyLimiterpydantic_ai.concurrencyMain limiter class with OTel observability
ConcurrencyLimitpydantic_ai.concurrencySerialisable config dataclass
AbstractConcurrencyLimiterpydantic_ai.concurrencyABC for custom (e.g. Redis-backed) limiters
AnyConcurrencyLimitpydantic_ai.concurrencyType alias: int | ConcurrencyLimit | AbstractConcurrencyLimiter | None
normalize_to_limiterpydantic_ai.concurrencyNormalise any AnyConcurrencyLimit → limiter instance or None
get_concurrency_contextpydantic_ai.concurrencyAsync context manager wrapping acquire/release
ConcurrencyLimitExceededpydantic_ai.exceptionsRaised when queue depth exceeds max_queued
limit_model_concurrency, ConcurrencyLimitedModelpydantic_ai.models.concurrency (also pydantic_ai)Wrap a model so its requests share a limiter
Agent(max_concurrency=...)pydantic_ai.agentLimit concurrent runs of one agent