Skip to content
LgPy

Chapter 8 — Agent Middleware

Customise a create_agent agent with middleware: hooks before and after model calls, wrappers around model and tool calls, and the built-in middleware for approvals, summarisation, limits, retries and PII.

What you’ll learn: how to change what a create_agent agent does on each step (trim history, pick a model, handle tool errors, stop the loop, pause for approval) with middleware, and which middleware ships with langchain.

Checked against langchain 1.5.0 and langgraph 1.2.22. Time: ~25 minutes.

Prereqs: Chapter 2 (create_agent) and Chapter 4 (tools).

Middleware is a langchain feature: create_agent and langchain.agents.middleware live in the langchain package (pip install langchain), and create_agent compiles to a LangGraph graph. If you build your own StateGraph instead, the equivalents are node policies and error handlers; see In your own graph below.

create_agent runs a loop: call the model, run any tools it asked for, repeat until the model answers without tool calls. Middleware attaches at fixed points in that loop.

HookRunsTypical use
before_agentOnce, when the run startsLoad user data, validate input
before_modelBefore every model call; can update stateTrim or summarise history
wrap_model_callAround every model call; gets the request and a handlerChange model, prompt, tools or messages for one call; retries; fallbacks; logging
after_modelAfter every model call; can update state or jumpGuardrails on the response, counters
wrap_tool_callAround every tool executionTool error handling, retries, approval
after_agentOnce, when the run endsClean-up, logging

With several middleware in middleware=[A, B], the before_* hooks run in list order (A then B), the after_* hooks in reverse (B then A), and the wrap_* hooks nest with A outermost.

Each decorator turns a function into a middleware object. This agent uses five of them:

from dataclasses import dataclass
from langchain.agents import create_agent, AgentState
from langchain.agents.middleware import (
before_model, after_model, dynamic_prompt, wrap_model_call, wrap_tool_call,
ModelRequest, ModelResponse,
)
from langchain.chat_models import init_chat_model
from langchain.messages import AIMessage, RemoveMessage, ToolMessage
from langchain.tools import tool
from langgraph.graph.message import REMOVE_ALL_MESSAGES
from langgraph.runtime import Runtime
@dataclass
class Context:
tier: str = "standard"
@tool
def lookup_order(order_id: str) -> str:
"""Look up an order by ID."""
if order_id == "0":
raise ValueError("order not found")
return f"Order {order_id}: shipped"
# 1. before_model: runs before every model call and can update state.
@before_model
def trim_history(state: AgentState, runtime: Runtime) -> dict | None:
messages = state["messages"]
if len(messages) <= 30:
return None
# Keep the first message and the 20 most recent ones (permanently).
return {"messages": [RemoveMessage(id=REMOVE_ALL_MESSAGES), messages[0], *messages[-20:]]}
# 2. dynamic_prompt: build the system prompt per call from context or state.
@dynamic_prompt
def prompt_for_tier(request: ModelRequest) -> str:
return f"You are a support agent. Customer tier: {request.runtime.context.tier}. Be brief."
# 3. wrap_model_call: wraps the model call itself; override() changes the request.
small_model = init_chat_model("anthropic:claude-haiku-5-5")
@wrap_model_call
def cheap_model_for_short_chats(request: ModelRequest, handler) -> ModelResponse:
if len(request.messages) < 4:
request = request.override(model=small_model)
return handler(request)
# 4. wrap_tool_call: wraps each tool execution.
@wrap_tool_call
def tool_errors_to_messages(request, handler):
try:
return handler(request)
except Exception as exc:
return ToolMessage(content=f"Tool failed: {exc}. Try another approach.",
tool_call_id=request.tool_call["id"])
# 5. after_model: inspect the response; jump_to="end" stops the loop.
@after_model(can_jump_to=["end"])
def block_secrets(state: AgentState, runtime: Runtime) -> dict | None:
last = state["messages"][-1]
if isinstance(last, AIMessage) and "API_KEY" in str(last.content):
return {"messages": [AIMessage("I can't share that.")], "jump_to": "end"}
return None
agent = create_agent(
"anthropic:claude-sonnet-5-5",
tools=[lookup_order],
middleware=[trim_history, prompt_for_tier, cheap_model_for_short_chats,
tool_errors_to_messages, block_secrets],
context_schema=Context,
)
result = agent.invoke(
{"messages": [{"role": "user", "content": "Where is order 42?"}]},
context=Context(tier="gold"),
)
print(result["messages"][-1].content)

Notes on the hooks used:

  • before_model / after_model return None (no change) or a state update. Updates go through the state’s reducers, which is why trimming uses RemoveMessage(id=REMOVE_ALL_MESSAGES) followed by the messages to keep.
  • jump_to is allowed only when the hook declares it: @after_model(can_jump_to=["end"]). Targets are "end", "model" and "tools".
  • wrap_model_call receives a ModelRequest (model, messages, system_message, tools, tool_choice, response_format, state, runtime, model_settings). request.override(...) returns a modified copy, so the change applies to this call only.
  • dynamic_prompt is a shortcut for a wrap_model_call that replaces the system prompt.
  • wrap_tool_call receives a ToolCallRequest (request.tool_call is the call dict) and returns a ToolMessage or a Command. For the common case of turning tool exceptions into messages, the built-in ToolErrorMiddleware does the same job with less code:
from langchain.agents.middleware import ToolErrorMiddleware
def describe(exc: Exception, request) -> str | None:
if isinstance(exc, ValueError):
return f"{request.tool_call['name']} failed: {exc}. Try another order ID."
return None # anything else propagates and fails the run
tool_errors = ToolErrorMiddleware(on_error=describe, tools=["lookup_order"])

Async agents (ainvoke, astream) use async functions with the same decorators.

Subclass AgentMiddleware when a middleware needs configuration, several hooks, or its own state keys. Declaring state_schema adds fields to the agent’s state:

from typing import Any
from typing_extensions import NotRequired
from langchain.agents import create_agent, AgentState
from langchain.agents.middleware import AgentMiddleware, ModelRequest, ModelResponse
from langgraph.runtime import Runtime
class UsageState(AgentState):
model_calls: NotRequired[int]
class UsageMiddleware(AgentMiddleware[UsageState]):
"""Counts model calls in state and shows the model only recent history."""
state_schema = UsageState
def __init__(self, window: int = 20) -> None:
super().__init__()
self.window = window
def wrap_model_call(self, request: ModelRequest, handler) -> ModelResponse:
# Trim what the model sees for this call only; state keeps every message.
return handler(request.override(messages=request.messages[-self.window:]))
def after_model(self, state: UsageState, runtime: Runtime) -> dict[str, Any] | None:
return {"model_calls": state.get("model_calls", 0) + 1}
agent = create_agent("anthropic:claude-sonnet-5-5", tools=[], middleware=[UsageMiddleware(window=10)])
result = agent.invoke({"messages": [{"role": "user", "content": "Hello"}]})
print(result["model_calls"]) # 1

All of these are in langchain.agents.middleware.

MiddlewareWhat it does
HumanInTheLoopMiddleware(interrupt_on={...})Pauses before chosen tools run, for approve / edit / reject / respond decisions. Needs a checkpointer.
SummarizationMiddleware(model, trigger=..., keep=...)Replaces older messages with a summary once a token, message-count or context-fraction threshold is reached
ContextEditingMiddleware(edits=[ClearToolUsesEdit(...)])Clears old tool results from the context instead of summarising
ModelCallLimitMiddleware(run_limit=..., thread_limit=...)Caps model calls per run or per thread; exit_behavior="end" or "error"
ToolCallLimitMiddleware(tool_name=..., run_limit=..., thread_limit=...)Caps calls to one tool (or all tools)
ModelRetryMiddleware(max_retries=...), ToolRetryMiddleware(...)Retries failed model or tool calls with exponential backoff
ModelFallbackMiddleware(first_model, *others)Tries other models in order when the main one fails
ToolErrorMiddleware(on_error=fn, tools=None)Calls fn(exc, request) for each exception a tool raises; return text to send it to the model as an error ToolMessage, or None to let it propagate. Argument-validation errors never reach it (ToolNode handles those first)
PIIMiddleware(pii_type, strategy=...)Detects e-mail addresses, card numbers, IPs, MAC addresses, URLs or a custom pattern, and blocks, redacts, masks or hashes them
LLMToolSelectorMiddleware(model=..., max_tools=...)Uses a small model to choose which tools to offer when there are many
ProviderToolSearchMiddleware(searchable_tools=[...])Defers the named tools behind the provider’s server-side tool search (Anthropic and OpenAI only), so their schemas are not sent on every call
ModelRefusalMiddleware()Raises ModelRefusalError when a response contains a standard refusal content block, instead of returning it as a normal answer. See Treating refusals as errors
ModelRouterMiddleware(choices=..., instructions=..., classifier=...)Beta. Classifies the latest human message once per run with a decision model and uses the chosen ModelChoice for every model call in that run. Classifier failures end the run rather than falling back silently
TodoListMiddleware()Gives the agent a write_todos planning tool
ShellToolMiddleware, FilesystemFileSearchMiddleware, LLMToolEmulatorShell and file-search tools; emulated tools for testing

A support agent that combines several:

from langchain.agents import create_agent
from langchain.agents.middleware import (
HumanInTheLoopMiddleware, ModelCallLimitMiddleware, ModelFallbackMiddleware,
ModelRetryMiddleware, PIIMiddleware, SummarizationMiddleware, ToolCallLimitMiddleware,
)
from langchain.tools import tool
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.types import Command
@tool
def send_email(to: str, body: str) -> str:
"""Send an email."""
return f"sent to {to}"
@tool
def search_docs(query: str) -> str:
"""Search internal documentation."""
return "Refunds take 5 working days."
agent = create_agent(
"anthropic:claude-sonnet-5-5",
tools=[send_email, search_docs],
middleware=[
# Summarise older messages once the history passes ~4,000 tokens; keep the last 20.
SummarizationMiddleware(model="anthropic:claude-haiku-5-5",
trigger=("tokens", 4000), keep=("messages", 20)),
# Mask card numbers in user input before the model sees them.
PIIMiddleware("credit_card", strategy="mask", apply_to_input=True),
# Retry failed model calls, then fall back to another model.
ModelRetryMiddleware(max_retries=2),
ModelFallbackMiddleware("anthropic:claude-haiku-5-5"),
# Hard limits: at most 10 model calls per run, 3 e-mails per thread.
ModelCallLimitMiddleware(run_limit=10, exit_behavior="end"),
ToolCallLimitMiddleware(tool_name="send_email", thread_limit=3),
# Pause for approval before send_email runs; search_docs runs freely.
HumanInTheLoopMiddleware(interrupt_on={
"send_email": {"allowed_decisions": ["approve", "edit", "reject"]},
"search_docs": False,
}),
],
checkpointer=InMemorySaver(), # required for the approval pause
)
config = {"configurable": {"thread_id": "t1"}}
result = agent.invoke(
{"messages": [{"role": "user", "content": "Email the refund policy to sam@example.com"}]},
config,
)
# The run pauses before send_email; the interrupt describes the pending call.
request = result["__interrupt__"][0].value
print(request["action_requests"][0]["name"], request["action_requests"][0]["args"])
# Resume with one decision per pending call: approve, edit or reject.
result = agent.invoke(Command(resume={"decisions": [{"type": "approve"}]}), config)
print(result["messages"][-1].content)

Each decision in resume={"decisions": [...]} is one of:

  • {"type": "approve"}: the call runs as the model wrote it.
  • {"type": "edit", "edited_action": {"name": "send_email", "args": {...}}}: the edited call runs instead. The tool result is prefixed with a note (edit_notice=, pass None to drop it) telling the model a reviewer changed the call.
  • {"type": "reject", "message": "Not now"}: the tool does not run; the model receives an error ToolMessage framed as the user’s rejection. Without message, it is told not to retry unless asked.
  • {"type": "respond", "message": "..."}: the tool does not run; your text comes back as a successful ToolMessage. Use it for “ask the user” tools whose real implementation is a human answer.

In the default interrupt_mode="batched", one interrupt covers every gated call from a model turn, and decisions must hold one entry per entry in action_requests, in the same order. interrupt_on={"tool": True} allows all four decisions.

By default a refusal comes back as an ordinary AIMessage. ModelRefusalMiddleware raises ModelRefusalError instead. List it after ModelFallbackMiddleware so the fallback can catch the error and try another model. It only reacts to explicit refusal content blocks; it does not infer refusals from text. With streaming, content may already have been emitted before the refusal is known.

from langchain.agents import create_agent
from langchain.agents.middleware import ModelFallbackMiddleware, ModelRefusalMiddleware
from langchain_core.exceptions import ModelRefusalError
# Without a fallback, the error reaches your code:
agent = create_agent("anthropic:claude-sonnet-5-5", tools=[], middleware=[ModelRefusalMiddleware()])
try:
agent.invoke({"messages": [("user", "...")]})
except ModelRefusalError as exc:
print("refused:", exc)
# With a fallback listed first, a refusal triggers the second model:
agent = create_agent(
"anthropic:claude-sonnet-5-5",
tools=[],
middleware=[ModelFallbackMiddleware("openai:gpt-6.1-sol"), ModelRefusalMiddleware()],
)

A wrap_model_call hook normally returns a ModelResponse. To also write state keys, return ExtendedModelResponse(model_response=response, command=Command(update={...})) (import it from langchain.agents.middleware.types). The command is applied after the model node through the graph’s reducers, so messages in it are added alongside the model’s reply, not replacing it. For plain keys, an outer middleware’s command overwrites an inner one’s.

from langchain.agents import AgentState
from langchain.agents.middleware import AgentMiddleware
from langchain.agents.middleware.types import ExtendedModelResponse
from langgraph.types import Command
from typing_extensions import NotRequired
class SeenState(AgentState):
calls_seen: NotRequired[int]
class Track(AgentMiddleware[SeenState]):
state_schema = SeenState
def wrap_model_call(self, request, handler):
response = handler(request)
return ExtendedModelResponse(model_response=response, command=Command(update={"calls_seen": 1}))

Approving only some calls, one interrupt per call

Section titled “Approving only some calls, one interrupt per call”

An InterruptOnConfig dict can also carry when, a predicate that receives the ToolCallRequest and returns False to auto-approve that call, and description, a string or a (tool_call, state, runtime) -> str function for the text the reviewer sees. With interrupt_mode="per_call", each gated call raises its own interrupt (value {"type": "tool_approval", "tool_call_id", "name", "args", "description", ...}), and you answer each one with a single decision keyed by interrupt id:

from langchain.agents import create_agent
from langchain.agents.middleware import HumanInTheLoopMiddleware
from langchain.tools import tool
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.types import Command
@tool
def send_email(to: str, body: str) -> str:
"""Send an email."""
return f"sent to {to}"
def external_recipient(request) -> bool:
# request is a ToolCallRequest; returning False auto-approves the call.
return not request.tool_call["args"]["to"].endswith("@example.com")
agent = create_agent(
"anthropic:claude-sonnet-5-5",
tools=[send_email],
middleware=[
HumanInTheLoopMiddleware(
interrupt_on={
"send_email": {
"allowed_decisions": ["approve", "reject"],
"when": external_recipient,
"description": lambda tool_call, state, runtime: f"Email to {tool_call['args']['to']}",
}
},
interrupt_mode="per_call",
)
],
checkpointer=InMemorySaver(),
)
config = {"configurable": {"thread_id": "mail-1"}}
result = agent.invoke(
{"messages": [{"role": "user", "content": "Tell ops the disk is at 91% and tell the customer their refund is on its way"}]},
config,
)
# Only calls to outside addresses pause; ops@example.com was sent already.
for intr in result.get("__interrupt__", []):
print(intr.value["name"], intr.value["args"]["to"], intr.value["description"])
# per_call mode: one Decision per interrupt id, no "decisions" list.
answers = {intr.id: {"type": "reject", "message": "Do not email customers yet."}
for intr in result.get("__interrupt__", [])}
if answers:
result = agent.invoke(Command(resume=answers), config)

With a scripted model that asks for one internal and one external e-mail, the internal one runs straight away, the external one pauses, and after the rejection the model receives an error ToolMessage that says the user rejected the send_email call and quotes the reason.

Chapter 7 covers interrupt() itself, which this middleware uses.

Middleware belongs to create_agent. In a hand-built StateGraph, a node is your own function, so the same needs are met differently:

NeedIn a StateGraph
Retry, timeout or cache a stepadd_node(..., retry_policy=, timeout=, cache_policy=) and set_node_defaults(...): see Node policies
Recover from a failed nodeadd_node(..., error_handler=fn) with a NodeError parameter: see Error handlers
Tool errorsToolNode(handle_tool_errors=...) or ToolNode(wrap_tool_call=...): see Chapter 4
Trim history, guardrailsDo it in the node, or add a node before or after the model node
Approvalinterrupt() in a node: see Chapter 7

You can also use a create_agent agent as a node (or subgraph) inside a larger StateGraph, and keep its middleware.

langgraph.prebuilt.create_react_agent is deprecated (it warns with LangGraphDeprecatedSinceV10 and is due for removal in 2.0). Its pre_model_hook and post_model_hook arguments map onto middleware:

create_react_agentcreate_agent
prompt=system_prompt= or @dynamic_prompt
pre_model_hook returning {"messages": ...}@before_model returning a state update
pre_model_hook returning {"llm_input_messages": ...} (model-only view)@wrap_model_call with request.override(messages=...)
post_model_hook@after_model
state_schema= (must include remaining_steps)state_schema= subclassing langchain.agents.AgentState, or a middleware state_schema
interrupt_before=["tools"]HumanInTheLoopMiddleware
response_format=response_format=

For reference, the legacy form looks like this:

from langchain_core.messages.utils import trim_messages, count_tokens_approximately
from langgraph.prebuilt import create_react_agent # deprecated: emits LangGraphDeprecatedSinceV10
def pre_model_hook(state):
trimmed = trim_messages(state["messages"], strategy="last", max_tokens=2000,
token_counter=count_tokens_approximately, start_on="human")
return {"llm_input_messages": trimmed} # what the model sees; state is unchanged
legacy = create_react_agent("anthropic:claude-sonnet-5-5", tools=[], pre_model_hook=pre_model_hook)
print(legacy.invoke({"messages": [{"role": "user", "content": "Hi"}]})["messages"][-1].content)

Next: Chapter 9 — Advanced patterns.