Chapter 8 — Agent Middleware
Customise a create_agent agent with middleware: hooks before and after model calls, wrappers around model and tool calls, and the built-in middleware for approvals, summarisation, limits, retries and PII.
What you’ll learn: how to change what a create_agent agent does on each step (trim history, pick a model, handle tool errors, stop the loop, pause for approval) with middleware, and which middleware ships with langchain.
Checked against langchain 1.5.0 and langgraph 1.2.22. Time: ~25 minutes.
Middleware is a langchain feature: create_agent and langchain.agents.middleware live in the langchain package (pip install langchain), and create_agent compiles to a LangGraph graph. If you build your own StateGraph instead, the equivalents are node policies and error handlers; see In your own graph below.
Where middleware runs
Section titled “Where middleware runs”create_agent runs a loop: call the model, run any tools it asked for, repeat until the model answers without tool calls. Middleware attaches at fixed points in that loop.
| Hook | Runs | Typical use |
|---|---|---|
before_agent | Once, when the run starts | Load user data, validate input |
before_model | Before every model call; can update state | Trim or summarise history |
wrap_model_call | Around every model call; gets the request and a handler | Change model, prompt, tools or messages for one call; retries; fallbacks; logging |
after_model | After every model call; can update state or jump | Guardrails on the response, counters |
wrap_tool_call | Around every tool execution | Tool error handling, retries, approval |
after_agent | Once, when the run ends | Clean-up, logging |
With several middleware in middleware=[A, B], the before_* hooks run in list order (A then B), the after_* hooks in reverse (B then A), and the wrap_* hooks nest with A outermost.
Writing middleware with decorators
Section titled “Writing middleware with decorators”Each decorator turns a function into a middleware object. This agent uses five of them:
from dataclasses import dataclass
from langchain.agents import create_agent, AgentStatefrom langchain.agents.middleware import ( before_model, after_model, dynamic_prompt, wrap_model_call, wrap_tool_call, ModelRequest, ModelResponse,)from langchain.chat_models import init_chat_modelfrom langchain.messages import AIMessage, RemoveMessage, ToolMessagefrom langchain.tools import toolfrom langgraph.graph.message import REMOVE_ALL_MESSAGESfrom langgraph.runtime import Runtime
@dataclassclass Context: tier: str = "standard"
@tooldef lookup_order(order_id: str) -> str: """Look up an order by ID.""" if order_id == "0": raise ValueError("order not found") return f"Order {order_id}: shipped"
# 1. before_model: runs before every model call and can update state.@before_modeldef trim_history(state: AgentState, runtime: Runtime) -> dict | None: messages = state["messages"] if len(messages) <= 30: return None # Keep the first message and the 20 most recent ones (permanently). return {"messages": [RemoveMessage(id=REMOVE_ALL_MESSAGES), messages[0], *messages[-20:]]}
# 2. dynamic_prompt: build the system prompt per call from context or state.@dynamic_promptdef prompt_for_tier(request: ModelRequest) -> str: return f"You are a support agent. Customer tier: {request.runtime.context.tier}. Be brief."
# 3. wrap_model_call: wraps the model call itself; override() changes the request.small_model = init_chat_model("anthropic:claude-haiku-5-5")
@wrap_model_calldef cheap_model_for_short_chats(request: ModelRequest, handler) -> ModelResponse: if len(request.messages) < 4: request = request.override(model=small_model) return handler(request)
# 4. wrap_tool_call: wraps each tool execution.@wrap_tool_calldef tool_errors_to_messages(request, handler): try: return handler(request) except Exception as exc: return ToolMessage(content=f"Tool failed: {exc}. Try another approach.", tool_call_id=request.tool_call["id"])
# 5. after_model: inspect the response; jump_to="end" stops the loop.@after_model(can_jump_to=["end"])def block_secrets(state: AgentState, runtime: Runtime) -> dict | None: last = state["messages"][-1] if isinstance(last, AIMessage) and "API_KEY" in str(last.content): return {"messages": [AIMessage("I can't share that.")], "jump_to": "end"} return None
agent = create_agent( "anthropic:claude-sonnet-5-5", tools=[lookup_order], middleware=[trim_history, prompt_for_tier, cheap_model_for_short_chats, tool_errors_to_messages, block_secrets], context_schema=Context,)
result = agent.invoke( {"messages": [{"role": "user", "content": "Where is order 42?"}]}, context=Context(tier="gold"),)print(result["messages"][-1].content)Notes on the hooks used:
before_model/after_modelreturnNone(no change) or a state update. Updates go through the state’s reducers, which is why trimming usesRemoveMessage(id=REMOVE_ALL_MESSAGES)followed by the messages to keep.jump_tois allowed only when the hook declares it:@after_model(can_jump_to=["end"]). Targets are"end","model"and"tools".wrap_model_callreceives aModelRequest(model,messages,system_message,tools,tool_choice,response_format,state,runtime,model_settings).request.override(...)returns a modified copy, so the change applies to this call only.dynamic_promptis a shortcut for awrap_model_callthat replaces the system prompt.wrap_tool_callreceives aToolCallRequest(request.tool_callis the call dict) and returns aToolMessageor aCommand. For the common case of turning tool exceptions into messages, the built-inToolErrorMiddlewaredoes the same job with less code:
from langchain.agents.middleware import ToolErrorMiddleware
def describe(exc: Exception, request) -> str | None: if isinstance(exc, ValueError): return f"{request.tool_call['name']} failed: {exc}. Try another order ID." return None # anything else propagates and fails the run
tool_errors = ToolErrorMiddleware(on_error=describe, tools=["lookup_order"])Async agents (ainvoke, astream) use async functions with the same decorators.
Class-based middleware
Section titled “Class-based middleware”Subclass AgentMiddleware when a middleware needs configuration, several hooks, or its own state keys. Declaring state_schema adds fields to the agent’s state:
from typing import Anyfrom typing_extensions import NotRequired
from langchain.agents import create_agent, AgentStatefrom langchain.agents.middleware import AgentMiddleware, ModelRequest, ModelResponsefrom langgraph.runtime import Runtime
class UsageState(AgentState): model_calls: NotRequired[int]
class UsageMiddleware(AgentMiddleware[UsageState]): """Counts model calls in state and shows the model only recent history."""
state_schema = UsageState
def __init__(self, window: int = 20) -> None: super().__init__() self.window = window
def wrap_model_call(self, request: ModelRequest, handler) -> ModelResponse: # Trim what the model sees for this call only; state keeps every message. return handler(request.override(messages=request.messages[-self.window:]))
def after_model(self, state: UsageState, runtime: Runtime) -> dict[str, Any] | None: return {"model_calls": state.get("model_calls", 0) + 1}
agent = create_agent("anthropic:claude-sonnet-5-5", tools=[], middleware=[UsageMiddleware(window=10)])result = agent.invoke({"messages": [{"role": "user", "content": "Hello"}]})print(result["model_calls"]) # 1Built-in middleware
Section titled “Built-in middleware”All of these are in langchain.agents.middleware.
| Middleware | What it does |
|---|---|
HumanInTheLoopMiddleware(interrupt_on={...}) | Pauses before chosen tools run, for approve / edit / reject / respond decisions. Needs a checkpointer. |
SummarizationMiddleware(model, trigger=..., keep=...) | Replaces older messages with a summary once a token, message-count or context-fraction threshold is reached |
ContextEditingMiddleware(edits=[ClearToolUsesEdit(...)]) | Clears old tool results from the context instead of summarising |
ModelCallLimitMiddleware(run_limit=..., thread_limit=...) | Caps model calls per run or per thread; exit_behavior="end" or "error" |
ToolCallLimitMiddleware(tool_name=..., run_limit=..., thread_limit=...) | Caps calls to one tool (or all tools) |
ModelRetryMiddleware(max_retries=...), ToolRetryMiddleware(...) | Retries failed model or tool calls with exponential backoff |
ModelFallbackMiddleware(first_model, *others) | Tries other models in order when the main one fails |
ToolErrorMiddleware(on_error=fn, tools=None) | Calls fn(exc, request) for each exception a tool raises; return text to send it to the model as an error ToolMessage, or None to let it propagate. Argument-validation errors never reach it (ToolNode handles those first) |
PIIMiddleware(pii_type, strategy=...) | Detects e-mail addresses, card numbers, IPs, MAC addresses, URLs or a custom pattern, and blocks, redacts, masks or hashes them |
LLMToolSelectorMiddleware(model=..., max_tools=...) | Uses a small model to choose which tools to offer when there are many |
ProviderToolSearchMiddleware(searchable_tools=[...]) | Defers the named tools behind the provider’s server-side tool search (Anthropic and OpenAI only), so their schemas are not sent on every call |
ModelRefusalMiddleware() | Raises ModelRefusalError when a response contains a standard refusal content block, instead of returning it as a normal answer. See Treating refusals as errors |
ModelRouterMiddleware(choices=..., instructions=..., classifier=...) | Beta. Classifies the latest human message once per run with a decision model and uses the chosen ModelChoice for every model call in that run. Classifier failures end the run rather than falling back silently |
TodoListMiddleware() | Gives the agent a write_todos planning tool |
ShellToolMiddleware, FilesystemFileSearchMiddleware, LLMToolEmulator | Shell and file-search tools; emulated tools for testing |
A support agent that combines several:
from langchain.agents import create_agentfrom langchain.agents.middleware import ( HumanInTheLoopMiddleware, ModelCallLimitMiddleware, ModelFallbackMiddleware, ModelRetryMiddleware, PIIMiddleware, SummarizationMiddleware, ToolCallLimitMiddleware,)from langchain.tools import toolfrom langgraph.checkpoint.memory import InMemorySaverfrom langgraph.types import Command
@tooldef send_email(to: str, body: str) -> str: """Send an email.""" return f"sent to {to}"
@tooldef search_docs(query: str) -> str: """Search internal documentation.""" return "Refunds take 5 working days."
agent = create_agent( "anthropic:claude-sonnet-5-5", tools=[send_email, search_docs], middleware=[ # Summarise older messages once the history passes ~4,000 tokens; keep the last 20. SummarizationMiddleware(model="anthropic:claude-haiku-5-5", trigger=("tokens", 4000), keep=("messages", 20)), # Mask card numbers in user input before the model sees them. PIIMiddleware("credit_card", strategy="mask", apply_to_input=True), # Retry failed model calls, then fall back to another model. ModelRetryMiddleware(max_retries=2), ModelFallbackMiddleware("anthropic:claude-haiku-5-5"), # Hard limits: at most 10 model calls per run, 3 e-mails per thread. ModelCallLimitMiddleware(run_limit=10, exit_behavior="end"), ToolCallLimitMiddleware(tool_name="send_email", thread_limit=3), # Pause for approval before send_email runs; search_docs runs freely. HumanInTheLoopMiddleware(interrupt_on={ "send_email": {"allowed_decisions": ["approve", "edit", "reject"]}, "search_docs": False, }), ], checkpointer=InMemorySaver(), # required for the approval pause)
config = {"configurable": {"thread_id": "t1"}}result = agent.invoke( {"messages": [{"role": "user", "content": "Email the refund policy to sam@example.com"}]}, config,)
# The run pauses before send_email; the interrupt describes the pending call.request = result["__interrupt__"][0].valueprint(request["action_requests"][0]["name"], request["action_requests"][0]["args"])
# Resume with one decision per pending call: approve, edit or reject.result = agent.invoke(Command(resume={"decisions": [{"type": "approve"}]}), config)print(result["messages"][-1].content)Each decision in resume={"decisions": [...]} is one of:
{"type": "approve"}: the call runs as the model wrote it.{"type": "edit", "edited_action": {"name": "send_email", "args": {...}}}: the edited call runs instead. The tool result is prefixed with a note (edit_notice=, passNoneto drop it) telling the model a reviewer changed the call.{"type": "reject", "message": "Not now"}: the tool does not run; the model receives an errorToolMessageframed as the user’s rejection. Withoutmessage, it is told not to retry unless asked.{"type": "respond", "message": "..."}: the tool does not run; your text comes back as a successfulToolMessage. Use it for “ask the user” tools whose real implementation is a human answer.
In the default interrupt_mode="batched", one interrupt covers every gated call from a model turn, and decisions must hold one entry per entry in action_requests, in the same order. interrupt_on={"tool": True} allows all four decisions.
Treating refusals as errors
Section titled “Treating refusals as errors”By default a refusal comes back as an ordinary AIMessage. ModelRefusalMiddleware raises ModelRefusalError instead. List it after ModelFallbackMiddleware so the fallback can catch the error and try another model. It only reacts to explicit refusal content blocks; it does not infer refusals from text. With streaming, content may already have been emitted before the refusal is known.
from langchain.agents import create_agentfrom langchain.agents.middleware import ModelFallbackMiddleware, ModelRefusalMiddlewarefrom langchain_core.exceptions import ModelRefusalError
# Without a fallback, the error reaches your code:agent = create_agent("anthropic:claude-sonnet-5-5", tools=[], middleware=[ModelRefusalMiddleware()])try: agent.invoke({"messages": [("user", "...")]})except ModelRefusalError as exc: print("refused:", exc)
# With a fallback listed first, a refusal triggers the second model:agent = create_agent( "anthropic:claude-sonnet-5-5", tools=[], middleware=[ModelFallbackMiddleware("openai:gpt-6.1-sol"), ModelRefusalMiddleware()],)Updating state from wrap_model_call
Section titled “Updating state from wrap_model_call”A wrap_model_call hook normally returns a ModelResponse. To also write state keys, return ExtendedModelResponse(model_response=response, command=Command(update={...})) (import it from langchain.agents.middleware.types). The command is applied after the model node through the graph’s reducers, so messages in it are added alongside the model’s reply, not replacing it. For plain keys, an outer middleware’s command overwrites an inner one’s.
from langchain.agents import AgentStatefrom langchain.agents.middleware import AgentMiddlewarefrom langchain.agents.middleware.types import ExtendedModelResponsefrom langgraph.types import Commandfrom typing_extensions import NotRequired
class SeenState(AgentState): calls_seen: NotRequired[int]
class Track(AgentMiddleware[SeenState]): state_schema = SeenState
def wrap_model_call(self, request, handler): response = handler(request) return ExtendedModelResponse(model_response=response, command=Command(update={"calls_seen": 1}))Approving only some calls, one interrupt per call
Section titled “Approving only some calls, one interrupt per call”An InterruptOnConfig dict can also carry when, a predicate that receives the ToolCallRequest and returns False to auto-approve that call, and description, a string or a (tool_call, state, runtime) -> str function for the text the reviewer sees. With interrupt_mode="per_call", each gated call raises its own interrupt (value {"type": "tool_approval", "tool_call_id", "name", "args", "description", ...}), and you answer each one with a single decision keyed by interrupt id:
from langchain.agents import create_agentfrom langchain.agents.middleware import HumanInTheLoopMiddlewarefrom langchain.tools import toolfrom langgraph.checkpoint.memory import InMemorySaverfrom langgraph.types import Command
@tooldef send_email(to: str, body: str) -> str: """Send an email.""" return f"sent to {to}"
def external_recipient(request) -> bool: # request is a ToolCallRequest; returning False auto-approves the call. return not request.tool_call["args"]["to"].endswith("@example.com")
agent = create_agent( "anthropic:claude-sonnet-5-5", tools=[send_email], middleware=[ HumanInTheLoopMiddleware( interrupt_on={ "send_email": { "allowed_decisions": ["approve", "reject"], "when": external_recipient, "description": lambda tool_call, state, runtime: f"Email to {tool_call['args']['to']}", } }, interrupt_mode="per_call", ) ], checkpointer=InMemorySaver(),)
config = {"configurable": {"thread_id": "mail-1"}}result = agent.invoke( {"messages": [{"role": "user", "content": "Tell ops the disk is at 91% and tell the customer their refund is on its way"}]}, config,)
# Only calls to outside addresses pause; ops@example.com was sent already.for intr in result.get("__interrupt__", []): print(intr.value["name"], intr.value["args"]["to"], intr.value["description"])
# per_call mode: one Decision per interrupt id, no "decisions" list.answers = {intr.id: {"type": "reject", "message": "Do not email customers yet."} for intr in result.get("__interrupt__", [])}if answers: result = agent.invoke(Command(resume=answers), config)With a scripted model that asks for one internal and one external e-mail, the internal one runs straight away, the external one pauses, and after the rejection the model receives an error ToolMessage that says the user rejected the send_email call and quotes the reason.
Chapter 7 covers interrupt() itself, which this middleware uses.
In your own graph
Section titled “In your own graph”Middleware belongs to create_agent. In a hand-built StateGraph, a node is your own function, so the same needs are met differently:
| Need | In a StateGraph |
|---|---|
| Retry, timeout or cache a step | add_node(..., retry_policy=, timeout=, cache_policy=) and set_node_defaults(...): see Node policies |
| Recover from a failed node | add_node(..., error_handler=fn) with a NodeError parameter: see Error handlers |
| Tool errors | ToolNode(handle_tool_errors=...) or ToolNode(wrap_tool_call=...): see Chapter 4 |
| Trim history, guardrails | Do it in the node, or add a node before or after the model node |
| Approval | interrupt() in a node: see Chapter 7 |
You can also use a create_agent agent as a node (or subgraph) inside a larger StateGraph, and keep its middleware.
Migrating from create_react_agent hooks
Section titled “Migrating from create_react_agent hooks”langgraph.prebuilt.create_react_agent is deprecated (it warns with LangGraphDeprecatedSinceV10 and is due for removal in 2.0). Its pre_model_hook and post_model_hook arguments map onto middleware:
create_react_agent | create_agent |
|---|---|
prompt= | system_prompt= or @dynamic_prompt |
pre_model_hook returning {"messages": ...} | @before_model returning a state update |
pre_model_hook returning {"llm_input_messages": ...} (model-only view) | @wrap_model_call with request.override(messages=...) |
post_model_hook | @after_model |
state_schema= (must include remaining_steps) | state_schema= subclassing langchain.agents.AgentState, or a middleware state_schema |
interrupt_before=["tools"] | HumanInTheLoopMiddleware |
response_format= | response_format= |
For reference, the legacy form looks like this:
from langchain_core.messages.utils import trim_messages, count_tokens_approximatelyfrom langgraph.prebuilt import create_react_agent # deprecated: emits LangGraphDeprecatedSinceV10
def pre_model_hook(state): trimmed = trim_messages(state["messages"], strategy="last", max_tokens=2000, token_counter=count_tokens_approximately, start_on="human") return {"llm_input_messages": trimmed} # what the model sees; state is unchanged
legacy = create_react_agent("anthropic:claude-sonnet-5-5", tools=[], pre_model_hook=pre_model_hook)print(legacy.invoke({"messages": [{"role": "user", "content": "Hi"}]})["messages"][-1].content)