Evaluation
Test agents programmatically with AgentEvaluator, EvalCase, EvalSet, and the built-in metric suite.
Verified against google-adk 2.13.0 (google/adk/evaluation/), checked 2026-11-19.
ADK ships a first-class evaluation framework built around three concepts: EvalCase (a single conversation to run), EvalSet (a collection of cases), and AgentEvaluator (the engine that runs cases against a live agent and scores the results). The framework integrates with pytest and supports custom metrics.
Minimal example
Section titled “Minimal example”import asyncioimport pytestfrom google.adk.evaluation.agent_evaluator import AgentEvaluatorfrom google.adk.evaluation.eval_case import EvalCase, Invocation, SessionInputfrom google.adk.evaluation.eval_set import EvalSetfrom google.adk.evaluation.eval_metrics import PrebuiltMetricsfrom google.adk.evaluation.eval_config import EvalConfigfrom google.genai import types
# Define a single-turn eval casecase = EvalCase( eval_id="add_two_numbers", conversation=[ Invocation( user_content=types.Content( role="user", parts=[types.Part(text="What is 15 + 27?")], ), final_response=types.Content( role="model", parts=[types.Part(text="42")], ), ) ],)
eval_set = EvalSet( eval_set_id="arithmetic_suite", eval_cases=[case],)
eval_config = EvalConfig( criteria={ PrebuiltMetrics.RESPONSE_MATCH_SCORE.value: 0.8, })
# Run — agent_module must expose `root_agent` or `get_agent_async`@pytest.mark.asyncioasync def test_arithmetic(): await AgentEvaluator.evaluate_eval_set( agent_module="my_package.agent", eval_set=eval_set, eval_config=eval_config, num_runs=1, )From the command line
Section titled “From the command line”adk eval runs eval set files against an agent folder (one whose agent.py defines root_agent) and prints a pass/fail summary per set. Results are also written under the agent’s .adk/eval_history/. From 2.13.0 the command exits with status 1 when any case fails (or when nothing ran) and 0 when all pass, so a CI step can rely on it; earlier releases always exited 0.
pip install "google-adk[eval]"
# Pass thresholds per metricecho '{"criteria": {"tool_trajectory_avg_score": 1.0, "response_match_score": 0.8}}' > my_agent/test_config.json
adk eval my_agent my_agent/tests/arithmetic_suite.evalset.json --config_file_path my_agent/test_config.jsonadk eval my_agent my_agent/tests/arithmetic_suite.evalset.json:add_two_numbers # one case onlyadk eval my_agent my_agent/tests/*.evalset.json --print_detailed_resultsadk eval_set create my_agent regression creates an empty regression.evalset.json in the agent folder, and adk eval_set add_eval_case / generate_eval_cases add cases from conversation scenarios. In the Dev UI (adk web), the Eval tab saves a live session as an eval case and runs sets interactively. Without --config_file_path, and with a single eval set file, ADK looks for a test_config.json next to that file; otherwise it uses its default criteria.
EvalCase
Section titled “EvalCase”The atomic unit. Defined in evaluation/eval_case.py.
from google.adk.evaluation.eval_case import ( EvalCase, Invocation, SessionInput, IntermediateData)from google.genai import types
case = EvalCase( eval_id="weather_lookup", # unique within an EvalSet session_input=SessionInput( # optional initial state app_name="weather_app", user_id="test_user", state={"preferred_units": "metric"}, ), conversation=[ Invocation( user_content=types.Content( role="user", parts=[types.Part(text="What's the weather in London?")], ), final_response=types.Content( role="model", parts=[types.Part(text="It's currently 18°C and partly cloudy.")], ), intermediate_data=IntermediateData( tool_uses=[ types.FunctionCall(name="get_weather", args={"city": "London"}), ], ), ), ], final_session_state={"last_city": "London"}, # optional; asserted after the run)Multi-turn conversation
Section titled “Multi-turn conversation”case = EvalCase( eval_id="two_turn_booking", conversation=[ Invocation( user_content=types.Content( role="user", parts=[types.Part(text="Book a table for 2 at 7pm.")], ), final_response=types.Content( role="model", parts=[types.Part(text="Which restaurant?")], ), ), Invocation( user_content=types.Content( role="user", parts=[types.Part(text="La Trattoria.")], ), final_response=types.Content( role="model", parts=[types.Part(text="Done! Table booked at La Trattoria for 2 at 7pm.")], ), intermediate_data=IntermediateData( tool_uses=[ types.FunctionCall( name="book_table", args={"restaurant": "La Trattoria", "covers": 2, "time": "19:00"}, ), ], ), ), ],)Invocation fields:
| Field | Type | Purpose |
|---|---|---|
user_content | types.Content | The user message for this turn |
final_response | types.Content | None | Expected final agent response (used by response metrics) |
intermediate_data | IntermediateData | None | Expected tool calls + responses (used by trajectory metrics) |
rubrics | list[Rubric] | None | Per-invocation rubrics (used by rubric_based_* metrics) |
app_details | AppDetails | None | Override app name / user id for this invocation |
EvalSet
Section titled “EvalSet”A collection of EvalCase objects. Defined in evaluation/eval_set.py.
from google.adk.evaluation.eval_set import EvalSet
eval_set = EvalSet( eval_set_id="full_regression", name="Full regression suite", description="Tests the booking and weather sub-agents.", eval_cases=[case], # replace with your list of EvalCase objects)
# Serialise to JSON file for reusewith open("eval_data/full_regression.evalset.json", "w") as f: f.write(eval_set.model_dump_json(indent=2))
# Load from JSON filefrom google.adk.evaluation.eval_set import EvalSetwith open("eval_data/full_regression.evalset.json") as f: eval_set = EvalSet.model_validate_json(f.read())EvalConfig and metrics
Section titled “EvalConfig and metrics”EvalConfig maps metric names to thresholds or criterion objects. Defined in evaluation/eval_config.py.
from google.adk.evaluation.eval_config import EvalConfigfrom google.adk.evaluation.eval_metrics import ( PrebuiltMetrics, BaseCriterion, LlmAsAJudgeCriterion, ToolTrajectoryCriterion, JudgeModelOptions,)
config = EvalConfig( criteria={ # Simple threshold — the metric must score >= value to pass PrebuiltMetrics.RESPONSE_MATCH_SCORE.value: 0.7,
# LLM-as-judge with custom model and sampling PrebuiltMetrics.RESPONSE_EVALUATION_SCORE.value: LlmAsAJudgeCriterion( threshold=0.8, judge_model_options=JudgeModelOptions( judge_model="gemini-4-argon", num_samples=3, ), ),
# Tool trajectory with ordered matching PrebuiltMetrics.TOOL_TRAJECTORY_AVG_SCORE.value: ToolTrajectoryCriterion( threshold=1.0, match_type=ToolTrajectoryCriterion.MatchType.IN_ORDER, ), })Available prebuilt metrics
Section titled “Available prebuilt metrics”All defined in evaluation/eval_metrics.py as PrebuiltMetrics:
| Metric key | Class | What it measures |
|---|---|---|
tool_trajectory_avg_score | ToolTrajectoryCriterion | Whether the agent called the expected tools (EXACT / IN_ORDER / ANY_ORDER) |
response_match_score | BaseCriterion | Lexical similarity between actual and expected final response |
response_evaluation_score | LlmAsAJudgeCriterion | LLM judge rating of response quality |
final_response_match_v2 | LlmAsAJudgeCriterion | Semantic match using an LLM judge (v2, more robust) |
safety_v1 | BaseCriterion | Safety (harmlessness) score; uses the Vertex Gen AI Eval SDK, so needs GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION |
hallucinations_v1 | HallucinationsCriterion | Detects factual hallucinations |
rubric_based_final_response_quality_v1 | RubricsBasedCriterion | Rubric-scored response quality |
rubric_based_tool_use_quality_v1 | RubricsBasedCriterion | Rubric-scored tool selection |
multi_turn_task_success_v1 | — | Whether a multi-turn task succeeded end-to-end |
multi_turn_trajectory_quality_v1 | — | Quality of the full multi-turn trajectory |
multi_turn_tool_use_quality_v1 | — | Tool use quality across all turns |
rubric_based_multi_turn_trajectory_quality_v1 | RubricsBasedCriterion | The multi-turn trajectory scored against your rubrics by an LLM judge |
per_turn_user_simulator_quality_v1 | LlmBackedUserSimulatorCriterion | Whether each simulated user message follows the conversation scenario |
tool_call_count_v1 | — | Informational: tool calls per invocation, averaged over the case |
inference_call_count_v1 | — | Informational: model calls per invocation |
token_usage_v1 | — | Informational: tokens per invocation, with an input/output breakdown |
invocation_duration_v1 | — | Informational: wall-clock seconds per invocation |
The four efficiency metrics at the end are added to every eval automatically and appear in results and CSV output with the status INFORMATIONAL; they never pass or fail. Leave them out of criteria: naming one there gives it a threshold, which these metrics reject, so the metric errors and the run fails with “was not evaluated”.
ToolTrajectoryCriterion match types
Section titled “ToolTrajectoryCriterion match types”from google.adk.evaluation.eval_metrics import ToolTrajectoryCriterion
# EXACT — actual calls must match expected calls preciselyToolTrajectoryCriterion(threshold=1.0, match_type=ToolTrajectoryCriterion.MatchType.EXACT)
# IN_ORDER — expected calls must appear in the actual trajectory in order# (extra calls are allowed between them)ToolTrajectoryCriterion(threshold=1.0, match_type=ToolTrajectoryCriterion.MatchType.IN_ORDER)
# ANY_ORDER — all expected calls must appear, order doesn't matterToolTrajectoryCriterion(threshold=1.0, match_type=ToolTrajectoryCriterion.MatchType.ANY_ORDER)AgentEvaluator
Section titled “AgentEvaluator”The engine. All methods are @staticmethod. Defined in evaluation/agent_evaluator.py (re-checked against 2.13.0).
evaluate_eval_set — programmatic, in-memory
Section titled “evaluate_eval_set — programmatic, in-memory”from google.adk.evaluation.agent_evaluator import AgentEvaluator
await AgentEvaluator.evaluate_eval_set( agent_module="my_package.agent", # must expose root_agent or get_agent_async eval_set=eval_set, eval_config=eval_config, num_runs=2, # run each case twice; results are averaged agent_name=None, # None → root_agent; set to sub-agent name if needed print_detailed_results=True, # print per-metric breakdown to stdout)num_runs=2 (the default) runs each case twice and averages the scores, improving reliability for non-deterministic models. Increase to 5 for stability-sensitive metrics.
How evaluate_eval_set works internally (source-verified):
- Loads the agent via
_get_agent_for_eval: imports the module (its name must end in.agent, or it must have anagentattribute), usesroot_agentif present and otherwise awaitsget_agent_async(). A module-levelappthat is anAppis used too, so its plugins run during the eval. - Creates an
InMemoryEvalSetsManagerand stores the eval set. - Runs each
EvalCasenum_runstimes — calls the agent via aRunnerfor each turn in the conversation. - After all runs, scores each metric using the registered
MetricEvaluatorRegistry. - Averages scores across runs with
statistics.mean(). - Asserts each metric against its threshold — raises
AssertionErrorif any fails. - Optionally prints a tabular report via pandas/tabulate.
evaluate — file-based
Section titled “evaluate — file-based”await AgentEvaluator.evaluate( agent_module="my_package.agent", eval_dataset_file_path_or_dir="tests/eval_data/", # .test.json or directory num_runs=2, initial_session_file="tests/initial_session.json",)eval_dataset_file_path_or_dir can be:
- A path to a single
.test.jsonfile (old format) or.evalset.jsonfile (newEvalSetformat). - A directory — ADK recursively finds all
*.test.jsonfiles. Note: directory scanning uses the old.test.jsonsuffix only; pass individual.evalset.jsonpaths explicitly.
Agent module conventions
Section titled “Agent module conventions”AgentEvaluator loads the module and looks for (in order):
get_agent_async— an async factory() -> BaseAgent. Checked first.root_agent— a module-levelBaseAgentinstance.
from google.adk.agents import LlmAgentfrom google.adk.tools import google_search
root_agent = LlmAgent( name="research_bot", model="gemini-3.8-flash", instruction="Answer questions using web search.", tools=[google_search],)Or with factory for dependency injection:
async def get_agent_async(): # Can connect to real DBs, inject credentials, etc. db = await create_db_pool() return LlmAgent( name="db_agent", tools=[make_db_tool(db)], )find_config_for_test_file — auto-load eval config
Section titled “find_config_for_test_file — auto-load eval config”When running file-based evals, AgentEvaluator can auto-discover a test_config.json file in the same folder:
# Structure:# test_config.json ← auto-discovered# my_suite.evalset.json
# tests/eval_data/test_config.json{ "criteria": { "tool_trajectory_avg_score": 1.0, "response_match_score": 0.7 }}# Load it manuallyconfig = AgentEvaluator.find_config_for_test_file("tests/eval_data/my_suite.evalset.json")print(config.criteria) # {"tool_trajectory_avg_score": 1.0, ...}Full end-to-end example with result capture
Section titled “Full end-to-end example with result capture”import asynciofrom google.adk.evaluation.agent_evaluator import AgentEvaluatorfrom google.adk.evaluation.eval_case import EvalCase, Invocation, IntermediateDatafrom google.adk.evaluation.eval_set import EvalSetfrom google.adk.evaluation.eval_config import EvalConfigfrom google.adk.evaluation.eval_metrics import PrebuiltMetrics, ToolTrajectoryCriterionfrom google.adk.evaluation.local_eval_set_results_manager import LocalEvalSetResultsManagerfrom google.genai import types
# --- Build eval cases --------------------------------------------------------cases = [ EvalCase( eval_id="unit_conversion", conversation=[ Invocation( user_content=types.Content( role="user", parts=[types.Part(text="Convert 100 Fahrenheit to Celsius.")], ), final_response=types.Content( role="model", parts=[types.Part(text="37.78")], ), intermediate_data=IntermediateData( tool_uses=[ types.FunctionCall( name="convert_temperature", args={"value": 100, "from_unit": "F", "to_unit": "C"}, ) ] ), ) ], ), EvalCase( eval_id="multi_turn_booking", conversation=[ Invocation( user_content=types.Content( role="user", parts=[types.Part(text="Book a flight to Paris.")], ), final_response=types.Content( role="model", parts=[types.Part(text="Which date would you like to travel?")], ), ), Invocation( user_content=types.Content( role="user", parts=[types.Part(text="June 15th.")], ), final_response=types.Content( role="model", parts=[types.Part(text="Done! Flight booked for June 15th to Paris.")], ), intermediate_data=IntermediateData( tool_uses=[ types.FunctionCall( name="book_flight", args={"destination": "Paris", "date": "2026-06-15"}, ) ] ), ), ], ),]
eval_set = EvalSet(eval_set_id="travel_agent_suite", eval_cases=cases)
eval_config = EvalConfig( criteria={ PrebuiltMetrics.TOOL_TRAJECTORY_AVG_SCORE.value: ToolTrajectoryCriterion( threshold=1.0, match_type=ToolTrajectoryCriterion.MatchType.IN_ORDER, ), PrebuiltMetrics.RESPONSE_MATCH_SCORE.value: 0.5, })
# --- Run evaluation ----------------------------------------------------------async def main(): await AgentEvaluator.evaluate_eval_set( agent_module="travel_bench.agent", eval_set=eval_set, eval_config=eval_config, num_runs=2, print_detailed_results=True, )
asyncio.run(main())evaluate_eval_set — CSV output, preloaded artifacts, results manager
Section titled “evaluate_eval_set — CSV output, preloaded artifacts, results manager”Source-verified against google.adk.evaluation.agent_evaluator.evaluate_eval_set (first verified in 2.7.1; re-checked against 2.13.0). Three params are worth calling out because they’re often ignored and each solves a real reporting problem.
import asynciofrom google.adk.evaluation.agent_evaluator import AgentEvaluatorfrom google.adk.evaluation.eval_set_results_manager import EvalSetResultsManagerfrom google.adk.evaluation.local_eval_set_results_manager import LocalEvalSetResultsManagerfrom google.adk.artifacts import InMemoryArtifactServicefrom google.genai import types
# ── artifact_service: preload files that an eval case will reference ─────────artifact_service = InMemoryArtifactService()
# ── eval_set_results_manager: persist evalset_result.json to disk ────────────results_manager: EvalSetResultsManager = LocalEvalSetResultsManager( agents_dir="./eval_results")
async def preload_artifacts() -> None: # Preload one artifact per eval-case session_id — the case must set # SessionInput.session_id to the same value for the load to hit. await artifact_service.save_artifact( app_name="travel_bench", user_id="eval", session_id="case-booking-01", filename="itinerary_template.txt", artifact=types.Part.from_text(text="Departure: {city}\nDate: {date}\n"), )
async def main() -> None: await preload_artifacts() await AgentEvaluator.evaluate_eval_set( agent_module="travel_bench.agent", eval_set=eval_set, eval_config=eval_config, num_runs=3, print_detailed_results=True,
# Per-invocation results (metric name, value, pass/fail) written as CSV. # Parent directory is created automatically. output_file="./eval_results/travel_bench_20260824.csv",
# Preloaded artifacts reach eval cases whose SessionInput.session_id matches # the id you used to save_artifact above. artifact_service=artifact_service,
# Required whenever eval_set_results_manager is provided; otherwise a # ValueError is raised at the top of evaluate_eval_set. app_name="travel_bench", eval_set_results_manager=results_manager, )
asyncio.run(main())What ends up on disk:
./eval_results/travel_bench_20260824.csv— one row per (eval case, metric, invocation), with the columnseval_set_id, eval_id, metric_name, threshold, score, eval_status, prompt, expected_response, actual_response, expected_tool_calls, actual_tool_calls. The file is appended to if it already exists (the header is written once), and writing it needs pandas (pip install "google-adk[eval]")../eval_results/travel_bench/.adk/eval_history/<result_id>.evalset_result.json— the persistedEvalSetResultwritten byLocalEvalSetResultsManager. The path is built as<agents_dir>/<app_name>/.adk/eval_history/<result_id>.evalset_result.json(see_get_eval_history_dirinlocal_eval_set_results_manager.py). This is what the ADK Web UI reads back when browsing historical runs.
Two hard rules from the source (verified in the docstring & body):
eval_configis required — passingNoneraisesValueError("eval_configis required.").- If you set
eval_set_results_managerbut leaveapp_nameNone, the same top-of-function check raisesValueError("app_name is required when eval_set_results_manager is provided.").
Legacy criteria= shortcut. Still accepted, still deprecated. Verified: AgentEvaluator auto-maps criteria={metric: threshold} → EvalConfig(criteria={metric: BaseCriterion(threshold=threshold)}) and logs a deprecation message through the google_adk logger (logger.warning, not a Python DeprecationWarning). Migrate anything you own by constructing EvalConfig explicitly.
Saving eval results
Section titled “Saving eval results”evaluate_eval_set returns None; to keep results, hand it a results manager and an app_name. LocalEvalSetResultsManager writes each run to <agents_dir>/<app_name>/.adk/eval_history/:
from google.adk.evaluation.local_eval_set_results_manager import ( LocalEvalSetResultsManager,)
results_manager = LocalEvalSetResultsManager(agents_dir="./agents")
await AgentEvaluator.evaluate_eval_set( agent_module="my_agent.agent", eval_set=eval_set, eval_config=eval_config, app_name="my_agent", eval_set_results_manager=results_manager,)Or use GcsEvalSetResultsManager to persist to Cloud Storage:
from google.adk.evaluation.gcs_eval_set_results_manager import GcsEvalSetResultsManager
results_manager = GcsEvalSetResultsManager( bucket_name="my-eval-results", project="my-gcp-project", # extra kwargs go to google.cloud.storage.Client)Custom metrics
Section titled “Custom metrics”A custom metric is a function that ADK imports by dotted path. It is called with the EvalMetric, the actual invocations, the expected invocations (or None) and the ConversationScenario (or None), and returns an EvaluationResult; it may be async. ADK clears the threshold on the copy of EvalMetric it passes in, so the function sets the pass/fail status itself, and the threshold in criteria is then checked against overall_score.
from typing import Optional
from google.adk.evaluation.conversation_scenarios import ConversationScenariofrom google.adk.evaluation.eval_case import Invocationfrom google.adk.evaluation.eval_metrics import EvalMetric, EvalStatusfrom google.adk.evaluation.evaluator import EvaluationResult, PerInvocationResult
MAX_CHARS = 100
def response_brevity( eval_metric: EvalMetric, actual_invocations: list[Invocation], expected_invocations: Optional[list[Invocation]], conversation_scenario: Optional[ConversationScenario],) -> EvaluationResult: """Scores 1.0 for each final response of at most MAX_CHARS characters.""" per_invocation = [] for actual in actual_invocations: parts = actual.final_response.parts if actual.final_response else [] text = "".join(p.text or "" for p in parts or []) score = 1.0 if len(text) <= MAX_CHARS else 0.0 per_invocation.append( PerInvocationResult( actual_invocation=actual, score=score, eval_status=EvalStatus.PASSED if score == 1.0 else EvalStatus.FAILED, ) ) overall = sum(r.score for r in per_invocation) / max(len(per_invocation), 1) return EvaluationResult( overall_score=overall, overall_eval_status=EvalStatus.PASSED if overall == 1.0 else EvalStatus.FAILED, per_invocation_results=per_invocation, )Register it under a name in custom_metrics and give that name a threshold in criteria:
from google.adk.agents.common_configs import CodeConfigfrom google.adk.evaluation.eval_config import CustomMetricConfig, EvalConfig
config = EvalConfig( criteria={"response_brevity": 1.0}, custom_metrics={ "response_brevity": CustomMetricConfig( code_config=CodeConfig(name="my_package.metrics.response_brevity"), description="Final response is at most 100 characters.", ), },)With a stub model replying 42, this metric passes; a 150-character reply fails with response_brevity for my_package.agent Failed. Expected 1.0, but got 0.0.
Rubric-based evaluation
Section titled “Rubric-based evaluation”Rubrics let you score responses against structured criteria instead of a single binary pass/fail:
from google.adk.evaluation.eval_rubrics import Rubric, RubricContentfrom google.adk.evaluation.eval_metrics import RubricsBasedCriterion
def rubric(rubric_id: str, text: str) -> Rubric: return Rubric(rubric_id=rubric_id, rubric_content=RubricContent(text_property=text))
rubrics = [ rubric("cites_source", "The response cites at least one source URL."), rubric("plain_english", "The response is written in plain English, without jargon."), rubric("concise", "The response is under 200 words."),]
config = EvalConfig( criteria={ "rubric_based_final_response_quality_v1": RubricsBasedCriterion( threshold=0.8, # minimum mean rubric score to pass rubrics=rubrics, ), })pytest integration
Section titled “pytest integration”import pytestfrom google.adk.evaluation.agent_evaluator import AgentEvaluatorfrom google.adk.evaluation.eval_set import EvalSetfrom google.adk.evaluation.eval_config import EvalConfigfrom google.adk.evaluation.eval_metrics import PrebuiltMetrics
EVAL_SET = EvalSet.model_validate_json( open("tests/eval_data/regression.evalset.json").read())
EVAL_CONFIG = EvalConfig( criteria={ PrebuiltMetrics.TOOL_TRAJECTORY_AVG_SCORE.value: 1.0, PrebuiltMetrics.RESPONSE_MATCH_SCORE.value: 0.7, })
@pytest.mark.asyncioasync def test_agent_regression(): await AgentEvaluator.evaluate_eval_set( agent_module="my_package.agent", eval_set=EVAL_SET, eval_config=EVAL_CONFIG, num_runs=2, )Run with:
pytest tests/test_agent.py -vFile-based eval format
Section titled “File-based eval format”For tooling compatibility, save eval cases as JSON: an EvalSet serialised with snake_case field names (EvalSet.model_dump_json() produces this; camelCase keys such as evalSetId fail validation):
{ "eval_set_id": "arithmetic_suite", "eval_cases": [ { "eval_id": "add_two_numbers", "conversation": [ { "user_content": { "role": "user", "parts": [{ "text": "What is 15 + 27?" }] }, "final_response": { "role": "model", "parts": [{ "text": "42" }] }, "intermediate_data": { "tool_uses": [] } } ] } ]}Save as tests/eval_data/arithmetic_suite.evalset.json. The old .test.json format is still accepted but will emit a migration warning — use AgentEvaluator.migrate_eval_data_to_new_schema() to convert.
Migrate old eval data
Section titled “Migrate old eval data”AgentEvaluator.migrate_eval_data_to_new_schema( old_eval_data_file="tests/eval_data/old_tests.test.json", new_eval_data_file="tests/eval_data/old_tests.evalset.json", initial_session_file="tests/initial_session.json",)Optimising the instruction with GEPA
Section titled “Optimising the instruction with GEPA”adk optimize searches for a better root-agent instruction, using your eval sets as the score. It needs the eval extra (pip install "google-adk[eval]", which brings in gepa). It takes two JSON files.
The sampler config names the eval sets (a LocalEvalSamplerConfig). app_name must equal the agent folder’s name, or the command stops with an error:
{ "eval_config": { "criteria": {"tool_trajectory_avg_score": 1.0, "response_match_score": 0.7} }, "app_name": "my_agent", "train_eval_set": "train", "validation_eval_set": "val"}validation_eval_set is optional; without it the training cases are also used for validation, which rewards overfitting. train_eval_case_ids and validation_eval_case_ids narrow either set to named cases.
The optimiser config is optional (a GEPARootAgentPromptOptimizerConfig):
{"max_metric_calls": 60, "reflection_minibatch_size": 3, "run_dir": "gepa_runs"}max_metric_calls (default 100) caps the number of evaluations and so the cost; each is a full agent run plus its metrics. optimizer_model (default gemini-2.5-flash) writes the candidate instructions. run_dir keeps intermediate results.
adk optimize my_agent \ --sampler_config_file_path sampler.json \ --optimizer_config_file_path optimizer.json \ --print_detailed_resultsThe command prints the best instruction and does not edit your files. Limits worth knowing:
- Only the root agent’s
instructionis changed. Sub-agents are left alone (the optimiser logs a warning), and the instruction must be a plain string, not an instruction provider function. - Overlap between training and validation case IDs is logged as a warning, because scores for the shared cases are no longer independent.
- The same pieces are available from Python:
LocalEvalSampler(config, eval_sets_manager)andGEPARootAgentPromptOptimizer(config).optimize(agent, sampler), which returns candidates inoptimized_agents, each anAgentWithScoreswithoptimized_agentandoverall_score. To use a different search strategy, subclassAgentOptimizer.
Review the winning instruction as you would any prompt change, and re-run the full eval suite on cases that were in neither set.
Patterns
Section titled “Patterns”1 — CI gate on tool trajectory
Section titled “1 — CI gate on tool trajectory”Record expected tool calls from a golden run. In CI, ToolTrajectoryCriterion(match_type=IN_ORDER, threshold=1.0) fails the build if the agent forgets a required tool or calls them out of order.
2 — LLM judge for quality
Section titled “2 — LLM judge for quality”Use RESPONSE_EVALUATION_SCORE or FINAL_RESPONSE_MATCH_V2 with num_samples=5 to get stable scores. Reserve expensive judge metrics for nightly runs; use RESPONSE_MATCH_SCORE (lexical) in fast PR checks.
3 — Rubric tiers
Section titled “3 — Rubric tiers”Three rubrics worth 1 point each. Threshold at 0.67 (≥ 2/3 criteria). Run with num_runs=3 to smooth out judge variance.
4 — Per-agent sub-eval
Section titled “4 — Per-agent sub-eval”Set agent_name="specialist_bot" on evaluate_eval_set to evaluate a sub-agent in isolation, bypassing the root agent’s routing.
5 — End-to-end state assertion
Section titled “5 — End-to-end state assertion”Populate final_session_state={"order_confirmed": True} in the EvalCase. ADK asserts the session state matches after the conversation completes. Combine with tool trajectory to verify both the path and the outcome.
Multi-turn evaluators
Section titled “Multi-turn evaluators”Three Vertex AI–backed evaluators score entire conversations holistically, rather than scoring individual turn responses. They delegate to the Vertex Gen AI Eval SDK, so they need a Google Cloud project: set GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION. An AI Studio GOOGLE_API_KEY is not enough.
| Class | PrebuiltMetrics key | What it scores |
|---|---|---|
MultiTurnTaskSuccessV1Evaluator | multi_turn_task_success_v1 | Did the agent achieve the user’s goal by the end of the conversation? |
MultiTurnToolUseQualityV1Evaluator | multi_turn_tool_use_quality_v1 | Did the agent call the right tools across all turns? |
MultiTurnTrajectoryQualityV1Evaluator | multi_turn_trajectory_quality_v1 | Was the path the agent took to reach the goal reasonable? |
All three are reference-free — they do not need final_response or intermediate_data in your EvalCase; the Vertex AI rubric model judges based on the conversation transcript and the ConversationScenario context you supply.
ConversationScenario
Section titled “ConversationScenario”ConversationScenario describes the task the agent was meant to solve. The multi-turn evaluators use it to judge whether the agent succeeded. It lives in google.adk.evaluation.conversation_scenarios with two required fields, starting_prompt (the first user message) and conversation_plan (the evaluator’s guide to what a successful conversation looks like), and an optional user_persona used by the simulated user.
from google.adk.evaluation.conversation_scenarios import ConversationScenario
scenario = ConversationScenario( starting_prompt="Book a restaurant table for 2 people at 7pm on Friday.", conversation_plan=( "User wants a table for 2 at 7pm Friday. " "A successful conversation ends with the agent confirming the booking details." ),)Attach it to an EvalCase using conversation_scenario. Important: EvalCase enforces an XOR constraint — you must set exactly one of conversation (static turns) or conversation_scenario (scenario-driven). Setting both, or neither, fails validation (a pydantic ValidationError, which subclasses ValueError).
from google.adk.evaluation.eval_case import EvalCase
case = EvalCase( eval_id="restaurant_booking", conversation_scenario=scenario, # ← scenario only; no conversation= here)Full multi-turn eval example
Section titled “Full multi-turn eval example”import asyncioimport osfrom google.adk.evaluation.agent_evaluator import AgentEvaluatorfrom google.adk.evaluation.conversation_scenarios import ConversationScenariofrom google.adk.evaluation.eval_case import EvalCasefrom google.adk.evaluation.eval_set import EvalSetfrom google.adk.evaluation.eval_config import EvalConfigfrom google.adk.evaluation.eval_metrics import PrebuiltMetrics
# Required for the Vertex AI evaluatorsos.environ["GOOGLE_CLOUD_PROJECT"] = "my-gcp-project"os.environ["GOOGLE_CLOUD_LOCATION"] = "us-central1"
# ConversationScenario uses starting_prompt + conversation_plan (source-verified)scenario = ConversationScenario( starting_prompt="Find me a flight from London to Tokyo for August 10th.", conversation_plan=( "User wants to book a direct flight from London to Tokyo on August 10th. " "They will pick an option from the agent's search results and confirm the booking. " "A successful conversation ends with a booking confirmation." ),)
# EvalCase enforces XOR: set conversation_scenario OR conversation, never bothcase = EvalCase( eval_id="flight_booking_multi_turn", conversation_scenario=scenario,)
eval_set = EvalSet(eval_set_id="flight_booking_suite", eval_cases=[case])
eval_config = EvalConfig( criteria={ # Reference-free multi-turn metrics — no expected responses needed PrebuiltMetrics.MULTI_TURN_TASK_SUCCESS_V1.value: 0.7, PrebuiltMetrics.MULTI_TURN_TOOL_USE_QUALITY_V1.value: 0.7, PrebuiltMetrics.MULTI_TURN_TRAJECTORY_QUALITY_V1.value: 0.7, })
async def main(): await AgentEvaluator.evaluate_eval_set( agent_module="flight_bench.agent", eval_set=eval_set, eval_config=eval_config, num_runs=1, # multi-turn rubric metrics are expensive; 1 run is typical print_detailed_results=True, )
asyncio.run(main())Configuring the simulated user
Section titled “Configuring the simulated user”A scenario-driven case is played out by a simulated user. Configure it with EvalConfig(user_simulator_config=LlmBackedUserSimulatorConfig(...)) from google.adk.evaluation.simulation.llm_backed_user_simulator:
| Field | Default | Purpose |
|---|---|---|
model | "gemini-2.5-flash" | LLM that plays the user. The default is a Gemini 2.5 model, which new Gemini API projects cannot use; set model="gemini-3.8-flash" or another model you have access to |
model_configuration | default GenerateContentConfig | Generation config for that LLM |
max_allowed_invocations | 20 | Turn cap before the simulation stops |
include_function_calls | False | Show the agent’s tool calls to the simulated user |
custom_instructions | None | Jinja2 template replacing the default simulator prompt |
ConversationScenario.user_persona takes a UserPersona (or a registered persona name) to give the simulated user a style, with rubrics the judge checks it against:
from google.adk.evaluation.simulation.llm_backed_user_simulator import LlmBackedUserSimulatorConfigfrom google.adk.evaluation.simulation.user_simulator_personas import UserBehavior, UserPersona
impatient = UserPersona( id="impatient_traveller", description="A frequent flyer who is time-pressed and expects quick answers.", behaviors=[UserBehavior( name="brief_responses", description="Gives very short answers.", behavior_instructions=["Keep your messages under 15 words."], violation_rubrics=["Response is longer than 15 words."], )],)scenario = ConversationScenario( starting_prompt="Book me a flight to Paris.", conversation_plan="Book a one-way flight LHR→CDG on 2025-05-01. Say yes to all prompts.", user_persona=impatient,)eval_config = EvalConfig( criteria={PrebuiltMetrics.MULTI_TURN_TASK_SUCCESS_V1.value: 0.7}, user_simulator_config=LlmBackedUserSimulatorConfig(max_allowed_invocations=15),)AgentEvaluator imports agent_module and reads its module-level root_agent, so keep the driver (with asyncio.run) in a separate script from the agent module.
Choosing between multi-turn and per-turn metrics
Section titled “Choosing between multi-turn and per-turn metrics”| Situation | Recommended approach |
|---|---|
| Fast CI checks — did the agent call the right tools? | TOOL_TRAJECTORY_AVG_SCORE (per-turn, cheap, reference-based) |
| Overnight quality gate — did the conversation end well? | MULTI_TURN_TASK_SUCCESS_V1 (holistic, reference-free, Vertex AI) |
| Investigating agent reasoning path | MULTI_TURN_TRAJECTORY_QUALITY_V1 (judges whether the path made sense) |
| Tool selection across a dialogue | MULTI_TURN_TOOL_USE_QUALITY_V1 (rubric over entire tool-call log) |
ConversationGenerationConfig and ScenarioGenerator
Section titled “ConversationGenerationConfig and ScenarioGenerator”Instead of writing EvalCase objects by hand, you can ask a Vertex AI model to generate them. This is synchronous (not async). It uses Vertex AI, so it requires GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION. The correct module path is google.adk.evaluation._vertex_ai_scenario_generation_facade.
ConversationGenerationConfig lives in google.adk.evaluation.conversation_scenarios with fields: count (required), model_name (required), generation_instruction (optional), environment_context (optional).
import osfrom google.adk.agents import LlmAgentfrom google.adk.evaluation.conversation_scenarios import ConversationGenerationConfigfrom google.adk.evaluation._vertex_ai_scenario_generation_facade import ScenarioGenerator
os.environ["GOOGLE_CLOUD_PROJECT"] = "my-project"os.environ["GOOGLE_CLOUD_LOCATION"] = "us-central1"
travel_agent = LlmAgent( name="travel_agent", model="gemini-3.8-flash", description="Books flights and checks weather.", tools=[], # your actual tools here)
gen_config = ConversationGenerationConfig( count=3, # number of scenarios to generate model_name="gemini-3.8-flash", environment_context=( "Available flights: LHR→NRT on 2026-08-10 at 09:00 (ANA, £620), 14:00 (BA, £580)." ),)
generator = ScenarioGenerator()
# generate_scenarios is synchronous — no await neededscenarios = generator.generate_scenarios(agent=travel_agent, config=gen_config)
for s in scenarios: print(s.starting_prompt) print(s.conversation_plan)Combine with EvalCase to build a full synthetic eval set:
from google.adk.evaluation.eval_case import EvalCasefrom google.adk.evaluation.eval_set import EvalSet
cases = [ EvalCase( eval_id=f"generated_{i}", conversation_scenario=s, # XOR constraint: no conversation= when using scenario ) for i, s in enumerate(scenarios)]eval_set = EvalSet(eval_set_id="synthetic_travel_suite", eval_cases=cases)AppDetails and AgentDetails
Section titled “AppDetails and AgentDetails”AppDetails and AgentDetails capture a lightweight snapshot of the agent tree at eval time. Both live in google.adk.evaluation.app_details. AppDetails has a single field: agent_details: dict[str, AgentDetails]. AgentDetails has: name: str, instructions: str, tool_declarations: list.
from google.adk.evaluation.app_details import AppDetails, AgentDetails
# Typically you don't construct these manually — AgentEvaluator populates them# automatically from the running agent tree. But you can supply them explicitly# when replaying recorded conversations for offline eval.snapshot = AppDetails( agent_details={ "travel_assistant": AgentDetails( name="travel_assistant", instructions="You are a travel booking assistant.", tool_declarations=[], # list of tool FunctionDeclarations if known ), "flight_sub_agent": AgentDetails( name="flight_sub_agent", instructions="Search and book flights.", tool_declarations=[], ), })Gotchas
Section titled “Gotchas”agent_modulemust be an importable dotted path (e.g."my_package.agent"), not a file path, and must be onsys.path. Its name must end in.agent, or the module must have anagentattribute; otherwise loading fails with aValueErrorsaying the module has no member namedagent.num_runs=1can produce flaky results for non-deterministic models. Usenum_runs=2(the default) or higher for metrics that use LLM judges.- The
criteriadict key must exactly match thePrebuiltMetrics.valuestring (e.g."tool_trajectory_avg_score") or a custom metric name registered incustom_metrics. RESPONSE_EVALUATION_SCOREis inherently unstable — the docstring in source says “this evaluation is not very stable”. Treat it as a soft signal, not a hard gate.- Old
.test.jsonfiles are accepted but emit a deprecation warning. Migrate toEvalSetJSON to suppress the warning. SessionInput.statesets the initial session state before the first turn. Mutations during the conversation are not reflected back tosession_input.- Multi-turn Vertex AI evaluators (
multi_turn_task_success_v1, etc.) incur Vertex AI API calls per evaluation run. Keep the results withLocalEvalSetResultsManagerso you can compare runs, and avoid running them on every PR commit. ScenarioGenerator.generate_scenarios()is synchronous (source-verified: plaindef, notasync def). It takesagentandconfigas positional/keyword arguments. The correct import path isgoogle.adk.evaluation._vertex_ai_scenario_generation_facade.ConversationScenariofields arestarting_promptandconversation_plan(fromgoogle.adk.evaluation.conversation_scenarios), notscenario_description/task_description.EvalCaseenforcesconversationXORconversation_scenario: passing both, or neither, fails validation at construction time.