Evaluation
Tests prove the code runs. Evals prove the agent behaves. This lesson covers the eval loop — the closed cycle of dataset, metric, harness, and iteration that lets you measure agent quality with the same rigor you measure latency or test coverage.
SDK focusoutputFormathooks.PostToolUseresult.subtypefallbackModelmodel routing
Evals vs tests
Both check correctness and can overlap; the useful distinction is what signal they optimize for:
| Tests | Evals | |
|---|---|---|
| Question | "Does the code run?" | "Does the agent behave?" |
| Typical truth | Deterministic assertions | Rubrics or statistical aggregates |
| Typical output | Per-case pass/fail | Rates, distributions, and segments |
| Inputs | Fixtures | Datasets that approximate reality |
| Failure shape | Stack trace | Behavior drift, regression on subgroup |
A unit test on file_read_tool proves the tool works. An eval on the agent proves it picks the right tool, calls it with sane arguments, and recovers when it doesn't. You need both.
The eval loop
A useful eval system is a closed loop, not a one-time test suite. Each cycle improves either the agent or the eval itself:
Each step is a discipline of its own. We'll walk through them in order.