Bản tiếng Việt: Đánh giá agent: làm sao biết một thay đổi có giúp ích. Part 5 of the series. Previous: Tools, Permissions and Guardrails. Next: Forward Deployed Engineer.

For the first six months of running agents in production, our release process was a senior engineer trying five prompts and saying "feels better". We shipped a model upgrade that way. It felt better. Two weeks later a customer reported that the extraction agent had started skipping the signature page of multi-party contracts, something it had handled correctly for months. There was no test that would have caught it, because there was no test. We had a product built on a component whose behaviour we could not measure, and we were changing that component every week.

Evals are what turn an agent from a demo into an engineering artifact: something you can change with confidence, because you will know within an hour whether the change helped, hurt, or did nothing. This post is how we built that, what I would do differently, and the code to start with.

Why agent evals are a different problem#

Evaluating a model on a benchmark is a solved shape: inputs, expected outputs, a score. Evaluating an agent is harder in four specific ways.

Runs are non-deterministic. The same task, the same model, the same harness, produce different trajectories. One number per task is a sample, not a measurement; you need several runs per task to say anything.

The thing you care about is the outcome, but the outcome is reached through side effects. "Fixed the bug" means the tests pass and the agent did not also delete the migration folder on the way. You have to grade the end state and the path.

Many outputs have no single correct answer. A summary, a draft reply, a refactor. For those you need a rubric and a judge, and the judge has to be calibrated or it will quietly prefer whatever is longer.

Every run costs real money and real minutes. An eval suite that takes four hours and sixty dollars will not run on every pull request, and an eval that does not run on every pull request is a report, not a guardrail.

The arithmetic that convinced my team to take this seriously is pass^k. Anthropic's SWE-bench write-up distinguishes pass@k (succeed at least once in k attempts) from pass^k (succeed every time in k attempts). Users experience pass^k. If an agent succeeds on a task 90% of the time, the chance it succeeds five times in a row is 0.9⁵ ≈ 59%. At 95% per run it is 77%. At 99% it is 95%. The gap between "demo works" and "customers trust it" is the gap between pass@1 and pass^k, and you only see it when you run each task more than once.

Build the task set from failures, not from imagination#

The eval set we wrote from scratch on a whiteboard was mostly useless: easy tasks the agent always passed, and hard tasks nobody could grade. The eval set that mattered was built from the trace log (part 1): every time the agent did something wrong in production, that run became a task. Fifty tasks from real failures beat five hundred synthetic ones, because they are weighted exactly like your users' problems.

A task has five parts, and I keep them in version control next to the harness code:

{
  "id": "contract-signature-page-003",
  "category": "extraction",
  "setup": {"fixture": "fixtures/multi-party-contract", "env": {"DOC_ID": "mp-0193"}},
  "prompt": "Extract all signatories with their roles and signing dates from document mp-0193.",
  "checks": [
    {"type": "json_path", "path": "$.signatories.length()", "equals": 4},
    {"type": "json_path", "path": "$.signatories[*].signing_date", "all_match": "^\\d{4}-\\d{2}-\\d{2}$"},
    {"type": "no_tool_called", "name": "crm_update_contact"},
    {"type": "max_steps", "value": 15},
    {"type": "judge", "rubric": "rubrics/extraction-faithfulness.md", "min_score": 4}
  ],
  "budget": {"steps": 25, "usd": 0.80}
}

The setup block is what makes the task reproducible: a fixture directory copied fresh for every run, so no run can pollute the next. The checks mix three kinds of grading, which is the heart of it.

Three kinds of graders, in order of trust#

Deterministic checks are the foundation. Tests pass. A file exists with a specific content. A JSON field equals a value. A forbidden tool was not called. The run took fewer than N steps and cost less than M dollars. These are cheap, exact, and immune to the judge's moods. Push as much grading into this category as you can, and design the agent's finish tool (part 1) to return structured output so that it is possible.

Trajectory checks grade the path. Did the agent read the signature page before answering? Did it run the tests before claiming done? Did it try the same failing command more than twice? These are also deterministic, over the trace rather than the output, and they catch the case where the answer is right by luck.

Judge checks use a model with a rubric for the parts that have no exact answer. Three rules keep them honest. The rubric is written down, specific, and scored on a small scale (1 to 5) with a description of each level. The judge sees the trace and the fixture, not just the final text, so it can check claims against reality. And the judge is calibrated: we labelled thirty outputs by hand, ran the judge on them, and tuned the rubric until agreement was above 85%. Before calibration our judge gave 5/5 to a summary that was fluent, confident and about the wrong contract.

Pairwise judging (is output A better than B?) is more reliable than absolute scoring and is the right tool for comparing two harness versions, but it does not give you a number to track over time, so we use both.

For reference, this is roughly what the rubric file in the task above looks like. The level descriptions are the part that took iteration; the first version just said "5 = excellent" and the judge agreed with us about 60% of the time.

# Extraction faithfulness (1–5)

You are given the agent's final JSON, the trace of tools it called, and the source document.
Score ONLY faithfulness: whether every extracted value is supported by the document.

5  Every value is present in the document, with the correct role attached. No invented values.
   The trace shows the signature page(s) were read before `finish`.
4  All values correct; one minor formatting issue (date format, trailing title) that a reviewer would fix in seconds.
3  One value is wrong or missing, OR a value is correct but the trace shows it was guessed (page not read).
2  Two or more values wrong or missing, or roles swapped between parties.
1  Output is about a different document, is empty, or invents parties.

Return a JSON object: {"score": <1-5>, "evidence": "<one sentence citing page/line for the most important finding>"}.

Two habits make rubrics like this work. The judge is asked to cite evidence, which makes its scores auditable and its drift visible. And the rubric scores one thing. "Quality" is not a rubric; faithfulness, completeness and format are three rubrics, and a task can require all three.

A runner you can put in CI#

Here is the eval runner we started with, trimmed to its essentials. It is deliberately independent of any agent framework: you give it a function run_task(prompt, workdir) -> (output, trace) and it does the rest. The stub agent at the bottom means the file runs as-is, so you can see the report format before wiring in a real agent.

# eval_runner.py — run every task k times in a fresh fixture, grade, and report pass@1 / pass^k
import json
import re
import shutil
import statistics
import tempfile
from pathlib import Path

TASKS = Path("evals/tasks")
FIXTURES = Path("evals")


def check(c: dict, output: dict, trace: list[dict]) -> bool:
    kind = c["type"]
    if kind == "json_path":                     # tiny subset of JSONPath: $.a.b, $.a[*].b, .length()
        value = resolve(c["path"], output)
        if "equals" in c:
            return value == c["equals"]
        if "all_match" in c:
            return all(re.fullmatch(c["all_match"], str(v)) for v in value)
    if kind == "no_tool_called":
        return not any(e["kind"] == "tool" and e["name"] == c["name"] for e in trace)
    if kind == "max_steps":
        return sum(e["kind"] == "model" for e in trace) <= c["value"]
    if kind == "tool_called_before_finish":
        names = [e["name"] for e in trace if e["kind"] == "tool"]
        return c["name"] in names
    if kind == "judge":
        return judge(c["rubric"], output, trace) >= c["min_score"]
    raise ValueError(f"unknown check {kind}")


def resolve(path: str, obj):
    wants_len = path.endswith(".length()")
    parts = path.removesuffix(".length()").lstrip("$.").split(".")
    cur = [obj]
    for part in parts:
        nxt = []
        for item in cur:
            if part.endswith("[*]"):
                nxt.extend(item.get(part[:-3], []))
            else:
                nxt.append(item.get(part))
        cur = nxt
    value = cur[0] if len(cur) == 1 else cur
    return len(value) if wants_len else value


def judge(rubric_path: str, output: dict, trace: list[dict]) -> int:
    """Replace with a model call that returns 1-5 against the rubric. Stubbed here."""
    return 5


def run_once(task: dict, run_task) -> dict:
    with tempfile.TemporaryDirectory() as tmp:
        workdir = Path(tmp) / "work"
        shutil.copytree(FIXTURES / task["setup"]["fixture"], workdir)
        output, trace = run_task(task["prompt"], workdir, task["setup"].get("env", {}), task["budget"])
        results = {c["type"] + ":" + c.get("name", c.get("path", "")): check(c, output, trace) for c in task["checks"]}
        steps = sum(e["kind"] == "model" for e in trace)
        usd = sum(e.get("usd", 0) for e in trace)
        return {"passed": all(results.values()), "checks": results, "steps": steps, "usd": usd}


def evaluate(run_task, k: int = 3, baseline: dict | None = None) -> dict:
    report = {}
    for path in sorted(TASKS.glob("*.json")):
        task = json.loads(path.read_text())
        runs = [run_once(task, run_task) for _ in range(k)]
        passes = sum(r["passed"] for r in runs)
        report[task["id"]] = {
            "pass_at_1": passes / k,
            "pass_pow_k": 1.0 if passes == k else 0.0,
            "steps": statistics.mean(r["steps"] for r in runs),
            "usd": statistics.mean(r["usd"] for r in runs),
            "failed_checks": sorted({name for r in runs for name, ok in r["checks"].items() if not ok}),
        }
    print_report(report, baseline)
    return report


def print_report(report: dict, baseline: dict | None):
    total = len(report)
    p1 = sum(r["pass_at_1"] for r in report.values()) / total
    pk = sum(r["pass_pow_k"] for r in report.values()) / total
    print(f"\n{total} tasks  pass@1={p1:.0%}  pass^k={pk:.0%}  "
          f"mean steps={statistics.mean(r['steps'] for r in report.values()):.1f}  "
          f"mean usd={statistics.mean(r['usd'] for r in report.values()):.3f}\n")
    print(f"{'task':40} {'p@1':>5} {'p^k':>5} {'steps':>6} {'usd':>7}  failed checks")
    for tid, r in report.items():
        flag = ""
        if baseline and tid in baseline and r["pass_at_1"] < baseline[tid]["pass_at_1"]:
            flag = "  REGRESSION"
        print(f"{tid:40} {r['pass_at_1']:>5.0%} {r['pass_pow_k']:>5.0%} {r['steps']:>6.1f} {r['usd']:>7.3f}  "
              f"{', '.join(r['failed_checks'])}{flag}")


if __name__ == "__main__":
    # Stub agent so the runner executes without an API key. Replace with your harness.
    def fake_agent(prompt, workdir, env, budget):
        trace = [{"kind": "model", "usd": 0.01}, {"kind": "tool", "name": "read_section", "usd": 0},
                 {"kind": "model", "usd": 0.01}]
        output = {"signatories": [{"name": f"Party {i}", "signing_date": "2026-01-1" + str(i)} for i in range(4)]}
        return output, trace

    TASKS.mkdir(parents=True, exist_ok=True)
    (FIXTURES / "fixtures/multi-party-contract").mkdir(parents=True, exist_ok=True)
    (TASKS / "contract-signature-page-003.json").write_text(json.dumps({
        "id": "contract-signature-page-003", "category": "extraction",
        "setup": {"fixture": "fixtures/multi-party-contract", "env": {"DOC_ID": "mp-0193"}},
        "prompt": "Extract all signatories with their roles and signing dates from document mp-0193.",
        "checks": [
            {"type": "json_path", "path": "$.signatories.length()", "equals": 4},
            {"type": "json_path", "path": "$.signatories[*].signing_date", "all_match": r"^\d{4}-\d{2}-\d{2}$"},
            {"type": "no_tool_called", "name": "crm_update_contact"},
            {"type": "max_steps", "value": 15},
            {"type": "judge", "rubric": "rubrics/extraction-faithfulness.md", "min_score": 4},
        ],
        "budget": {"steps": 25, "usd": 0.80},
    }, indent=2))
    evaluate(fake_agent, k=3)

The important design choices: a fresh fixture per run, k runs per task, both pass@1 and pass^k reported, cost and steps tracked as first-class numbers, and a baseline comparison that prints the word REGRESSION so that a CI job can grep for it and fail.

What runs when#

Three cadences have worked for us. On every pull request that touches the prompt, tools, hooks or harness: a smoke subset of about fifteen tasks at k=2, under ten minutes and a couple of dollars, failing the build on any regression against the merged baseline. Nightly: the full set at k=5, with a report posted to the team channel, which is where we notice slow drift. And on every model version change: the full set, pinned to the new version, compared against the old, before any traffic moves. That last one is the test that would have caught the signature-page regression in an afternoon instead of two weeks.

Pin the model version in evals explicitly. "Latest" aliases are convenient in development and a source of unexplained diffs in evals.

Production is the other half#

Evals are offline. The loop that keeps them honest is online: traces from real runs, sampled and labelled, flowing back into the task set. We sample every failed run (the user hit "wrong", the agent stopped on budget, a check in the finish tool failed) and 2% of successful ones, and someone spends an hour a week turning the interesting ones into tasks. The eval set grows by five to ten tasks a week and stays representative, because it is literally made of what users are doing.

The production dashboard that pairs with this has fewer numbers than you would expect: success rate by task category, cost per successful task, the distribution of steps per run, approval and denial counts from the policy layer (part 4), and how many times the same file was read per run (part 3). A 30% rise in median steps with a flat success rate is the earliest signal we have found for "something upstream changed", usually a tool that started returning more than it used to.

Mistakes we made so you can make different ones#

We overfit to the eval set. After a quarter of tuning prompts against the same fifty tasks, pass^k was 96% on the set and users were not noticeably happier. The fix was the weekly intake of fresh failures, and holding out a slice of tasks that nobody tunes against.

We let the judge see only the output. It scored confident wrong answers highly. The judge now sees the trace and the fixture.

We graded "done" by the agent's own finish call. Agents learn to call finish early if that ends the pain. Now finish is where grading starts: the checks run against the state, not the claim.

We ran evals in a shared environment. A leftover file from run three made run four pass. Fresh fixture per run, always.

We measured accuracy and ignored cost. A change that raised pass@1 by two points and doubled cost per task shipped, and the margin on that product went negative for a month. Cost per successful task is the number that goes on the dashboard now.

None of this is sophisticated. It is the same discipline we already apply to code (tests, CI, monitoring), adapted to a component that is probabilistic. The difference between teams that ship agents and teams that demo them is mostly whether that discipline exists.

References#

• Anthropic, Raising the bar on SWE-bench Verified (pass@k versus pass^k)
• Anthropic, Building effective agents (December 2024)
• Hamel Husain, Your AI product needs evals (2024)
• OpenAI, Harness engineering (February 2026), on turning review feedback into mechanical checks