An AI agent benchmark is trustworthy when it defines success precisely, runs each agent many times, keeps every failed run in the count, reports its uncertainty and tests under conditions that resemble the work you need done. A single headline score answers none of those points on its own. The ten questions below form a checklist for reading any agent benchmark, whether it comes from a research lab, a vendor or your own team.

TL;DR: Research on agentic benchmarks has found that flaws in task setup or scoring can under or overstate agent performance by up to 100% in relative terms (Zhu et al., 2025). Before trusting a result, check what counted as success, how many attempts sat behind it, what was left out of the denominator and whether the gap between two agents is larger than the noise. For media buying, also check whether the agent was scored on the contract it signed, not only on whether it signed one.

What are the 10 questions to ask of any AI agent benchmark?

What exactly counted as success?
Is it testing the model or the whole agent?
How many times was each agent run?
Were failed and interrupted runs kept in the count?
Is the gap between agents bigger than the noise?
Who, or what, sat on the other side of the deal?
Does the test environment look like the market you trade in?
Could the agent have been tuned to the test?
Does it report cost and time alongside the score?
Who ran it, and is there a baseline?

1. What exactly counted as success?

The scoring rule is the benchmark. A test that checks only whether an agent finished a task will rate a careless finish the same as a careful one, and the authors of the Agentic Benchmark Checklist found that weak task setup or reward design can shift measured performance by up to 100% in relative terms. Applying their checklist to one complex benchmark, CVE-Bench, cut its performance overestimation by 33%, according to the same paper.

A worked example from media buying shows why this matters. In Alkimi's Concourse Bench v1, 55 of 96 registered runs completed a buy, yet 95 of the 110 contracts signed in those completed buys promised a viewability floor below 70%. As a completion score, that looks like progress; as a measure of buyer protection, it shows agents closing deals that left the buyer weakly covered.

Ask for the scoring rule in writing, and check whether it separates finishing the task from doing it well. One published split uses four measures: execution, commercial quality, reliability and measured efficiency.

2. Is it testing the model or the whole agent?

An agent is a model plus the software around it: the prompts, tools, memory and checks that turn text into actions. A benchmark that swaps the model but keeps that software fixed tells you about models, while one that lets each builder bring its own software tells you about products. Sayash Kapoor and colleagues argue in AI Agents That Matter that the benchmarking needs of model developers and downstream developers have been conflated, which makes it hard to identify which agent suits a particular application.

If you are choosing a product, you want results for the full system you would deploy, a point covered in Is your AI buying agent better than GPT or Claude?

3. How many times was each agent run?

One successful run proves an agent can do something, not that it will do it reliably. The tau-bench paper by Shunyu Yao and colleagues introduced a metric called pass^k to measure whether an agent succeeds consistently across repeated trials of the same task. In that study, even the strongest function-calling agents tested succeeded on fewer than half of tasks, and their pass^8 score in the retail domain fell below 25%.

Ask how many attempts sat behind each figure. A benchmark that reports only the best attempt is a showreel, not a measurement.

4. Were failed and interrupted runs kept in the count?

Denominators are where results quietly improve. If runs that crashed, timed out or hit a technical fault are dropped, the success rate is calculated on a friendlier population than the one the agent will face in production. Ask how many runs were started, how many were scored and what happened to the rest; if the answer is vague, treat the headline rate as an upper bound.

5. Is the gap between agents bigger than the noise?

Two agents a few points apart on a leaderboard may not differ at all. Anthropic's research note on adding error bars to evals recommends reporting the standard error of the mean alongside each score, with a 95% confidence interval calculated as the mean plus or minus 1.96 standard errors. The same note warns that when test items arrive in related clusters, standard errors "can be over three times as large as naive standard errors", and that ignoring this "may lead researchers to inadvertently detect a difference in model capabilities when in fact none exists".

Media-buying tests are often clustered by design, with several attempts per brief, so ask whether the report shows confidence intervals that account for it.

6. Who, or what, sat on the other side of the deal?

A buying agent is only as tested as its counterparty. Negotiating against a fixed software seller that follows a set policy is a different task from negotiating against another adaptive agent, and results from one do not automatically carry over to the other. Concourse v1, for example, states that its sellers were fixed software policies and leaves model-driven sellers to a later study.

Ask the benchmark owner to name the counterparty and say what the results therefore do not cover. Teams that want a buying agent assessed against questions like these can request a Concourse scoping conversation.

7. Does the test environment look like the market you trade in?

Benchmarks age. The authors of AD-Bench, an advertising analytics benchmark built from real requests on a production advertising platform, note that ground-truth answers "quickly become obsolete as data and platform rules evolve", and built a pipeline to regenerate them. Their results also show how much difficulty matters: the top Pass@1 score fell from 76.9% overall to 61.4% on the hardest tier of requests.

Shared standards narrow the gap between test and market. IAB Tech Lab's AAMP (Agentic Advertising Management Protocols) initiative argues that standards are needed "so the agents are trained to execute the action asked in a natural language with repeatable accuracy". A benchmark whose messages and deal objects follow the standards your counterparties use tells you more than one built on an invented format.

8. Could the agent have been tuned to the test?

Any agent that has seen the test, or been tuned against it many times, will score better than it performs. Kapoor and colleagues found that many agent benchmarks have inadequate holdout sets, sometimes none at all, which has led to fragile agents that "take shortcuts and overfit to the benchmark in various ways". Ask whether the tasks used for scoring were kept apart from those used during development, and whether the benchmark is versioned so that a changed test is reported as a new benchmark rather than a rerun.

9. Does it report cost and time alongside the score?

An agent that matches another's success rate while using far more compute is not equally good. The same paper argues that a narrow focus on accuracy has produced agents that are "needlessly complex and costly", and proposes optimising accuracy and cost together.

Look for model usage and elapsed time reported for every run, including failed ones. A cost per success means little if the failed runs were left out of the calculation.

10. Who ran it, and is there a baseline?

Ask who built the benchmark, whether its method and records are published and whether an outside party could reproduce the result. Kapoor and colleagues describe a "pervasive lack of reproducibility" in agent evaluation, which they trace to a lack of standardisation in evaluation practices.

Human baselines are still rare. Concourse states that it does not yet have one, and that a human comparison is planned before its completion rates can be read as good or bad rather than simply recorded. Treat any claim that an agent beats a human buyer with care unless both ran under the same conditions.

What should you do with a benchmark result once you have checked it?

Use the result to decide how much autonomy an agent earns, not whether to switch it on. One ladder runs from Observe to Recommend, then Draft, then Human-approved action, and only then Bounded automatic action, with each step backed by evidence from tests like these. How to test an AI media buyer before you deploy one sets out a practical testing sequence, and the Concourse methodology explainer covers the design choices behind one media-buying benchmark.

Alkimi publishes Concourse results with their limits stated, including unassessed runs and the missing human baseline, because a benchmark without its caveats is a sales sheet. Teams that want their own buying agent evaluated can request a scoping conversation.

Frequently asked questions

What makes an AI agent benchmark trustworthy?

A trustworthy benchmark publishes its scoring rule, runs each agent several times, keeps failed runs in the denominator and reports its uncertainty. It also states what it did not test, such as the type of counterparty or the absence of a human baseline.

Is a high completion rate enough to trust an AI media buyer?

No, because completion only shows that the agent finished the workflow. Contract quality shows whether the terms it agreed actually protect the buyer, so the two should be scored separately.

What is pass^k in agent benchmarks?

Pass^k is a reliability metric introduced in the tau-bench paper that measures whether an agent succeeds consistently across repeated attempts at the same task. It tends to fall well below single-attempt success, which is why one good run proves little.

How many runs does a benchmark need?

There is no fixed number, because it depends on how small a difference you want to detect. Anthropic's guidance on eval statistics recommends power analysis to work out the sample size needed before the test is run.

How to Read an AI Agent Benchmark: 10 Questions Before You Trust the Results