Evaluating language models
AI agent benchmarks, explained
What SWE-bench, τ-bench (tau-bench) and GAIA ask an AI agent to do, how each scores success, what the best systems scored when each paper appeared, and what later studies found these benchmarks miss.
Oct 22, 20265 min read6 sources
Why it matters
An agent that acts on files, databases or the web is judged by whether the job gets done, and at publication the best model resolved 1.96 percent of SWE-bench's real software issues.
The short version
- 1SWE-bench scores a code patch by running the repository's tests; tau-bench checks the final database state; GAIA checks one short factual answer.
- 2On τ-bench, the best agent succeeded on 61.2 percent of retail tasks in one try, but its chance of succeeding on all eight tries of a task fell below 25 percent.
- 3A manual review of one top SWE-bench agent found the fix already given in the issue for 32.67 percent of its successes and weak tests behind 31.08 percent.
An AI agent benchmark gives a language model tools and a goal, then scores whether the goal was reached, not whether the reply reads well. Three benchmarks show the range. SWE-bench asks for fixes to real software issues. τ-bench, also written tau-bench, asks an agent to serve a simulated customer under a written policy. GAIA asks questions that need browsing, files, and tools. When each paper appeared, the best system resolved 1.96 percent of SWE-bench issues, succeeded on under half of τ-bench tasks, and answered 15 percent of GAIA questions, against 92 percent for people.
SWE-bench: fixing real software issues
Jimenez and colleagues, at the International Conference on Learning Representations (ICLR) in 2024, built 2,294 tasks from real GitHub issues and the pull requests that resolved them, across 12 popular Python repositories. The model gets the issue text and the whole codebase and must produce a patch. The patch counts as a fix only if it applies and all the task's tests pass. Every task has at least one test that failed before the real fix and passed after it.
With a standard keyword retriever, BM25, choosing which files to show the model, Claude 2 resolved 1.96 percent of issues. ChatGPT-3.5 resolved 0.17 percent. GPT-4, run on a random 25 percent of tasks for budget reasons, resolved none. Shown the files that the real fix had edited, an "oracle" setting the authors call less realistic, Claude 2 reached 4.8 percent.
The authors name the limits. Every task is in Python. Passing tests is not enough to guarantee reliable code, since model patches were often less comprehensive, efficient, or readable than human ones. The project documentation lists later datasets, including SWE-bench Lite and SWE-bench Verified. Verified, released August 13, 2024, holds 500 problems that engineers confirmed solvable.
τ-bench: serving a customer under a policy
Yao and colleagues, at ICLR 2025, give an agent database tools and a domain policy, and let a language model play the customer. The retail domain has 115 tasks over 500 users, 50 products, and 1,000 orders. The airline domain has 50 tasks over 300 flights and 2,000 reservations. A task succeeds if the database at the end matches the one correct outcome and the agent told the user what was needed.
The paper adds a reliability measure, pass^k: the chance that all k independent tries of a task succeed, averaged across tasks. The best agent, gpt-4o using function calling, succeeded on 61.2 percent of retail tasks and 35.2 percent of airline tasks in one try. In retail, pass^8 fell below 25 percent.
One test checks whether agents use the policy at all. Removing it from the prompt cost gpt-4o only 4.4 points in retail, which the authors read as success mostly from common-sense tool use. In airline, with more complex rules, the drop was 22.4 points. The authors list limits of the simulated user: its instructions can contain typos or ambiguities, and the simulating model has limited capacity. They also note some implicit bias, because gpt-4-turbo was used to tune the user prompts.
GAIA: easy for people, hard for assistants
Mialon and colleagues, also at ICLR 2024, wrote 466 questions with one short factual answer each, scored by near-exact match. Answers to 300 are held back for a leaderboard. Level 1 questions need at most one tool and five steps; Level 2 needs roughly 5 to 10 steps and several tools; Level 3 needs arbitrarily long action sequences.
| Level (questions) | Human annotators | GPT-4 with plugins |
|---|---|---|
| Level 1 (146) | 93.9% | 30.3% |
| Level 2 (245) | 91.8% | 9.7% |
| Level 3 (75) | 87.3% | 0% |
The plugins were chosen by hand for each question, so the authors treat that score as an oracle that cannot be reproduced exactly. Their stated limits: GAIA does not grade the path to the answer, every question is in standard English, and closed systems change over time, so a result holds for one point in time. They expect the benchmark to decay through contamination of training data and the disappearance of web pages it relies on.
What agent benchmarks miss
Kapoor and colleagues, in AI Agents That Matter (July 2024), argue that agent benchmarks report accuracy without cost. On the HumanEval coding benchmark, simple baselines such as retrying a model matched complex agent designs. For similar accuracy, costs differed by almost two orders of magnitude, and one agent cost over 50 times more than a simple baseline. Weak holdout sets invite shortcuts: the top agent on the WebArena web benchmark, at 35.8 percent, hardcoded policies for its specific tasks, such as appending a fixed path to reach a user's profile page.
Test quality matters as much. In an October 2024 preprint, Aleithan and colleagues checked by hand the SWE-bench fixes credited to the top agent of the time. In 32.67 percent, the solution was already in the issue report or its comments. Another 31.08 percent were suspect because the tests were too weak to verify the patch. Filtering those out dropped the agent's rate from 12.47 percent to 3.97 percent. Over 94 percent of the issues predated the models' knowledge cutoff dates, a possible source of data leakage.
What to ask
- Which benchmark version produced the score (full, Lite, or Verified), and on what date?
- Could a wrong answer pass the check, through weak tests or a solution in the task text?
- How many tries per task were run, and what is pass^k, not only the average?
- What did a run cost, and did a simple baseline at the same cost do as well?
- Were the tasks held out from development and written after the model's training data was collected?
Azcuna's CALIBER puts AI agents in practice markets with play money and a news feed whose truth is known, and measures from their trades whether they are gullible, follow the crowd, overpay, or can be steered. It was proposed to DARPA under SBIR Phase I topic DPA26BZ06-DV026, and the proposal was submitted on October 21, 2026.
Sources
- 1SWE-bench: Can Language Models Resolve Real-world Github Issues? · International Conference on Learning Representations 2024 (Jimenez et al.)
- 2SWE-bench documentation · SWE-bench project site (read October 2026)
- 3τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains · International Conference on Learning Representations 2025 (Yao, Shinn, Razavi, and Narasimhan)
- 4GAIA: a benchmark for General AI Assistants · International Conference on Learning Representations 2024 (Mialon et al.)
- 5AI Agents That Matter · arXiv (Kapoor, Stroebl, Siegel, Nadgir, and Narayanan; July 2024)
- 6SWE-Bench+: Enhanced Coding Benchmark for LLMs · arXiv (Aleithan et al.; October 2024 preprint)
Where this shows up in our work
- CALIBER, the market testbed for AI agents measures from an AI agent's trades whether it is gullible, herds, overpays, or can be steered.
- ELLMENT, the offline LLM for E-2D crews answers a crew's question from the aircraft's own manuals and cites the page, with no network.