LLM-as-a-judge, explained

LLM-as-a-judge is the practice of having a large language model (LLM) grade or compare other models' answers in place of human raters. Zheng and colleagues tested the approach in a 2023 paper for the NeurIPS Datasets and Benchmarks track. GPT-4, acting as judge, reached over 80 percent agreement with human preferences, the same level at which the humans agreed with each other. The same paper documented position, verbosity, and self-enhancement biases. Later studies found that agreement depends on the task and on who the human raters are, which decides where a model judge can stand in for people.

How a model judge is set up

Zheng describes three variations:

• Pairwise comparison: the judge sees a question and two answers and picks the better one or declares a tie.
• Single answer grading: the judge assigns a score to one answer.
• Reference-guided grading: the judge also sees a reference solution.

Each has a cost. The number of pairs grows quadratically with the number of models compared. Single scores may fluctuate more than pairwise results if the judge model changes.

The paper introduced two benchmarks. MT-Bench has 80 multi-turn questions, 10 in each of 8 categories: writing, roleplay, extraction, reasoning, math, coding, knowledge of science, technology, engineering, and mathematics (STEM), and knowledge of the humanities and social sciences. Chatbot Arena collects votes from users who chat with two anonymous models at once; it gathered about 30,000 votes in its first month. The authors name two benefits of a model judge: scale, and explanations that accompany each verdict.

Agreement with human raters

For MT-Bench, six models answered all 80 questions, and 58 expert-level raters, mostly graduate students, cast about 3,000 votes. Agreement is the probability that a randomly chosen judge of each type agrees on a randomly chosen question.

On first-turn votes without ties, GPT-4 comparing pairs agreed with the humans 85 percent of the time, against 81 percent between humans; two random judges would agree 50 percent of the time. With ties and order-inconsistent verdicts counted as ties, the figures were 66 and 63 percent, against a random 33 percent.

Agreement was not uniform. It rose from 70 percent to nearly 100 percent as the performance gap between the two models grew. In the authors' reading, GPT-4 aligns with humans better when the models differ significantly, so the judge is weakest on close comparisons.

Position and length bias

To test position bias, the authors built two similar answers for each first-turn question and swapped their order. GPT-4 gave the same verdict both times in 65.0 percent of cases, GPT-3.5 in 46.2 percent, and Claude-v1 in 23.8 percent; most judges favored the first position. Few-shot examples raised GPT-4 to 77.5 percent but made each call four times as expensive. The authors' conservative fix is to judge twice with the order swapped and count a win only when both verdicts agree.

Wang and colleagues showed how far order alone can move a result. In their 2023 study, with ChatGPT as judge, Vicuna-13B beat ChatGPT on 66 of 80 queries when its answer was placed second. Placed first, its win rate was 2.5 percent.

For verbosity, Zheng padded 23 answers with a reworded copy of their own lists. Claude-v1 and GPT-3.5 preferred the padded answer 91.3 percent of the time, GPT-4 8.7 percent. Dubois and colleagues found the same pull on the AlpacaEval benchmark. In their length-control study, one model's win rate ranged from 22.9 to 64.3 percent depending only on a prompt asking for more or less detail. Adjusting for length narrowed the range to 41.9 to 51.6 percent and raised the benchmark's Spearman correlation with Chatbot Arena from 0.94 to 0.98.

Self-preference and grading errors

In Zheng's data, GPT-4 gave itself a 10 percent higher win rate than the humans did, and Claude-v1 25 percent; the authors could not determine whether this was bias, in part because a response cannot easily be rewritten in another model's style without changing its quality. Panickssery and colleagues tested the question directly. GPT-4 told its own summaries from others' with 73.5 percent accuracy, and after fine-tuning, the strength of self-preference was linearly correlated with this self-recognition.

Judges also misgrade answers they could have solved. On 10 math questions, each judged in both orders, GPT-4 accepted a wrong answer in 14 of 20 judgments; the authors show it misled by the answers on a problem it solved correctly when asked alone. Having it solve the question before grading cut failures to 6 of 20, though it often made the same mistake as the answers. Showing it its own separately generated answer as a reference cut them to 3 of 20.

What a model judge can and cannot replace

Bavaresco and colleagues tested 11 models as judges on 20 datasets with human labels. Some tasks, such as instruction following, gave reliable agreement. Toxicity and safety gave the lowest. On graded judgments, every model correlated better with non-expert raters than with experts, and chain-of-thought prompting did not consistently help. The authors recommend validating a judge against task-specific human judgments before use, and note that judge and humans can agree simply by sharing a bias.

Zheng's own limits point the same way: the study centered on helpfulness, largely neglected safety, and folded accuracy, relevance, and creativity into one score. Taken together, a model judge can screen many open-ended answers quickly, run in both orders and controlled for length. The studies above give no basis for using it unchecked to grade safety, expert-domain correctness, close comparisons, or its own outputs.

What to ask

• Was each pairwise comparison run in both orders, and how often did the verdict change?
• Was the judge checked against human raters on this task, and were those raters domain experts?
• Is the judge the same model, or the same family, as a system being graded?
• Were answer lengths controlled or reported alongside the scores?
• For factual or numerical questions, did the judge have a reference answer?

Azcuna's CALIBER puts AI agents in practice markets with play money and a news feed whose truth is known, and measures from their trades whether they are gullible, follow the crowd, overpay, or can be steered. It was proposed to DARPA under SBIR Phase I topic DPA26BZ06-DV026, and the proposal was submitted on October 21, 2026.

Sources

1. 1Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · arXiv (Zheng et al.; NeurIPS 2023 Datasets and Benchmarks Track)
2. 2Large Language Models are not Fair Evaluators · arXiv (Wang et al., May 2023, revised August 2023)
3. 3Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators · arXiv (Dubois, Galambosi, Liang, and Hashimoto; COLM 2024)
4. 4LLM Evaluators Recognize and Favor Their Own Generations · arXiv (Panickssery, Bowman, and Feng, April 2024)
5. 5LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks · arXiv (Bavaresco et al.; ACL 2025)

Where this shows up in our work

• CALIBER, the market testbed for AI agents measures from an AI agent's trades whether it is gullible, herds, overpays, or can be steered.
• ELLMENT, the offline LLM for E-2D crews answers a crew's question from the aircraft's own manuals and cites the page, with no network.