RAG evaluation, explained

RAG evaluation scores a retrieval-augmented generation (RAG) system in parts: whether retrieval returned passages that bear on the question, whether the answer stays within them, and whether the answer addresses the question. Two frameworks set the common terms. RAGAS, by Es and colleagues at the European Chapter of the Association for Computational Linguistics (EACL) in March 2024, prompts a language model and needs no reference answers. ARES, by Saad-Falcon and colleagues at the association's North American Chapter (NAACL) in June 2024, trains small judge models and corrects them with a few hundred human labels. Both report close agreement with people in their own tests. An outside study found far weaker agreement.

Three scores with the same names

Both papers use three scores: faithfulness, answer relevance, and context relevance. The names match. The measurements do not.

Score	RAGAS (EACL 2024 paper)	ARES (NAACL 2024 paper)
Faithfulness	A model splits the answer into statements and checks each against the context; the score is the share supported	A trained classifier labels each question, passage and answer triple as faithful or not
Answer relevance	A model writes questions from the answer; the score is their average embedding similarity to the original question	A classifier judges whether the answer is relevant to the question and passage
Context relevance	The share of context sentences the model extracts as needed to answer	A classifier judges whether the passage is relevant to the question

RAGAS states that its answer relevance score does not take factuality into account. An answer can score as relevant and still be wrong.

How RAGAS scores without reference answers

RAGAS ran every prompt in the paper on gpt-3.5-turbo-16k, a commercial model reached through an application programming interface (API). To test the scores, the authors built WikiEval: questions on 50 Wikipedia pages about events since the start of 2022, judged by two annotators. The annotators agreed in about 95 percent of cases on faithfulness and context relevance and about 90 percent on answer relevance.

On pairwise choices, RAGAS agreed with the annotators 0.95 of the time on faithfulness, 0.78 on answer relevance, and 0.70 on context relevance. A baseline that asked ChatGPT for a score from 0 to 10 reached 0.72, 0.52, and 0.63. The authors found context relevance hardest; the model struggled to pick the needed sentences, especially from longer contexts.

The authors list two limits. RAGAS relies heavily on the model used as evaluator. Repeated runs can differ, because of an undocumented change behind an API or ordinary randomness.

The library has also changed. The current documentation lists six metrics under retrieval-augmented generation, including context precision, context recall, and noise sensitivity, beside faithfulness and response relevancy. The faithfulness page keeps the paper's method: identify the claims, check each against the retrieved context, and divide supported claims by all claims.

How ARES trains judges and bounds their error

ARES needs three inputs: in-domain passages, a human-labeled validation set of about 150 examples or more, and at least five example questions and answers. A generator model writes synthetic questions and answers from the passages. Questions that cannot retrieve their own source passage are dropped. A 304-million-parameter classifier is then fine-tuned for each of the three scores.

Judges make errors, so ARES applies prediction-powered inference (PPI), a method by Angelopoulos and colleagues. PPI combines a small labeled set with many machine predictions to give valid confidence intervals, with no assumptions about the model that made the predictions. More accurate predictions give narrower intervals.

The main ARES tests used mock systems: nine splits of public datasets with known success rates from 70 to 90 percent, 2.5 points apart. On those, ARES beat RAGAS on context relevance accuracy by 59.3 points according to the introduction and 59.9 points according to the results section, and on answer relevance by 14.4 points. On ranking real systems, the average Kendall's tau, a rank correlation where 1.0 is perfect, was 0.91 for context relevance and 0.97 for answer relevance. The 95 percent intervals averaged 7.4 and 6.1 points wide.

Stated limits and an outside check

The ARES authors name their limits. The 150 to 300 labels often need an annotator familiar with the domain, and fields such as law, medicine, and finance may need specialists. Fine-tuning and data generation needed graphics processors (GPUs) with about 32GB of memory. Every dataset was in English. Judges trained on Natural Questions dropped to a Kendall's tau of 0.33 on other languages, 0.28 on code, and 0.38 on entity extraction. Faithfulness was not tested on the main datasets, for lack of human-labeled hallucinated answers; on the Attributable to Identified Sources (AIS) benchmark, ARES came within 2.5 points of the true scores.

RAGChecker, by Ru and colleagues at the 2024 Neural Information Processing Systems (NeurIPS) conference, is a competing framework, and its authors tested the earlier metrics against 280 pairs of answers labeled by humans. On overall preference, the paper reports Pearson correlation scores of 7.83 for RAGAS faithfulness, 8.80 for ARES faithfulness, 10.27 for RAGAS answer relevance, and 17.81 for ARES answer relevance. Agreement between the human annotators scored 70.09. The authors caution that the metrics were built for different aspects, so an exact comparison of scores does not apply. The humans judged correctness, completeness, and overall quality, not faithfulness alone.

What to ask

• Which framework and which version produced the score? RAGAS today lists six retrieval metrics; the ARES comparison used version 0.0.18.
• Which model served as judge, and was it held fixed between runs?
• How many human labels from the system's own documents checked the automated score, and is there a confidence interval?
• Does a high answer relevance score also mean the answer is correct? In RAGAS it does not.
• Were the judges tested in the language and the document type of the system being evaluated?

Azcuna's ELLMENT, the E-2D Large Language Model Entity, was proposed to NAVAIR PMA-231 under Navy SBIR Phase I topic DON26BZ01-NV010. The prototype, built at private expense on publicly available doctrine, answers with a page-level citation and refuses when the documents do not support an answer. The proposal has been submitted.

Sources

1. 1RAGAs: Automated Evaluation of Retrieval Augmented Generation · ACL Anthology (Es, James, Espinosa-Anke, and Schockaert; EACL 2024 System Demonstrations)
2. 2ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems · ACL Anthology (Saad-Falcon, Khattab, Potts, and Zaharia; NAACL 2024)
3. 3Prediction-Powered Inference · arXiv (Angelopoulos, Bates, Fannjiang, Jordan, and Zrnic; January 2023, revised November 2023)
4. 4List of available metrics · Ragas documentation (stable version, read October 2026)
5. 5Faithfulness · Ragas documentation (stable version, read October 2026)
6. 6RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation · Advances in Neural Information Processing Systems 37, Datasets and Benchmarks Track (Ru et al., NeurIPS 2024)

Where this shows up in our work

• CALIBER, the market testbed for AI agents measures from an AI agent's trades whether it is gullible, herds, overpays, or can be steered.
• ELLMENT, the offline LLM for E-2D crews answers a crew's question from the aircraft's own manuals and cites the page, with no network.