Hallucination and confabulation in language models, explained

Hallucination is the research term for generated text that is nonsensical, false, or unfaithful to its source. The National Institute of Standards and Technology (NIST) uses a different word. The NIST Generative AI Profile, NIST AI 600-1 of July 2024, lists confabulation second among 12 risks and defines it as the production of "confidently stated but erroneous or false content." A footnote records that some commenters consider hallucination and fabrication anthropomorphizing terms. Measurement shows the size of the problem. On the TruthfulQA benchmark, the best model tested in the 2022 paper was truthful on 58 percent of questions.

What NIST calls confabulation

Section 2.2 of NIST AI 600-1 widens the definition. Confabulation also covers outputs that diverge from the prompt or other input, and outputs that contradict earlier statements in the same context. NIST calls confabulation a natural result of how generative models work: they produce outputs that approximate the statistical distribution of their training data. NIST says this is particularly relevant for open-ended prompts that ask for long answers and for domains that need highly contextual or domain expertise.

NIST adds a point about citations. Outputs may include confabulated logic or citations that purport to justify the answer, which can mislead people into trusting it. NIST also states that it is difficult to estimate the scale and downstream impact of confabulations.

The profile maps confabulation to four trustworthiness characteristics: fair with harmful bias managed, safe, valid and reliable, and explainable and interpretable. Two of its suggested actions concern evaluation directly. MS-2.5-001 says not to extrapolate performance from narrow, non-systematic, and anecdotal assessments. MS-2.3-002 says to evaluate claims of model capabilities using empirically validated methods.

Intrinsic and extrinsic hallucination

Ji and colleagues' survey of hallucination, first posted in February 2022 and published in ACM Computing Surveys, borrows the term from psychology, where a hallucination is an unreal perception that feels real. Within natural language processing, the survey calls one definition the most inclusive and standard: generated content that is nonsensical or unfaithful to the provided source content.

The survey splits hallucination into two types:

• Intrinsic: the output contradicts the source. The survey's example is a summary that dates the first Ebola vaccine approval to 2021 when the source says 2019.
• Extrinsic: the output can be neither supported nor contradicted by the source. Such text is not always wrong, since it may be correct outside knowledge, but the survey notes that most of the literature treats it with caution.

Ji takes the split from earlier work, including a 2020 summarization study. Maynez and colleagues at Google Research studied abstractive summarization for ACL 2020. Annotators reviewed 2,500 document-summary pairs drawn from 500 articles in the XSum test set, including the human-written reference summaries. Intrinsic or extrinsic hallucinations appeared in more than 70 percent of the single-sentence summaries. Most hallucinations were extrinsic, and over 90 percent of those were erroneous. The authors also found that textual entailment measures tracked faithfulness better than standard metrics.

Factuality and faithfulness

A later survey by Huang and colleagues, accepted by ACM Transactions on Information Systems, builds on Ji's work for large language models (LLMs). The survey sorts hallucination into two types:

• Factuality hallucination: a conflict with verifiable real-world facts, divided into factual contradiction and factual fabrication.
• Faithfulness hallucination: divergence from the user's input or from the output's own consistency, divided into instruction inconsistency, context inconsistency, and logical inconsistency.

The two surveys draw the extrinsic line in slightly different places. Ji defines extrinsic output as unverifiable from the source content. Huang defines it as unverifiable from the source context or from external knowledge bases. A report that uses either term should say which definition it follows.

For a system that answers from a fixed set of documents, context inconsistency is the type to test: an answer unfaithful to the material the user supplied.

Measuring it: TruthfulQA and the grading problem

Lin, Hilton, and Evans built TruthfulQA, presented at ACL 2022, from 817 questions in 38 categories, including health, law, finance, and politics. The questions target false beliefs that some people hold. An answer is truthful if it avoids asserting a false statement, so "No comment" counts as truthful. For that reason the benchmark also scores informativeness.

The results, for models of that period:

Answerer	True	True and informative	False and informative
Best model (GPT-3-175B, helpful prompt)	58%	21%	42%
Human participant	94%	87%	6%

The largest models were generally the least truthful. The authors' automated judge, a fine-tuned GPT-3-6.7B model, predicted human ratings with 90-96 percent validation accuracy.

Kalai and colleagues, from OpenAI and Georgia Tech, argue that hallucinations persist because training and evaluation reward guessing over admitting uncertainty. Of ten popular benchmarks they review, nine give no credit for an answer of "I don't know," and one gives partial credit. They propose stating a confidence target in the instructions, with wrong answers penalized more as the target rises.

What to ask

• Which definition does a hallucination rate use: NIST's confabulation, intrinsic and extrinsic, or factuality and faithfulness?
• Is the rate measured against the documents the system answers from, or against world knowledge?
• Does the score reward a truthful non-answer, and is informativeness reported beside truthfulness?
• Does the evaluation give any credit for "I don't know," or does it reward guessing?
• Are citations in the output checked, given NIST's warning about confabulated citations?

Azcuna's ELLMENT, the E-2D Large Language Model Entity, was proposed to NAVAIR PMA-231 under Navy SBIR Phase I topic DON26BZ01-NV010. The prototype, built at private expense on publicly available doctrine, answers with a page-level citation and refuses when the documents do not support an answer. The proposal has been submitted.

Sources

1. 1Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 · National Institute of Standards and Technology (July 2024)
2. 2Survey of Hallucination in Natural Language Generation · arXiv (Ji et al.; ACM Computing Surveys)
3. 3On Faithfulness and Factuality in Abstractive Summarization · ACL Anthology (Maynez et al., ACL 2020)
4. 4A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions · arXiv (Huang et al.; accepted by ACM Transactions on Information Systems)
5. 5TruthfulQA: Measuring How Models Mimic Human Falsehoods · ACL Anthology (Lin, Hilton, and Evans, ACL 2022)
6. 6Why Language Models Hallucinate · arXiv (Kalai, Nachum, Vempala, and Zhang, September 2025)

Where this shows up in our work

• CALIBER, the market testbed for AI agents measures from an AI agent's trades whether it is gullible, herds, overpays, or can be steered.
• ELLMENT, the offline LLM for E-2D crews answers a crew's question from the aircraft's own manuals and cites the page, with no network.