Semantic entropy, explained

Semantic entropy measures how much a language model's answers to one question disagree in meaning, across several samples. Farquhar, Kossen, Kuhn, and Gal used it in a paper published in Nature on June 19, 2024 to detect confabulations, a kind of hallucination in which a model gives answers that are wrong and arbitrary. If the samples agree in meaning, the model is probably not guessing. If they scatter, it probably is. Averaged over 30 combinations of tasks and models, semantic entropy scored 0.790 on a standard detection measure, against 0.691 for plain entropy over the words. The price is sampling many answers to every question.

What the method detects

The paper defines confabulations as claims that are both wrong and arbitrary, meaning the answer changes with irrelevant details such as the random seed. The authors' example is a medical question about the target of the drug Sotorasib. The same model, given the same instructions, sometimes answered KRAS G12C, which is correct, and sometimes G12D, which is not.

The authors separate this from errors with other causes: a model trained on a common misconception, a model that misleads in pursuit of a reward, and systematic failures of reasoning. Semantic entropy does not address those. The paper states plainly that the method does not guarantee factuality, because it does not help when outputs are systematically bad. A model that gives the same wrong answer every time shows low entropy.

The approach began in an earlier paper by Kuhn, Gal, and Farquhar, a spotlight at the International Conference on Learning Representations (ICLR) in 2023. A related idea, SelfCheckGPT by Manakul and colleagues at the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), also samples several responses, on the reasoning that facts a model knows stay consistent across samples while hallucinated ones diverge.

How it works

The method has three steps.

• Sample. The model answers the same question M times at temperature 1, with nucleus sampling (P = 0.9) and top-K sampling (K = 50). One more answer at low temperature, 0.1, serves as the answer whose accuracy is judged.
• Cluster. Answers go in the same group only if each entails the other. The authors' example: "Paris," "It's Paris," and "France's capital Paris" share one meaning in answer to a question about the capital of France. The main results use GPT-3.5 with a short prompt to judge entailment; a smaller model trained for entailment is the alternative.
• Measure. Entropy is computed over the groups, not the individual strings. High entropy means the answers disagree in meaning.

A discrete version estimates each group's share from the count of samples in it, so it works without the model's token probabilities. The authors used it for GPT-4, which did not expose probabilities at the time.

The entailment step was checked against people. In the supplementary information, two raters labeled pairs of answers to 100 questions from each of three datasets, 600 answers in all. They agreed with each other 87 percent of the time and with GPT-4 87 percent of the time. GPT-3.5, the judge in the main results, agreed with them 83 percent of the time.

What the Nature paper reported

The tests covered trivia (TriviaQA), general knowledge (SQuAD 1.1), life sciences (BioASQ), real search queries (NQ-Open), and math word problems (SVAMP), with 400 test questions each. The authors withheld the context passages that some of these datasets include, to induce confabulations. Six open models from three families, from 7 to 70 billion parameters, wrote the answers. GPT-4 judged each answer correct or not against the dataset's reference.

The main measure was the area under the receiver operating characteristic curve (AUROC) for spotting wrong answers, where 1 is perfect and 0.5 is no better than chance. Semantic entropy averaged 0.790. Plain entropy scored 0.691, a method that asks the model whether its own answer is true scored 0.698, and a trained classifier on the model's internal state scored 0.687. Semantic entropy stayed between 0.78 and 0.81 across model families and sizes.

For longer text, the authors split GPT-4 biographies of 21 people into 150 factual claims, labeled by hand. The discrete version outperformed both baselines overall. When the 20 percent of claims judged riskiest were set aside, one baseline had a narrow edge in accuracy on the rest.

The cost of sampling

Every question needs extra answers. The paper used ten samples for sentence-length and short-phrase answers; the supplement reports diminishing returns after about five, though ten still helped. Clustering can take up to M squared comparisons, each made by the entailment model. The authors argue this cost is small next to generating the answers, because the number of meaning groups is usually low and identical strings can reuse earlier results.

A June 2024 paper by Kossen and colleagues, with two of the Nature paper's authors among them, puts the added computation at a 5-to-10-fold increase and names it as an obstacle to practical use. Their proposed fix, semantic entropy probes, estimates the score from the model's internal state on a single answer, which needs access to that state. Karin Verspoor's Nature News & Views piece summarizes the original design: a second model groups the outputs, and a third evaluates the method.

What to ask

• Is the system meant to catch arbitrary errors, consistent errors, or both? Semantic entropy addresses only the first.
• How many answers are sampled per question, and what does that do to response time on the hardware in service?
• Which model judges entailment, and was its agreement with human raters measured?
• Was the method tested with the documents in the prompt, as the system will run? The Nature tests withheld context passages.
• Who decided which answers were correct in the evaluation: people, or another model?

Azcuna's ELLMENT, the E-2D Large Language Model Entity, was proposed to NAVAIR PMA-231 under Navy SBIR Phase I topic DON26BZ01-NV010. The prototype, built at private expense on publicly available doctrine, answers with a page-level citation and refuses when the documents do not support an answer. The proposal has been submitted.

Sources

1. 1Detecting hallucinations in large language models using semantic entropy · Nature 630, 625-630 (Farquhar, Kossen, Kuhn, and Gal; published June 19, 2024)
2. 2Supplementary information: Detecting hallucinations in large language models using semantic entropy · Nature (supplementary material in the format provided by the authors, 2024)
3. 3Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation · arXiv (Kuhn, Gal, and Farquhar; ICLR 2023)
4. 4SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models · ACL Anthology (Manakul, Liusie, and Gales; EMNLP 2023)
5. 5Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs · arXiv (Kossen, Han, Razzak, Schut, Malik, and Gal; June 2024)
6. 6'Fighting fire with fire' - using LLMs to combat LLM hallucinations · Nature News & Views (Verspoor, 2024)

Where this shows up in our work

• CALIBER, the market testbed for AI agents measures from an AI agent's trades whether it is gullible, herds, overpays, or can be steered.
• ELLMENT, the offline LLM for E-2D crews answers a crew's question from the aircraft's own manuals and cites the page, with no network.