Calibration in language models, explained

Calibration is the match between a model's stated confidence and how often it is right. Guo and colleagues, in a 2017 International Conference on Machine Learning (ICML) paper, give the standard illustration: of 100 predictions made with confidence 0.8, about 80 should be correct. The paper found modern neural networks poorly calibrated, and a one-parameter fix, temperature scaling, cut one network's calibration error from 16.53 percent to 1.26 percent. For a system that answers from documents, calibration matters at the refusal step. A system told to refuse below a confidence threshold refuses well only if its confidence means what it says.

How calibration is measured

Guo defines perfect calibration as accuracy equal to confidence at every confidence level. A finite sample cannot show that directly, so the paper sorts predictions into M bins of width 1/M and compares each bin's accuracy with its average confidence. A reliability diagram plots the two. The paper notes that the diagram does not show how many samples fall in each bin.

Expected calibration error (ECE), which Guo credits to Naeini and colleagues (2015), averages the bin gaps, weighted by each bin's share of the samples. Guo reports ECE with 15 bins.

ECE is not computed the same way everywhere. Kadavath and colleagues, in a 2022 study of language models, use 10 bins that hold equal numbers of predictions, not equally spaced bins. A 2019 study of calibration measures by Nixon and colleagues lists more problems. ECE looks only at the probability of the predicted class. Confident models crowd their predictions into a few bins, typically one or two, which then dominate the score. Overconfident and underconfident predictions in the same bin can cancel. In their experiments, the choice of measure drastically changed the ranking of recalibration methods.

Modern networks and temperature scaling

Guo compares a 5-layer LeNet from 1998 with a 110-layer ResNet from 2016 on CIFAR-100, an image set with 100 classes. The ResNet had lower error, 30.6 against 44.9, but its average confidence was substantially higher than its accuracy. Depth, width, weight decay, and batch normalization all affected calibration. In one training run, test error fell from 29 percent to 27 percent while the network overfit its loss: accuracy gained, in the authors' words, at the expense of well-modeled probabilities.

Temperature scaling divides the network's raw output scores (logits) by a single number T, fitted on a held-out validation set. Because T does not change which class scores highest, accuracy is unchanged. In the paper's table it brought ECE for that ResNet on CIFAR-100 from 16.53 percent to 1.26 percent, and for a DenseNet on ImageNet from 6.28 percent to 1.99 percent.

The method assumes training, validation, and test data come from the same distribution. Ovadia and colleagues tested what happens when they do not. In their NeurIPS 2019 benchmark, temperature scaling kept ECE low under small shifts, but ECE rose significantly as the shift grew. Calibration on the validation set, they conclude, does not guarantee calibration under distributional shift.

What language models report about their own answers

Kadavath and colleagues studied language models of 800 million to 52 billion parameters. On multiple-choice questions with visible lettered options, the larger models were well calibrated, and calibration improved with model size and with few-shot examples. Replacing one option with "none of the above" reduced accuracy and calibration significantly.

The paper also asks models about their own output. P(True) is the probability a model assigns that its own sampled answer is correct. P(IK), for "I know," is a trained estimate that the model will answer a question correctly. P(IK) partly generalized to new tasks but was poorly calibrated on them. It rose when relevant source material was in the context: for one obscure question, from 18 percent to 78 percent once a Wikipedia article was added.

Models tuned with reinforcement learning from human feedback (RLHF) appeared very miscalibrated, and one temperature, 2.5, largely fixed this on several evaluations. The authors caution that fine-tuned models will not be so well calibrated. The GPT-4 Technical Report points the same way: on a subset of the MMLU benchmark, ECE was 0.007 for the pre-trained model and 0.074 after post-training, which the report says hurt calibration significantly.

Calibration and the refusal threshold

A refusal rule answers above a confidence threshold and abstains below it. Geifman and El-Yaniv, at NIPS 2017, built such a rule for neural network classifiers and tested it on images. From a labeled sample, a target error rate, and a confidence level, their method finds the threshold that keeps error on answered inputs under the target. On ImageNet it guaranteed a 2 percent top-5 error with probability 99.9 percent while answering almost 60 percent of test inputs. The method assumes only that the score ranks inputs and that the sample comes from the same distribution as later inputs.

Two properties matter here. Ranking means confident answers are right more often than unconfident ones; Kadavath notes that the area under the receiver operating characteristic curve (AUROC) measures this and is indifferent to calibration. Calibration means the number itself reads as a probability. A threshold tuned on labeled questions from the documents in service needs ranking. A threshold read off the number alone, such as "refuse below 0.9," also needs calibration.

Kalai and colleagues give the arithmetic in a September 2025 paper. If a wrong answer costs t/(1 - t) points, a right one earns 1, and "I don't know" earns 0, answering beats abstaining only when the probability of being correct exceeds t. At t = 0.9 the penalty is 9 points. They also warn that calibration error is not a usefulness measure: a model that is always wrong and always reports 0 percent confidence has zero calibration error.

What to ask

• How was calibration measured: how many bins, equally spaced or equally filled, and on the predicted answer only?
• Was calibration measured on questions from the system's own documents, or only on a public benchmark?
• Was the model fine-tuned after pre-training, and was calibration measured again afterward?
• Where does the refusal threshold come from: a labeled sample with a stated target error, or the raw score?
• At that threshold, what share of questions is answered, and how often are those answers wrong?

Azcuna's ELLMENT, the E-2D Large Language Model Entity, was proposed to NAVAIR PMA-231 under Navy SBIR Phase I topic DON26BZ01-NV010. The prototype, built at private expense on publicly available doctrine, answers with a page-level citation and refuses when the documents do not support an answer. The proposal has been submitted.

Sources

1. 1On Calibration of Modern Neural Networks · Proceedings of the 34th International Conference on Machine Learning, PMLR 70 (Guo, Pleiss, Sun, and Weinberger, 2017)
2. 2Measuring Calibration in Deep Learning · arXiv (Nixon et al., 2019, revised 2020)
3. 3Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift · arXiv (Ovadia et al.; NeurIPS 2019)
4. 4Language Models (Mostly) Know What They Know · arXiv (Kadavath et al., July 2022, revised November 2022)
5. 5GPT-4 Technical Report · arXiv (OpenAI, March 2023, revised March 2024)
6. 6Selective Classification for Deep Neural Networks · Advances in Neural Information Processing Systems 30 (Geifman and El-Yaniv, NIPS 2017)
7. 7Why Language Models Hallucinate · arXiv (Kalai, Nachum, Vempala, and Zhang, September 2025)

Where this shows up in our work

• CALIBER, the market testbed for AI agents measures from an AI agent's trades whether it is gullible, herds, overpays, or can be steered.
• ELLMENT, the offline LLM for E-2D crews answers a crew's question from the aircraft's own manuals and cites the page, with no network.