What is perplexity?

Perplexity is the standard intrinsic score for a language model: a single number that says how well the model predicted a test text. The Hugging Face Transformers documentation defines it as the exponentiated average negative log-likelihood of a sequence, and Jurafsky and Martin's textbook Speech and Language Processing defines the same quantity as the inverse probability of the test set normalized by its length. Lower is better. The number is cheap to compute and appears in model cards, quantization tables, and scaling-law papers, but it is tied to the tokenizer and the test text it was measured on, and a low perplexity does not by itself show that a model is useful for a task.

The perplexity formula

Start with a tokenized sequence X of t tokens. A language model assigns each token a probability given the tokens before it, p(x_i | x_<i). The Transformers documentation writes perplexity as

PPL(X) = exp( -(1/t) * sum over i of log p(x_i | x_<i) )

The quantity inside the exponential is the average negative log-likelihood per token, which is the cross-entropy loss that the model was trained to minimize. Perplexity is that loss with the logarithm undone. If a model's average loss on a test set is 3.0 nats per token, its perplexity is e to the 3.0, about 20.1. If the loss is 2.0, the perplexity is about 7.4. The documentation notes that this is equivalent to exponentiating the cross-entropy between the data and the model's predictions.

Jurafsky and Martin reach the same formula from the other direction. In their Section 3.3, the perplexity of a model on a test set W of N words is P(w_1 ... w_N) raised to the power -1/N, which they expand by the chain rule into the Nth root of the product of 1 over P(w_i | w_1 ... w_{i-1}). Because of the inverse, the higher the probability the model assigns to the sequence, the lower the perplexity, and minimizing perplexity is the same as maximizing the probability of the test set. Their Section 3.7 gives the information-theoretic form: perplexity equals 2 raised to the cross-entropy H(W) of the model on the text, when the logarithm is taken in base 2. Whether the base is e or 2, the same loss gives the same perplexity, because the exponential and the logarithm use the same base.

A worked example from the textbook shows the scale. The authors trained unigram, bigram, and trigram models on 38 million words of Wall Street Journal text and measured each on a 1.5-million-word test set. The perplexities were 962, 170, and 109. The trigram model, which conditions on two previous words, was the least surprised by the test text and therefore the best predictor of it.

What perplexity measures

Jurafsky and Martin describe perplexity as a weighted average branching factor: the effective number of equally likely choices the model faces at each step. In their toy example, a language with 3 colors in which any color can follow any other has a branching factor of 3, and a model that assigns each color probability 1/3 scores a perplexity of exactly 3 on any test string. A second model trained on text where red appeared 80 percent of the time scores lower on the test string "red red red red blue", because most of its predictions were confident and correct, even though the branching factor is still 3. Perplexity therefore rewards confident probability on the tokens that actually occur and punishes probability spread across tokens that do not.

Read that way, a perplexity of 20 says the model was, on average, as uncertain as if it were choosing uniformly among 20 tokens at each step. A perplexity of 7.16 means fewer effective choices. That figure is the one the Chinchilla paper reports for its 70-billion-parameter model on WikiText-103, against 7.75 for the 280-billion-parameter Gopher. The Pile paper observes the general pattern: larger language models attain lower perplexity than smaller ones, and the GPT-3 family follows a scaling-law line on the Pile.

What perplexity does not measure is whether any of those predictions are correct in the sense a user cares about. Perplexity is computed over every token of a fixed text, including function words, punctuation, and formatting, with no notion of a question or an answer. A model can be an excellent predictor of what text usually looks like and still produce a wrong answer with high confidence. The textbook puts this as the difference between an intrinsic metric and an extrinsic one, discussed below.

What a perplexity figure depends on

A perplexity number comes with conditions attached, and comparing two numbers measured under different conditions is the most common misreading.

Tokenizer and vocabulary

Perplexity is per token, and the number of tokens in a text depends on the tokenizer. The Transformers documentation says the tokenization procedure "has a direct impact on a model's perplexity" and should always be taken into account when comparing models. Jurafsky and Martin state that the perplexity of two language models is only comparable if they use identical vocabularies. A tokenizer that splits a sentence into 30 pieces and one that splits it into 20 are being asked different numbers of questions about the same text, and the per-question average is not the same quantity. The Pile paper handles this explicitly: it notes that GPT-2 and GPT-3 share the same tokenizer and vocabulary, which is what made their perplexity scores directly comparable in that study. The post on tokenizers and language cost shows how far token counts differ across languages for the same sentence, which is the same effect from another angle.

Context window and stride

Neural models have a maximum context, and how the test text is fed through it changes the result. The Transformers documentation walks through the case of GPT-2, whose largest version has a fixed length of 1,024 tokens. Breaking a long text into disjoint 1,024-token chunks and scoring each one independently gives the model little context at the start of each chunk and yields a higher, worse perplexity. A sliding window that moves by a smaller stride gives the model more preceding tokens for each prediction at the cost of more forward passes. On the WikiText-2 test set with gpt2-large, the documentation reports a perplexity of 19.44 with a stride of 1,024 and no overlap, close to the 19.93 in the GPT-2 paper, and 16.44 with a stride of 512. Same model, same text, a 3-point difference from the evaluation procedure alone. The Pile paper adds another variable: it scores each document independently rather than concatenating all documents, which is the more common practice.

The test text

Perplexity is a property of a model and a text together. Jurafsky and Martin say the model must be built without any knowledge of the test set, "or else the perplexity will be artificially low." The Chinchilla paper gives the real-world version of that caution: when comparing its model with Gopher on language-modeling benchmarks, the authors note that Chinchilla was trained on 4 times more data, so train-test leakage may have artificially enhanced the results, and they place more weight on other evaluations. The post on held-out test sets covers how leakage and contamination invalidate a score.

Bits per byte, the comparable alternative

Because per-token perplexity cannot be compared across tokenizers, the Pile paper reports its results in bits per UTF-8 encoded byte (BPB). The paper says bits per byte is preferred over bits per character or perplexity when using the Pile as a metric, because of its invariance to different tokenization schemes and the ambiguity of measuring characters in Unicode. The conversion from a model's negative log-likelihood loss l, in nats per token, is

BPB = (L_T / L_B) * l / ln(2)

where L_T is the length of the dataset in tokens and L_B is its length in UTF-8 bytes. The ratio L_T / L_B is the tokenizer's tokens per byte on that text; across the Pile, the GPT-2 tokenizer gives 0.29335 tokens per byte, and the paper tabulates the ratio for each component. Worked through: a model with a loss of 2.0 nats per token on text it tokenizes at 0.29335 tokens per byte has a BPB of 0.29335 times 2.0 divided by 0.6931, about 0.846. A model with a different tokenizer and a different per-token loss on the same bytes can be compared on this scale, because the bytes are the same for both.

The measure has become standard for cross-model reporting. The Chinchilla paper reports bits per byte on every Pile subset and states that Chinchilla outperformed Gopher on all of them, while Jurassic-1 outperformed Chinchilla on two subsets, DM Mathematics and Ubuntu IRC. Bits per byte is still a prediction score, so everything said above about test-text leakage applies to it; what it removes is the tokenizer from the comparison.

Perplexity in scaling laws

The scaling-law literature works with the loss that perplexity is the exponential of, and the two are interchangeable for a fixed tokenizer. Kaplan and colleagues' Scaling Laws for Neural Language Models (January 23, 2020) studied the cross-entropy loss, in nats per token, of transformers trained on WebText2 with a byte-pair-encoding vocabulary of 50,257 tokens and a 1,024-token context. The paper reports that the loss scales as a power law with model size, dataset size, and training compute, with some trends spanning more than seven orders of magnitude, while architectural details such as width or depth had minimal effect within a wide range. The three fitted laws are L(N) = (N_c / N) to the 0.076 with N_c about 8.8 times 10 to the 13 non-embedding parameters; L(D) = (D_c / D) to the 0.095 with D_c about 5.4 times 10 to the 13 tokens; and L(C_min) = (C_c / C_min) to the 0.050 with C_c about 3.1 times 10 to the 8 petaflop-days.

Hoffmann and colleagues' Training Compute-Optimal Large Language Models (March 29, 2022) revisited the question with over 400 models from 70 million to over 16 billion parameters trained on 5 to 500 billion tokens. The paper concluded that for every doubling of model size the number of training tokens should also double, and that models of the time were significantly undertrained because they had been scaled in parameters while training on roughly 300 billion tokens. The paper's parametric fit is L(N, D) = E + A / N to the 0.34 + B / D to the 0.28, with E = 1.69, A = 406.4, and B = 410.7; the constant E is the loss the fit predicts would remain with unlimited parameters and data. The test case was Chinchilla, 70 billion parameters trained on 1.4 trillion tokens, which outperformed the 280-billion-parameter Gopher trained on 300 billion tokens on the paper's downstream tasks and reached 67.5 percent on MMLU. The paper's Table 3 projects that a 67-billion-parameter model should see about 1.5 trillion tokens and a 175-billion-parameter model about 3.7 trillion.

For a reader of these papers, the point is that every result is a statement about loss on a held-out distribution, which is perplexity on a log scale. The curves are smooth and predictable, which is what makes them useful for planning training runs. Whether the predicted loss translates into task performance is a separate question, which the papers address with separate downstream evaluations.

Why low perplexity does not mean a good assistant

Jurafsky and Martin state the general caution: an intrinsic improvement in perplexity does not guarantee an extrinsic improvement in a task such as speech recognition or machine translation, and a perplexity improvement should be confirmed by an end-to-end evaluation on a real task when possible. Because perplexity usually correlates with task improvements, it remains a convenient metric. The question for an assistant model is whether the correlation holds once the model is being trained to answer rather than to continue text.

The LIMA paper (Zhou et al., May 18, 2023) reports the case where it did not. The authors fine-tuned a 65-billion-parameter LLaMA model on 1,000 curated prompts and responses. In the body of the paper they write that perplexity does not correlate with generation quality, and Appendix B explains why they say so. While fine-tuning, they measured perplexity on 2,000 held-out Stack Exchange examples and found it negatively correlated with the model's ability to produce quality responses: as perplexity rose with more training steps, which would normally be read as overfitting, the quality of the generations, scored by ChatGPT, increased. Their Figure 9 plots validation perplexity against generation quality across training for the 65B model, and the authors state they observed similar trends for 7B and 30B models and across different training mixtures. Lacking an intrinsic measure they could trust, they selected checkpoints by hand on a 50-example validation set.

The mechanism is not mysterious. A fine-tuned assistant is being pushed toward a particular style of response, and a held-out text written in a different style becomes less probable under it, even as the responses themselves get better. The loss measures agreement with a text; the user measures whether the answer was right, complete, and grounded. The two can move in opposite directions. Perplexity still has a role in detecting damage: the post on model quantization shows how the GPTQ paper used WikiText-2 perplexity to show that 4-bit weights changed a 175-billion-parameter model's score only from 8.34 to 8.37. That is a comparison of one model with itself on one text, which is exactly the setting where perplexity is reliable.

Perplexity as a contamination and membership test

The same property that makes perplexity a weak judge of usefulness makes it a useful detector. Text that a model saw during training tends to receive an unusually low perplexity, and that signal can be turned on the model to ask what it was trained on.

Carlini and colleagues' Extracting Training Data from Large Language Models (first posted December 14, 2020) built an attack on GPT-2 around this. The authors generated 200,000 samples from the 1.5-billion-parameter model, sorted them by the model's perplexity, and inspected the lowest. Because plain perplexity also flags trivially common text, such as counting from 1 to 100, and repeated strings, the paper added five comparison metrics: the ratio of the large model's log-perplexity to that of the Small (124 million parameter) and Medium (334 million) GPT-2 models, the ratio of the perplexity to the text's zlib compression entropy, the ratio of perplexity before and after lowercasing the sample, and the minimum perplexity over any sliding window of 50 tokens. Across all strategies, 604 of 1,800 manually checked candidates were confirmed as memorized training examples, a true positive rate of 33.5 percent, with the best single variant at 67 percent. The paper also reports that the 1.5-billion-parameter model memorized over 18 times as much content as the 124-million-parameter one in one setting.

Later work turned the idea into a test for benchmark contamination. Shi and colleagues' Detecting Pretraining Data from Large Language Models (October 25, 2023) proposed Min-K% Prob, which scores a text by the probability of its k percent lowest-probability tokens, on the reasoning that unseen text tends to contain a few outlier words the model finds very unlikely, while seen text does not. The method needs no access to the training corpus and no reference model, and the authors report a 7.4 percent improvement over prior methods on their WikiMIA benchmark, which is built from text written before and after a model's training date so that the ground truth is known. Li's Estimating Contamination via Perplexity (September 19, 2023) applies perplexity directly to benchmark samples to estimate memorization without the training data, and reports significant memorization by several recent models on popular reading comprehension and summarization benchmarks, with multiple-choice benchmarks appearing less contaminated.

For an evaluator, the practical reading is that an unusually low perplexity on a benchmark's test items is evidence that the benchmark score is inflated, not evidence that the model is good.

What to check

• Which tokenizer and vocabulary produced a quoted perplexity, and are the models being compared using the same one? If not, ask for bits per byte.
• What test text was used, and is there any evidence it appeared in the training data?
• How was the text fed through the context window, and what stride was used, since the Transformers documentation shows a 3-point swing from that choice alone?
• Was perplexity used to compare one model with itself, as in a quantization table, or to compare different models, where it is far less reliable?
• Has the model's quality on the actual task been measured end to end, since perplexity can rise while answers improve?
• Were the benchmark items checked for low perplexity or a Min-K% Prob signal before the score was accepted?

Azcuna's ELLMENT, the E-2D Large Language Model Entity, was proposed to NAVAIR PMA-231 under Navy SBIR Phase I topic DON26BZ01-NV010. The prototype, built at private expense on publicly available doctrine, answers with a page-level citation and refuses when the documents do not support an answer. The proposal has been submitted.

Questions people ask

What is perplexity in a language model?

Perplexity is a score for how well a language model predicts a test text. Jurafsky and Martin's Speech and Language Processing defines it as the inverse probability of the test set, normalized by the number of words or tokens by taking the Nth root. The Hugging Face Transformers documentation gives the equivalent form used for neural models: the exponential of the average negative log-likelihood of each token given the tokens before it. A lower perplexity means the model assigned the text a higher probability.

What is the perplexity formula?

For a tokenized sequence of t tokens, the Transformers documentation gives PPL = exp of minus one over t times the sum over i of log p(x_i given x_<i), where p is the model's probability for token i given the preceding tokens. Jurafsky and Martin write the same quantity as the probability of the whole sequence raised to the power minus 1/N, and in Section 3.7 as 2 raised to the cross-entropy of the model on the text. A model that assigns every token probability 1 has perplexity 1.

What does a lower perplexity mean?

A lower perplexity means the model was less surprised by the test text: it assigned a higher probability to the tokens that actually occurred. Jurafsky and Martin describe perplexity as a weighted average branching factor, the effective number of equally likely next words at each step. In their example, a unigram, bigram, and trigram model trained on 38 million words of Wall Street Journal text scored 962, 170, and 109 on the same 1.5-million-word test set, the trigram model being the better predictor.

Can perplexity be compared across models with different tokenizers?

No. The Transformers documentation says the tokenization procedure has a direct impact on perplexity and should always be taken into consideration when comparing models, and Jurafsky and Martin state that perplexities are only comparable if the models use identical vocabularies. Per-token perplexity depends on how many tokens the text was cut into. The Pile paper notes that GPT-2 and GPT-3 share one tokenizer and vocabulary, which is what made their perplexity scores directly comparable in that study.

What is bits per byte?

Bits per byte (BPB) is the model's loss expressed per UTF-8 encoded byte of the test text rather than per token. The Pile paper (Gao et al., 2020) defines it as the ratio of tokens to bytes, L_T over L_B, times the negative log-likelihood loss divided by the natural log of 2, and prefers it over perplexity because it is invariant to the tokenization scheme. Across the Pile, the GPT-2 tokenizer produced 0.29335 tokens per byte. The Chinchilla paper reports its Pile results in bits per byte.

How is perplexity used in scaling laws?

Scaling-law papers work with the cross-entropy loss that perplexity is the exponential of. Kaplan et al. (2020) measured loss in nats per token on WebText2 with a 50,257-token vocabulary and found it falls as a power law in parameters, data, and compute, with exponents of about 0.076, 0.095, and 0.050. Hoffmann et al. (2022) fit loss as E plus A over N to the 0.34 plus B over D to the 0.28, with E = 1.69, and concluded that parameters and training tokens should grow equally.

Does low perplexity mean a model is accurate?

Not by itself. Jurafsky and Martin state that an intrinsic improvement in perplexity does not guarantee an extrinsic improvement on a real task and should be confirmed by an end-to-end evaluation. The LIMA paper (Zhou et al., 2023) reports the sharper case: during fine-tuning of a 65-billion-parameter model, perplexity on 2,000 held-out Stack Exchange examples rose with more training steps while generation quality, as judged by ChatGPT, improved, so the authors selected checkpoints manually on a 50-example validation set.

How is perplexity used to detect training data contamination?

Text a model has seen in training tends to get an unusually low perplexity. Carlini et al. (2021) sorted 200,000 GPT-2 samples by perplexity and related ratios and confirmed 604 of 1,800 candidates as memorized training text. Shi et al. (2023) proposed Min-K% Prob, which scores a text by its lowest-probability tokens, and reported a 7.4 percent improvement on their WikiMIA benchmark. Li (2023) used perplexity on benchmark samples to estimate contamination without access to the training data.

Why is perplexity not defined for BERT?

The Transformers documentation states that perplexity applies to classical, autoregressive language models and is not well defined for masked language models like BERT. Perplexity needs a probability for each token given the tokens before it, multiplied along the sequence. A masked model predicts a hidden token from context on both sides, so it does not assign one probability to the whole sequence in that way, and the per-token average that perplexity exponentiates does not exist in the same form.

Sources

1. 1Speech and Language Processing, Chapter 3: N-gram Language Models (draft of August 19, 2026) · Stanford University (Jurafsky and Martin)
2. 2Perplexity of fixed-length models · Hugging Face Transformers documentation
3. 3Scaling Laws for Neural Language Models · arXiv (Kaplan et al.)
4. 4Training Compute-Optimal Large Language Models · arXiv (Hoffmann et al.)
5. 5The Pile: An 800GB Dataset of Diverse Text for Language Modeling · arXiv (Gao et al.)
6. 6LIMA: Less Is More for Alignment · arXiv (Zhou et al.)
7. 7Extracting Training Data from Large Language Models · arXiv (Carlini et al.)
8. 8Detecting Pretraining Data from Large Language Models · arXiv (Shi et al.)
9. 9Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation · arXiv (Li)

Where this shows up in our work

• CALIBER, the market testbed for AI agents measures from an AI agent's trades whether it is gullible, herds, overpays, or can be steered.
• ELLMENT, the offline LLM for E-2D crews answers a crew's question from the aircraft's own manuals and cites the page, with no network.