Evaluating language models
Long-context evaluation, explained
How the needle-in-a-haystack test works, what the RULER benchmark added and found about effective versus claimed context length, and what later studies show a long context window does not guarantee.
Oct 22, 20265 min read5 sources
Why it matters
A system that loads whole manuals into a model's input relies on the model using all of it, and RULER found only half of 17 models claiming 32K tokens or more held up at 32K.
The short version
- 1Nearly every model RULER tested scored almost perfectly on the plain needle test, yet most fell below the benchmark's threshold before reaching the context length they claimed.
- 2RULER calls a model's effective length the longest input at which it still beats a fixed score of 85.6 percent; for one model claiming 1 million tokens, that was under 4K.
- 3When the question shares no words with the needle, 11 of 13 models fell below half their short-input score at 32K tokens.
Long-context evaluation tests whether a language model can find and use information placed anywhere in a long input. The context length a model advertises is the most it accepts, not the most it uses well. The best-known check, the needle-in-a-haystack test, hides one fact in filler text and asks for it back. The RULER benchmark of Hsieh and colleagues, presented at the Conference on Language Modeling (COLM) in 2024, found that nearly all 17 models tested passed that test almost perfectly. Yet although all of them claimed context sizes of 32K tokens or more, only half kept satisfactory performance at 32K.
The needle-in-a-haystack test
The test comes from Gregory Kamradt's Needle In A Haystack repository, whose README dates the original runs on two commercial models to November 8 and November 21, 2023. The tool runs a sweep of context length against needle depth, meaning how far into the text the fact sits, and scores each cell. The default haystack is a set of Paul Graham essays. RULER notes that Kamradt's needle was a sentence about San Francisco.
The test, and a simpler variant called passkey retrieval, became a common way for model developers to show long-context ability. RULER's authors observe that these evaluations were used inconsistently across reports and reveal only the ability to retrieve.
A passed needle test shows one thing: the model can find a string and copy it back. The authors call that a superficial form of long-context understanding. In their appendix, almost all models scored perfectly on the plain needle test at their claimed length.
What RULER adds
RULER is synthetic, so the authors can set the input length and the difficulty. It has 13 tasks in four groups:
- Retrieval. Needle variants with several keys, several values, or several queries, where extra needles act as hard distractors.
- Multi-hop tracing. Variable tracking: a value passes along a chain of assignments, such as X2 = X1, and the model must list every name bound to it.
- Aggregation. Find the most common or most frequent words in a long list, a stand-in for summarization.
- Question answering. A paragraph that holds the answer is mixed into paragraphs from the same dataset.
Each model got 500 examples per task at each length from 4K to 128K tokens. The authors chose tasks that most models handle well at 4K, so the score measures how performance holds up as the input grows.
The error analysis, run on one 34-billion-parameter model, shows how models fail. With the haystack full of distractor needles, its score dropped by about 40 points at 256K. At 128K, over 80 percent of its output on one aggregation task was a string copied from the example at the top of the prompt. On question answering, its accuracy fell toward its score with no documents at all.
Claimed length and effective length
RULER sets one fixed bar: 85.6 percent, the score that a 7-billion-parameter baseline model with a 4K window reached at 4K. A model's effective length is the longest tested input at which it still beats that bar. The August 2024 version of the paper reports, among others:
| Model | Claimed length | Effective length |
|---|---|---|
| Gemini-1.5-Pro | 1M | over 128K |
| GPT-4 | 128K | 64K |
| GLM4 (9B) | 1M | 64K |
| Yi (34B) | 200K | 32K |
| DBRX (36B/132B) | 32K | 8K |
| LWM (7B) | 1M | under 4K |
The paper states its own limits. RULER reports one number per length, with no breakdown by the position of the key information. The proxy tasks have not been checked against realistic long-document work, and the authors write that RULER should not be preferred over more realistic settings. A good score at 4K should not be misread as perfect capability at 4K.
What a long window does not guarantee
A larger window is not better use of it. Liu and colleagues, in Transactions of the Association for Computational Linguistics (TACL) volume 12, compared models with their extended-context versions. When an input fit in both windows, performance was nearly identical. In an open-domain question answering case study, giving a model 50 retrieved documents instead of 20 improved accuracy by only about 1.5 percent for one model and about 1 percent for another, while raising the input length, and with it latency and cost.
Reasoning degrades well before the window is full. Levy, Jacoby, and Goldberg, in a paper that received an Outstanding Paper Award at the 2024 meeting of the Association for Computational Linguistics (ACL), padded the same reasoning questions to lengths from about 250 to 3,000 tokens. Averaged over the models tested, accuracy fell from 0.92 to 0.68 at 3,000 tokens. Chain-of-thought prompting did not close the gap.
Needle tests also allow word matching. NoLiMa, from Modarressi and colleagues at the 2025 International Conference on Machine Learning (ICML), writes needles that share almost no words with the question. One needle says a character lives next to the Semper Opera House; the question asks which character has been to Dresden. Of 13 models claiming at least 128K tokens, 11 fell below half their short-input score at 32K. One top model went from 99.3 percent to 69.7 percent.
What to ask
- Was long-context performance shown on a plain needle test or on harder tasks such as tracing, aggregation, or question answering with distractors?
- What is the model's effective length on a stated threshold, and how far is it from the claimed length?
- Did the test questions repeat the words of the passage that answers them, or require an inference to find it?
- Was accuracy measured with the key passage at the start, the middle, and the end of the input?
- Does a larger window improve answers on the system's own documents, or only raise the input length and the wait?
Azcuna's ELLMENT, the E-2D Large Language Model Entity, was proposed to NAVAIR PMA-231 under Navy SBIR Phase I topic DON26BZ01-NV010. The prototype, built at private expense on publicly available doctrine, answers with a page-level citation and refuses when the documents do not support an answer. The proposal has been submitted.
Sources
- 1RULER: What's the Real Context Size of Your Long-Context Language Models? · arXiv (Hsieh et al., COLM 2024; version 3, August 6, 2024)
- 2Needle In A Haystack · GitHub (Kamradt; README for version 2, read October 2026)
- 3Lost in the Middle: How Language Models Use Long Contexts · Transactions of the Association for Computational Linguistics, volume 12 (Liu et al., 2024)
- 4Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models · ACL Anthology (Levy, Jacoby, and Goldberg, ACL 2024)
- 5NoLiMa: Long-Context Evaluation Beyond Literal Matching · arXiv (Modarressi et al., ICML 2025)
Where this shows up in our work
- CALIBER, the market testbed for AI agents measures from an AI agent's trades whether it is gullible, herds, overpays, or can be steered.
- ELLMENT, the offline LLM for E-2D crews answers a crew's question from the aircraft's own manuals and cites the page, with no network.