Chunking: how documents are cut up for retrieval

Retrieval systems split documents into chunks because the unit they index, retrieve, and show to a language model is a passage, not a whole file. Where the cuts fall decides what the system can find and whether a retrieved passage still carries the context it needs. On a maintenance procedure, a bad cut returns an action without its hazard, and a passage with no record of its source cannot be checked against the page. That matters most where the documents cannot leave a secure room.

Retrieval indexes passages, not files

The original retrieval-augmented generation (RAG) paper by Lewis and colleagues pairs a pretrained sequence-to-sequence model with a dense vector index of Wikipedia, searched by a neural retriever. To build the index, each article was split into disjoint 100-word chunks, giving 21 million documents.

Dense passage retrieval, published the same year, used the same 100-word blocks as its basic retrieval units and ended with 21,015,324 passages. The preprocessing removed semi-structured content such as tables, info-boxes, and lists. The trained retriever beat a strong Lucene-BM25 baseline, a keyword-matching method, by 9 to 19 percentage points in top-20 passage accuracy.

Those benchmarks used encyclopedia text. A procedure manual leans on lists and tables, the kind of material that preprocessing step removed.

Size matters in both directions. A 2026 preprint on chunking methods notes that large chunks may contain irrelevant information, while small chunks may lack information the reader needs.

Fixed-size and structure-aware splitting

Fixed-size chunking cuts by count, regardless of content. It often uses overlapping windows to reduce mid-sentence splits and to preserve context at chunk edges, yet it still frequently fragments semantic dependencies. Fixed-size splitting nonetheless stayed a solid baseline in the 2026 preprint's results and was the most computationally efficient method it tested.

Structure-aware chunking uses the document's own layout, element types, or syntax trees to place boundaries. A study of oil and gas enterprise documents compared fixed-size sliding windows, recursive splitting, semantic splitting, and structure-aware splitting on a proprietary corpus of text-heavy manuals, table-heavy specifications, and piping and instrumentation diagrams. Structure-aware chunking performed best overall, with the clearest gains in top-K metrics, which score the highest-ranked results, and at lower computational cost. All four methods did poorly on the diagrams, which the authors attribute to a limit of text-only retrieval.

No single method wins everywhere. A taxonomy and evaluation from February 2026 finds the best strategy depends on the task: simple structure-based methods beat language-model-guided ones for retrieval across a corpus, while a different guided method did best for finding a passage inside one document. Chunk size relates moderately to effectiveness within a document and only weakly to effectiveness across a corpus.

The unit of indexing can also shrink below a passage. The Dense X Retrieval authors report that the retrieval unit significantly affects retrieval and downstream tasks, and that indexing by propositions, which are atomic, standalone expressions each encapsulating one factoid, outperformed passage-level units.

What a bad cut loses

Consider a procedure in which a warning box sits above step 4, and the step reads only "disconnect the connector." If a cut falls between them, a retriever that returns the step supplies an action without its hazard. One that returns the warning supplies a hazard without the action it governs. Neither chunk is wrong on its own, and neither is complete.

Overlap lowers the chance that a cut lands exactly there, though fixed-size splitting still breaks dependencies even with overlap. Structure-aware splitting can keep the warning and the step together when the source marks the relationship. The oil and gas authors conclude that preserving document structure is essential for specialized domains.

A cut through a table can likewise separate a row from the header that gives its cells meaning.

Every chunk needs a provenance record

The Lewis paper names provenance as an open research problem for models that hold knowledge only in their parameters. It presents retrieval as a way to make accessed knowledge inspectable and interpretable.

That benefit depends on the chunk. The generator sees only the retrieved chunks, so a citation can point only as far back as the chunk's own record. A useful record holds the source file and its version, the location inside it, such as page and section, and the position of the chunk's text. With those fields, a reviewer can open the original page, read the text on either side of the cut, and judge whether the chunk was a fair excerpt.

Without them, a chunk is an unattributed paragraph. A system can still answer from it, but no one can check where it came from. For a foreign-language document, the same record lets a reader set the original passage beside any translation of it.

What to ask

• Are the cuts placed by a fixed count, or by the document's own headings, lists, and tables?
• Does a warning stay in the same chunk as the step it governs, and a table row with its header?
• Was the chunking tested on manuals like these, or only on encyclopedia-style text?
• Does each chunk record the source file and version, the page and section, and the position of its text?
• Can a reviewer open the original page and read the text on either side of the cut?

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

The long read on air-gapped RAG goes further.

Sources

1. 1Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · arXiv (Lewis et al., NeurIPS 2020)
2. 2Dense Passage Retrieval for Open-Domain Question Answering · arXiv (Karpukhin et al., EMNLP 2020)
3. 3Dense X Retrieval: What Retrieval Granularity Should We Use? · arXiv (Chen et al.)
4. 4Chunking Methods on Retrieval-Augmented Generation: Effectiveness Evaluation Against Computational Cost and Limitations · arXiv (Śmigielski et al.)
5. 5Evaluating Chunking Strategies For Retrieval-Augmented Generation in Oil and Gas Enterprise Documents · arXiv (Taiwo and Yusoff)
6. 6Beyond Chunk-Then-Embed: A Comprehensive Taxonomy and Evaluation of Document Chunking Strategies for Information Retrieval · arXiv (Zhou et al.)