Document layout analysis, explained

Document layout analysis is the task of finding the regions on a page and naming what each one is: a heading, a paragraph, a list item, a table, a picture, a caption. Most current systems treat it as object detection and draw labeled boxes on an image of the page. The DocLayNet paper, presented at the 28th Conference on Knowledge Discovery and Data Mining (KDD) in August 2022, shows that models trained only on scientific articles lose accuracy on more varied layouts, such as technical manuals and company reports. On a technical manual, those boxes decide whether a warning stays with its step.

The task and how it is scored

The DocLayNet authors describe object detection as the most common current approach. A model receives a page image and outputs boxes, each with a class label. Ground truth is a set of boxes drawn by people or generated from a source file.

Scores are reported as mean average precision (mAP) over ten overlap thresholds, from 0.5 to 0.95 in steps of 0.05, written mAP@0.5-0.95. A predicted box earns credit only if it overlaps the true box closely enough, so a box that clips a table's last row or swallows its caption can lose credit at the stricter thresholds.

The ceiling is not 100. DocLayNet had some pages annotated two or three times and measured how far people agree. Overall human agreement was 82 to 83 mAP. The authors note that complex layouts can have different but equally acceptable interpretations, so human agreement sets a natural upper bound on accuracy.

PubLayNet and DocBank: scale from automatic labels

PubLayNet, a 2019 dataset from Zhong, Tang, and Jimeno Yepes of IBM Research Australia, labels pages automatically. The authors matched the Extensible Markup Language (XML) and Portable Document Format (PDF) versions of over 1 million PubMed Central articles and produced over 360,000 annotated page images in five classes: text, title, list, table, and figure. On the test set, Mask R-CNN reached a macro-average mAP of 0.907, with 0.955 for figures, 0.947 for tables, and 0.828 for titles, the hardest class because titles are small. Models pretrained on PubLayNet reached state-of-the-art table detection on a 2013 competition set of European government documents with only 170 training pages.

DocBank, presented at the 28th International Conference on Computational Linguistics (COLING) in 2020, took the same route with 500,000 pages and token-level labels drawn from LaTeX sources on arXiv.

The DocLayNet authors name the limit of this approach: automatic labels need a structured source file, and scientific articles are typeset in uniform publisher templates. Technical manuals, they write, have very different and partially unique layouts.

DocLayNet: varied pages, human labels

DocLayNet holds 80,863 pages annotated by hand in 11 classes, including Section-header, List-item, Table, Picture, Caption, Page-header, and Page-footer. The pages come from six categories, and Manuals is one of the two largest. Close to 95% are in English, and scanned documents were excluded wherever possible.

The best tested model, YOLOv5x6, scored 76.8 mAP overall, against 72.4 to 73.5 for Mask R-CNN and Faster R-CNN. Per class, the YOLO model's gaps varied. Page-footer scored 61.1 against human agreement of 93 to 94, and Section-header 74.6 against 83 to 84. Tables scored 86.3, above the human range of 77 to 81.

Two results matter for anyone reading a vendor's numbers. On the labels the datasets share, a Mask R-CNN model trained on PubLayNet scored 93 on PubLayNet's own test set and 30 on DocLayNet's; one trained on DocLayNet scored 78 at home and 59 on PubLayNet. And splitting the data by page instead of by document, so that pages of one document land in both training and test sets, raised mAP by about 10 points.

Table structure recognition

A box labeled Table says nothing about rows, columns, or headers. PubTables-1M, by Smock, Pesala, and Abraham of Microsoft at the 2022 Conference on Computer Vision and Pattern Recognition (CVPR), splits table extraction into detection, structure recognition, and functional analysis, which recognizes a table's keys and values. The dataset holds 947,642 tables for structure recognition, 52.7% of them complex, with at least one cell spanning rows or columns.

A canonicalization step, aimed at oversegmentation, where a spanning header cell is incorrectly split into several grid cells, adjusted the annotations of 34.7% of the tables. With the Detection Transformer (DETR), every cell's text matched exactly for 81.38% of test tables: 94.68% of simple tables and 69.44% of complex ones. Faster R-CNN managed 10.39%. Tables spanning more than one page were out of scope. The project repository adds that the model needs text extraction, from optical character recognition (OCR) or directly from the PDF, as a separate input.

Why layout errors matter in a technical manual

In a manual, meaning lives in order. MIL-STD-38784B, the Department of Defense standard practice for technical manual style dated 16 November 2020, says that for procedural steps "the warning or caution shall precede the paragraph or step," unless the acquiring activity specifies otherwise. Warnings precede cautions, which precede notes. The standard defines a warning as highlighting a procedure, practice, or condition that, if not strictly observed, could result in injury to or death of personnel, or long-term health hazards. The standard's foreword limits its use for Naval Air Systems Command (NAVAIR) programs to legacy data and manuals.

A layout model returns a box for the warning and a box for the step, and no label for the link between them. If the reading order puts a page footer between them, or a chunker cuts between them, the extracted text holds a step whose hazard sits elsewhere.

What to check

• Was the layout model tested on manual pages, with training and test data split by document, not by page?
• How does its score for each class compare with the human agreement range for that class?
• Does table output include the cell structure and headers, and how are tables continued across pages handled?
• Where does the text inside a detected table come from: the PDF itself or OCR?
• Does extracted text keep each warning, caution, and note directly before the step it governs, in the order MIL-STD-38784B sets?

Azcuna's TRACE is one offline viewer for S1000D, MIL-STD-3001 XML, and PDF technical manuals. It keeps the source location and the markings of everything it shows and never modifies the source. It was proposed to NAVAIR under Navy SBIR Phase I topic DON26BZ05-NV078, and the proposal was submitted on September 23, 2026.

Sources

1. 1DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis · arXiv (Pfitzmann et al., KDD 2022)
2. 2PubLayNet: largest dataset ever for document layout analysis · arXiv (Zhong, Tang, and Jimeno Yepes, 2019)
3. 3DocBank: A Benchmark Dataset for Document Layout Analysis · ACL Anthology (Li et al., COLING 2020)
4. 4PubTables-1M: Towards Comprehensive Table Extraction From Unstructured Documents · CVF Open Access (Smock, Pesala, and Abraham, CVPR 2022)
5. 5microsoft/table-transformer · Microsoft, GitHub repository
6. 6MIL-STD-38784B, Department of Defense Standard Practice: General Style and Format Requirements for Technical Manuals (16 November 2020) · Department of Defense, via EverySpec

Where this shows up in our work

• TRACE, the offline S1000D and IETM viewer opens S1000D, MIL-STD-3001 and PDF technical manuals on a tablet with every marking intact.
• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.