LayoutLM, LayoutLMv2, and LayoutLMv3, explained

LayoutLM is a family of document-understanding models from Microsoft that read a scanned page using the words on it, where those words sit, and, from the second version on, what the page looks like. The first paper, LayoutLM: Pre-training of Text and Layout for Document Image Understanding, was posted on December 31, 2019 and presented at KDD 2020; LayoutLMv2 followed at ACL 2021 and LayoutLMv3 at ACM Multimedia 2022. Each version is a pre-trained Transformer that is then fine-tuned to label the fields on a form, classify a page, or answer a question about it. None of them reads pixels into text: optical character recognition (OCR) runs first, and the model works on its output. This page explains what each version adds, what the papers report on the standard benchmarks, what the licenses allow, and what the models are good for on a technical manual.

What LayoutLM adds to BERT: text plus layout

A plain language model such as BERT sees a page as a sequence of words. A form is not a sequence; the value "12 Nm" means something because it sits beside the label "Torque" and under the heading for a particular fastener. The LayoutLM paper's starting observation is that pre-training methods for text mostly ignored this layout and style information.

The fix is small and effective. LayoutLM is BERT with extra input embeddings. For each word, the OCR bounding box gives four coordinates, x0, y0, x1, y1, and the paper scales each to a virtual range of 0 to 1,000 so pages of any size share one coordinate space. The four coordinates share two embedding tables, one for x values and one for y values, and the four embeddings are added to the usual word and position embeddings. The model is then pre-trained on scanned pages with a Masked Visual-Language Model (MVLM) objective: 15 percent of the words are masked, their 2-D positions are kept, and the model predicts the words from the surrounding text and the positions. A second objective, Multi-label Document Classification (MDC), uses the document tags that come with the pre-training collection. Image features, cropped from a Faster R-CNN with a ResNet-101 backbone for each word's box, are added at fine-tuning in the paper's best configuration.

The pre-training data is the IIT-CDIP Test Collection 1.0, which the paper describes as more than 6 million documents and more than 11 million scanned page images. The collection supplies text but not word positions, so the authors re-ran OCR on every page with Tesseract and stored the words and boxes in hOCR format. The base model has about 113 million parameters, the large model about 343 million. The paper states that its scope is scanned documents; digital-born documents, which need no OCR, are set aside.

LayoutLMv2: image, layout, and text in one Transformer

LayoutLMv2, by Xu and colleagues, posted December 29, 2020, moves the image into pre-training. The page is resized to 224 by 224 pixels and passed through a ResNeXt-FPN convolutional encoder whose output is flattened into a sequence of visual tokens. Those tokens are concatenated with the text tokens, and both carry layout embeddings, so a single Transformer attends across text, image, and position together.

Two further changes matter. The attention mechanism becomes spatial-aware: learnable biases for the relative 1-D and 2-D positions of tokens are added to the attention scores, so the model can learn that a label tends to sit to the left of or above its value. And two pre-training tasks join MVLM. Text-Image Alignment (TIA) covers some text lines in the image and asks the model to predict, per token, whether its line was covered, which forces the text and image streams to line up. Text-Image Matching (TIM) asks whether the text and image come from the same page. Pre-training again used the 11 million IIT-CDIP pages, this time with words and boxes from the Microsoft Read API. The visual encoder raises the size: about 200 million parameters for base and 426 million for large.

LayoutLMv3: unified masking and no CNN

LayoutLMv3, by Huang, Lv, Cui, Lu, and Wei, posted April 18, 2022, simplifies the architecture and unifies the training. The convolutional backbone is gone. The page image is cut into 16 by 16 pixel patches, each patch is projected linearly, as in a Vision Transformer, and the patch sequence is fed to the Transformer alongside the text. Layout embeddings move from word level to segment level: words that the OCR grouped into one line or block share one bounding box.

The three pre-training objectives are symmetric. Masked Language Modeling (MLM) masks 30 percent of text tokens with span masking. Masked Image Modeling (MIM) masks about 40 percent of image patches blockwise and predicts, not pixels, but discrete tokens from an image tokenizer with a vocabulary of 8,192. Word-Patch Alignment (WPA) asks, for each unmasked text token, whether the image patches under it are masked or not, which teaches the model which patches belong to which words. The paper's title names the idea: unified text and image masking. The base model has 133 million parameters and the large 368 million, smaller than v2 because the CNN is gone. Pre-training used about 11 million IIT-CDIP images, with text from what the paper calls an off-the-shelf OCR toolkit; for RVL-CDIP and DocVQA the authors used the Microsoft Read API.

The benchmarks: FUNSD, CORD, SROIE, RVL-CDIP, DocVQA

The papers report on a shared set of public datasets, each standing for a task.

FUNSD stands for form understanding. The FUNSD paper (Jaume, Ekenel, and Thiran, ICDAR 2019 workshop) describes 199 real, fully annotated scanned forms, sampled from RVL-CDIP after a manual review of 25,000 form images, split 149 for training and 50 for testing. They carry 31,485 words, 9,743 semantic entities labeled header, question, answer, or other, and 5,312 links between entities. The paper notes that Tesseract performed poorly on these forms, likely because they fall below its 300 dpi minimum, and that Google Vision did best. The task scored in the LayoutLM papers is entity labeling, reported as F1.

CORD stands for receipt understanding. The CORD repository describes Indonesian receipts collected from shops and restaurants, more than 11,000 in all, of which 1,000 have been released, split 800, 100, and 100, with OCR boxes and text per line and multi-level semantic labels such as menu, subtotal, and total. The dataset is under CC BY 4.0 and the paper appeared at the NeurIPS 2019 Document Intelligence Workshop. SROIE and Kleister-NDA are two further extraction sets in the v2 paper, for scanned receipts and legal agreements.

RVL-CDIP stands for document image classification. The dataset page describes 400,000 grayscale images in 16 classes, 25,000 per class, split 320,000, 40,000, and 40,000, each sized so the largest dimension does not exceed 1,000 pixels. It is a subset of IIT-CDIP, which is itself a subset of the Legacy Tobacco Document Library, and the citation is Harley, Ufkes, and Derpanis, ICDAR 2015. The score is accuracy.

DocVQA stands for document visual question answering. The DocVQA paper (Mathew, Karatzas, and Jawahar, WACV 2021) describes 50,000 questions on 12,767 images from 6,071 documents in the UCSF Industry Documents Library. Because no OCR is perfect, the authors made Average Normalized Levenshtein Similarity (ANLS) the primary metric, which gives partial credit for an answer that differs from the reference by a few characters. Their best BERT-large baseline, fine-tuned on SQuAD 1.1 before DocVQA, scored 0.665 ANLS, and human performance was 0.981 ANLS, or 94.36 percent accuracy.

The reported figures, side by side

The table below collects the figures from the three papers. The v1 and v2 papers write F1 and ANLS as decimals; the v3 paper writes them as percentages, and the table keeps each paper's own form.

Three readings follow. First, layout alone did most of the work on FUNSD: adding 2-D positions to BERT-base lifted F1 from 0.6026 to 0.7866, while adding the image at fine-tuning added less than a point more. Second, the image mattered most for classification: LayoutLM-base went from 91.78 to 94.42 percent on RVL-CDIP once image features were included, and the later versions added about 1 to 1.5 points more. Third, the v2 paper's SROIE figure of 0.9781 is marked as excluding OCR mismatches; the unadjusted figure is 0.9661, a reminder that part of every score on these benchmarks is the OCR's, not the model's.

LayoutLMv3 also reports document layout analysis. On PubLayNet, with a Cascade R-CNN detector on top of the base model, the paper gives 95.1 mean average precision over the five classes, and the unilm layoutlmv3 README breaks that down as text 94.5, title 90.6, list 95.5, table 97.9, and figure 97.0. The document layout analysis page covers that task and its datasets.

LayoutLM vs OCR: the model needs OCR first

A searcher comparing LayoutLM with OCR is comparing two stages of one pipeline. OCR turns pixels into words with positions. LayoutLM takes those words and positions and decides what they mean. Nothing in any of the three models recognizes characters.

The Hugging Face LayoutLMv2 documentation makes the dependency concrete. The model depends on detectron2, torchvision, and tesseract. The processor that prepares inputs has an apply_ocr setting, true by default, documented as whether to apply the Tesseract OCR engine to get words plus normalized bounding boxes, and it uses PyTesseract under the hood; a user who prefers another OCR engine sets apply_ocr to false and supplies the words and boxes. Each box must be normalized to the 0 to 1,000 scale, and the image must be 224 by 224 with BGR channel order, because the Detectron2 backbone was trained that way. The LayoutLMv3 documentation keeps the same apply_ocr default, adds ocr_lang to set the Tesseract language and tesseract_config to pass flags such as --psm 6, and switches to RGB channels and byte-pair encoding, since there is no CNN. The Tesseract page explains what those settings control.

Two consequences follow for anyone evaluating a system built on these models. The first is that OCR errors pass straight through. A misread digit arrives as a token the model has no reason to doubt, so the quality floor is set by the engine and the scan, as the page on OCR on scanned manuals describes. The second is that the choice of OCR engine changes the score. The LayoutLM paper used Tesseract for pre-training and ground-truth OCR for SROIE; LayoutLMv2 and v3 used the Microsoft Read API for the DocVQA fine-tuning data, and v3 also for RVL-CDIP. A deployment that runs Tesseract on 200 dpi scans is not operating under the conditions behind the published figures, and the FUNSD authors recorded Tesseract doing poorly on exactly these forms.

Licenses: MIT for LayoutLM, CC BY-NC-SA 4.0 for v2 and v3

The three versions do not share a license, and the difference decides which one a program can field. The original LayoutLM code lives in the unilm repository, whose root LICENSE file is the MIT License, copyright Microsoft Corporation; the unilm/layoutlm README points to that file, and the layoutlm-base-uncased model card on Hugging Face lists the license as MIT with 113 million parameters.

LayoutLMv2 and LayoutLMv3 are different. The unilm/layoutlmv2 README and the unilm/layoutlmv3 README each state that the content of the project is licensed under the Attribution-NonCommercial-ShareAlike 4.0 International license, CC BY-NC-SA 4.0, and the layoutlmv2-base-uncased and layoutlmv3-base model cards carry the same license field. The license deed says the material may not be used for commercial purposes, and that anyone who remixes, transforms, or builds upon it must distribute those contributions under the same license as the original. A contractor delivering a fielded system cannot rely on v2 or v3 weights under those terms, and a checkpoint fine-tuned from them builds upon the original. The datasets have terms of their own: CORD is CC BY 4.0, and RVL-CDIP and FUNSD derive from tobacco industry document collections, whose terms the RVL-CDIP page refers readers to.

What they are good for on technical manuals

The benchmarks are forms, receipts, letters, and questions about business documents. A technical manual shares some of their structure and not all of it.

Where the models fit: fine-tuned for token classification, a LayoutLM model labels the fields on a scanned maintenance form, a parts requisition, or an inspection record, the same task as FUNSD, with the layout embeddings carrying the label-to-value relationship that a text-only model misses. Fine-tuned for classification, the same model sorts a mixed scan batch into page types, the RVL-CDIP task, which is useful when a box of scanned pages arrives without a table of contents. Fine-tuned for question answering, it finds a value on a page in response to a question, the DocVQA task, within the limits of OCR quality and of a model that reads one page at a time.

Where they do not fit: a manual is hundreds of pages with cross-references, warnings tied to steps, and tables whose meaning depends on headings several lines up, and none of the benchmarks tests that. The models see one page, resized to 224 by 224 pixels for the image stream, with text truncated to the Transformer's sequence limit. Procedures, numbered steps, and the reading order of a two-column page are the job of layout analysis and chunking, not of a page-level field labeler. On a born-digital PDF, where the words and positions come from the file rather than from OCR, the LayoutLM paper's own scope statement applies: the models were built for scanned documents, and the page on why a PDF manual is hard to read by machine explains what a PDF does and does not supply.

The honest summary is that LayoutLM-family models are strong field extractors for scanned, form-like pages, that their published figures were measured with specific OCR engines on specific datasets, and that only the first version can be deployed commercially without a separate license.

What to check

• Which version is in use, and if it is LayoutLMv2 or LayoutLMv3, does the program have a license for commercial use beyond CC BY-NC-SA 4.0?
• Which OCR engine supplies the words and boxes, at what scan resolution, and is it the same engine the reported figures were measured with?
• Was the model fine-tuned on pages from these manuals and forms, or are the quoted numbers the paper's FUNSD, CORD, or DocVQA scores?
• Does the system keep each extracted value tied to its word boxes and page, so a reader can check the field against the scan?
• How are multi-page procedures, tables, and two-column pages handled, given that the model reads one page at a time?
• If the manuals are born-digital PDFs, are words and positions taken from the file, and was the model tested on that input?

Azcuna's TRACE is one offline viewer for S1000D, MIL-STD-3001 XML, and PDF technical manuals. It keeps the source location and the markings of everything it shows and never modifies the source. It was proposed to NAVAIR under Navy SBIR Phase I topic DON26BZ05-NV078, and the proposal was submitted on September 23, 2026.

Questions people ask

What is LayoutLM?

LayoutLM is a document-understanding model from Microsoft, described in the paper LayoutLM: Pre-training of Text and Layout for Document Image Understanding (Xu et al., first posted December 31, 2019, KDD 2020). It takes a BERT model and adds 2-D position embeddings, the coordinates of each word's bounding box scaled to a 0 to 1,000 range, so the model learns where words sit on the page as well as what they say. It was pre-trained on more than 11 million scanned pages from the IIT-CDIP collection.

What is the difference between LayoutLM, LayoutLMv2, and LayoutLMv3?

Each version adds a modality or simplifies one. LayoutLM models text plus layout, with image features added only at fine-tuning. LayoutLMv2 (Xu et al., ACL 2021) puts the page image into pre-training through a ResNeXt-FPN visual encoder and adds spatial-aware self-attention and two text-image alignment tasks. LayoutLMv3 (Huang et al., ACM Multimedia 2022) drops the CNN, feeds 16 by 16 image patches directly, and pre-trains with unified masking of text and image plus a word-patch alignment objective.

Does LayoutLM do OCR?

No. All three models take words and bounding boxes as input and need an OCR engine to produce them. The LayoutLM paper re-ran Tesseract on the IIT-CDIP pages to get words and positions in hOCR format. The LayoutLMv2 paper used the Microsoft Read API. The Hugging Face LayoutLMv2 and LayoutLMv3 processors have an apply_ocr setting that defaults to True and calls Tesseract through PyTesseract; set it to False to supply words and boxes from another engine.

What is the FUNSD benchmark?

FUNSD, from the paper FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents (Jaume, Ekenel, and Thiran, ICDAR 2019 workshop), is 199 real scanned forms sampled from the RVL-CDIP collection, split into 149 training and 50 test forms. The forms carry 31,485 words, 9,743 semantic entities labeled header, question, answer, or other, and 5,312 links between entities. The common score is entity-level F1, which the LayoutLM papers report as 0.6026 for BERT-base and 92.08 for LayoutLMv3-large.

What is LayoutLMv3 used for?

The LayoutLMv3 paper reports results on form understanding (FUNSD), receipt understanding (CORD), document image classification (RVL-CDIP), document visual question answering (DocVQA), and document layout analysis (PubLayNet). The base model reached 90.29 F1 on FUNSD, 96.56 on CORD, 95.44 percent accuracy on RVL-CDIP, 78.76 ANLS on DocVQA, and 95.1 mAP on PubLayNet, where the table class scored 97.9 average precision. Fine-tuned, it labels fields, classifies pages, and answers questions about a page.

Is LayoutLM free for commercial use?

The versions differ. The original LayoutLM model card on Hugging Face lists the MIT license, and the unilm repository that hosts the code carries an MIT LICENSE file. The LayoutLMv2 and LayoutLMv3 READMEs in that repository and their Hugging Face model cards state that the content is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0), which excludes commercial use. A program that needs a commercial license has only the first version.

How does LayoutLMv2 score on DocVQA?

The LayoutLMv2 paper reports Average Normalized Levenshtein Similarity (ANLS) on the DocVQA test set. Trained on the training set alone, LayoutLM-base scored 0.6979 and LayoutLM-large 0.7259, while LayoutLMv2-base scored 0.7808 and LayoutLMv2-large 0.8348. Adding the development set and a question-generation pre-fine-tuning step raised the large model to 0.8672, the figure in the abstract. The DocVQA paper's best baseline, a BERT-large model fine-tuned on SQuAD 1.1 and then DocVQA, scored 0.665, and human performance was 0.981.

What is RVL-CDIP?

RVL-CDIP is a document image classification dataset of 400,000 grayscale images in 16 classes, 25,000 per class, split into 320,000 training, 40,000 validation, and 40,000 test images, sized so the largest dimension is at most 1,000 pixels. The dataset page describes it as a subset of the IIT-CDIP Test Collection, itself drawn from the Legacy Tobacco Document Library, and cites Harley, Ufkes, and Derpanis, ICDAR 2015. LayoutLM reached 94.42 percent accuracy, LayoutLMv2-large 95.64, and LayoutLMv3-large 95.93.

How many parameters do the LayoutLM models have?

The LayoutLM paper gives about 113 million parameters for the base model and 343 million for large. The LayoutLMv2 paper gives about 200 million for base and 426 million for large, the increase coming from the visual encoder. The LayoutLMv3 paper lists 133 million for base and 368 million for large, smaller than v2 because the CNN backbone was replaced with a linear patch embedding. All three families publish base and large checkpoints.

Sources

1. 1LayoutLM: Pre-training of Text and Layout for Document Image Understanding · arXiv (Xu, Li, Cui, Huang, Wei, and Zhou; KDD 2020)
2. 2LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding · arXiv (Xu et al.; ACL 2021)
3. 3LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking · arXiv (Huang, Lv, Cui, Lu, and Wei; ACM Multimedia 2022)
4. 4FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents · arXiv (Jaume, Ekenel, and Thiran; ICDAR 2019 OST workshop)
5. 5CORD: A Consolidated Receipt Dataset for Post-OCR Parsing · Clova AI (GitHub)
6. 6RVL-CDIP Dataset · Adam W. Harley (Ryerson Vision Lab)
7. 7DocVQA: A Dataset for VQA on Document Images · arXiv (Mathew, Karatzas, and Jawahar; WACV 2021)
8. 8microsoft/layoutlm-base-uncased model card · Hugging Face
9. 9microsoft/layoutlmv2-base-uncased model card · Hugging Face
10. 10microsoft/layoutlmv3-base model card · Hugging Face
11. 11unilm/layoutlm README · Microsoft (GitHub)
12. 12unilm/layoutlmv2 README · Microsoft (GitHub)
13. 13unilm/layoutlmv3 README · Microsoft (GitHub)
14. 14unilm LICENSE · Microsoft (GitHub)
15. 15Attribution-NonCommercial-ShareAlike 4.0 International deed · Creative Commons
16. 16LayoutLMv2 (Transformers documentation) · Hugging Face
17. 17LayoutLMv3 (Transformers documentation) · Hugging Face

Where this shows up in our work

• TRACE, the offline S1000D and IETM viewer opens S1000D, MIL-STD-3001 and PDF technical manuals on a tablet with every marking intact.
• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.