ColBERT and late interaction, explained

ColBERT is a neural retrieval model that keeps one embedding per token instead of one per passage, and scores a query against a passage by matching each query token to its best-matching passage token. Omar Khattab and Matei Zaharia introduced it in ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, posted to arXiv on April 27, 2020 and accepted at SIGIR 2020. The paper's name for the design is late interaction: the query and the document are encoded independently, as a bi-encoder would, and the comparison between them is delayed to a cheap final step that still works at the level of individual tokens. The result sits between the two standard architectures of neural search, and the two follow-on papers, ColBERTv2 in 2021 and PLAID in 2022, are about making its index small enough and its search fast enough to use at scale.

Where it sits between bi-encoders and cross-encoders

Figure 2 of the ColBERT paper lays out four ways a neural model can match a query q to a document d. Representation-based similarity computes one embedding for q and one for d and takes their dot product. Query-document interaction models match word- and phrase-level relationships across q and d with a neural network. All-to-all interaction feeds q and d through a deep language model together, so that every term attends to every other. Late interaction, the paper's proposal, encodes q and d separately into bags of token embeddings and compares the bags with a MaxSim operation.

The first and third are the bi-encoder and the cross-encoder of everyday usage. The bi-encoder is the design of Sentence-BERT, whose 2019 paper by Reimers and Gurevych motivates it with a cost figure: finding the most similar pair among 10,000 sentences takes about 65 hours when both sentences go through BERT together and about 5 seconds with precomputed sentence embeddings. The cross-encoder is the design of Nogueira and Cho's Passage Re-ranking with BERT, which fed each query-passage pair through BERT and improved the MS MARCO leaderboard's MRR@10 by 27 percent relative to the previous best. The ColBERT paper's framing is that interaction-based models are more effective and representation-based models are cheaper, because only the latter can precompute document representations offline, and that late interaction keeps the fine-grained matching of the former with the precomputation of the latter. The post on what a reranker is covers the cross-encoder side of that trade in more depth.

The MaxSim operator

ColBERT uses one BERT model for both queries and documents, distinguished by a special token prepended to the input: [Q] for queries and [D] for documents, placed right after BERT's [CLS]. The query encoder tokenizes the query with WordPiece, pads it to a fixed Nq of 32 tokens with BERT's [mask] token, or truncates it to the first 32, and runs it through BERT. The paper calls the padding query augmentation, intends it as a soft mechanism for learning to expand the query or re-weigh its terms, and its ablation finds it essential to ColBERT's effectiveness. The document encoder does not pad with masks and filters out the embeddings of punctuation symbols after encoding. In both cases the BERT outputs pass through a linear layer with no activation that reduces each token's vector to m dimensions, 128 in the paper's main experiments, and the vectors are L2-normalized so that a dot product is a cosine similarity.

Scoring is Equation 3 of the paper. For a query with embeddings Eq and a document with embeddings Ed, the relevance score is the sum over every query embedding of the maximum dot product between that embedding and any document embedding. Each query token softly searches the document for its best match, and the matches are summed. This is the MaxSim operator: a maximum over document tokens inside a sum over query tokens. It has no trainable parameters, so training only tunes the BERT encoders, the linear layer and the [Q] and [D] marker embeddings, using a pairwise softmax cross-entropy loss over triples of a query, a positive passage and a negative one.

The paper gives two reasons for choosing MaxSim over richer interactions. The first is cost: the operation is a batch of matrix products followed by a max and a sum, and the paper puts ColBERT's re-ranking at 7 billion FLOPs per query. The second is that it is amenable to pruning. Because each query token's contribution is a maximum over document tokens, a vector-similarity index over the document tokens can find, for each query token, the documents containing its nearest tokens, and those documents are the only candidates worth scoring in full. Section 4.4 of the paper ablates the alternatives and finds MaxSim-based late interaction ahead of them.

Per-token embeddings and storage cost

Late interaction's cost is in the index. A single-vector model stores one embedding per passage; ColBERT stores one per token, and the paper's Section 3.4 describes writing them to disk as 32-bit or 16-bit values per dimension. Table 4 of the paper gives the space for the MS MARCO passage collection of 8.8 million passages under several settings.

The paper's comment is that the most space-efficient setting, 24 dimensions at 2 bytes, is only 1 percent worse in MRR@10 than the most space-consuming one while needing 27 GiB. The arithmetic behind the table comes from the token count: the PLAID paper's Table 1 gives MS MARCO v1 as 597.9 million tokens across 8.8 million passages, about 68 tokens per passage, and 597.9 million vectors of 128 dimensions at 2 bytes is 153 GB. For comparison, the ColBERTv2 paper notes that storing one 768-dimensional float vector for each of the same 9 million passages comes to a little over 25 GiB, before any nearest-neighbor index. Uncompressed late interaction is therefore about 6 times the storage of a single-vector index for the same collection, and the ratio grows with passage length.

Indexing time follows the same logic, since every token must pass through BERT. The paper reports indexing MS MARCO in about three hours with multiple GPUs, and notes that any BERT-based model that precomputes document representations pays the same encoding cost.

Re-ranking and end-to-end retrieval

The 2020 paper evaluates ColBERT two ways. In the re-ranking setting it rescores the official BM25 top-1000 passages per query, loading each candidate's embedding matrix and applying MaxSim on a GPU. Table 1 gives the MS MARCO development set results.

The paper summarizes the comparison as over 170 times lower latency and 13,900 times fewer FLOPs than BERT-base at the same model size, measured on a single Tesla V100 GPU. On TREC CAR, BM25 plus ColBERT reached a MAP of 31.3 against 31.0 for BM25 plus BERT-base and 33.5 for BERT-large.

In the end-to-end setting ColBERT retrieves from the whole collection. Section 3.6 describes a two-stage procedure. First, every query token embedding is sent to a Faiss vector index over all document token embeddings; the paper used an IVFPQ index with 2,000 partitions, probing the 10 nearest to each query embedding, under squared L2 distance because the Faiss index was faster that way. The documents containing the returned tokens form the candidate set. Second, those candidates are re-ranked exhaustively with MaxSim. Table 2 reports MRR@10 of 36.0 for end-to-end ColBERT, above its own re-ranking result because recall is no longer capped by BM25, and the paper observes that ColBERT's Recall@50 exceeded the official BM25's Recall@1000. The companion post on vector search and Faiss covers the IVFPQ index type the paper used.

ColBERTv2: residual compression and denoised training

The storage figures above are the problem ColBERTv2 solves. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts and Matei Zaharia posted it on December 2, 2021, and it appeared at NAACL 2022. The abstract states that existing late interaction systems "inflates the space footprint" by an order of magnitude over single-vector models, and that ColBERTv2 reduces that footprint by 6 to 10 times while improving quality.

The compression rests on an observation the paper tests in its Appendix A: ColBERT's token vectors cluster into regions that capture context-aware word sense, so a vector can be described well by the cluster it belongs to. ColBERTv2 runs k-means over a sample of token embeddings to get a set of centroids C. Each token vector v is then stored as the index t of its nearest centroid plus a quantized residual r, which approximates v minus Ct, with every dimension of the residual encoded in 1 or 2 bits. At search time the vector is reconstructed as Ct plus r. With 128-dimensional vectors the paper uses 4 bytes for the centroid index, enough for up to 2 to the 32 centroids, and 16 or 32 bytes for the residual, so each token costs 20 or 36 bytes against the 256 bytes of a 16-bit ColBERT vector. The paper describes the scheme as a natural extension of product quantization to multi-vector representations. Section 5.3 reports the result: where ColBERT needs 154 GiB for the MS MARCO index, ColBERTv2 needs 16 GiB with 1-bit residuals or 25 GiB with 2-bit, including 4.5 GiB for the inverted list, which the paper says matches the storage of a typical single-vector index.

The centroids also drive retrieval. Each passage's tokens are grouped by centroid into an inverted list, and a query finds the nprobe nearest centroids to each of its token embeddings, gathers the passages whose tokens fall under them, scores them approximately from the compressed embeddings, and re-ranks the top candidates with full MaxSim.

The second change is supervision. ColBERTv2 is trained by distillation from a 22 million parameter MiniLM cross-encoder, the public cross-encoder/ms-marco-MiniLM-L-6-v2 checkpoint, using a KL-divergence loss over a positive and several hard-negative passages per query, plus in-batch negatives. Table 4 of the paper gives MS MARCO development set MRR@10 of 39.7 for ColBERTv2 against 36.0 for the original ColBERT, 36.8 for SPLADEv2 and 38.8 for RocketQAv2, with Recall@50 of 86.8 and Recall@1000 of 98.4.

BEIR figures

BEIR, from Thakur, Reimers, Ruckle, Srivastava and Gurevych and accepted at the NeurIPS 2021 Datasets and Benchmarks track, collects 18 retrieval datasets across domains and evaluates models zero-shot, without training on the target data. The BEIR paper compared 10 systems spanning lexical, sparse, dense, late-interaction and re-ranking architectures, found BM25 to be a robust baseline, and found that re-ranking and late-interaction models achieved the best average zero-shot performance, at high computational cost, while dense and sparse models were cheaper but often behind. ColBERT was the late-interaction entry.

The ColBERTv2 paper's Table 5 reports nDCG@10 on the BEIR tasks it evaluated, copying the original ColBERT, ANCE and TAS-B results from the BEIR paper and the SPLADEv2 results from its own paper.

The paper's reading of its own table is that ColBERTv2 led SPLADEv2 on six BEIR tasks and tied on two, with the largest gains on NQ, TREC-COVID and FiQA-2018, all of which have natural search queries, while SPLADEv2 led on five, with its largest gains on Climate-FEVER, whose queries are sentences making climate claims, and HotpotQA, whose queries were written by crowdworkers who could see the target passages. Across its full out-of-domain evaluation, which adds Wikipedia open-domain question answering and the paper's own LoTTE benchmark of 12 StackExchange test sets, the paper reports ColBERTv2 reaching the highest quality on 22 of 28 tests.

PLAID: making late interaction fast

Compression made the ColBERTv2 index small but not quick to search, because every candidate's residuals had to be looked up and decompressed. PLAID, the Performance-optimized Late Interaction Driver, posted May 19, 2022 by Santhanam, Khattab, Potts and Zaharia and published at CIKM 2022 per the ColBERT repository, attacks that cost. The paper's Figure 2 shows that unoptimized ColBERTv2 spends most of its query time on index lookup and residual decompression.

PLAID's observation is that the centroid ids alone are a good enough description of a passage to rule most passages out. Centroid interaction treats each passage as a bag of centroids, scores it with MaxSim against the query using the centroid vectors instead of the reconstructed token vectors, and drops the low scorers before any residual is touched. Centroid pruning goes further by discarding centroids whose similarity to every query token is below a threshold, which sparsifies the bag. Figure 5 of the paper lays out the four-stage pipeline: candidate generation from the nprobe nearest centroids per query token, a centroid pruning stage that returns ndocs candidates, a centroid interaction stage that keeps a quarter of them, and finally residual decompression and exact MaxSim scoring of the survivors. Optimized C++ kernels on the CPU and a CUDA kernel for decompression on the GPU handle the data movement.

Table 3 of the paper gives MS MARCO v1 results. Unoptimized ColBERTv2 reached MRR@10 of 39.7 and Recall@100 of 90.4 at 3,485 milliseconds per query on a single CPU thread, 922 milliseconds on 8 threads and 53 milliseconds on a GPU. PLAID ColBERTv2 with k of 1,000 reached 39.8 and 91.3 at 352, 101 and 38 milliseconds, and with k of 100 the same 39.8 at 222, 53 and 20 milliseconds. The paper summarizes this as speedups of 6.8 times on a GPU and 45 times on a CPU with no loss in MRR@10 or Recall@100, and reports that the largest collection it tested, MS MARCO v2 with 138 million passages and 9.4 billion tokens, searched at 136 to 251 milliseconds on 8 CPU threads with an index of 202 GiB. For reference, the same table lists BM25 under the PISA engine at 8.3 milliseconds and SPLADEv2 under PISA at 220 milliseconds single-threaded, both copied from prior work on different hardware.

The ColBERT repository ships ColBERTv2 and PLAID together on its stable branch, with a colbertv2.0 checkpoint trained on MS MARCO passage ranking, an indexing example at nbits of 2, and a note that a binarization-based compression branch was found less robust than residual compression.

What to check

• How many tokens does the collection hold, and at the configured bytes per token embedding, 256 uncompressed or 20 to 36 under ColBERTv2 compression, does the index fit where it has to run?
• Is the system re-ranking a lexical shortlist or retrieving end to end, and if the latter, what are nprobe, the candidate counts and the final MaxSim depth?
• Which checkpoint is in use, and does its training domain match the documents? The published checkpoint was trained on MS MARCO.
• Is the query length cap of 32 tokens appropriate for the queries the system will see, since longer queries are truncated?
• Were the retrieval figures measured with the same engine, hardware and thread count as the deployment, given that vanilla ColBERTv2 ran about 65 times slower on one CPU thread than on a GPU in the PLAID paper's Table 3?
• How is quality being measured on this collection, as distinct from MS MARCO or BEIR scores reported in the papers?

Azcuna's TRACE is one offline viewer for S1000D, MIL-STD-3001 XML, and PDF technical manuals. It keeps the source location and the markings of everything it shows and never modifies the source. It was proposed to NAVAIR under Navy SBIR Phase I topic DON26BZ05-NV078, and the proposal was submitted on September 23, 2026.

Questions people ask

What is ColBERT?

ColBERT is a neural retrieval model introduced by Omar Khattab and Matei Zaharia in a 2020 paper accepted at SIGIR. The name stands for Contextualized Late Interaction over BERT. Instead of compressing a query or passage into one vector, ColBERT encodes each into a bag of token-level embeddings using BERT, stores the passage embeddings offline, and scores a query against a passage with a cheap late interaction step. The paper reports quality competitive with BERT re-rankers at two orders of magnitude lower latency.

What is late interaction in retrieval?

Late interaction is the design in which the query and the document are encoded separately into sets of token embeddings and only then compared. The ColBERT paper contrasts it with representation-based similarity, which compares one vector per text, and with all-to-all interaction, which feeds the query and document through BERT together. Delaying the interaction lets document representations be precomputed and indexed, while keeping it token-level preserves fine-grained matching that a single vector loses.

What is the MaxSim operator?

MaxSim is the scoring step of ColBERT. Equation 3 of the 2020 paper defines the relevance of a document to a query as the sum, over every query token embedding, of the maximum dot product between that embedding and any of the document's token embeddings. Because embeddings are normalized, the dot product is a cosine similarity. The operator has no trainable parameters, and the paper notes it is friendly to pruning, which is what makes end-to-end retrieval with a vector index possible.

How much storage does ColBERT need?

One vector per token. The ColBERT paper's Table 4 gives 154 GiB for the MS MARCO passage collection with 128-dimensional embeddings at 2 bytes per dimension and 286 GiB at 4 bytes, against 27 GiB for 24-dimensional vectors, which cost 1 point of MRR@10. The ColBERTv2 paper reports that a single 768-dimensional float vector per passage for the same 9 million passages comes to a little over 25 GiB, so uncompressed late interaction is roughly 6 times larger.

What does ColBERTv2 change?

The ColBERTv2 paper, from Santhanam, Khattab, Saad-Falcon, Potts and Zaharia at NAACL 2022, adds residual compression and denoised supervision. Each token embedding is stored as the id of its nearest k-means centroid plus a residual quantized to 1 or 2 bits per dimension, 20 or 36 bytes per vector instead of 256. Training distills scores from a MiniLM cross-encoder with hard negatives. The paper reports MRR@10 of 39.7 on MS MARCO and a 6 to 10 times smaller footprint.

What is PLAID?

PLAID, the Performance-optimized Late Interaction Driver, is the retrieval engine described in a 2022 paper by Santhanam, Khattab, Potts and Zaharia. It discards low-scoring passages using centroid interaction, which scores a passage from the centroid ids of its tokens without decompressing residuals, and centroid pruning, which drops unimportant centroids. The paper reports latency cut by up to 7 times on a GPU and 45 times on a CPU against plain ColBERTv2, with no loss in quality, on collections up to 140 million passages.

Is ColBERT a bi-encoder or a cross-encoder?

Neither, by the ColBERT paper's own taxonomy. A bi-encoder, such as Sentence-BERT, encodes each text to one vector and compares them, which is fast but coarse. A cross-encoder, such as the BERT re-ranker of Nogueira and Cho, processes the query and passage together and cannot precompute anything. ColBERT encodes the two separately like a bi-encoder but compares them token by token like an interaction model. The paper presents late interaction as a new paradigm alongside three existing ones.

How does ColBERT do on BEIR?

The BEIR paper by Thakur and colleagues, covering 18 datasets, found that re-ranking and late-interaction models achieved the best average zero-shot performance among 10 systems, at high computational cost. The ColBERTv2 paper's Table 5 reports nDCG@10 of 56.2 on NQ, 73.8 on TREC-COVID and 35.6 on FiQA for ColBERTv2, against 52.1, 71.0 and 33.6 for SPLADEv2, and says ColBERTv2 led SPLADEv2 on six BEIR tasks and tied on two, while SPLADEv2 led on five, with its largest gains on Climate-FEVER and HotpotQA.

Can ColBERT be used as a reranker?

Yes, and that was its first evaluation. In the 2020 paper's Table 1, ColBERT re-ranked the official BM25 top-1000 on MS MARCO to MRR@10 of 34.9 in 61 milliseconds per query on a Tesla V100, where BERT-base reached 34.7 in 10,700 milliseconds and BERT-large 36.5 in 32,900. The paper also shows end-to-end retrieval, using a Faiss IVFPQ index over the token embeddings to shortlist passages before exhaustive MaxSim scoring, reaching MRR@10 of 36.0.

Sources

1. 1ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT · arXiv (Khattab and Zaharia)
2. 2ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction · arXiv (Santhanam, Khattab, Saad-Falcon, Potts and Zaharia)
3. 3PLAID: An Efficient Engine for Late Interaction Retrieval · arXiv (Santhanam, Khattab, Potts and Zaharia)
4. 4BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models · arXiv (Thakur, Reimers, Ruckle, Srivastava and Gurevych)
5. 5Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks · arXiv (Reimers and Gurevych)
6. 6Passage Re-ranking with BERT · arXiv (Nogueira and Cho)
7. 7ColBERT repository README · Stanford Future Data Systems on GitHub

Where this shows up in our work

• TRACE, the offline S1000D and IETM viewer opens S1000D, MIL-STD-3001 and PDF technical manuals on a tablet with every marking intact.
• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.