What is an embedding model?

An embedding model is a neural network that turns a piece of text into a fixed-length list of numbers, a vector, built so that texts with similar meaning produce vectors that lie close together. The idea in its modern form comes from Sentence-BERT (Reimers and Gurevych, submitted August 27, 2019), which modified BERT so that each sentence maps to one vector that can be compared with cosine similarity. The E5 paper defines text embeddings as low-dimensional vector representations for texts of arbitrary length. In a document system the embedding model decides which passages are near a question; the model that writes the answer is a different model, chosen separately.

What a text embedding is

A text embedding is a dense vector, typically a few hundred to a few thousand floating-point numbers, that stands in for a sentence, a paragraph, or a chunk of a document. Dense means every position holds a value, unlike a sparse keyword representation such as TF-IDF, where most positions are zero. The E5 paper notes that, compared with sparse representations such as TF-IDF, text embeddings have the potential to overcome the lexical mismatch issue, where a question and a relevant passage use different words.

The Sentence-BERT paper explains why a separate embedding model was needed. BERT scored sentence pairs by reading both sentences together, which the paper says made finding the most similar pair among 10,000 sentences cost about 50 million inference computations, or roughly 65 hours. Computing 10,000 sentence embeddings once and comparing them with cosine similarity took about 5 seconds for the embeddings and about 0.01 seconds for the comparisons. The paper also showed that BERT's raw outputs were not good embeddings: on the semantic textual similarity (STS) tasks in its Table 1, the BERT CLS vector reached an average Spearman correlation of 29.19 and averaged BERT vectors reached 54.81, both below averaged GloVe word vectors at 61.32. After Sentence-BERT fine-tuning, the base model reached 74.89.

Sentence-BERT adds a pooling step to BERT's output to get one fixed-size vector per sentence. The paper tested three strategies, taking the CLS token, taking the mean of all output vectors, and taking the element-wise maximum, and made mean pooling the default. The E5 paper likewise uses average pooling over the output layer.

Cosine similarity, dot product, and normalization

Once two texts are vectors, similarity is arithmetic. Cosine similarity is the inner product of the two vectors divided by the product of their lengths. The result is the cosine of the angle between them: 1 when they point the same way, 0 when they are at right angles. Because the lengths are divided out, cosine ignores how long a vector is and keeps only its direction. The Sentence-BERT paper reports all of its STS results as the Spearman correlation between cosine similarity and the human labels, and notes that negative Manhattan and Euclidean distances gave roughly the same results.

The Sentence Transformers documentation lists cosine similarity as the default similarity function for a loaded model, with dot product, negative Euclidean distance, and negative Manhattan distance as the alternatives. The function can be set when the model is created or changed after loading, and model.similarity returns a matrix of scores for every pair between two sets of embeddings.

Dot product and cosine are the same ranking when every vector has length 1. The Faiss wiki's page on metric types states that normalizing the query and database vectors beforehand turns maximum inner product search into a cosine ranking. OpenAI's embeddings guide says its embeddings are normalized to length 1, so cosine similarity can be computed slightly faster with just a dot product, and cosine similarity and Euclidean distance give identical rankings. The bge-large-en-v1.5 model card sets normalize_embeddings=True in its examples so that dot products give cosine similarity. The same card warns that a score above 0.5 does not mean two sentences are similar, and that a threshold has to be chosen from the user's own data. The absolute value of a similarity score belongs to one model; only the ranking carries over.

How embedding models are trained

Most embedding models are trained with a contrastive objective: paired texts are pulled closer, unpaired texts are pushed apart.

Sentence-BERT trained on the Stanford Natural Language Inference (SNLI) corpus, 570,000 sentence pairs labeled contradiction, entailment, or neutral, plus 430,000 MultiNLI pairs. The paper defined three objectives and fine-tuned on the NLI data with the first: a classification objective that feeds the two sentence vectors u and v and their element-wise difference |u - v| to a softmax classifier; a regression objective that trains the cosine similarity of u and v toward the gold score with mean squared error; and a triplet objective that pushes an anchor's distance to a positive sentence below its distance to a negative one by a margin.

E5 scaled the data up and dropped the labels. The paper's name expands to EmbEddings from bidirEctional Encoder rEpresentations. The training set, CCPairs, came from sources such as community question answering, Common Crawl, and scientific papers, and a model trained on the 1.3 billion noisy pairs filtered them down to about 270 million for contrastive pre-training. The loss is InfoNCE with in-batch negatives, in which passages paired with other queries in the same batch serve as negatives, so the batch size was set to 32,768. The score is cosine similarity divided by a temperature, and the prefixes query: and passage: mark which side a text is on. The result was the first unsupervised model to beat BM25 on the BEIR retrieval benchmark: averaged over the 15 BEIR datasets with public downloads, BM25 scored an nDCG@10 of 41.7, the pre-trained E5 base model 42.9, and the large model 44.2. Fine-tuning on labeled data such as MS MARCO and Natural Questions raised the base model to 48.7, with the paper noting that those two datasets are then in-domain rather than zero-shot.

BGE, from the C-Pack paper (Xiao et al., first submitted September 14, 2023), stands for BAAI General Embeddings. The recipe has three steps: pre-training the encoder with RetroMAE, in which a lightweight decoder reconstructs clean text from the embedding of a corrupted copy; contrastive learning on unlabeled pairs with in-batch negatives and a batch size as large as 19,200; and task-specific fine-tuning with a verbal instruction on the query side and one mined hard negative per pair. The Chinese models come in small (24 million parameters), base (102 million), and large (326 million) sizes, and the large English BGE model reached an average of 64.23 across the 56 English MTEB datasets, which the paper describes as 1.1 points above the prior state of the art.

Smaller models follow the same pattern. The all-MiniLM-L6-v2 model card says the model was fine-tuned on 1,170,060,424 sentence pairs with a contrastive objective, picking the true pair out of the other sentences in a batch of 1,024, and produces 384-dimensional vectors.

The MTEB leaderboard and how to read it

The Massive Text Embedding Benchmark (MTEB, Muennighoff et al., first submitted October 13, 2022) exists because embedding models were being evaluated on a few STS datasets and then used for everything else. MTEB spans 8 task types across 58 datasets and 112 languages: bitext mining, classification, clustering, pair classification, reranking, retrieval, STS, and summarization. Each has a main metric. Bitext mining uses F1; classification uses accuracy; clustering uses v-measure; pair classification uses average precision based on cosine similarity; reranking uses mean average precision; retrieval uses nDCG@10; STS and summarization use Spearman correlation of cosine similarity with human scores. The paper benchmarked 33 models and found that no particular method dominated across all tasks. Models geared toward STS performed badly on retrieval, and retrieval models performed badly on STS. The project README runs any model with mteb.get_model, mteb.get_tasks, and mteb.evaluate, and the results feed the public leaderboard on the Hugging Face Hub.

Three observations from the paper shape how to read a leaderboard row. First, performance strongly correlated with model size, and most tasks were dominated by multi-billion-parameter models. Second, the latency-performance plot sorted models into three clusters: word-embedding models such as GloVe at maximum speed, the large GTR-XXL, ST5-XXL, and SGPT-5.8B models at maximum performance but slow, and fine-tuned MPNet and MiniLM models in between. Third, the overall average hides task differences: the paper notes that the MPNet model, almost 50 times smaller than ST5-XXL, was on par with it on clustering.

The benchmark has since grown. The MMTEB paper (Enevoldsen et al., first submitted February 19, 2025) expands it to more than 500 tasks in over 250 languages, adds instruction following, long-document retrieval, and code retrieval, and reports that while billion-parameter models led some language subsets, the best publicly available model was multilingual-e5-large-instruct with 560 million parameters. When reading the leaderboard, pick the benchmark that matches the languages in use, read the column for the task that matters (retrieval nDCG@10 for a document system), and check the model's dimensions, maximum tokens, and parameter count, which the leaderboard reports alongside the scores.

Dimensions and Matryoshka embeddings

An embedding's dimension is the length of the vector. Published models vary widely: 384 for all-MiniLM-L6-v2, 768 for base-size BERT models, 1024 for bge-large-en-v1.5 and multilingual-e5-large, and 1536 or 3072 for OpenAI's text-embedding-3-small and text-embedding-3-large. Dimension sets storage. The Faiss wiki's index table gives a flat index as 4 times d bytes per vector, so 1,000,000 passages at 1024 dimensions occupy about 4.1 gigabytes uncompressed, and 3072 dimensions triples that. The post on vector search and Faiss covers the index types that trade memory for approximation.

Matryoshka Representation Learning (MRL, Kusupati et al., first submitted May 26, 2022) addresses the choice of dimension by removing it. The method trains one model so that the first m dimensions of its output, for a nested set of values of m, are each usable on their own. For a ResNet50 with a 2048-dimensional output the paper optimizes m in {8, 16, 32, 64, 128, 256, 512, 1024, 2048}; for BERT-Base with 768 dimensions it uses {12, 24, 48, 96, 192, 384, 768}. The nesting is in the training loss, so at inference a vector is simply cut to the length wanted. The paper's adaptive retrieval procedure shortlists 200 candidates with a 16-dimensional prefix and re-ranks them with the full 2048-dimensional vector, and reports that this was as accurate as single-shot search with 2048 dimensions while being about 128 times cheaper in floating-point operations and about 14 times faster in wall-clock time on ImageNet-1K. The appendix describes MRL applied to BERT masked language modeling as within 0.5 percent of the ordinary model's validation accuracy; its Table 7 shows a largest gap of 0.54 points, at 768 dimensions. OpenAI's embeddings guide exposes the same idea as a dimensions parameter that shortens the native 1536 or 3072 output.

Multilingual embedding models

A multilingual embedding model places many languages in one vector space, so that a query in one language is near a passage in another. The LaBSE paper (Language-agnostic BERT Sentence Embedding, Feng et al., first submitted July 3, 2020) is the standard reference. The model is a dual encoder: a BERT Base transformer with 12 layers and 768 hidden units, shared across all languages, whose sentence embedding is the L2-normalized CLS token. The vocabulary is a custom wordpiece set of 501,153 entries built to counter the imbalance of data across languages. Training used 17 billion monolingual sentences for pre-training and a translation corpus of 6 billion sentence pairs for the translation ranking task, with additive margin softmax and in-batch negatives at a global batch size of 4,096. On the Tatoeba bitext retrieval task, LaBSE reached 83.7 percent accuracy over 112 languages against 65.5 percent for the prior state of the art, performed well on more than 30 Tatoeba languages for which it had no training data, and was released for 109 or more languages. The MTEB paper found LaBSE dominated its bitext mining tasks.

The multilingual E5 report (Wang et al., February 8, 2024) describes three sizes initialized from multilingual MiniLM, xlm-roberta-base, and xlm-roberta-large, contrastively pre-trained on about 1 billion multilingual text pairs with a batch size of 32k for 30,000 steps, then fine-tuned on about 1.6 million labeled examples. On the MIRACL multilingual retrieval benchmark, averaged over 16 languages, the report gives nDCG@10 of 39.3 for BM25, 41.5 for mDPR, 60.8 for the small model, 62.3 for base, 66.5 for large, and 65.7 for the instruction-tuned large model. The multilingual-e5-large model card gives 24 layers, 1024-dimensional output, the 100 languages of XLM-RoBERTa with a warning that low-resource languages may degrade, and a requirement that every input start with query: or passage: because the model was trained that way. The bge-m3 model card describes a model covering more than 100 working languages that produces dense, sparse, and multi-vector representations from one encoder. The post on cross-lingual retrieval covers what studies report about script and language bias in these shared spaces.

Context length limits

Every embedding model has a maximum input length, and text past it is dropped. The all-MiniLM-L6-v2 card says input longer than 256 word pieces is truncated by default. The multilingual-e5-large card states that long texts are truncated to at most 512 tokens, and bge-large-en-v1.5 also takes 512. bge-m3 raises the limit to 8192 tokens, and OpenAI's text-embedding-3 models accept 8192 tokens. The limit is counted in the model's own tokens, not words or characters, so the same page costs a different number of tokens in different languages. The E5 paper adds a deeper caution: most research focuses on short texts, and for long documents it remains an open research question whether a fixed-length embedding can encode all the information.

Truncation is silent: a 20-page document fed to a 512-token model yields a vector of roughly its first page, and the rest is never indexed. The MTEB paper states that the benchmark imposes no limit on input length and leaves truncation to the model, so a high leaderboard score does not say how a model handles long passages. This is the main reason documents are cut into pieces before indexing; the post on chunking documents for retrieval covers the strategies and what public tests found about size and overlap.

Embedding model versus the language model that answers

The embedding model and the LLM are not the same thing; in most systems they are two different models with different training. BERT, the starting point for Sentence-BERT, E5, and LaBSE, is described in the Transformers documentation as a bidirectional transformer pretrained to predict masked tokens, reading text to the left and right of each position. A generator reads left to right and predicts the next token. The embedding models above then add a contrastive stage whose only output is a vector.

The split is built into retrieval-augmented generation. The RAG paper (Lewis et al., first submitted May 22, 2020) combined a pre-trained neural retriever over a dense vector index of Wikipedia with a pre-trained seq2seq generator, two models with different jobs. The retriever it built on, Dense Passage Retrieval (Karpukhin et al., first submitted April 10, 2020), is a dual encoder trained on a small number of question-passage pairs that outperformed a Lucene BM25 system by 9 to 19 points absolute in top-20 passage retrieval accuracy. Nothing in that design requires the two to share weights, a vocabulary, or a vendor.

Two consequences follow. First, the embedding model can be far smaller than the generator: the E5 and BGE models above have tens to hundreds of millions of parameters. Second, the embedding model is fixed at indexing time. Every stored vector came from one model, with one tokenizer and one pooling step, and queries have to be encoded by that same model to land in the same space; the E5 recipe even trains the query and passage sides together with their prefixes. Changing the embedding model means re-encoding the whole collection, which is a cost that a change of generator does not carry.

Running embedding models offline

The open models discussed here run without a network. E5, BGE, and all-MiniLM-L6-v2 are published on the Hugging Face Hub, under the MIT license for the E5 and BGE cards and Apache-2.0 for all-MiniLM-L6-v2, and load through the Sentence Transformers library, whose documentation counts more than 25,000 pretrained models on the Hub and installs with pip install -U sentence-transformers. The Transformers installation page describes the offline procedure: download a model repository ahead of time with snapshot_download, then set the environment variable HF_HUB_OFFLINE=1 to prevent HTTP calls to the Hub, or pass local_files_only=True to from_pretrained with a local path. The same page says downloaded models are cached by default under ~/.cache/huggingface/hub, with the location controlled by the HF_HUB_CACHE variable, so the cache directory is what has to be carried to a disconnected machine. The Sentence Transformers page recommends Python 3.10 or later and PyTorch 1.11.0 or later.

Encoding is the expensive step and happens once per passage; the Sentence-BERT paper's figure of about 5 seconds for 10,000 sentences was measured on a V100 GPU, and its Table 7 gives SBERT-base with smart batching at 2,042 sentences per second on that GPU against 83 on a CPU. Storage follows the dimension at 4 bytes per value in a flat index. After that, search is a comparison of the query vector against the stored ones.

What to check

• Which embedding model, at which version and dimension, produced the stored vectors, and is the same model encoding the queries?
• What is the model's maximum input in tokens, and how were passages longer than that handled before indexing?
• Which languages does the model card list, and was retrieval measured on the languages and scripts in the actual collection?
• Which MTEB or MMTEB benchmark and which task column does a quoted score come from, and were any of the evaluation datasets used in fine-tuning?
• Are vectors normalized, and does the similarity function in the index match the one the model was trained for?
• What is the plan for re-encoding the collection when the embedding model changes?

Azcuna's TRACE is one offline viewer for S1000D, MIL-STD-3001 XML, and PDF technical manuals. It keeps the source location and the markings of everything it shows and never modifies the source. It was proposed to NAVAIR under Navy SBIR Phase I topic DON26BZ05-NV078, and the proposal was submitted on September 23, 2026.

Questions people ask

What is an embedding model?

An embedding model is a neural network that maps a piece of text to a fixed-length vector of numbers, built so that texts with similar meaning produce vectors that lie close together. The Sentence-BERT paper (Reimers and Gurevych, 2019) describes deriving such vectors from BERT with a pooling step and training the network with siamese and triplet structures so the vectors can be compared with cosine similarity. The E5 paper calls text embeddings low-dimensional vector representations for texts of arbitrary length.

What is the difference between an embedding model and an LLM?

An embedding model outputs a vector and stops; a large language model outputs text. The two are trained for different jobs. The E5 and BGE papers train embedding models with a contrastive loss that pulls paired texts together, while a generator is trained to predict the next token. The 2020 retrieval-augmented generation paper used two separate pre-trained models, a neural retriever over a dense vector index and a seq2seq generator.

How is cosine similarity used with embeddings?

Cosine similarity is the inner product of two vectors divided by the product of their lengths, so it measures the angle between them and ignores their magnitude. The Sentence Transformers documentation lists cosine as the default similarity function, with dot product, Euclidean, and Manhattan as alternatives. The Faiss wiki notes that if every vector is normalized to length 1 beforehand, maximum inner product search returns the cosine ranking, which is why many systems store normalized vectors and use a dot product.

How are embedding models trained?

Mostly with contrastive objectives. Sentence-BERT fine-tuned BERT on 570,000 SNLI and 430,000 MultiNLI sentence pairs with a softmax classification objective, and also defined regression and triplet objectives. E5 pre-trained on about 270 million filtered text pairs from the web with an InfoNCE loss and in-batch negatives at a batch size of 32,768, then fine-tuned on labeled data. BGE used RetroMAE pre-training, contrastive learning with batches as large as 19,200, and fine-tuning with one mined hard negative and a task instruction per pair.

What is the MTEB leaderboard?

MTEB, the Massive Text Embedding Benchmark (Muennighoff et al., 2022), evaluates embedding models on 8 task types across 58 datasets and 112 languages, and the project runs a public leaderboard on the Hugging Face Hub. Each task type has a main metric, for example nDCG@10 for retrieval and Spearman correlation of cosine scores for semantic textual similarity. The paper found that no single method dominated every task, so the column that matches the intended use matters more than the overall average.

How many dimensions should an embedding have?

Published models range widely: all-MiniLM-L6-v2 outputs 384 dimensions, bge-large-en-v1.5 and multilingual-e5-large output 1024, and OpenAI's text-embedding-3-large outputs 3072 unless shortened. Each extra dimension costs 4 bytes per stored vector in a flat index, according to the Faiss wiki. The Matryoshka Representation Learning paper (Kusupati et al., 2022) trains models so the first m dimensions of a vector work on their own, which lets a system truncate vectors to save space and recover accuracy by re-ranking with the full vector.

What are Matryoshka embeddings?

Matryoshka Representation Learning, from a 2022 paper by Kusupati and colleagues, trains one embedding so that nested prefixes of it, such as the first 8, 16, 32, and so on up to all dimensions, are each usable representations. The paper reports that shortlisting candidates with 16 dimensions and re-ranking with the full 2048 was as accurate as full search while being about 128 times cheaper in theory and about 14 times faster in practice on ImageNet-1K. OpenAI's embeddings guide exposes the idea as a dimensions parameter.

What is a multilingual embedding model?

A multilingual embedding model places text from many languages in one vector space, so a query in one language retrieves a passage in another. LaBSE (Feng et al., 2020) is a dual encoder trained on 6 billion translation pairs that reached 83.7 percent bitext retrieval accuracy over 112 Tatoeba languages, and its released model covers 109 or more languages. The multilingual E5 report (Wang et al., 2024) describes three model sizes contrastively pre-trained on about 1 billion multilingual text pairs.

What is the input limit of an embedding model?

Each model has a maximum input length and truncates anything longer. The all-MiniLM-L6-v2 model card says text beyond 256 word pieces is truncated by default; the multilingual-e5-large card says long texts are truncated to at most 512 tokens; bge-large-en-v1.5 also takes 512 tokens; bge-m3 accepts up to 8192 tokens; and OpenAI's text-embedding-3 models accept 8192 tokens. The MTEB paper imposes no limit on input length and leaves truncation to the model, so a benchmark score can hide the cut.

Can an embedding model run offline?

Yes. The open models discussed here, such as the E5, BGE, and all-MiniLM families, are released on the Hugging Face Hub under MIT or Apache-2.0 licenses and run through the Sentence Transformers library. The Transformers installation documentation describes an offline mode: download the model repository ahead of time with snapshot_download, then set HF_HUB_OFFLINE=1 or pass local_files_only=True so that loading makes no network calls. The vectors can then be stored and searched in a local index.

Sources

1. 1Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks · arXiv (Reimers and Gurevych)
2. 2Text Embeddings by Weakly-Supervised Contrastive Pre-training · arXiv (Wang et al.)
3. 3C-Pack: Packed Resources For General Chinese Embeddings · arXiv (Xiao et al.)
4. 4MTEB: Massive Text Embedding Benchmark · arXiv (Muennighoff et al.)
5. 5MMTEB: Massive Multilingual Text Embedding Benchmark · arXiv (Enevoldsen et al.)
6. 6Matryoshka Representation Learning · arXiv (Kusupati et al.)
7. 7Language-agnostic BERT Sentence Embedding · arXiv (Feng et al.)
8. 8Multilingual E5 Text Embeddings: A Technical Report · arXiv (Wang et al.)
9. 9Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · arXiv (Lewis et al.)
10. 10Dense Passage Retrieval for Open-Domain Question Answering · arXiv (Karpukhin et al.)
11. 11embeddings-benchmark/mteb README · MTEB project (GitHub)
12. 12MTEB Leaderboard · MTEB project (Hugging Face Space)
13. 13intfloat/multilingual-e5-large model card · Hugging Face Hub (intfloat)
14. 14BAAI/bge-m3 model card · Hugging Face Hub (BAAI)
15. 15BAAI/bge-large-en-v1.5 model card · Hugging Face Hub (BAAI)
16. 16sentence-transformers/all-MiniLM-L6-v2 model card · Hugging Face Hub (sentence-transformers)
17. 17Sentence Transformers documentation · Sentence Transformers project
18. 18Semantic Textual Similarity · Sentence Transformers documentation
19. 19Installation (offline mode) · Hugging Face Transformers documentation
20. 20BERT · Hugging Face Transformers documentation
21. 21Vector embeddings · OpenAI developer documentation
22. 22MetricType and distances · Faiss project wiki (facebookresearch/faiss)
23. 23Faiss indexes · Faiss project wiki (facebookresearch/faiss)

Where this shows up in our work

• TRACE, the offline S1000D and IETM viewer opens S1000D, MIL-STD-3001 and PDF technical manuals on a tablet with every marking intact.
• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.