Matryoshka embeddings, explained

A Matryoshka embedding is a vector trained so that its first few dimensions are a usable embedding on their own, its first few more are a better one, and the whole vector is the best. The method is Matryoshka Representation Learning, or MRL, from the 2022 paper of that name by Aditya Kusupati and ten co-authors, first posted to arXiv on May 26, 2022. An ordinary embedding model spreads information across all of its output dimensions, so cutting a 768-dimensional vector to 128 throws away information at random. A Matryoshka-trained model packs the most important information into the leading dimensions, so the same cut keeps most of the accuracy. The practical payoff is that one model and one set of stored vectors can serve several memory and speed budgets, and that a search can be run cheaply on short vectors and then corrected with the long ones.

Nested representations

The paper's Section 3 states the mechanism. A model F maps an input x to a d-dimensional representation z. For a chosen set M of nesting sizes, for example 8, 16, 32, 64, 128, 256, 512, 1024 and 2048 for a 2,048-dimensional ResNet50, MRL asks that the first m dimensions of z, for every m in M, be a good representation. During training, each prefix gets its own linear classifier, the usual loss is computed on each prefix separately, and the losses are added together after scaling by relative importance weights. Everything else in the pipeline is unchanged. The paper says MRL "minimally modifies existing representation learning pipelines" and "imposes no additional cost during inference and deployment".

Two details matter. First, the paper only optimizes O(log d) sizes, roughly doubling at each step, yet its Figure 5 shows the accuracy at sizes in between interpolates smoothly, so a prefix of 100 or 300 dimensions works even though it was never trained for directly. Second, there is a cheaper variant, MRL-E for Efficient, that ties the classifier weights across sizes so the m-dimensional head is just the first m columns of the full head. The paper notes that MRL applied to BERT's masked language modeling reduces to MRL-E because of weight tying in the output layer.

The paper demonstrates the method across modalities: ResNet50 on ImageNet-1K with 2,048 dimensions, ViT-B/16 on JFT-300M and BERT-Base with 768 dimensions nested at 12, 24, 48, 96, 192, 384 and 768, and the ALIGN vision-language model. For every one, the claim tested is the same: is the m-dimensional prefix of a Matryoshka representation as good as an m-dimensional embedding trained independently at that size, which the paper calls a fixed-feature or FF baseline?

Truncating dimensions

Using a Matryoshka embedding is just slicing. Take the first m values of the vector and discard the rest. One step follows: because similarity search uses cosine similarity or inner product on unit vectors, the truncated vector must be re-normalized to unit length, since dropping dimensions shortens it. Every implementation shows the step. The nomic-embed-text-v1.5 model card applies a layer normalization, slices to the chosen dimension and then L2-normalizes. OpenAI's embeddings guide says that when a developer shortens an embedding by hand, "you need to be sure to normalize the dimensions of the embedding", shows code that truncates to 256 and L2-normalizes, and recommends asking the API for the shorter size through the dimensions parameter instead. Snowflake's arctic-embed-m-v1.5 card describes its compression as "Truncation and renormalization to 256 dimensions".

Truncating a model that was not trained this way is a different matter. Nothing orders its dimensions by importance, so a prefix is an arbitrary projection and its quality is unpredictable. The question to ask of any model before cutting its vectors is whether its card or paper states Matryoshka training and lists the sizes it was trained at.

The Sentence Transformers Matryoshka guide adds a caution about what truncation does not buy. A Matryoshka model is not faster to train or to run, and not smaller or more memory-efficient, than a standard model of the same size; the encoder still produces every dimension. Only the processing and storage of the resulting embeddings get cheaper.

The accuracy versus dimension curve

The paper's Figure 2 plots ImageNet-1K linear classification accuracy against representation size for ResNet50 models, and its Figure 3 plots 1-nearest-neighbor accuracy. In both, the MRL model is at least as accurate as the independently trained fixed-feature model at every size, and ahead of it at the small sizes. Figure 7 repeats the comparison for retrieval, measured as mAP@10, and finds MRL better across all representation sizes. The paper's headline numbers are in the abstract: up to 14 times smaller embeddings for ImageNet-1K classification at the same accuracy, up to 14 times real-world speedup for large-scale retrieval on ImageNet-1K and ImageNet-4K, and up to 2 percent better accuracy on long-tail few-shot classification.

The language result is in the paper's Appendix D.2, Table 7, which reports masked language modeling validation accuracy for a BERT-Base model trained normally and with MRL at each nesting size. The paper describes the two as within 0.5 percent; the largest gap in its table is 0.54 points, at 768 dimensions.

The paper calls this "a promising initial result that could help with large-scale adaptive document retrieval", and the open text embedding models trained since have published the retrieval version of the curve. The nomic-embed-text-v1.5 card gives MTEB averages at a sequence length of 8,192 tokens.

The shape is what MRL predicts: a third of a point lost at 512, 1.2 points at 256, 3 points at 128 and 6 points at 64, which is one twelfth of the storage. The Sentence Transformers guide reports a similar shape on STSBenchmark for a model it trained: "Even at 8.3% of the embedding size, the Matryoshka model preserves 98.37% of the performance", against 96.46 percent for a standard model truncated the same way. OpenAI's guide makes the same point for its models, stating that on MTEB a text-embedding-3-large embedding shortened to 256 dimensions still outperforms an unshortened text-embedding-ada-002 embedding of 1,536.

Adoption in open models

The paper's method has been adopted by the text embedding models most often used in document search, and the model cards say so. nomic-embed-text-v1.5, released under Apache 2.0, says it "utilizes Matryoshka Representation Learning" to let developers trade embedding size for "a negligible reduction in performance", and publishes the table above. The card also requires a task prefix on every input, search_query or search_document for retrieval, and supports sequences past 2,048 tokens.

mixedbread's mxbai-embed-large-v1, also Apache 2.0, states that the model "supports Matryoshka Representation Learning and binary quantization" and that the two can be combined; its examples use truncate_dim of 512 in Sentence Transformers and a dimensions argument of 512 in its API, alongside int8 and binary output formats. The card reports an MTEB average of 64.68 over 56 datasets at full size.

Snowflake's arctic-embed-m-v1.5, a 109 million parameter model, is designed around compressibility and gives the most complete table of the three.

The card describes the last row as 24 times compression against 768 float32 dimensions at 98 percent of the score, and gives the uniform scalar quantization ranges it uses, minus 0.18 to plus 0.18 for 4-bit and minus 0.3 to plus 0.3 for 8-bit. It also compares retrieval at 256 dimensions across MRL-trained models: 54.2 for its own, 52.4 for Google's gecko, 51.7 for OpenAI's text-embedding-3-large and 50.8 for nomic-embed-text-v1.5.

The jina-embeddings-v3 paper, posted September 16, 2024, describes a 570 million parameter multilingual model with an 8,192-token context and a default output of 1,024 dimensions that MRL allows to be reduced to as low as 32. OpenAI's guide states that both text-embedding-3 models were trained so developers can shorten embeddings by removing numbers from the end, with native sizes of 1,536 for text-embedding-3-small and 3,072 for text-embedding-3-large.

On the training side, the Sentence Transformers guide describes MatryoshkaLoss, which wraps a base loss and applies it to truncated portions of the embeddings at each size in matryoshka_dims, 768, 512, 256, 128 and 64 in its example, adding the results together, optionally with weights. A Matryoshka2dLoss combines this with an adaptive-layer loss so that shallower models can be used as well.

Adaptive retrieval: shortlist on short vectors, rerank on full

The paper's most useful idea for search is in Section 4.3.1. Adaptive retrieval runs nearest-neighbor search over the database using a short prefix of the query embedding, Ds, to produce a shortlist of K candidates, then re-ranks those K with a longer prefix, Dr. On ImageNet-1K, with a database of about 1.3 million images and 50,000 queries, the paper uses K of 200, Ds of 16 and Dr of 2,048, and reports that this setting "is as accurate as single-shot retrieval with d = 2048" while being about 128 times more efficient in theory and about 14 times faster in practice, the practical figure measured with HNSW on the same hardware. On the harder ImageNet-4K, with about 4.2 million images, Ds had to rise to 64, giving about 32 times and 6 times. The re-ranking step is cheap: the paper notes that re-ranking 200 images with 2,048 dimensions costs 400 thousand FLOPs, against 2.6 billion FLOPs for an exact full-dimension search over ImageNet-1K.

The paper also reports that the shortlisting search can use an approximate index without penalty: HNSW with 32 neighbors for shortlisting did not decrease retrieval accuracy. All adaptive retrieval settings the paper tried sat above the Pareto frontier of single-shot retrieval at any fixed dimension, and an ablation on K found accuracy gains stopped after a point. The paper proposes a further variant, funnel retrieval, which re-ranks in several stages with increasing dimensions and a shrinking shortlist, to avoid having to pick a single Ds and Dr.

In a Faiss or vector database workflow this is a two-index design. The Faiss wiki's index table puts a flat index at 4 times d bytes per vector and the guidelines put an HNSW index at d times 4 plus M times 2 times 4, so a shortlist index over 64-dimensional prefixes of a 768-dimensional model stores 256 bytes per vector instead of 3,072, with the graph links added on top. The full vectors are kept for the second stage, and the Faiss wiki's Pre- and post-processing page describes the re-ranking wrapper: IndexRefineFlat fetches k_factor times k candidates from the fast index, 4 times 10 in the page's example, recomputes real distances and returns the best k, with the warning that it has to store the full vectors and so is not memory-efficient. Matryoshka prefixes change what the fast index holds, not the pattern. The companion post on vector search and Faiss covers the index types the two stages can use.

Two practical consequences follow. The stored full vectors can be truncated later if the budget changes, without re-encoding the collection, which is the main operational advantage over picking a smaller model. And the shortlist index can run on short vectors on a small machine while the full vectors sit on disk, since only K of them are read per query.

Where the gains come from and what they do not cover

MRL's gains are relative to a fixed-dimension baseline of the same size, trained independently. The paper's claim is that the m-dimensional prefix is as good as, and often better than, an m-dimensional model, not that it is as good as the full d-dimensional model. The curves above show the full vector always wins; the point is how slowly accuracy falls.

The curve is model-specific. nomic-embed-text-v1.5 loses 1.2 MTEB points at 256 dimensions; Snowflake reports 0.9 nDCG@10 points on retrieval at 256; a model that was not trained this way has no published curve to consult. Benchmarks on MTEB or ImageNet also say nothing about a particular document collection, where the difference between 256 and 768 dimensions may be larger or smaller. The paper's own adaptive retrieval experiments had to raise Ds from 16 to 64 when moving from ImageNet-1K to ImageNet-4K, which is the same lesson: the right prefix length depends on the data and has to be measured on it.

Finally, truncation is one of two orthogonal compressions, and the Snowflake and mixedbread cards treat them together. Scalar or binary quantization reduces the bytes per dimension; Matryoshka truncation reduces the dimensions. The 128-byte arctic-embed vector is both at once.

What to check

• Does the embedding model's card or paper state Matryoshka training, and which prefix lengths does it list?
• Are truncated vectors re-normalized before they are stored and before they are compared?
• What is the accuracy at the chosen prefix on this collection, measured against passages a person marked relevant, not only the model card's MTEB curve?
• If adaptive retrieval is used, what are the shortlist size K, the shortlist dimension and the re-rank dimension, and were they checked against a full-dimension search on the same queries?
• Where do the full vectors live for the re-ranking stage, and is their storage counted in the system's memory and disk budget?
• Is truncation being combined with scalar or binary quantization, and was the combined loss measured rather than assumed from each alone?

Azcuna's TRACE is one offline viewer for S1000D, MIL-STD-3001 XML, and PDF technical manuals. It keeps the source location and the markings of everything it shows and never modifies the source. It was proposed to NAVAIR under Navy SBIR Phase I topic DON26BZ05-NV078, and the proposal was submitted on September 23, 2026.

Questions people ask

What are Matryoshka embeddings?

Matryoshka embeddings are vectors trained so that any leading prefix of the dimensions is itself a usable embedding. The 2022 paper Matryoshka Representation Learning by Kusupati and colleagues optimizes the training loss at O(log d) nested sizes within one vector, for example 8, 16, 32 and so on up to the full 2,048 dimensions, so that the first m dimensions carry the most important information. The name comes from the nested Russian dolls, since each shorter embedding sits inside the longer one.

What is Matryoshka Representation Learning?

Matryoshka Representation Learning, or MRL, is the training method from the 2022 paper of that name. For each chosen prefix length m, the model's loss is computed using only the first m dimensions of the embedding, with a separate output head per length, and the losses are added together with relative importance weights. The paper describes it as a minimal change to an existing representation learning pipeline with no extra inference cost, and shows it on ResNet50, ViT-B/16, ALIGN and BERT.

Can you truncate embedding dimensions?

Only if the model was trained for it. With an ordinary embedding model, the dimensions are not ordered by importance and cutting them discards information unpredictably. With a Matryoshka-trained model, the model card or paper states which prefix lengths work; nomic-embed-text-v1.5 lists 768, 512, 256, 128 and 64, and OpenAI's guide says text-embedding-3-large can be shortened to 256. After truncation the vector must be re-normalized, which the Nomic card, the OpenAI guide and the Snowflake card all show.

How much accuracy do you lose by truncating a Matryoshka embedding?

The model cards give the curve. nomic-embed-text-v1.5 reports MTEB averages of 62.28 at 768 dimensions, 61.96 at 512, 61.04 at 256, 59.34 at 128 and 56.10 at 64. Snowflake's arctic-embed-m-v1.5 card reports its MTEB Retrieval nDCG@10 going from 55.1 at 768 dimensions to 54.2 at 256, 99 percent of the baseline. The Sentence Transformers guide reports a Matryoshka model keeping 98.37 percent of its STSBenchmark performance at 8.3 percent of the embedding size.

What is adaptive retrieval with Matryoshka embeddings?

Adaptive retrieval, in Section 4.3.1 of the Matryoshka Representation Learning paper, searches the database with a short prefix of the query embedding to get a shortlist, then re-ranks the shortlist with the full embedding. The paper shortlists 200 candidates using 16 dimensions and re-ranks with 2,048 on ImageNet-1K, and reports accuracy equal to single-shot search at 2,048 dimensions while being about 128 times cheaper in FLOPs and about 14 times faster in wall-clock time with HNSW shortlisting.

Which embedding models use Matryoshka training?

Model cards and papers that state it include nomic-embed-text-v1.5 from Nomic, whose card says it uses Matryoshka Representation Learning; mixedbread's mxbai-embed-large-v1, whose card says the model supports Matryoshka Representation Learning and binary quantization; Snowflake's arctic-embed-m-v1.5, whose card describes truncation and renormalization to 256 dimensions; and jina-embeddings-v3, whose paper says its 1,024-dimension output can be reduced to as low as 32 through MRL. OpenAI's guide says its text-embedding-3 models were trained to allow shortening.

Does Matryoshka training make a model faster?

No. The Sentence Transformers Matryoshka guide states that a Matryoshka model is not faster to train or run and is not smaller or more memory-efficient; the encoder still produces the full vector. What gets cheaper is everything after encoding: storing the vectors, comparing them, and indexing them. The MRL paper says the same, describing the method as adding no inference or deployment cost while enabling cheaper downstream classification and retrieval.

How do you train a Matryoshka embedding model?

The Sentence Transformers library provides MatryoshkaLoss, which wraps any base loss and applies it to truncated copies of the embeddings at each length in matryoshka_dims, for example 768, 512, 256, 128 and 64, adding the results together, optionally with weights. At inference the truncate_dim argument cuts the output. The library also offers Matryoshka2dLoss, which combines prefix truncation with an adaptive-layer loss so that fewer transformer layers can be used too. The original paper's code is at the RAIVNLab MRL repository.

Can you combine Matryoshka truncation with quantization?

Yes, and two model cards document it. Snowflake's arctic-embed-m-v1.5 card describes truncating to 256 dimensions and then applying 4-bit uniform scalar quantization in the range minus 0.18 to plus 0.18, for 128 bytes per vector, 24 times smaller than 768 float32 values, at 98 percent of the full retrieval score. mixedbread's mxbai-embed-large-v1 card says the model supports both MRL and binary quantization and that the two can be combined to reduce memory use significantly.

How does Matryoshka relate to FAISS or a vector database?

A flat index stores 4 times d bytes per float32 vector, by the Faiss index table, so truncating d cuts the index in proportion. The Matryoshka paper's adaptive retrieval maps onto Faiss directly: a flat or HNSW index over short prefixes produces a shortlist, and a refinement step recomputes distances with full vectors, which is what the Faiss wiki's IndexRefineFlat does with its k_factor parameter. The paper notes HNSW shortlisting with 32 neighbors did not reduce accuracy.

Sources

1. 1Matryoshka Representation Learning · arXiv (Kusupati et al.)
2. 2nomic-embed-text-v1.5 model card · Nomic AI on Hugging Face
3. 3mxbai-embed-large-v1 model card · mixedbread on Hugging Face
4. 4snowflake-arctic-embed-m-v1.5 model card · Snowflake on Hugging Face
5. 5jina-embeddings-v3: Multilingual Embeddings With Task LoRA · arXiv (Sturua et al.)
6. 6Vector embeddings · OpenAI developer documentation
7. 7Matryoshka Embeddings · Sentence Transformers documentation
8. 8Pre- and post-processing · Faiss wiki
9. 9Guidelines to choose an index · Faiss wiki
10. 10Faiss indexes · Faiss wiki

Where this shows up in our work

• TRACE, the offline S1000D and IETM viewer opens S1000D, MIL-STD-3001 and PDF technical manuals on a tablet with every marking intact.
• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.