Pith. sign in

REVIEW 10 cited by

Matryoshka Representation Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2205.13147 v4 pith:BG3T7W4E submitted 2022-05-26 cs.LG cs.CV

Aditya Kusupati , Gantavya Bhatt , Aniket Rege , Matthew Wallingford , Aditya Sinha , Vivek Ramanujan , William Howard-Snyder , Kaifeng Chen
show 3 more authors
Sham Kakade Prateek Jain Ali Farhadi
This is my paper · ORCID
classification cs.LGcs.CV
keywords representationsdownstreamrepresentationcomputationallearningmatryoshkatasksaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Learned representations are a central component in modern ML systems, serving a multitude of downstream tasks. When training such representations, it is often the case that computational and statistical constraints for each downstream task are unknown. In this context rigid, fixed capacity representations can be either over or under-accommodating to the task at hand. This leads us to ask: can we design a flexible representation that can adapt to multiple downstream tasks with varying computational resources? Our main contribution is Matryoshka Representation Learning (MRL) which encodes information at different granularities and allows a single embedding to adapt to the computational constraints of downstream tasks. MRL minimally modifies existing representation learning pipelines and imposes no additional cost during inference and deployment. MRL learns coarse-to-fine representations that are at least as accurate and rich as independently trained low-dimensional representations. The flexibility within the learned Matryoshka Representations offer: (a) up to 14x smaller embedding size for ImageNet-1K classification at the same level of accuracy; (b) up to 14x real-world speed-ups for large-scale retrieval on ImageNet-1K and 4K; and (c) up to 2% accuracy improvements for long-tail few-shot classification, all while being as robust as the original representations. Finally, we show that MRL extends seamlessly to web-scale datasets (ImageNet, JFT) across various modalities -- vision (ViT, ResNet), vision + language (ALIGN) and language (BERT). MRL code and pretrained models are open-sourced at https://github.com/RAIVNLab/MRL.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 25 citations worldwide. Full citation record

  1. Bekko Embedding: Parameter-Efficient Multilingual Retrieval with Ultra-Compact Encoders

    cs.IR 2026-07 conditional novelty 7.0 of 10

    Bekko a8m, with 7.7M active parameters, scores 56.2 on MMTEB Multilingual v2 Retrieval, beating mE5 models and BGE-M3, while a25m reaches 57.5, on par with gte-multilingual-base.

  2. Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Trained connectors and audio-only gated adapters integrate audio into a frozen vision-language embedding space, preserving base outputs bit-exactly and yielding emergent audio-image retrieval.

  3. Discriminative Axis, Not Data Volume: What a Contrastive Corpus Teaches an Audio Embedding

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A contrastive audio embedding learns an attribute only when in-batch negatives cannot be separated without it, so corpus structure, not size or caption vocabulary, controls what is encoded.

  4. MosaicJoin: Compact Semantic Sketches for Value-Level Join Discovery

    cs.DB 2026-07 conditional novelty 6.0 of 10

    MosaicJoin finds semantically joinable columns in data lakes fast by scoring fixed-size k-center sketches of value embeddings with a Chamfer-style similarity and sampling query values.

  5. Sticking to the Mean: Detecting Sticky Tokens in Text Embedding Models

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Sticky tokens, defined as vocabulary tokens that pull sentence similarity toward the model's mean, are detected across 40 text embedding models and can degrade retrieval and clustering by up to 50%.

  6. Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Prompt-based text embeddings can be truncated to a small fraction of their dimensions with little performance loss on classification and clustering, but retrieval and STS degrade faster; the difference tracks lower in...

  7. Training Sparse Mixture Of Experts Text Embedding Models

    cs.CL 2025-02 reject novelty 6.0 of 10

    Nomic Embed v2 applies sparse mixture-of-experts upcycling to a multilingual biencoder, reporting competitive BEIR and MIRACL scores with fewer active parameters than dense models of similar size.

  8. Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders

    cs.CL 2026-07 conditional novelty 5.5 of 10

    Portuguese embedding quality is task-dependent; multilingual leaderboards misrank models, while Portuguese contrastive fine-tuning plus MRL improves STS and retrieval under compact dimensions.

  9. vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models

    cs.NI 2026-02 conditional novelty 5.0 of 10

    vLLM Semantic Router routes LLM requests by composing thirteen signal types into Boolean decision policies, with safety, caching, and model-selection plugin chains.

  10. Specializing General-purpose LLM Embeddings for Implicit Hate Speech Detection across Datasets

    cs.CL 2025-08 conditional novelty 5.0 of 10

    Fine-tuning large general-purpose text embeddings with a simple instruction yields state-of-the-art implicit hate speech detection, with up to 20.35 point cross-dataset F1 gains.

Pith tools