Pith. sign in

REVIEW 4 major objections 6 minor 23 references

2D Matryoshka Training for Information Retrieval

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read A single encoder trained with 2D Matryoshka embeddings outperforms full-size fine-tuning on sub-layer, sub-dimension outputs, but not separately trained small models—and the same holds in retrieval.

desk verdict Useful reproducibility study with a clear engineering takeaway, but the claim that 2DMSE-V2 underperforms separately trained BERT-S rests on AnglE defaults and no reported variance, so the abstract overstates the point. read the letter →

arxiv 2411.17299 v1 pith:VZL7VZUF submitted 2024-11-26 cs.IR cs.CL

classification cs.IRcs.CL
keywords 2DMatryoshkatrainingrepresentationlearningdenseretrievalsentenceembeddingssemantictextsimilarityreproducibilitystudyMSMARCOBEIR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether 2D Matryoshka training—a method that trains one encoder to produce usable embeddings from many layer-depth and dimension-size combinations at once—keeps its promised edge when scrutinized and when moved from sentence similarity to search. Reproducing both published versions of the method, the authors find that each beats standard Matryoshka training and full-model fine-tuning on sub-layer, sub-dimension embeddings, and that the advantage carries over to passage retrieval on MS MARCO and zero-shot retrieval on BEIR. The catch: neither version beats models trained separately for one specific layer-and-dimension size, so a single 2D model is a strong generalist but not the accuracy ceiling. The paper then shows retrieval-specific tweaks—training full-dimension outputs at every layer and training a wider set of target dimensions—recover some of the lost ground at small sizes, while naively fixing document encoders to the full model does not help.

What carries the argument

The object is the 2D Matryoshka training objective, which nests two truncation axes: layer depth and embedding dimension. Version 1 draws a random sub-layer, trains last-layer and sub-layer embeddings, and aligns their distributions with KL divergence; Version 2 applies a logarithmically weighted loss to every layer's sub-dimension embeddings and adds a PCA-driven mean-squared-error plus KL alignment to a target dimension of 128. The mechanism is that matryoshka-style truncated outputs are trained as first-class citizens at every depth, so early-layer small embeddings inherit usable semantics, letting one encoder serve many size budgets. The paper's key analytical move is comparing this single multi-size model against BERT-S, a separately trained model per size—the comparison that reveals the accuracy cost of multi-size flexibility.

What would settle it

Release the original V2 training configuration; if a faithfully retrained V2 matches or beats BERT-S across the layer-dimension grid, the paper's claim that dedicated small models remain the accuracy ceiling is overturned.

Watch

Extended reading notes

Core claim

The central discovery is that the two published implementations of 2D Matryoshka training reproduce the original qualitative trend: a single model trained across layers and dimensions dominates both raw BERT fine-tuning and last-layer-only Matryoshka training when embeddings are taken from early layers and small dimensions. Version 2 is the better design, nearly matching separately trained small models and beating them in some STS sub-layer cases, but the original claim that 2DMSE surpasses dedicated small models does not survive the broader comparison: across almost all layer-dimension setups, BERT-S, a model fine-tuned separately for that exact size, remains more effective. The same ordering transfers to retrieval: on MS MARCO and BEIR, 2DMSE beats MSE and BERT except at the last layer, yet loses to BERT-S everywhere. The paper also establishes that two loss modifications—adding full-dimension loss from each layer and training more target dimensions—improve retrieval results, while a fixed full-size document encoder does not.

Load-bearing premise

The conclusion that 2DMSE-V2 loses to separately trained small models assumes the reproduction's training settings match the original V2, whose hyperparameters were never reported; the authors substitute default BERT fine-tuning parameters from a third-party library.

Editorial extensions

If this is right

  • A single 2D Matryoshka model can replace a family of separately trained encoders with a modest accuracy loss, making it cheaper to serve multiple embedding-size budgets from one checkpoint.
  • The qualitative results transfer from sentence similarity to both supervised and zero-shot retrieval, so the multi-size tradeoff is not an artifact of STS evaluation.
  • Training full-dimension sub-layer outputs and covering more target dimensions shrinks the low-dimension gap, indicating that loss design, not architecture, drives much of the remaining deficit.
  • Fixing the document encoder to the full-model output does not consistently pay for its extra computation, so paired small encoders remain the sensible efficiency point for asymmetric retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the 48-cell layer-dimension grid implies a measurable size-accuracy frontier, and a practitioner could use those measured gaps to decide when one elastic model suffices versus when per-size training is worth the extra compute.
  • Beyond the paper: if the abrupt retrieval collapse below dimension 128 stems from training only at that target size, then serving small vectors safely requires explicit training at those dimensions—a hypothesis directly testable by evaluating the +DIMS checkpoints on BEIR.
  • Beyond the paper: the fact that the PCA-based dimension loss helps STS but trails dedicated small models in retrieval hints that the PCA projection may discard retrieval-specific signal; a variant that aligns truncated embeddings with a contrastive rather than reconstruction objective could close the gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This reproducibility study re-implements the two published versions of 2D Matryoshka Sentence Embedding training (2DMSE-V1 and V2) on BERT-base and evaluates them on semantic textual similarity (STS) benchmarks, then extends the approach to passage retrieval on MS MARCO and zero-shot BEIR. The paper reports that both versions outperform traditional Matryoshka training (MSE) and full-model baselines on sub-layer/sub-dimension configurations, that 2DMSE-V2 generally outperforms V1, and that neither version consistently beats models separately trained for each target layer-dimension setup. For retrieval, the authors find that the original 2DMSE-V2 is less effective than separately trained models, and they explore loss modifications (scoring loss, full-dimension loss, more target dimensions) as well as a fixed full-size document encoder.

Significance. If the findings hold, the paper provides a useful reproducibility result: a single 2D Matryoshka model is a strong multi-size generalist, but dedicated small models remain the accuracy ceiling at a fixed size. The work also contributes the first retrieval extension of 2DMSE and an openly available reproduction codebase. The main strengths are the breadth of evaluation (STS, MS MARCO, 13 BEIR datasets, 48 layer-dimension configurations) and the authors' transparency about the unreported V2 hyperparameters. However, the central negative claim regarding V2 rests on an external hyperparameter assumption, the experiments are single-run with no variance estimates, and one headline statement in the abstract is contradicted by the paper's own V1 results. These issues currently limit the reliability of the contribution.

major comments (4)
  1. [Section 3.1, footnote 5] The V2-versus-BERT-S comparison, which underlies the central claim that 2DMSE does not beat separately trained models, depends on the AnglE example BERT fine-tuning configuration because the original 2DMSE-V2 hyperparameters were never reported. Several V2-versus-BERT-S gaps are small (e.g., layer 12 at dimensions 64-256 in Figure 1(b)), so a different learning rate, batch size, epoch count, warmup, or loss weighting could change the ordering. The paper should add a sensitivity analysis over training hyperparameters and/or repeated-seed results, or explicitly restrict the claim to the AnglE-default configuration.
  2. [Abstract; Section 4.1] The abstract states that 'both versions ... do not outperform models trained separately on specific sub-layer and sub-dimension setups,' but Section 4.1 reports that 2DMSE-V1 surpasses BERT-S in the sub-layers (layers 2, 4, and 6) and at embedding size 8. This is a direct internal contradiction that should be corrected by qualifying the abstract to refer to V2 and to the tested configurations.
  3. [Figures 1(b), 2; Section 4.3] All comparisons are single-run and report no error bars, confidence intervals, or significance tests. The small observed differences, such as V2 versus BERT-S at the last layer and dimensions 64-256 in Figure 1(b), and the 'significant improvement' claimed for FULL-DIM in Section 4.3.2, cannot be established as robust orderings on the basis of one seed per configuration. At minimum, repeated seeds or bootstrap intervals should be provided for the key comparisons, and the claims should be softened otherwise.
  4. [Section 3; Figure 1 caption] The training recipe for BERT-S is not described in sufficient detail: the paper does not specify the number of steps, learning rate, batch size, or whether each separately trained model uses the same data, loss, and schedule as the 2DMSE models. Because BERT-S is the reference point for the 'do not outperform' conclusion, the paper should either report these details or confirm that they are fully specified in the repository and identical across all BERT-S configurations.
minor comments (6)
  1. [Figure 5 caption] The caption uses 'paird-doc' where 'paired-doc' is intended.
  2. [Section 4.2] The sentence 'performance abruptly deteriorateeas for dimensions lower than 128' contains a typo ('deteriorateeas' should be 'deteriorates').
  3. [Section 1] The contributions paragraph spells 'Matryoska' instead of 'Matryoshka'.
  4. [Section 4.3.1] The SCORE modification is described only as 'Initial findings suggest...' without quantitative support; either report the relevant numbers or specify exactly which configurations in Figure 4 support the statement.
  5. [Section 3.1] The backbone model name 'bert-base-uncased' should be set in code font for consistency.
  6. [References] Reference [13] is cited as the source of 2D Matryoshka training but is titled 'ESE: Espresso sentence embeddings'; please clarify the relationship between the ESE paper and the 2D Matryoshka method, or cite the exact title if different.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical reproducibility study with no fitted-input predictions or self-citation load-bearing arguments.

full rationale

This paper performs no derivations that reduce to their inputs, and it fits no parameters to the results it reports. The central claims are empirical comparisons of 2DMSE-V1, 2DMSE-V2, Matryoshka (MSE), full-model BERT, and separately trained BERT-S models on held-out STS and retrieval benchmarks (MSMARCO dev, BEIR), so every effectiveness number is measured rather than constructed from the method definition. The only potential concern raised by the paper is reproduction fidelity: 'We use BERT training parameters as the original 2DMSE-V2 configurations have not been reported' (Section 3.1, footnote 5), and the paper explicitly states its goal is to 'confirm the observed trends' rather than reproduce absolute metrics (Section 4.1). That limitation concerns external validity of the reproduction, not circular reasoning, because the training configuration is inherited from an external library (AnglE) rather than chosen to force the paper's conclusions. The paper cites two prior works by its own authors ([19] and [21]), but only in the related-work section for background context on dense retrievers; these citations are not load-bearing for any of the paper's experimental findings or design choices. No 'uniqueness theorem' or prior author-derived constraint is invoked to forbid alternatives, and no known empirical pattern is renamed as an organizing principle. The claimed improvements from modified losses (FULL-DIM, +DIMS) are also empirical outcomes, not consequences of how the losses are defined relative to the evaluation. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

This is an empirical reproducibility paper. It introduces no invented entities and no fitted parameters. All hyperparameters are inherited from the original work or set to defaults. The main assumptions are about fidelity of the reproduction and the validity of the benchmarks.

free parameters (3)
  • alpha, beta = 1 (default)
    Scaling weights in Eq. 11 combining layer-specific and dimension-specific losses; set to 1 as in the original AnglE implementation, not tuned by the authors.
  • target dimension ki = 128
    The sub-dimension used for V2 training and for defining the dimension losses, same as in the original study; all other dimensions are not directly trained in the base 2DMSE-V2.
  • lambda (KL scaling) = not reported
    Scaling factor for KL divergence in the V1 loss (Eq. 7); the paper does not give its value, so V1 reproduction depends on an unstated choice.
assumptions (3)
  • domain assumption The reproduced implementations of 2DMSE-V1 and V2 match the methods described in the two versions of the original paper.
    The entire comparison stands or falls on this; the paper notes discrepancies between the two implementations but cannot verify the original code beyond what is available.
  • domain assumption The evaluation benchmarks (SemEval STS, MS MARCO, BEIR) are valid measures of embedding and retrieval quality.
    Standard benchmarks in the field, but no evidence is given that they capture all relevant aspects of the tasks.
  • standard math PCA and KL divergence are used as in the original work to align sub-layer embeddings.
    Assumed mathematical tools, not re-derived in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 2D Matryoshka Training for Information Retrieval." pith.science (2026). https://pith.science/paper/VZL7VZUF

@misc{pith2026241117299,
  author       = {Pith},
  title        = {Pith review of: 2D Matryoshka Training for Information Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VZL7VZUF}},
  note         = {Machine review of arXiv:2411.17299}
}
read the original abstract

2D Matryoshka Training is an advanced embedding representation training approach designed to train an encoder model simultaneously across various layer-dimension setups. This method has demonstrated higher effectiveness in Semantic Text Similarity (STS) tasks over traditional training approaches when using sub-layers for embeddings. Despite its success, discrepancies exist between two published implementations, leading to varied comparative results with baseline models. In this reproducibility study, we implement and evaluate both versions of 2D Matryoshka Training on STS tasks and extend our analysis to retrieval tasks. Our findings indicate that while both versions achieve higher effectiveness than traditional Matryoshka training on sub-dimensions, and traditional full-sized model training approaches, they do not outperform models trained separately on specific sub-layer and sub-dimension setups. Moreover, these results generalize well to retrieval tasks, both in supervised (MSMARCO) and zero-shot (BEIR) settings. Further explorations of different loss computations reveals more suitable implementations for retrieval tasks, such as incorporating full-dimension loss and training on a broader range of target dimensions. Conversely, some intuitive approaches, such as fixing document encoders to full model outputs, do not yield improvements. Our reproduction code is available at https://github.com/ielab/2DMSE-Reproduce.

Figures

Figures reproduced from arXiv: 2411.17299 by the authors.

Figure 1
Figure 1. Comparison of model effectiveness across different layer and di [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Comparison of Model Effectiveness in different layer and dimension [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Comparison of Model Effectiveness in different layer and dimension [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of model effectiveness with different loss computations, [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Comparison of model effectiveness when a different document en [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 9 canonical work pages

  1. [1]

    17 mantic textual similarity, monolingual and cross-lingual evalua- tion

    Agirre, E., Banea, C., Cer, D., Diab, M., Gonzalez-Agirre, A., Mihalcea, R., Rigau, G., Wiebe, J.: SemEval-2016 task 1: Se- 9This figure does not account for initial trials that resulted in errors. 17 mantic textual similarity, monolingual and cross-lingual evalua- tion. In: Bethard, S., Carpuat, M., Cer, D., Jurgens, D., Nakov, P., Zesch, T. (eds.) Proce...

  2. [2]

    Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Ma- jumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenberg, M., Song, X., Stoica, A., Tiwary, S., Wang, T.: MS MARCO: A hu- man generated machine reading comprehension dataset (2018), https: //arxiv.org/abs/1611.09268

  3. [3]

    In: M` arquez, L., Callison-Burch, C., Su, J

    Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large an- notated corpus for learning natural language inference. In: M` arquez, L., Callison-Burch, C., Su, J. (eds.) Proceedings of the 2015 Conference on Empirical Methods in Natural Language Process- ing. pp. 632–642. Association for Computational Linguistics, Lisbon, Portugal (Sep 2015). https://d...

  4. [4]

    In: Sadat Moosavi, N., Gurevych, I., Hou, Y., Kim, G., Kim, Y.J., Schuster, T., Agrawal, A

    Campos, D., Magnani, A., Zhai, C.: Quick dense retrievers con- sume KALE: Post training KullbackLeibler alignment of embeddings for asymmetrical dual encoders. In: Sadat Moosavi, N., Gurevych, I., Hou, Y., Kim, G., Kim, Y.J., Schuster, T., Agrawal, A. (eds.) Proceedings of The Fourth Workshop on Simple and Effi- cient Natural Language Processing (SustaiNL...

  5. [5]

    In: International Con- ference on Learning Representations

    Carlsson, F., Gyllensten, A.C., Gogoulou, E., Hellqvist, E.Y., Sahlgren, M.: Semantic re-tuning with contrastive tension. In: International Con- ference on Learning Representations

  6. [6]

    In: Bethard, S., Carpuat, M., Apidianaki, 18 M., Mohammad, S.M., Cer, D., Jurgens, D

    Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., Specia, L.: SemEval- 2017 task 1: Semantic textual similarity multilingual and crosslin- gual focused evaluation. In: Bethard, S., Carpuat, M., Apidianaki, 18 M., Mohammad, S.M., Cer, D., Jurgens, D. (eds.) Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval- 2017). pp. 1–14. Asso...

  7. [7]

    Devvrit, F., Kudugunta, S., Kusupati, A., Dettmers, T., Chen, K., Dhillon, I., Tsvetkov, Y., Hajishirzi, H., Kakade, S., Farhadi, A., Jain, P.: Matformer: Nested transformer for elastic inference. In: Work- shop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (W ANT@NeurIPS 2023) (2023), https://openr...

  8. [8]

    In: Moens, M.F., Huang, X., Specia, L., Yih, S.W.t

    Gao, L., Callan, J.: Condenser: a pre-training architecture for dense retrieval. In: Moens, M.F., Huang, X., Specia, L., Yih, S.W.t. (eds.) Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 981–993. Association for Compu- tational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/1...

Show all 23 references
  1. [9]

    In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

    Gao, L., Ma, X., Lin, J., Callan, J.: Tevatron: An effi- cient and flexible toolkit for neural retrieval. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 3120–3124. SIGIR ’23, Association for Computing Ma...

  2. [10]

    In: Webber, B., Cohn, T., He, Y., Liu, Y

    Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., Yih, W.t.: Dense passage retrieval for open-domain question answer- ing. In: Webber, B., Cohn, T., He, Y., Liu, Y. (eds.) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Proc...

  3. [11]

    In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A

    Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanu- jan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., Farhadi, 19 A.: Matryoshka representation learning. In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neu...

  4. [12]

    In: Ku, L.W., Martins, A., Srikumar, V

    Li, X., Li, J.: AoE: Angle-optimized embeddings for semantic tex- tual similarity. In: Ku, L.W., Martins, A., Srikumar, V. (eds.) Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1825–

  5. [13]

    Li, X., Li, Z., Li, J., Xie, H., Li, Q.: ESE: Espresso sentence embeddings (2024), https://arxiv.org/abs/2402.14776

  6. [14]

    In: Proceedings of the 47th International ACM SIGIR Conference on Research and De- velopment in Information Retrieval

    Ma, X., Wang, L., Yang, N., Wei, F., Lin, J.: Fine-tuning llama for multi-stage text retrieval. In: Proceedings of the 47th International ACM SIGIR Conference on Research and De- velopment in Information Retrieval. pp. 2421–2425. SIGIR ’24, Association for Computing Machinery,...

  7. [15]

    In: Calzolari, N., Choukri, K., Declerck, T., Loftsson, H., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., Piperidis, S

    Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., Zam- parelli, R.: A SICK cure for the evaluation of compositional distri- butional semantic models. In: Calzolari, N., Choukri, K., Declerck, T., Loftsson, H., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., ...

  8. [16]

    In: Inui, K., Jiang, J., Ng, V., Wan, X

    Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings us- ing Siamese BERT-networks. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint 20 Conference o...

  9. [17]

    In: Vanschoren, J., Ye- ung, S

    Thakur, N., Reimers, N., R¨ uckl´ e, A., Srivastava, A., Gurevych, I.: BEIR: A heterogeneous benchmark for zero-shot evalua- tion of information retrieval models. In: Vanschoren, J., Ye- ung, S. (eds.) Proceedings of the Neural Information Process- ing Systems Track on Dataset...

  10. [18]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Wang, F., Liu, H.: Understanding the behaviour of contrastive loss. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2495–2504 (2021)

  11. [19]

    In: Proceedings of the 46th Interna- tional ACM SIGIR Conference on Research and Development in Infor- mation Retrieval

    Wang, S., Zuccon, G.: Balanced topic aware sampling for effective dense retriever: A reproducibility study. In: Proceedings of the 46th Interna- tional ACM SIGIR Conference on Research and Development in Infor- mation Retrieval. pp. 2542–2551 (2023)

  12. [20]

    In: Walker, M., Ji, H., Stent, A

    Williams, A., Nangia, N., Bowman, S.: A broad-coverage challenge corpus for sentence understanding through inference. In: Walker, M., Ji, H., Stent, A. (eds.) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: H...

  13. [21]

    Zhuang, S., Ma, X., Koopman, B., Lin, J., Zuccon, G.: PromptReps: Prompting large language models to generate dense and sparse represen- tations for zero-shot document retrieval (2024), https://arxiv.org/abs/ 2404.18424 21

  14. [511]

    https://doi.org/10.18653/v1/S16-1081, https:// aclanthology.org/S16-1081

    Association for Computational Linguistics, San Diego, Cal- ifornia (Jun 2016). https://doi.org/10.18653/v1/S16-1081, https:// aclanthology.org/S16-1081

  15. [1839]

    https://doi.org/10.18653/v1/2024.acl-long.101, https:// aclanthology.org/2024.acl-long.101

    Association for Computational Linguistics, Bangkok, Thailand (Aug 2024). https://doi.org/10.18653/v1/2024.acl-long.101, https:// aclanthology.org/2024.acl-long.101

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.