REVIEW 4 major objections 6 minor 23 references
2D Matryoshka Training for Information Retrieval
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A single encoder trained with 2D Matryoshka embeddings outperforms full-size fine-tuning on sub-layer, sub-dimension outputs, but not separately trained small models—and the same holds in retrieval.
desk verdict Useful reproducibility study with a clear engineering takeaway, but the claim that 2DMSE-V2 underperforms separately trained BERT-S rests on AnglE defaults and no reported variance, so the abstract overstates the point. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The object is the 2D Matryoshka training objective, which nests two truncation axes: layer depth and embedding dimension. Version 1 draws a random sub-layer, trains last-layer and sub-layer embeddings, and aligns their distributions with KL divergence; Version 2 applies a logarithmically weighted loss to every layer's sub-dimension embeddings and adds a PCA-driven mean-squared-error plus KL alignment to a target dimension of 128. The mechanism is that matryoshka-style truncated outputs are trained as first-class citizens at every depth, so early-layer small embeddings inherit usable semantics, letting one encoder serve many size budgets. The paper's key analytical move is comparing this single multi-size model against BERT-S, a separately trained model per size—the comparison that reveals the accuracy cost of multi-size flexibility.
What would settle it
Release the original V2 training configuration; if a faithfully retrained V2 matches or beats BERT-S across the layer-dimension grid, the paper's claim that dedicated small models remain the accuracy ceiling is overturned.
Extended reading notes
Core claim
The central discovery is that the two published implementations of 2D Matryoshka training reproduce the original qualitative trend: a single model trained across layers and dimensions dominates both raw BERT fine-tuning and last-layer-only Matryoshka training when embeddings are taken from early layers and small dimensions. Version 2 is the better design, nearly matching separately trained small models and beating them in some STS sub-layer cases, but the original claim that 2DMSE surpasses dedicated small models does not survive the broader comparison: across almost all layer-dimension setups, BERT-S, a model fine-tuned separately for that exact size, remains more effective. The same ordering transfers to retrieval: on MS MARCO and BEIR, 2DMSE beats MSE and BERT except at the last layer, yet loses to BERT-S everywhere. The paper also establishes that two loss modifications—adding full-dimension loss from each layer and training more target dimensions—improve retrieval results, while a fixed full-size document encoder does not.
Load-bearing premise
The conclusion that 2DMSE-V2 loses to separately trained small models assumes the reproduction's training settings match the original V2, whose hyperparameters were never reported; the authors substitute default BERT fine-tuning parameters from a third-party library.
Editorial extensions
If this is right
- A single 2D Matryoshka model can replace a family of separately trained encoders with a modest accuracy loss, making it cheaper to serve multiple embedding-size budgets from one checkpoint.
- The qualitative results transfer from sentence similarity to both supervised and zero-shot retrieval, so the multi-size tradeoff is not an artifact of STS evaluation.
- Training full-dimension sub-layer outputs and covering more target dimensions shrinks the low-dimension gap, indicating that loss design, not architecture, drives much of the remaining deficit.
- Fixing the document encoder to the full-model output does not consistently pay for its extra computation, so paired small encoders remain the sensible efficiency point for asymmetric retrieval.
Reading between the lines
- Beyond the paper: the 48-cell layer-dimension grid implies a measurable size-accuracy frontier, and a practitioner could use those measured gaps to decide when one elastic model suffices versus when per-size training is worth the extra compute.
- Beyond the paper: if the abrupt retrieval collapse below dimension 128 stems from training only at that target size, then serving small vectors safely requires explicit training at those dimensions—a hypothesis directly testable by evaluating the +DIMS checkpoints on BEIR.
- Beyond the paper: the fact that the PCA-based dimension loss helps STS but trails dedicated small models in retrieval hints that the PCA projection may discard retrieval-specific signal; a variant that aligns truncated embeddings with a contrastive rather than reconstruction objective could close the gap.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This reproducibility study re-implements the two published versions of 2D Matryoshka Sentence Embedding training (2DMSE-V1 and V2) on BERT-base and evaluates them on semantic textual similarity (STS) benchmarks, then extends the approach to passage retrieval on MS MARCO and zero-shot BEIR. The paper reports that both versions outperform traditional Matryoshka training (MSE) and full-model baselines on sub-layer/sub-dimension configurations, that 2DMSE-V2 generally outperforms V1, and that neither version consistently beats models separately trained for each target layer-dimension setup. For retrieval, the authors find that the original 2DMSE-V2 is less effective than separately trained models, and they explore loss modifications (scoring loss, full-dimension loss, more target dimensions) as well as a fixed full-size document encoder.
Significance. If the findings hold, the paper provides a useful reproducibility result: a single 2D Matryoshka model is a strong multi-size generalist, but dedicated small models remain the accuracy ceiling at a fixed size. The work also contributes the first retrieval extension of 2DMSE and an openly available reproduction codebase. The main strengths are the breadth of evaluation (STS, MS MARCO, 13 BEIR datasets, 48 layer-dimension configurations) and the authors' transparency about the unreported V2 hyperparameters. However, the central negative claim regarding V2 rests on an external hyperparameter assumption, the experiments are single-run with no variance estimates, and one headline statement in the abstract is contradicted by the paper's own V1 results. These issues currently limit the reliability of the contribution.
major comments (4)
- [Section 3.1, footnote 5] The V2-versus-BERT-S comparison, which underlies the central claim that 2DMSE does not beat separately trained models, depends on the AnglE example BERT fine-tuning configuration because the original 2DMSE-V2 hyperparameters were never reported. Several V2-versus-BERT-S gaps are small (e.g., layer 12 at dimensions 64-256 in Figure 1(b)), so a different learning rate, batch size, epoch count, warmup, or loss weighting could change the ordering. The paper should add a sensitivity analysis over training hyperparameters and/or repeated-seed results, or explicitly restrict the claim to the AnglE-default configuration.
- [Abstract; Section 4.1] The abstract states that 'both versions ... do not outperform models trained separately on specific sub-layer and sub-dimension setups,' but Section 4.1 reports that 2DMSE-V1 surpasses BERT-S in the sub-layers (layers 2, 4, and 6) and at embedding size 8. This is a direct internal contradiction that should be corrected by qualifying the abstract to refer to V2 and to the tested configurations.
- [Figures 1(b), 2; Section 4.3] All comparisons are single-run and report no error bars, confidence intervals, or significance tests. The small observed differences, such as V2 versus BERT-S at the last layer and dimensions 64-256 in Figure 1(b), and the 'significant improvement' claimed for FULL-DIM in Section 4.3.2, cannot be established as robust orderings on the basis of one seed per configuration. At minimum, repeated seeds or bootstrap intervals should be provided for the key comparisons, and the claims should be softened otherwise.
- [Section 3; Figure 1 caption] The training recipe for BERT-S is not described in sufficient detail: the paper does not specify the number of steps, learning rate, batch size, or whether each separately trained model uses the same data, loss, and schedule as the 2DMSE models. Because BERT-S is the reference point for the 'do not outperform' conclusion, the paper should either report these details or confirm that they are fully specified in the repository and identical across all BERT-S configurations.
minor comments (6)
- [Figure 5 caption] The caption uses 'paird-doc' where 'paired-doc' is intended.
- [Section 4.2] The sentence 'performance abruptly deteriorateeas for dimensions lower than 128' contains a typo ('deteriorateeas' should be 'deteriorates').
- [Section 1] The contributions paragraph spells 'Matryoska' instead of 'Matryoshka'.
- [Section 4.3.1] The SCORE modification is described only as 'Initial findings suggest...' without quantitative support; either report the relevant numbers or specify exactly which configurations in Figure 4 support the statement.
- [Section 3.1] The backbone model name 'bert-base-uncased' should be set in code font for consistency.
- [References] Reference [13] is cited as the source of 2D Matryoshka training but is titled 'ESE: Espresso sentence embeddings'; please clarify the relationship between the ESE paper and the 2D Matryoshka method, or cite the exact title if different.
Circularity Check
No circularity: the paper is an empirical reproducibility study with no fitted-input predictions or self-citation load-bearing arguments.
full rationale
This paper performs no derivations that reduce to their inputs, and it fits no parameters to the results it reports. The central claims are empirical comparisons of 2DMSE-V1, 2DMSE-V2, Matryoshka (MSE), full-model BERT, and separately trained BERT-S models on held-out STS and retrieval benchmarks (MSMARCO dev, BEIR), so every effectiveness number is measured rather than constructed from the method definition. The only potential concern raised by the paper is reproduction fidelity: 'We use BERT training parameters as the original 2DMSE-V2 configurations have not been reported' (Section 3.1, footnote 5), and the paper explicitly states its goal is to 'confirm the observed trends' rather than reproduce absolute metrics (Section 4.1). That limitation concerns external validity of the reproduction, not circular reasoning, because the training configuration is inherited from an external library (AnglE) rather than chosen to force the paper's conclusions. The paper cites two prior works by its own authors ([19] and [21]), but only in the related-work section for background context on dense retrievers; these citations are not load-bearing for any of the paper's experimental findings or design choices. No 'uniqueness theorem' or prior author-derived constraint is invoked to forbid alternatives, and no known empirical pattern is renamed as an organizing principle. The claimed improvements from modified losses (FULL-DIM, +DIMS) are also empirical outcomes, not consequences of how the losses are defined relative to the evaluation. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- alpha, beta =
1 (default)
- target dimension ki =
128
- lambda (KL scaling) =
not reported
assumptions (3)
- domain assumption The reproduced implementations of 2DMSE-V1 and V2 match the methods described in the two versions of the original paper.
- domain assumption The evaluation benchmarks (SemEval STS, MS MARCO, BEIR) are valid measures of embedding and retrieval quality.
- standard math PCA and KL divergence are used as in the original work to align sub-layer embeddings.
Cite this review
Pith. "Pith review of 2D Matryoshka Training for Information Retrieval." pith.science (2026). https://pith.science/paper/VZL7VZUF
@misc{pith2026241117299,
author = {Pith},
title = {Pith review of: 2D Matryoshka Training for Information Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/VZL7VZUF}},
note = {Machine review of arXiv:2411.17299}
}
read the original abstract
2D Matryoshka Training is an advanced embedding representation training approach designed to train an encoder model simultaneously across various layer-dimension setups. This method has demonstrated higher effectiveness in Semantic Text Similarity (STS) tasks over traditional training approaches when using sub-layers for embeddings. Despite its success, discrepancies exist between two published implementations, leading to varied comparative results with baseline models. In this reproducibility study, we implement and evaluate both versions of 2D Matryoshka Training on STS tasks and extend our analysis to retrieval tasks. Our findings indicate that while both versions achieve higher effectiveness than traditional Matryoshka training on sub-dimensions, and traditional full-sized model training approaches, they do not outperform models trained separately on specific sub-layer and sub-dimension setups. Moreover, these results generalize well to retrieval tasks, both in supervised (MSMARCO) and zero-shot (BEIR) settings. Further explorations of different loss computations reveals more suitable implementations for retrieval tasks, such as incorporating full-dimension loss and training on a broader range of target dimensions. Conversely, some intuitive approaches, such as fixing document encoders to full model outputs, do not yield improvements. Our reproduction code is available at https://github.com/ielab/2DMSE-Reproduce.
Figures
Reference graph
Works this paper leans on
-
[1]
17 mantic textual similarity, monolingual and cross-lingual evalua- tion
Agirre, E., Banea, C., Cer, D., Diab, M., Gonzalez-Agirre, A., Mihalcea, R., Rigau, G., Wiebe, J.: SemEval-2016 task 1: Se- 9This figure does not account for initial trials that resulted in errors. 17 mantic textual similarity, monolingual and cross-lingual evalua- tion. In: Bethard, S., Carpuat, M., Cer, D., Jurgens, D., Nakov, P., Zesch, T. (eds.) Proce...
work page 2016
-
[2]
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Ma- jumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenberg, M., Song, X., Stoica, A., Tiwary, S., Wang, T.: MS MARCO: A hu- man generated machine reading comprehension dataset (2018), https: //arxiv.org/abs/1611.09268
arXiv 2018
-
[3]
In: M` arquez, L., Callison-Burch, C., Su, J
Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large an- notated corpus for learning natural language inference. In: M` arquez, L., Callison-Burch, C., Su, J. (eds.) Proceedings of the 2015 Conference on Empirical Methods in Natural Language Process- ing. pp. 632–642. Association for Computational Linguistics, Lisbon, Portugal (Sep 2015). https://d...
-
[4]
In: Sadat Moosavi, N., Gurevych, I., Hou, Y., Kim, G., Kim, Y.J., Schuster, T., Agrawal, A
Campos, D., Magnani, A., Zhai, C.: Quick dense retrievers con- sume KALE: Post training KullbackLeibler alignment of embeddings for asymmetrical dual encoders. In: Sadat Moosavi, N., Gurevych, I., Hou, Y., Kim, G., Kim, Y.J., Schuster, T., Agrawal, A. (eds.) Proceedings of The Fourth Workshop on Simple and Effi- cient Natural Language Processing (SustaiNL...
-
[5]
In: International Con- ference on Learning Representations
Carlsson, F., Gyllensten, A.C., Gogoulou, E., Hellqvist, E.Y., Sahlgren, M.: Semantic re-tuning with contrastive tension. In: International Con- ference on Learning Representations
-
[6]
In: Bethard, S., Carpuat, M., Apidianaki, 18 M., Mohammad, S.M., Cer, D., Jurgens, D
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., Specia, L.: SemEval- 2017 task 1: Semantic textual similarity multilingual and crosslin- gual focused evaluation. In: Bethard, S., Carpuat, M., Apidianaki, 18 M., Mohammad, S.M., Cer, D., Jurgens, D. (eds.) Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval- 2017). pp. 1–14. Asso...
-
[7]
Devvrit, F., Kudugunta, S., Kusupati, A., Dettmers, T., Chen, K., Dhillon, I., Tsvetkov, Y., Hajishirzi, H., Kakade, S., Farhadi, A., Jain, P.: Matformer: Nested transformer for elastic inference. In: Work- shop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (W ANT@NeurIPS 2023) (2023), https://openr...
work page 2023
-
[8]
In: Moens, M.F., Huang, X., Specia, L., Yih, S.W.t
Gao, L., Callan, J.: Condenser: a pre-training architecture for dense retrieval. In: Moens, M.F., Huang, X., Specia, L., Yih, S.W.t. (eds.) Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 981–993. Association for Compu- tational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/1...
Show all 23 references
-
[9]
In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval
Gao, L., Ma, X., Lin, J., Callan, J.: Tevatron: An effi- cient and flexible toolkit for neural retrieval. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 3120–3124. SIGIR ’23, Association for Computing Ma...
2023
-
[10]
In: Webber, B., Cohn, T., He, Y., Liu, Y
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., Yih, W.t.: Dense passage retrieval for open-domain question answer- ing. In: Webber, B., Cohn, T., He, Y., Liu, Y. (eds.) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Proc...
2020 doi
-
[11]
In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A
Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanu- jan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., Farhadi, 19 A.: Matryoshka representation learning. In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neu...
2022
-
[12]
In: Ku, L.W., Martins, A., Srikumar, V
Li, X., Li, J.: AoE: Angle-optimized embeddings for semantic tex- tual similarity. In: Ku, L.W., Martins, A., Srikumar, V. (eds.) Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1825–
-
[13]
Li, X., Li, Z., Li, J., Xie, H., Li, Q.: ESE: Espresso sentence embeddings (2024), https://arxiv.org/abs/2402.14776
2024 arXiv
-
[14]
In: Proceedings of the 47th International ACM SIGIR Conference on Research and De- velopment in Information Retrieval
Ma, X., Wang, L., Yang, N., Wei, F., Lin, J.: Fine-tuning llama for multi-stage text retrieval. In: Proceedings of the 47th International ACM SIGIR Conference on Research and De- velopment in Information Retrieval. pp. 2421–2425. SIGIR ’24, Association for Computing Machinery,...
2024
-
[15]
In: Calzolari, N., Choukri, K., Declerck, T., Loftsson, H., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., Piperidis, S
Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., Zam- parelli, R.: A SICK cure for the evaluation of compositional distri- butional semantic models. In: Calzolari, N., Choukri, K., Declerck, T., Loftsson, H., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., ...
2014
-
[16]
In: Inui, K., Jiang, J., Ng, V., Wan, X
Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings us- ing Siamese BERT-networks. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint 20 Conference o...
2019 doi
-
[17]
In: Vanschoren, J., Ye- ung, S
Thakur, N., Reimers, N., R¨ uckl´ e, A., Srivastava, A., Gurevych, I.: BEIR: A heterogeneous benchmark for zero-shot evalua- tion of information retrieval models. In: Vanschoren, J., Ye- ung, S. (eds.) Proceedings of the Neural Information Process- ing Systems Track on Dataset...
2021
-
[18]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Wang, F., Liu, H.: Understanding the behaviour of contrastive loss. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2495–2504 (2021)
2021
-
[19]
In: Proceedings of the 46th Interna- tional ACM SIGIR Conference on Research and Development in Infor- mation Retrieval
Wang, S., Zuccon, G.: Balanced topic aware sampling for effective dense retriever: A reproducibility study. In: Proceedings of the 46th Interna- tional ACM SIGIR Conference on Research and Development in Infor- mation Retrieval. pp. 2542–2551 (2023)
2023
-
[20]
In: Walker, M., Ji, H., Stent, A
Williams, A., Nangia, N., Bowman, S.: A broad-coverage challenge corpus for sentence understanding through inference. In: Walker, M., Ji, H., Stent, A. (eds.) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: H...
2018 doi
-
[21]
Zhuang, S., Ma, X., Koopman, B., Lin, J., Zuccon, G.: PromptReps: Prompting large language models to generate dense and sparse represen- tations for zero-shot document retrieval (2024), https://arxiv.org/abs/ 2404.18424 21
2024 arXiv
-
[511]
https://doi.org/10.18653/v1/S16-1081, https:// aclanthology.org/S16-1081
Association for Computational Linguistics, San Diego, Cal- ifornia (Jun 2016). https://doi.org/10.18653/v1/S16-1081, https:// aclanthology.org/S16-1081
2016 doi
-
[1839]
https://doi.org/10.18653/v1/2024.acl-long.101, https:// aclanthology.org/2024.acl-long.101
Association for Computational Linguistics, Bangkok, Thailand (Aug 2024). https://doi.org/10.18653/v1/2024.acl-long.101, https:// aclanthology.org/2024.acl-long.101
2024 doi
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.