{"id":"9399b39f-d2ed-4fe3-beff-181687d02804","arxiv_id":"2411.17299","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"2D Matryoshka-trained embeddings beat standard Matryoshka and full-model baselines at small sizes, but lose to models trained separately for each specific layer and dimension, in both STS and retrieval.","lead":"One model trained with 2D Matryoshka training can produce many sizes of text embeddings from different layers and dimensions. This study shows it beats standard baselines at small sizes, but a dedicated model trained for one size is still better, in both similarity and search tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reproduction fidelity of 2DMSE-V2 is the load-bearing risk: using AnglE defaults when original V2 hyperparameters are unknown may drive the 'does not beat BERT-S' conclusion.","rationale":"The reader's conditional verdict is appropriate. The V2 reproduction fidelity is indeed the weakest load-bearing assumption because the abstract's strongest claim is comparative: 2DMSE-V2 does not beat separately trained models. The paper is honest about the unknown hyperparameters, but that honesty does not remove the risk; it localizes it. A sensitivity analysis would settle whether the ordering is stable. I did not find a more fundamental flaw: the code release, clear experimental setup, and honest limitations support the paper's claims at face value. No change to the reader's conditional verdict is needed.","tokens_in":11167,"tokens_out":4818,"duration_ms":44612,"concrete_test":"Run a hyperparameter sensitivity grid for 2DMSE-V2 on the STS setup: vary learning rate in {1e-5, 2e-5, 3e-5}, epochs in {3, 5, 10}, and batch size in {32, 64, 128} around the AnglE defaults; for each configuration, train 2DMSE-V2 and the matched BERT-S baseline, then compare Spearman correlation at the 48 layer-dimension points. If any plausible configuration yields 2DMSE-V2 at least as effective as BERT-S on a majority of points, the 'do not outperform' claim is an artifact of the chosen defaults. If no configuration closes the gap, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states: 'We use BERT training parameters as the original 2DMSE-V2 configurations have not been reported' (footnote 5, referring to AnglE issue #80). The paper's headline claim that 2DMSE-V2 does not outperform separately trained BERT-S therefore rests on the assumption that AnglE's example BERT fine-tuning schedule is representative of the original V2 recipe. If the original V2 used a different learning rate, epoch count, batch size, warmup, or loss weighting, the observed gap in Figures 1(b) and 2 could shrink or reverse. This risk is heightened by the paper's own statement in Section 4.1 that it aims to confirm trends rather than reproduce absolute metrics, and by the absence of repeated-seed variance: several V2-versus-BERT-S differences are small (e.g., last layer at dimensions 64-256), so a single seed cannot establish a robust ordering. The concern is external validity, not internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This reproducibility study re-implements the two published versions of 2D Matryoshka Sentence Embedding training (2DMSE-V1 and V2) on BERT-base and evaluates them on semantic textual similarity (STS) benchmarks, then extends the approach to passage retrieval on MS MARCO and zero-shot BEIR. The paper reports that both versions outperform traditional Matryoshka training (MSE) and full-model baselines on sub-layer/sub-dimension configurations, that 2DMSE-V2 generally outperforms V1, and that neither version consistently beats models separately trained for each target layer-dimension setup. For retrieval, the authors find that the original 2DMSE-V2 is less effective than separately trained models, and they explore loss modifications (scoring loss, full-dimension loss, more target dimensions) as well as a fixed full-size document encoder.","tokens_in":11460,"tokens_out":4689,"duration_ms":44875,"significance":"If the findings hold, the paper provides a useful reproducibility result: a single 2D Matryoshka model is a strong multi-size generalist, but dedicated small models remain the accuracy ceiling at a fixed size. The work also contributes the first retrieval extension of 2DMSE and an openly available reproduction codebase. The main strengths are the breadth of evaluation (STS, MS MARCO, 13 BEIR datasets, 48 layer-dimension configurations) and the authors' transparency about the unreported V2 hyperparameters. However, the central negative claim regarding V2 rests on an external hyperparameter assumption, the experiments are single-run with no variance estimates, and one headline statement in the abstract is contradicted by the paper's own V1 results. These issues currently limit the reliability of the contribution.","major_comments":[{"comment":"The V2-versus-BERT-S comparison, which underlies the central claim that 2DMSE does not beat separately trained models, depends on the AnglE example BERT fine-tuning configuration because the original 2DMSE-V2 hyperparameters were never reported. Several V2-versus-BERT-S gaps are small (e.g., layer 12 at dimensions 64-256 in Figure 1(b)), so a different learning rate, batch size, epoch count, warmup, or loss weighting could change the ordering. The paper should add a sensitivity analysis over training hyperparameters and/or repeated-seed results, or explicitly restrict the claim to the AnglE-default configuration.","section":"Section 3.1, footnote 5"},{"comment":"The abstract states that 'both versions ... do not outperform models trained separately on specific sub-layer and sub-dimension setups,' but Section 4.1 reports that 2DMSE-V1 surpasses BERT-S in the sub-layers (layers 2, 4, and 6) and at embedding size 8. This is a direct internal contradiction that should be corrected by qualifying the abstract to refer to V2 and to the tested configurations.","section":"Abstract; Section 4.1"},{"comment":"All comparisons are single-run and report no error bars, confidence intervals, or significance tests. The small observed differences, such as V2 versus BERT-S at the last layer and dimensions 64-256 in Figure 1(b), and the 'significant improvement' claimed for FULL-DIM in Section 4.3.2, cannot be established as robust orderings on the basis of one seed per configuration. At minimum, repeated seeds or bootstrap intervals should be provided for the key comparisons, and the claims should be softened otherwise.","section":"Figures 1(b), 2; Section 4.3"},{"comment":"The training recipe for BERT-S is not described in sufficient detail: the paper does not specify the number of steps, learning rate, batch size, or whether each separately trained model uses the same data, loss, and schedule as the 2DMSE models. Because BERT-S is the reference point for the 'do not outperform' conclusion, the paper should either report these details or confirm that they are fully specified in the repository and identical across all BERT-S configurations.","section":"Section 3; Figure 1 caption"}],"minor_comments":[{"comment":"The caption uses 'paird-doc' where 'paired-doc' is intended.","section":"Figure 5 caption"},{"comment":"The sentence 'performance abruptly deteriorateeas for dimensions lower than 128' contains a typo ('deteriorateeas' should be 'deteriorates').","section":"Section 4.2"},{"comment":"The contributions paragraph spells 'Matryoska' instead of 'Matryoshka'.","section":"Section 1"},{"comment":"The SCORE modification is described only as 'Initial findings suggest...' without quantitative support; either report the relevant numbers or specify exactly which configurations in Figure 4 support the statement.","section":"Section 4.3.1"},{"comment":"The backbone model name 'bert-base-uncased' should be set in code font for consistency.","section":"Section 3.1"},{"comment":"Reference [13] is cited as the source of 2D Matryoshka training but is titled 'ESE: Espresso sentence embeddings'; please clarify the relationship between the ESE paper and the 2D Matryoshka method, or cite the exact title if different.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The authors are transparent about the missing V2 hyperparameters in footnote 5, so the main issue is external validity rather than misreporting. The abstract overstatement and the absence of variance estimates are fixable with a robustness study. The paper is within scope as a reproducibility study for IR and the code release is a concrete asset. I would not reject, but the headline negative result should not be published without sensitivity analysis or a clearly scoped claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is a genuinely useful reproducibility study: it reproduces both published versions of 2D Matryoshka training, reconciles the two implementations, and extends the method to passage retrieval on MSMARCO and BEIR. Second, the headline ordering - one 2DMSE model is a good generalist, but separately trained per-size models remain the ceiling for a fixed budget - is plausible but not proven for V2. The original V2 hyperparameters were never reported, and the authors fell back to AnglE library defaults (Section 3.1, footnote 5). That matters because some V2-versus-BERT-S differences are small, especially at the last layer and dimensions 64-256.\n\nCredit where it's due. The paper does several things well. The loss-modification experiments (FULL-DIM, +DIMS, fix-doc) are a real contribution: FULL-DIM recovers high-dimension effectiveness, +DIMS smooths the low-dimension cliff at some cost to high dimensions, and fix-doc does not help. Those are actionable findings for anyone building elastic embedding models. The code is released, the limitations are stated plainly, and the authors explicitly say they aim to confirm trends rather than reproduce absolute metrics. That transparency is exactly what a reproducibility study should look like.\n\nSoft spots, in proportion. The V2 reproduction fidelity is the load-bearing risk. If the original V2 used a different learning rate, epoch count, or loss weighting, the observed gap could shrink or reverse. The paper's own goal of trend confirmation mitigates this, but it still means the V2-vs-BERT-S ordering is not on solid ground without repeated seeds. Relatedly, there are no error bars, no significance testing, only one backbone (bert-base-uncased), and six of twelve layers evaluated. These are standard limitations in this line of work, not fatal ones; the larger gaps (low layers, low dimensions) are likely robust.\n\nThere is also an internal inconsistency worth flagging: the abstract says both versions \"do not outperform models trained separately,\" but Section 4.1 reports that 2DMSE-V1 surpasses BERT-S on layers 2, 4, and 6 at dimension 8. The abstract overstates the finding and should be tightened.\n\nWho should read this: people building efficient embedding models, especially those considering Matryoshka-style training for retrieval, and anyone working on reproducibility of contrastive learning papers. It deserves a serious referee. I would send it out with a request for repeated runs or a narrowed claim on the V2 comparison.","headline":"Useful reproducibility study with a clear engineering takeaway, but the claim that 2DMSE-V2 underperforms separately trained BERT-S rests on AnglE defaults and no reported variance, so the abstract overstates the point.","tokens_in":781,"tokens_out":1056,"would_cite":true,"duration_ms":33965,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single encoder trained with 2D Matryoshka embeddings outperforms full-size fine-tuning on sub-layer, sub-dimension outputs, but not separately trained small models—and the same holds in retrieval.","keywords":["2D Matryoshka training","Matryoshka representation learning","dense retrieval","sentence embeddings","semantic text similarity","reproducibility study","MS MARCO","BEIR"],"falsifier":"Release the original V2 training configuration; if a faithfully retrained V2 matches or beats BERT-S across the layer-dimension grid, the paper's claim that dedicated small models remain the accuracy ceiling is overturned.","tokens_in":10968,"feed_emoji":"🪆","tokens_out":5004,"duration_ms":43795,"temperature":0.7,"pith_summary":"This paper tests whether 2D Matryoshka training—a method that trains one encoder to produce usable embeddings from many layer-depth and dimension-size combinations at once—keeps its promised edge when scrutinized and when moved from sentence similarity to search. Reproducing both published versions of the method, the authors find that each beats standard Matryoshka training and full-model fine-tuning on sub-layer, sub-dimension embeddings, and that the advantage carries over to passage retrieval on MS MARCO and zero-shot retrieval on BEIR. The catch: neither version beats models trained separately for one specific layer-and-dimension size, so a single 2D model is a strong generalist but not the accuracy ceiling. The paper then shows retrieval-specific tweaks—training full-dimension outputs at every layer and training a wider set of target dimensions—recover some of the lost ground at small sizes, while naively fixing document encoders to the full model does not help.","feed_headline":"2D Matryoshka models beat full BERT, lose to dedicated small ones","feed_subtitle":"Reproducing both published versions shows the multi-size trick works for retrieval, and two loss tweaks narrow the gap.","key_machinery":"The object is the 2D Matryoshka training objective, which nests two truncation axes: layer depth and embedding dimension. Version 1 draws a random sub-layer, trains last-layer and sub-layer embeddings, and aligns their distributions with KL divergence; Version 2 applies a logarithmically weighted loss to every layer's sub-dimension embeddings and adds a PCA-driven mean-squared-error plus KL alignment to a target dimension of 128. The mechanism is that matryoshka-style truncated outputs are trained as first-class citizens at every depth, so early-layer small embeddings inherit usable semantics, letting one encoder serve many size budgets. The paper's key analytical move is comparing this single multi-size model against BERT-S, a separately trained model per size—the comparison that reveals the accuracy cost of multi-size flexibility.","core_discovery":"The central discovery is that the two published implementations of 2D Matryoshka training reproduce the original qualitative trend: a single model trained across layers and dimensions dominates both raw BERT fine-tuning and last-layer-only Matryoshka training when embeddings are taken from early layers and small dimensions. Version 2 is the better design, nearly matching separately trained small models and beating them in some STS sub-layer cases, but the original claim that 2DMSE surpasses dedicated small models does not survive the broader comparison: across almost all layer-dimension setups, BERT-S, a model fine-tuned separately for that exact size, remains more effective. The same ordering transfers to retrieval: on MS MARCO and BEIR, 2DMSE beats MSE and BERT except at the last layer, yet loses to BERT-S everywhere. The paper also establishes that two loss modifications—adding full-dimension loss from each layer and training more target dimensions—improve retrieval results, while a fixed full-size document encoder does not.","pith_inferences":["Beyond the paper: the 48-cell layer-dimension grid implies a measurable size-accuracy frontier, and a practitioner could use those measured gaps to decide when one elastic model suffices versus when per-size training is worth the extra compute.","Beyond the paper: if the abrupt retrieval collapse below dimension 128 stems from training only at that target size, then serving small vectors safely requires explicit training at those dimensions—a hypothesis directly testable by evaluating the +DIMS checkpoints on BEIR.","Beyond the paper: the fact that the PCA-based dimension loss helps STS but trails dedicated small models in retrieval hints that the PCA projection may discard retrieval-specific signal; a variant that aligns truncated embeddings with a contrastive rather than reconstruction objective could close the gap."],"forward_implications":["A single 2D Matryoshka model can replace a family of separately trained encoders with a modest accuracy loss, making it cheaper to serve multiple embedding-size budgets from one checkpoint.","The qualitative results transfer from sentence similarity to both supervised and zero-shot retrieval, so the multi-size tradeoff is not an artifact of STS evaluation.","Training full-dimension sub-layer outputs and covering more target dimensions shrinks the low-dimension gap, indicating that loss design, not architecture, drives much of the remaining deficit.","Fixing the document encoder to the full-model output does not consistently pay for its extra computation, so paired small encoders remain the sensible efficiency point for asymmetric retrieval."],"supporting_citations":[{"why":"introduces 2D Matryoshka training in two versions and supplies the central claims being reproduced","marker":"[13]"},{"why":"defines Matryoshka representation learning, the MSE baseline the paper compares against","marker":"[11]"},{"why":"provides the second version's training procedure and the default BERT hyperparameters adopted because the original configurations were unreported","marker":"[12]"},{"why":"is the toolkit used to fine-tune and evaluate the retrieval models on MS MARCO","marker":"[9]"},{"why":"supplies the MS MARCO passage ranking collection used for supervised retrieval training and evaluation","marker":"[2]"},{"why":"supplies the BEIR benchmark used for zero-shot retrieval evaluation","marker":"[17]"},{"why":"supplies the SemEval STS benchmarks used to measure sentence similarity effectiveness","marker":"[1]"},{"why":"adds the STS Benchmark to the sentence similarity evaluation suite","marker":"[6]"},{"why":"with the SNLI corpus, provides the All-NLI training data used to reproduce the original STS training setup","marker":"[20]"}],"fun_headline_variants":["2D Matryoshka training outperforms BERT on retrieval, but not BERT-S","Reproducing 2D Matryoshka: version 2 shines, dedicated small models still win","Loss tweaks make 2D Matryoshka more retrieval-friendly","2D Matryoshka's multi-size edge vanishes at full layer and dimension","2D Matryoshka: both versions beat MSE, lose to size-specific training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that 2DMSE-V2 loses to separately trained small models assumes the reproduction's training settings match the original V2, whose hyperparameters were never reported; the authors substitute default BERT fine-tuning parameters from a third-party library.","fun_headline_variants_meta":{"raw":{"variants":["2D Matryoshka training outperforms BERT on retrieval, but not BERT-S","Reproducing 2D Matryoshka: version 2 shines, dedicated small models still win","Loss tweaks make 2D Matryoshka more retrieval-friendly","2D Matryoshka's multi-size edge vanishes at full layer and dimension","2D Matryoshka: both versions beat MSE, lose to size-specific training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000651,"raw_usage":{"total_tokens":3000,"prompt_tokens":977,"completion_tokens":2023,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":1910}},"tokens_in":593,"tokens_out":2023,"duration_ms":14704,"temperature":1.0,"reasoning_tokens":1910,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:14:57.696215+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Release the original V2 training configuration; if a faithfully retrained V2 matches or beats BERT-S across the layer-dimension grid, the paper's claim that dedicated small models remain the accuracy ceiling is overturned.","supporting_citations":[{"cited_title":"In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A","cited_arxiv_id":null,"evidence_quote":"defines Matryoshka representation learning, the MSE baseline the paper compares against"},{"cited_title":"In: Ku, L.W., Martins, A., Srikumar, V","cited_arxiv_id":null,"evidence_quote":"provides the second version's training procedure and the default BERT hyperparameters adopted because the original configurations were unreported"},{"cited_title":"In: Vanschoren, J., Ye- ung, S","cited_arxiv_id":null,"evidence_quote":"supplies the BEIR benchmark used for zero-shot retrieval evaluation"},{"cited_title":"17 mantic textual similarity, monolingual and cross-lingual evalua- tion","cited_arxiv_id":null,"evidence_quote":"supplies the SemEval STS benchmarks used to measure sentence similarity effectiveness"}],"review_version":1}