{"id":"dbc61d06-fea7-4744-ad71-dc1bdbf497b4","arxiv_id":"2506.19743","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A nested embedding training recipe improves e-commerce retrieval accuracy at small embedding sizes.","lead":"This paper applies Matryoshka Representation Learning, a known embedding technique, to e-commerce product search, claiming that smaller embedding dimensions can retrieve and rank products as accurately as full-size embeddings. The authors report up to 12x smaller embeddings with comparable or better accuracy on four test sets of challenging queries.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control for the two extra fine-tuning epochs means the improved 64-d metrics in Tables 2/4 may be caused by continued training, not by the nested objective; this breaks the paper's central attribution.","rationale":"The reader's weakest assumption identifies exactly the load-bearing gap: the paper attributes all improvements in Tables 2 and 4 to the nested embedding objective, but the NEAR2 models received two additional epochs of continued fine-tuning on the IGR training data while the comparison baselines did not. This is not a minor omission; it is the difference between testing NEAR2 and testing 'more fine-tuning.' The paper's own ablation in Section 6 does not include a same-epoch no-NEAR2 arm, and Table 5 shows that adding a different loss (OCL) can produce much larger gains (+32%) than the nesting effect claimed elsewhere, which strongly suggests the reported deltas are confounded by training signal and duration rather than by Matryoshka nesting. The central claim about 12x efficiency with improved accuracy therefore cannot be evaluated from the current evidence. I also note the inverted MNRL equation and the internally inconsistent memory-usage table as additional correctness problems; however, the missing control is the single most load-bearing issue because even a fully corrected write-up of the loss and memory numbers would leave the empirical attribution untested. The paper may describe a useful practical recipe, but as presented it does not establish the causal role of NEAR2, so rejection is appropriate pending a matched-duration control and a cleaned-up methodology section.","tokens_in":13401,"tokens_out":7510,"duration_ms":81283,"concrete_test":"Run a matched-duration control for each model in Tables 2/4: continue fine-tuning eBERT-siam, eBERT-UCO, and eBERT-siam-UCO for the same 2 epochs on the same IGR query-title relevance pairs with the same batch size (32), AdamW, learning rate (5e-05), margin (0.75), and the same MNRL/OCL loss, but with the MRL nested-loss term removed. Evaluate the resulting 768-dimensional model on CQ, CQ-balanced, CQ-common-str, and CQ-alphanum, and compare its deltas against the same Saadany et al. baselines. If the no-MRL control reproduces or exceeds the Table 2 deltas at 768 dimensions, and if NEAR2@64 does not match the control's 768-dimensional metrics, then the reported accuracy gains and the '12x with no loss' claim cannot be attributed to NEAR2; if NEAR2@64 matches or beats the control, the attribution survives and should be reported with confidence intervals.","verdict_should_be":"REJECT","load_bearing_attack":"NEAR2's headline claim—12x smaller embeddings with improved accuracy and no extra training cost—rests on deltas in Tables 2 and 4 computed against baselines from Saadany et al. [3] that did not receive the two additional epochs of continued fine-tuning described in Section 4.2. Section 4.2 states that the authors 'continued training the PTLMs and the UCO models ... for 2 epochs' on IGR data with the nested objective, while the comparison models in Tables 2, A.1, and 5 are the pre-continuation models. Any two epochs of in-domain fine-tuning on graded-relevance data, with or without the Matryoshka loss, can be expected to raise retrieval and ranking metrics. Table 5 makes the confound visible: applying NEAR2 after OCL-only fine-tuning produces +32.09% NDCG@5, while after MNRL+OCL it produces +3.34%; the dominant factor is which loss the extra epochs add, not the nesting itself. Without a matched-duration control (same data, epochs, batch size, learning rate, margin, with the MRL term disabled), the central attribution is untested. Secondary internal inconsistencies—Equation 2 is inverted if f is cosine similarity, and Table 3's memory figures are non-monotonic (512→2.77 MB, 256→4.09 MB, 64→1.56 MB)—further block verification, but even fixing those, the missing control leaves the empirical claim unestablished.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NEAR^2, a nested embedding approach based on Matryoshka representation learning (MRL) combined with a multiple negative ranking loss (MNRL) for e-commerce product retrieval and ranking. The method is applied to several encoder-based Transformers (BERT, eBERT, eBERT-siam, and their UCO variants) by continuing training on eBay's internal graded relevance data for two epochs. The paper claims up to 12x smaller embedding dimensions (768→64) with improved retrieval/ranking metrics on four test sets, as well as more than 100x smaller memory usage at inference, while introducing no extra training cost. The evaluation reports percentage improvements over baselines from Saadany et al. [3] and includes an ablation study over loss combinations.","tokens_in":13738,"tokens_out":5833,"duration_ms":54643,"significance":"If the central claims held, this work would offer a practical way to reduce embedding storage and inference cost in large-scale e-commerce search without sacrificing accuracy, which is directly relevant to a serious IR venue. The combination of MRL with a ranking loss for product retrieval is a sensible idea and the experimental scope covers several models and challenging query types. However, the paper currently suffers from fundamental issues in the description of the loss function, a missing matched-duration control experiment that leaves the performance gains un-attributed, and a clearly erroneous memory-usage table. These issues are load-bearing for the main conclusions.","major_comments":[{"comment":"The MNRL loss as written is inverted. For cosine similarity f, the standard ranking loss should penalize cases where f(q,p) is not greater than f(q,n)+margin, i.e., max(0, f(q,n) − f(q,p) + margin). Equation (2) instead uses max(0, f(q,p) − f(q,n) + margin), which penalizes exactly the desired ordering. The accompanying text makes the same error: it states the goal is to 'minimize the similarity between (q,p_i)'. If the implementation actually used the standard MNRL, the paper's description must be corrected to make the method reproducible; if Eq. (2) was actually used, the reported improvements are inexplicable. This is a central, correctness-relevant defect in the methodology section.","section":"§3.1, Eq. (2)"},{"comment":"The paper's central attribution is untested because there is no matched-duration control. NEAR2 models are produced by continuing training for 2 epochs on the IGR data with the nested objective, but the baseline numbers in Tables 2 and 4 are taken from Saadany et al. [3] for models that did not receive these extra epochs. Any additional in-domain fine-tuning on graded-relevance data could improve retrieval and ranking metrics, independent of the nested structure. The ablation in Table 5 makes this confound visible: adding NEAR2 after OCL-only fine-tuning gives +32.09% NDCG@5, while after MNRL+OCL it is only +3.34%. A control model trained for the same number of epochs on the same data with the MRL term disabled (e.g., standard MNRL only) is necessary to isolate the effect of the nesting loss. Without it, the headline claim of improved accuracy at smaller dimensions is not established.","section":"§4.2, Tables 2, 4, and 5"},{"comment":"The memory-usage figures are non-monotonic in embedding size: 768D→398.03 MB, 512D→2.77 MB, 256D→4.09 MB, 128D→0.55 MB, 64D→1.56 MB. This is impossible for any storage that scales with embedding dimensionality, and it directly undermines the abstract's claim of '100× smaller memory usage'. The values appear to suffer from a units or arithmetic error; the table must be corrected or the memory-related claim removed.","section":"Table 3"},{"comment":"The abstract and conclusion claim that NEAR2 introduces 'no extra cost in training', but §4.2 explicitly states that the authors continued training the PTLMs and UCO models for 2 epochs on the IGR data. If the baselines from [3] did not receive these epochs, then NEAR2 incurs additional training cost; if the baselines also received the extra epochs, that must be described. As written, the claim is contradicted by the experimental setup and must be clarified or corrected.","section":"Abstract and §4.2"}],"minor_comments":[{"comment":"Footnote 2 states that BERT and eBERT 'were not fine-tuned on e-commerce data', but §3.2 says eBERT was 'pre-trained on a corpus of approximately three billion product titles'. Please clarify the intended distinction between fine-tuning and pre-training to avoid apparent contradiction.","section":"§5, footnote and §3.2"},{"comment":"The importance weights c_m in the MRL loss are introduced but never specified (no values or tuning procedure are reported). State how these weights were set.","section":"§3.1, Eq. (1)"},{"comment":"The name 'NEAR2' is rendered inconsistently as 'NEAR2', 'NEAR 2', and 'NEAR^2'; use a single notation consistently.","section":"Throughout"},{"comment":"The deltas are reported relative to baselines from a prior paper; it would be useful to also provide the absolute baseline metric values in an appendix to make the comparisons interpretable.","section":"Tables 2 and A.1"},{"comment":"The sentence 'excluding pairs with a score of 3' relative to a 1–5 relevance scale is ambiguous: specify whether score 3 ('Good') is treated as neutral/removed and why.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript leans heavily on the authors' own prior work [3] for both the UCO models and the test sets, which is legitimate but heightens the need for the missing matched-duration control. The inverted MNRL equation is the most serious flaw; if the authors cannot clarify that the implementation used a correct loss, the paper should be rejected. In its current state the central empirical claim is not supported, but the issues are potentially fixable with a control experiment and careful corrections, so major revision is appropriate rather than outright rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper claims a 12x embedding-size reduction with better retrieval accuracy using nested embeddings (MRL) combined with ranking losses, but the supporting evidence has two internal errors and one missing control that undermine the headline result.\n\nWhat's genuinely useful: the paper applies Matryoshka representation learning to e-commerce product retrieval and ranking, and it reports an ablation showing that applying the nested objective *after* an IR task loss works better than applying it before. That training-order finding is a real empirical observation, and the four-test-set evaluation is reasonably thorough. The qualitative analysis of similarity-score distributions also gives a plausible mechanism for the improvement.\n\nThe soft spots are substantial. First, the MNRL loss in Eq. 2 is written as max(0, f(q,p_i) - f(q,n_j) + margin), which for cosine similarity would penalize ranking the positive above the negative. The text in Section 3.1 even says the goal is to \"minimize the similarity between (q, p_i)\"—the opposite of what retrieval needs. That may be a typo, but as written it makes the method description self-contradictory. Second, Table 3's memory usage numbers are non-monotonic (512 dims: 2.77 MB, 256 dims: 4.09 MB, 64 dims: 1.56 MB), which no storage model I know can explain.\n\nThe deeper problem is the missing control for training duration. Section 4.2 says the models were \"continued training... for 2 epochs\" on IGR data. All baselines from Saadany et al. are the pre-continuation models. So any gain could come from the extra in-domain fine-tuning, not from the nested objective. Table 5 makes this confound visible: NEAR2 applied after OCL-only fine-tuning gives +32% NDCG@5, while after MNRL+OCL it gives +3.3%. The dominant factor is which loss the extra epochs add, not the nesting itself. Without a matched-duration control (same data, epochs, batch size, learning rate, with the MRL term disabled), the central claim is untested.\n\nWho this is for: practitioners exploring MRL for efficient retrieval might find the idea worth testing, but the current evidence doesn't support the claimed 12x efficiency or improved accuracy. I'd recommend rejecting in current form; the authors need to fix the loss definition, correct the memory table, and rerun with a proper control. If they do that, the empirical finding might be salvageable.\n\nFor peer review, I would not accept this version, but it's not a desk reject—it's worth a referee's time to confirm the errors and ask for the control. Send it to review with the expectation of major revision.","headline":"A claimed 12x embedding-size win is undermined by an inverted MNRL loss, a non-monotonic memory table, and a missing control for extra fine-tuning epochs.","tokens_in":14259,"tokens_out":5321,"would_cite":false,"duration_ms":50486,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 64-dimension embedding beats the full 768-dimension model in product search.","keywords":["product retrieval","ranking","nested embeddings","Matryoshka representation learning","embedding efficiency","multiple negative ranking loss","online contrastive loss","e-commerce search"],"falsifier":"Train a control model with identical hyperparameters, data, and two-epoch schedule, using the same loss but no nested heads, and compare its full-dimension metrics with the NEAR2 model at 64 dimensions. If the control's full-dimension metrics match or exceed the nested 64-dimension result, the nested objective is not the source of the improvement. A second check is to evaluate nested sizes the model never saw in training, such as 32, to test whether the coarse-to-fine property generalises beyond the trained prefix sizes.","tokens_in":13240,"feed_emoji":"🔍","tokens_out":5490,"duration_ms":52000,"temperature":0.7,"pith_summary":"This paper argues that a nested training objective lets e-commerce retrieval models answer queries with 64-dimensional embeddings instead of 768 while matching or beating the full-size model on accuracy. The authors continue fine-tuning encoder-based Transformers on ranked product-title data, optimizing multiple prefix lengths of the same embedding at once with ranking losses. On four test sets spanning short, implicit, common-string, and alphanumeric queries, the 64-dimension version of models such as eBERT and eBERT-siam outperforms the same models at full dimension. If true, search systems could serve large product catalogs with roughly twelve times less embedding storage and over a hundred times less memory, at no added inference cost. The paper also reports that the effect holds across multiple negative ranking loss and online contrastive loss.","feed_headline":"Nested embeddings shrink product-search vectors 12x and still win","feed_subtitle":"A 64-dimension embedding beats full-size models on four hard query sets, cutting memory and latency.","key_machinery":"The load-bearing mechanism is the nested embedding objective: a weighted sum of task losses applied to the first $m$ dimensions of the embedding for $m \\in \\{64,128,256,512,768\\}$, with the multiple negative ranking loss or online contrastive loss as the task loss. During training, each prefix head is scored by its own ranking loss, so the model learns to make every prefix itself a usable retrieval embedding. At inference the system simply truncates the embedding to 64 dimensions and computes cosine similarity, which is what delivers the claimed 12× size reduction and 100× memory reduction. The ablations show the choice of loss ordering matters: applying the nested objective on top of a model already fine-tuned on MNRL or OCL improves results, while applying an MRL loss first and then MNRL+OCL slightly degrades them.","core_discovery":"NEAR2 applies a Matryoshka-style nested representation objective to product retrieval and ranking: while fine-tuning a 768-dimension encoder, it jointly optimises the 64-, 128-, 256-, 512-, and 768-dimension prefixes of the same embedding as separate output heads, summing their ranking losses. After training, only the small prefix needs to be stored and compared at inference. On the CQ, CQ-balanced, CQ-common-str, and CQ-alphanum test sets, the 64-dimension embeddings improve precision, recall, NDCG, and MRR over the same model fine-tuned without the nested objective at full dimension; for example, eBERT-siam's NDCG@5 rises by about 11% on the CQ set with the small vectors. The authors interpret this as evidence that nested supervision packs the information needed for retrieval into a compact prefix, making similarity scores more discriminative than those of the baseline models.","pith_inferences":["Because no equal-duration control without the nested objective was run, the cleanest attribution of the gains to nesting itself is not yet established; a matched fine-tuning control could change the size of the reported effect.","The qualitative score-distribution shift—nested models spread similarity scores out while baselines cluster near one value—suggests the objective acts partly as a calibration or sharpening mechanism, which could explain gains on hard queries beyond mere compression.","A testable extension is to evaluate prefix lengths outside the trained set, such as 32, to check whether the coarse-to-fine property generalises beyond the sizes seen in training."],"forward_implications":["Retrieval latency and storage scale down: a 64-dimension embedding needs 12× less space than 768, and the paper reports over 100× smaller memory usage at inference, so larger product catalogs can be served with the same infrastructure.","The gain is not tied to one loss function: NEAR2 improves models fine-tuned with multiple negative ranking loss and with online contrastive loss, and works on BERT, eBERT, and eBERT-siam backbones.","The method handles hard query types: accuracy improves on test sets built for short, implicit, common-string, and alphanumeric queries, where exact-match and semantic methods typically struggle.","Deployment can be adaptive: the same trained model can be truncated to different embedding sizes for different latency budgets without retraining."],"supporting_citations":[{"why":"Provides the user-intent centrality models, the four CQ test sets, and the baselines that all delta tables compare against.","marker":"[3]"},{"why":"Supplies the Matryoshka representation learning method that the nested embedding objective builds on.","marker":"[13]"},{"why":"Defines the multiple negative ranking loss used as the task loss inside the nested objective.","marker":"[15]"},{"why":"Provides the BERT encoder architecture used as one of the backbone models.","marker":"[8]"},{"why":"Provides the contrastive loss basis for the online contrastive loss used in the ablations.","marker":"[17]"}],"fun_headline_variants":["Nested embeddings: 12x smaller vectors, better retrieval accuracy","NEAR2: Shrink product-search vectors 12x and still beat full-size","12x smaller product-search vectors, better ranking with nested training","Nested embeddings pack more into fewer dimensions for product search","NEAR2: 12x efficiency gain with more accurate product ranking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains are attributed to the nested objective, but no control model was fine-tuned for the same extra epochs on the same data without the nested loss, so part or all of the improvement could come from continued fine-tuning alone.","fun_headline_variants_meta":{"raw":{"variants":["Nested embeddings: 12x smaller vectors, better retrieval accuracy","NEAR2: Shrink product-search vectors 12x and still beat full-size","12x smaller product-search vectors, better ranking with nested training","Nested embeddings pack more into fewer dimensions for product search","NEAR2: 12x efficiency gain with more accurate product ranking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000952,"raw_usage":{"total_tokens":4036,"prompt_tokens":894,"completion_tokens":3142,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":3049}},"tokens_in":510,"tokens_out":3142,"duration_ms":22090,"temperature":1.0,"reasoning_tokens":3049,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:25:19.910189+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a control model with identical hyperparameters, data, and two-epoch schedule, using the same loss but no nested heads, and compare its full-dimension metrics with the NEAR2 model at 64 dimensions. If the control's full-dimension metrics match or exceed the nested 64-dimension result, the nested objective is not the source of the improvement. A second check is to evaluate nested sizes the model never saw in training, such as 32, to test whether the coarse-to-fine property generalises beyond the trained prefix sizes.","supporting_citations":[{"cited_title":"Saadany, S","cited_arxiv_id":null,"evidence_quote":"Provides the user-intent centrality models, the four CQ test sets, and the baselines that all delta tables compare against."},{"cited_title":"Kusupati, G","cited_arxiv_id":null,"evidence_quote":"Supplies the Matryoshka representation learning method that the nested embedding objective builds on."},{"cited_title":"Carlsson, A","cited_arxiv_id":null,"evidence_quote":"Provides the contrastive loss basis for the online contrastive loss used in the ablations."}],"review_version":2}