REVIEW 4 major objections 5 minor 23 references
NEAR$^2$: A Nested Embedding Approach to Efficient Product Retrieval and Ranking
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A 64-dimension embedding beats the full 768-dimension model in product search.
desk verdict A claimed 12x embedding-size win is undermined by an inverted MNRL loss, a non-monotonic memory table, and a missing control for extra fine-tuning epochs. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the nested embedding objective: a weighted sum of task losses applied to the first $m$ dimensions of the embedding for $m \in \{64,128,256,512,768\}$, with the multiple negative ranking loss or online contrastive loss as the task loss. During training, each prefix head is scored by its own ranking loss, so the model learns to make every prefix itself a usable retrieval embedding. At inference the system simply truncates the embedding to 64 dimensions and computes cosine similarity, which is what delivers the claimed 12× size reduction and 100× memory reduction. The ablations show the choice of loss ordering matters: applying the nested objective on top of a model already fine-tuned on MNRL or OCL improves results, while applying an MRL loss first and then MNRL+OCL slightly degrades them.
What would settle it
Train a control model with identical hyperparameters, data, and two-epoch schedule, using the same loss but no nested heads, and compare its full-dimension metrics with the NEAR2 model at 64 dimensions. If the control's full-dimension metrics match or exceed the nested 64-dimension result, the nested objective is not the source of the improvement. A second check is to evaluate nested sizes the model never saw in training, such as 32, to test whether the coarse-to-fine property generalises beyond the trained prefix sizes.
Extended reading notes
Core claim
NEAR2 applies a Matryoshka-style nested representation objective to product retrieval and ranking: while fine-tuning a 768-dimension encoder, it jointly optimises the 64-, 128-, 256-, 512-, and 768-dimension prefixes of the same embedding as separate output heads, summing their ranking losses. After training, only the small prefix needs to be stored and compared at inference. On the CQ, CQ-balanced, CQ-common-str, and CQ-alphanum test sets, the 64-dimension embeddings improve precision, recall, NDCG, and MRR over the same model fine-tuned without the nested objective at full dimension; for example, eBERT-siam's NDCG@5 rises by about 11% on the CQ set with the small vectors. The authors interpret this as evidence that nested supervision packs the information needed for retrieval into a compact prefix, making similarity scores more discriminative than those of the baseline models.
Load-bearing premise
The reported gains are attributed to the nested objective, but no control model was fine-tuned for the same extra epochs on the same data without the nested loss, so part or all of the improvement could come from continued fine-tuning alone.
Editorial extensions
If this is right
- Retrieval latency and storage scale down: a 64-dimension embedding needs 12× less space than 768, and the paper reports over 100× smaller memory usage at inference, so larger product catalogs can be served with the same infrastructure.
- The gain is not tied to one loss function: NEAR2 improves models fine-tuned with multiple negative ranking loss and with online contrastive loss, and works on BERT, eBERT, and eBERT-siam backbones.
- The method handles hard query types: accuracy improves on test sets built for short, implicit, common-string, and alphanumeric queries, where exact-match and semantic methods typically struggle.
- Deployment can be adaptive: the same trained model can be truncated to different embedding sizes for different latency budgets without retraining.
Reading between the lines
- Because no equal-duration control without the nested objective was run, the cleanest attribution of the gains to nesting itself is not yet established; a matched fine-tuning control could change the size of the reported effect.
- The qualitative score-distribution shift—nested models spread similarity scores out while baselines cluster near one value—suggests the objective acts partly as a calibration or sharpening mechanism, which could explain gains on hard queries beyond mere compression.
- A testable extension is to evaluate prefix lengths outside the trained set, such as 32, to check whether the coarse-to-fine property generalises beyond the sizes seen in training.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes NEAR^2, a nested embedding approach based on Matryoshka representation learning (MRL) combined with a multiple negative ranking loss (MNRL) for e-commerce product retrieval and ranking. The method is applied to several encoder-based Transformers (BERT, eBERT, eBERT-siam, and their UCO variants) by continuing training on eBay's internal graded relevance data for two epochs. The paper claims up to 12x smaller embedding dimensions (768→64) with improved retrieval/ranking metrics on four test sets, as well as more than 100x smaller memory usage at inference, while introducing no extra training cost. The evaluation reports percentage improvements over baselines from Saadany et al. [3] and includes an ablation study over loss combinations.
Significance. If the central claims held, this work would offer a practical way to reduce embedding storage and inference cost in large-scale e-commerce search without sacrificing accuracy, which is directly relevant to a serious IR venue. The combination of MRL with a ranking loss for product retrieval is a sensible idea and the experimental scope covers several models and challenging query types. However, the paper currently suffers from fundamental issues in the description of the loss function, a missing matched-duration control experiment that leaves the performance gains un-attributed, and a clearly erroneous memory-usage table. These issues are load-bearing for the main conclusions.
major comments (4)
- [§3.1, Eq. (2)] The MNRL loss as written is inverted. For cosine similarity f, the standard ranking loss should penalize cases where f(q,p) is not greater than f(q,n)+margin, i.e., max(0, f(q,n) − f(q,p) + margin). Equation (2) instead uses max(0, f(q,p) − f(q,n) + margin), which penalizes exactly the desired ordering. The accompanying text makes the same error: it states the goal is to 'minimize the similarity between (q,p_i)'. If the implementation actually used the standard MNRL, the paper's description must be corrected to make the method reproducible; if Eq. (2) was actually used, the reported improvements are inexplicable. This is a central, correctness-relevant defect in the methodology section.
- [§4.2, Tables 2, 4, and 5] The paper's central attribution is untested because there is no matched-duration control. NEAR2 models are produced by continuing training for 2 epochs on the IGR data with the nested objective, but the baseline numbers in Tables 2 and 4 are taken from Saadany et al. [3] for models that did not receive these extra epochs. Any additional in-domain fine-tuning on graded-relevance data could improve retrieval and ranking metrics, independent of the nested structure. The ablation in Table 5 makes this confound visible: adding NEAR2 after OCL-only fine-tuning gives +32.09% NDCG@5, while after MNRL+OCL it is only +3.34%. A control model trained for the same number of epochs on the same data with the MRL term disabled (e.g., standard MNRL only) is necessary to isolate the effect of the nesting loss. Without it, the headline claim of improved accuracy at smaller dimensions is not established.
- [Table 3] The memory-usage figures are non-monotonic in embedding size: 768D→398.03 MB, 512D→2.77 MB, 256D→4.09 MB, 128D→0.55 MB, 64D→1.56 MB. This is impossible for any storage that scales with embedding dimensionality, and it directly undermines the abstract's claim of '100× smaller memory usage'. The values appear to suffer from a units or arithmetic error; the table must be corrected or the memory-related claim removed.
- [Abstract and §4.2] The abstract and conclusion claim that NEAR2 introduces 'no extra cost in training', but §4.2 explicitly states that the authors continued training the PTLMs and UCO models for 2 epochs on the IGR data. If the baselines from [3] did not receive these epochs, then NEAR2 incurs additional training cost; if the baselines also received the extra epochs, that must be described. As written, the claim is contradicted by the experimental setup and must be clarified or corrected.
minor comments (5)
- [§5, footnote and §3.2] Footnote 2 states that BERT and eBERT 'were not fine-tuned on e-commerce data', but §3.2 says eBERT was 'pre-trained on a corpus of approximately three billion product titles'. Please clarify the intended distinction between fine-tuning and pre-training to avoid apparent contradiction.
- [§3.1, Eq. (1)] The importance weights c_m in the MRL loss are introduced but never specified (no values or tuning procedure are reported). State how these weights were set.
- [Throughout] The name 'NEAR2' is rendered inconsistently as 'NEAR2', 'NEAR 2', and 'NEAR^2'; use a single notation consistently.
- [Tables 2 and A.1] The deltas are reported relative to baselines from a prior paper; it would be useful to also provide the absolute baseline metric values in an appendix to make the comparisons interpretable.
- [§4.2] The sentence 'excluding pairs with a score of 3' relative to a 1–5 relevance scale is ambiguous: specify whether score 3 ('Good') is treated as neutral/removed and why.
Circularity Check
No circularity: NEAR2 is an empirical application of the external Matryoshka objective; the missing training-time control is a validity concern, not a circular derivation.
full rationale
The paper's derivation chain is empirical rather than deductive: NEAR2 applies the MRL loss (Eq. 1) from Kusupati et al. [13] with MNRL (Eq. 2) as the task loss, continues training on eBay IGR data, and evaluates retrieval/ranking metrics on the four CQ test sets from Saadany et al. [3]. The headline efficiency claim (12x smaller embeddings, 100x smaller memory) is an observed outcome of evaluating at dimension 64, not a quantity that was fitted to the test data and then reported as a prediction. The comparisons in Tables 2, 4, A.1, and A.2 are deltas against the prior results in Saadany et al. [3]; those baselines are external empirical numbers, not defined in terms of NEAR2's output. Self-citations to [3] and [16] supply the UCO models and test sets, but the central NEAR2 result is not derived from those citations; it is a new fine-tuning procedure followed by evaluation. There is no self-definitional step, no fitted input renamed as a prediction, no imported uniqueness theorem, and no ansatz smuggled through a self-citation. The absence of a matched-duration control without the MRL loss is a genuine threat to attribution of the observed gains, because the extra two epochs of continued fine-tuning described in Section 4.2 could explain part of the improvement; however, that is a methodological confound, not a circularity of the kind where a claimed prediction is equivalent to its inputs by construction. Similarly, the apparent sign issue in Eq. 2 and the non-monotonic memory figures in Table 3 are correctness/reproducibility concerns, not circular reasoning. Therefore the paper's core claim has independent empirical content and the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- MRL importance weights c_m
- MNRL margin =
0.75
- Nested dimension set M =
{64, 128, 256, 512, 768}
- Continued training epochs =
2
assumptions (3)
- domain assumption Matryoshka Representation Learning produces nested embeddings whose prefixes match the accuracy of independently trained embeddings.
- domain assumption The IGR graded relevance annotations provide a reliable training signal for query-title relevance.
- domain assumption The CQ test sets are representative of e-commerce retrieval challenges such as short, implicit, and alphanumeric queries.
Cite this review
Pith. "Pith review of NEAR$^2$: A Nested Embedding Approach to Efficient Product Retrieval and Ranking." pith.science (2026). https://pith.science/paper/VBXBG2E4
@misc{pith2026250619743,
author = {Pith},
title = {Pith review of: NEAR$^2$: A Nested Embedding Approach to Efficient Product Retrieval and Ranking},
year = {2026},
howpublished = {\url{https://pith.science/paper/VBXBG2E4}},
note = {Machine review of arXiv:2506.19743}
}
abstract
E-commerce information retrieval (IR) systems struggle to simultaneously achieve high accuracy in interpreting complex user queries and maintain efficient processing of vast product catalogs. The dual challenge lies in precisely matching user intent with relevant products while managing the computational demands of real-time search across massive inventories. In this paper, we propose a Nested Embedding Approach to product Retrieval and Ranking, called NEAR$^2$, which can achieve up to $12$ times efficiency in embedding size at inference time while introducing no extra cost in training and improving performance in accuracy for various encoder-based Transformer models. We validate our approach using different loss functions for the retrieval and ranking task, including multiple negative ranking loss and online contrastive loss, on four different test sets with various IR challenges such as short and implicit queries. Our approach achieves an improved performance over a smaller embedding dimension, compared to any existing models.
Figures
Reference graph
Works this paper leans on
-
[3]
H. Saadany, S. Bhosale, S. Agrawal, D. Kanojia, C. Orasan, Z. Wu, Centrality-aware product retrieval and ranking, in: F. Dernoncourt, D. Preo¸ tiuc-Pietro, A. Shimorina (Eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, Association for Computational Linguistics, Miami, Florida, US, 2024, pp. 215...
work page 2024
-
[1]
S. Li, F. Lv, T. Jin, G. Lin, K. Yang, X. Zeng, X.-M. Wu, Q. Ma, Embedding-based product retrieval in taobao search, in: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD ’21, Association for Computing Machinery, New York, NY , USA, 2021, p. 3181–3189. URL: https://doi.org/10.1145/3447548.3467101. doi:10.1145/3447548.3467101
arXiv 2021
-
[2]
K. Keyvan, J. X. Huang, How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges, ACM Comput. Surv. 55 (2022). URL: https: //doi.org/10.1145/3534965. doi:10.1145/3534965
-
[4]
D. N. Mhawi, H. W. Oleiwi, N. H. Saeed, H. L. Al-Taie, An efficient information retrieval system using evolutionary algorithms, Network 2 (2022) 583–605. URL: https://www.mdpi.com/ 2673-8732/2/4/34. doi:10.3390/network2040034
-
[5]
K. A. Hambarde, H. Proença, Information retrieval: Recent advances and beyond, IEEE Access 11 (2023) 76581–76604. doi:10.1109/ACCESS.2023.3295776
arXiv 2023
-
[6]
Y . Zhu, H. Yuan, S. Wang, J. Liu, W. Liu, C. Deng, H. Chen, Z. Liu, Z. Dou, J.-R. Wen, Large language models for information retrieval: A survey, arXiv preprint (2023).arXiv:2308.07107
arXiv 2023
-
[7]
Vaswani, N
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Curran Associates Inc., Red Hook, NY , USA, 2017, p. 6000–6010
2017
-
[8]
Devlin, M.-W
J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transform- ers for language understanding, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, V olume 1 (Long and Short Papers), ...
2019
Show all 23 references
-
[9]
J. Guo, Y . Fan, Q. Ai, W. B. Croft, A deep relevance matching model for ad-hoc retrieval, in: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM ’16, Association for Computing Machinery, New York, NY , USA, 2016, p. 55–64. UR...
2016
-
[10]
Mitra, F
B. Mitra, F. Diaz, N. Craswell, Learning to match using local and distributed representations of text for web search, in: Proceedings of the 26th International Conference on World Wide Web, WWW ’17, International World Wide Web Conferences Steering Committee, Republic and Cant...
2017
-
[11]
S. Kim, A. S. Rawat, M. Zaheer, S. Jayasumana, V . Sadhanala, W. Jitkrittum, A. K. Menon, R. Fergus, S. Kumar, Embeddistill: A geometric knowledge distillation for information retrieval,
-
[12]
H. Wan, S. S. Patel, J. W. Murdock, S. Potdar, S. Joshi, Fast and light-weight answer text retrieval in dialogue systems, in: A. Loukina, R. Gangadharaiah, B. Min (Eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Ling...
2022 doi
-
[13]
Kusupati, G
A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha, V . Ramanujan, W. Howard-Snyder, K. Chen, S. Kakade, P. Jain, et al., Matryoshka representation learning, in: Advances in Neural Information Processing Systems, 2022
2022
-
[14]
X. Li, Z. Li, J. Li, H. Xie, Q. Li, ESE: Espresso sentence embeddings, arXiv preprint (2024). arXiv:2402.14776
2024 arXiv
-
[15]
Henderson, R
M. Henderson, R. Al-Rfou, B. Strope, Y .-H. Sung, L. Lukács, R. Guo, S. Kumar, B. Miklos, R. Kurzweil, Efficient natural language response suggestion for smart reply, arXiv preprint arXiv:1705.00652 (2017)
2017 arXiv
-
[16]
Saadany, S
H. Saadany, S. Bhosale, S. Agrawal, Z. Wu, C. Or˘asan, D. Kanojia, Product retrieval and ranking for alphanumeric queries, in: Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM ’24, Association for Computing Machinery, New York,...
2024
-
[17]
Carlsson, A
F. Carlsson, A. C. Gyllensten, E. Gogoulou, E. Y . Hellqvist, M. Sahlgren, Semantic re-tuning with contrastive tension, in: International Conference on Learning Representations, 2021. URL: https://openreview.net/forum?id=Ov_sMNau-PF
2021
-
[18]
Jiang, Y
Y . Jiang, Y . Shang, R. Li, W.-Y . Yang, G. Tang, C. Ma, Y . Xiao, E. Zhao, A unified neural network approach to e-commerce relevance learning, in: Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data, DLP-KDD ’19, Associati...
2019
-
[19]
D. Kang, W. Jang, Y . Park, Evaluation of e-commerce websites using fuzzy hierarchical topsis based on e-s-qual, Applied Soft Computing 42 (2016) 53–65. URL: https://www.sciencedirect.com/ science/article/pii/S1568494616300047. doi:https://doi.org/10.1016/j.asoc.2016.01.017
2016 doi
-
[20]
Loshchilov, F
I. Loshchilov, F. Hutter, Decoupled weight decay regularization, in: International Conference on Learning Representations, 2019. URL: https://openreview.net/forum?id=Bkg6RiCqY7
2019
-
[21]
Järvelin, J
K. Järvelin, J. Kekäläinen, Cumulated gain-based evaluation of ir techniques, ACM Trans. Inf. Syst. 20 (2002) 422–446. URL: https://doi.org/10.1145/582415.582418. doi:10.1145/582415.582418
2002
-
[22]
plants” with eBERT. Based on the gold label, the expected product title should include “potted plants
C. Lee, R. Roy, M. Xu, J. Raiman, M. Shoeybi, B. Catanzaro, W. Ping, Nv-embed: Improved techniques for training llms as generalist embedding models, 2024. URL: https://arxiv.org/abs/2405. 17428.arXiv:2405.17428. A. Additional Figures and Tables Model Precision@𝑘Recall@𝑘NDCG@𝑘M...
2024 arXiv
-
[2023]
URL: https://openreview.net/forum?id=BT03V9Re9a
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.