{"id":"247b24dc-6149-4e26-b315-a1382d55f165","arxiv_id":"2505.07105","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Distilling a 7B LLM teacher into a BERT-base student with Margin-MSE loss on 170M teacher-labeled pairs yields a small student that matches or slightly beats the teacher on NDCG and improves Walmart's tail-query search metrics online.","lead":"This paper describes a production system that uses a 7-billion-parameter language model to label over a hundred million product-query pairs, then trains a small BERT model to imitate the large model's relevance judgments. The small model, deployed at Walmart, improves tail-query search relevance and engagement metrics in online tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Golden test set may overlap with llm-judged training data; Sec. 4.1 only rules out leakage from human-judged data, not from teacher-labeled search-log data.","rationale":"The paper's central contribution is that a BERT-base student trained via Margin-MSE on 170M teacher-labeled search-log pairs reaches teacher-comparable relevance and improves tail-query engagement. The only offline evidence for this is the golden test set. The reader identified the golden-test/llm-judged overlap as the weakest assumption; the full text supports this. Section 4.1's leakage sentence addresses only human evaluation data and is silent on llm-judged data, despite both coming from search traffic. This is not a disagreement with consensus; it is an internal gap in the evaluation protocol. The teacher-vs-student comparison also lacks error bars, and the abstract's 'outperform' claim is stronger than the single NDCG@5 result, but those are secondary. The overlap check is concrete and can be run internally by the authors without new data collection. If overlap is zero, the concern is resolved and the conditional acceptance can stand. If overlap is material, the headline offline comparison must be recomputed on a disjoint subset and the paper's central claim is unsupported as written. Therefore we concur with CONDITIONAL, pending the overlap disclosure.","tokens_in":13894,"tokens_out":7004,"duration_ms":64892,"concrete_test":"Compute exact and query-level overlap between the golden test set (2,354 queries, 32,586 QIPs) and each llm-judged_v1/v2/v3 dataset using normalized query strings and item IDs. Report the overlap counts and the fraction of golden queries present in llm-judged_v3. Then re-run XE v2's evaluation restricted to golden queries with zero overlap with llm-judged_v3 and compare NDCG@5/R@P=95% to the full-set numbers. If the disjoint-subset lifts fall below the teacher's (+1.33% NDCG@5, +13.27% R@P=95%) or the confidence intervals overlap, the 'student outperforms teacher' claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that XE v2 reaches teacher-comparable relevance on tail queries rests on a golden test set of 2,354 tail queries. Section 4.1 says the golden test data 'was generated after all human evaluation data (used for training data) was collected, ensuring no risk of data leakage.' That statement addresses only the human-judged_v1/v2 datasets. It does not address the llm-judged_v1/v2/v3 datasets, which are created by sampling unlabeled query-item pairs from search-log data and labeling them with the Mistral-7B teacher (Sections 3.1 and 4.1). Since the golden queries are themselves 'randomly selected from the tail segment of our search traffic,' and llm-judged_v3 contains 5.2M unique queries from the same traffic source, exact or query-level overlap is plausible. If any golden query appears in llm-judged_v3, the student was trained on teacher scores for that query; its Margin-MSE objective includes the same query's item pairs. The golden set is then not a held-out evaluation, and the reported XE v2 NDCG@5 lift (+1.5% vs. teacher +1.33%) can reflect query-level memorization of the teacher's scores rather than a generalizable distillation gain. Even if overlap does not inflate the student's score, it invalidates the claim that the student generalizes to unseen tail queries. The paper should disclose overlap statistics or evaluate on a provably disjoint subset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a knowledge-distillation framework for e-commerce search relevance at Walmart. A 7B-parameter LLM teacher (Llama2-7B or Mistral-7B) is fine-tuned with soft relevance labels on human editorial data, then used to label large-scale unlabeled query-item pairs extracted from search logs. A BERT-base cross-encoder student is trained with a Margin-MSE loss on up to 170M such teacher-labeled pairs. Offline experiments on a golden tail-query test set show that Margin-MSE outperforms pointwise cross-entropy, that increasing teacher-labeled data improves NDCG, and that the student approaches or slightly exceeds the teacher on NDCG despite being much smaller. The student (XE v2) was deployed as a ranking feature, with significant lifts in manual evaluation, interleaving ATC, and an online AB test.","tokens_in":14162,"tokens_out":6629,"duration_ms":59156,"significance":"If validated, this is a practically significant industrial contribution: it demonstrates a concrete path to deploying LLM-level relevance judgments in a latency-constrained reranking system, with production-scale evidence (offline golden test plus online AB/interleaving). The paper's strengths are its clear ablation isolating loss function and data scale, and its honest reporting of deployment results. However, the headline comparison to the teacher is currently supported only by small offline metric differences without confidence intervals, and two methodological controls (golden/training overlap and teacher training-data asymmetry) must be resolved before the central claims can be taken at face value.","major_comments":[{"comment":"The statement that the golden test set 'was generated after all human evaluation data ... was collected' addresses only human-judged datasets; it does not address the llm-judged_v1/v2/v3 training sets, which are sampled from the same tail search traffic (Table 3). Because llm-judged_v3 contains 5.2M unique queries, exact query-level overlap with the 2,354 golden queries is plausible. If any golden query appears in llm-judged_v3, XE v2 is trained with Margin-MSE on teacher scores for the very query-item pairs used for evaluation, and the reported NDCG@5 lift (+1.5% vs. teacher +1.33%) could reflect memorization rather than generalization. Please report overlap statistics or evaluate on a provably disjoint subset.","section":"§4.1 (Test Data)"},{"comment":"The RQ3 comparison between the student and the teacher is confounded by training-data asymmetry. The Mistral-7B teacher is fine-tuned only on human-judged_v1 (6.0M QIPs), while XE v2 and XE v1.4 are initialized from XE v1.1, which was trained on human-judged_v2 (10M QIPs). Thus the student has access to more human-annotated data than the teacher in addition to the llm-judged data, so the apparent NDCG advantage over the teacher cannot be attributed solely to distillation at scale. To support the abstract's claim that 'with enough augmented data the student can outperform the teacher,' either train the teacher on human-judged_v2 as well, or initialize the student from a checkpoint trained only on human-judged_v1, or add an ablation that isolates this factor.","section":"§4.2.1 and Table 5"},{"comment":"The offline metrics are reported without confidence intervals or significance tests, and the differences that support the 'student outperforms teacher' claim are small (NDCG@5 +1.50% vs +1.33%; R@P=95% +12.84% vs +13.27%). These gaps are within the range where sampling noise could change the qualitative conclusion. Please report bootstrap confidence intervals over the 2,354 golden queries or paired significance tests for the comparisons in RQ1-RQ3.","section":"§4.3.3 and Tables 4-5"}],"minor_comments":[{"comment":"The text refers to 'Table 8' for the AB test results, but the table is numbered Table 7.","section":"§5.3"},{"comment":"The table marks all four metrics as statistically significant, but p-values are only given in the text for ATC rate per visitor; please report p-values or confidence intervals for all four metrics.","section":"§5.3/Table 7"},{"comment":"The phrase 'softRank Adaptation (LoRA)' should read 'Low-Rank Adaptation (LoRA)'.","section":"§3.2"},{"comment":"The baseline rows are all listed as 0%, which likely means the reported numbers are relative lifts over XE v1; please state this explicitly and include the absolute baseline values for the golden test set.","section":"§4.3/Tables 4-5"},{"comment":"There is a typo in 'we present our our enhanced'.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The two main technical risks are the unresolved overlap between the golden test set and the llm-judged training data, and the confounded student-vs-teacher comparison caused by different human-judged training sets. Both are fixable with additional analysis or experiments, but the paper should not be accepted until they are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The genuinely useful thing is the scaling experiment: a 110M BERT student, trained with Margin-MSE on teacher-labeled unlabeled data, reaches or slightly exceeds the Mistral-7B teacher's NDCG on tail queries. The extension of Margin-MSE to all pairs on unlabeled data is real and useful, and the ablations—Margin-MSE vs CE, dataset size 50M→170M—are clean and support the main claims. The online A/B test on live traffic is a rare and valuable addition; the +1.2% ATC/visitor is small but significant.\n\nThe soft spot is the one flagged in the stress-test: the golden test set may overlap with the llm-judged training data. The paper only rules out leakage from human-judged data, but llm-judged data is sampled from the same tail search traffic, and llm-judged_v3 has 5.2M unique queries. Without an explicit exclusion or overlap stats, the student-vs-teacher NDCG comparison is uncertain. That said, the scaling trend (XE 1.3→1.4→2) holds across three dataset sizes, and the online results are based on live traffic, not the golden set, so the core distillation story likely survives. Still, this is the first thing I'd ask the authors to fix.\n\nMinor issues: no confidence intervals on offline metrics, and the abstract's 'outperform the teacher' overstates what is a mixed result (better NDCG, worse R@P). Also, the paper is written for an e-commerce search audience; the method is not deeply novel but the empirical scaling result is worth knowing.\n\nThis paper deserves a serious referee: the question of how far a small student can go with large-scale teacher-labeled data is important, and this is one of the few production-validated data points. I'd send it to review, but with an explicit request to address the test-set overlap and tone down the abstract.","headline":"Solid industrial distillation scaling study; the margin-MSE-on-unlabeled-data extension is useful, but the student-vs-teacher comparison undercuts itself by not ruling out golden-test overlap with teacher-labeled training data.","tokens_in":14754,"tokens_out":2599,"would_cite":true,"duration_ms":25438,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a BERT-base student distilled from a 7B LLM teacher matches or slightly exceeds it on ranking metrics when trained on 170M teacher-labeled query-item pairs.","keywords":["LLM","knowledge distillation","e-commerce search","Margin-MSE loss","cross-encoder","search relevance","tail queries","production deployment"],"falsifier":"Inspect the 2,354 golden test queries against the 5.2M unique queries in the llm-judged datasets. If a meaningful fraction of test queries appear in the teacher-labeled training data, retrain the student on data with those queries removed and re-measure the NDCG@5 lift over the baseline; the central claim predicts the lift will remain near +1.5%, while a large drop would indicate leakage-driven inflation.","tokens_in":13680,"feed_emoji":"🛒","tokens_out":5842,"duration_ms":54536,"temperature":0.7,"pith_summary":"The paper claims that a small, deployable search-relevance model can absorb the ranking ability of a 7-billion-parameter LLM through knowledge distillation, provided the student is trained on a much larger set of teacher-labeled data than the teacher itself used. The student is a BERT-base cross-encoder with 110M parameters, trained with a Margin-MSE loss that matches the predicted relevance gaps between pairs of products for a query. On a golden tail-query test set, the student reaches roughly the same gains over the production baseline as the Mistral-7B teacher on classification-oriented recall, and slightly larger gains on NDCG. The paper reports that the student was deployed in production and improved tail-query engagement metrics such as add-to-cart rate per visitor (+1.2%, p<0.05).","feed_headline":"Distilled BERT base matches 7B LLM teacher in search relevance","feed_subtitle":"A 110M-parameter student trained on 170M LLM-labeled pairs lifts add-to-cart rate per visitor by 1.2% in production.","key_machinery":"The mechanism is an extension of Margin-MSE distillation. For each query $q$, the teacher produces scores $t(q,d)$ and the student produces scores $s(q,d)$ for documents $d$; the loss averages the squared difference $(\\Delta^t_{q,d_i,d_j} - \\Delta^s_{q,d_i,d_j})^2$ over all unordered document pairs, so the student learns the teacher's relative gaps between products rather than absolute scores. This margin matching removes the need for true labels, which is what lets the framework convert large-scale unlabeled search-log data into 50M, 110M, and 170M teacher-labeled query-item pairs for student training.","core_discovery":"The central claim is that a BERT-base student model, distilled from a Mistral-7B teacher on 170M teacher-labeled query-item pairs, performs comparably to or slightly better than the teacher on ranking metrics, despite being more than 60 times smaller and using fewer input fields. Teacher and student share a cross-encoder architecture, but the student is trained on an expanded unlabeled dataset labeled entirely by the teacher, with the loss defined over all product-pair score margins for each query. The authors interpret the results as evidence that larger model capacity and the item-description field are not necessary to reproduce the teacher's relevance judgments, and that scaling teacher-labeled unlabeled data is the main driver of the student's gains.","pith_inferences":["An extension the paper leaves implicit: the same data-generation loop could be applied to head and torso queries, where the paper's stated future work points, since the limiting factor appears to be labeled-data volume rather than model capacity.","A testable extension: because the all-pairs margin loss scales quadratically with the number of documents per query, sampling more than the reported 10 candidates per query may change how quickly performance plateaus.","A risk the paper does not discuss: training on 170M teacher-labeled pairs copies whatever bias the teacher has, such as overconfidence on popular brands, so calibrating teacher scores before labeling is a natural next check."],"forward_implications":["A 110M-parameter student can replace a 7B LLM in latency-sensitive ranking, making LLM-level relevance feasible in real-time e-commerce search.","Increasing teacher-labeled unlabeled data continues to improve student ranking quality, so more search-log data can be converted into training signal without human annotation.","Learning margins between pairs is more effective than pointwise score matching even when student and teacher share the same architecture.","The student's comparable performance without item descriptions means production inputs can be shortened while retaining teacher-level relevance.","Deploying the distilled student can yield measurable engagement gains on tail queries, including higher add-to-cart rates and lower session abandonment."],"supporting_citations":[{"why":"Introduces the Margin-MSE loss that the paper extends to unlabeled all-pairs data.","marker":"[7]"},{"why":"Establishes the cross-encoder knowledge-distillation setup and soft-target formulation the student training builds on.","marker":"[31]"},{"why":"Shows LLMs can judge product relevance and supplies the teacher fine-tuning hyperparameters and the item-description benefit.","marker":"[18]"},{"why":"Provides the Mistral-7B checkpoint used as the teacher model whose predictions label the augmented datasets.","marker":"[9]"},{"why":"Provides the BERT-base architecture used as the student model.","marker":"[11]"},{"why":"Defines knowledge distillation as the teacher-to-student transfer mechanism the framework relies on.","marker":"[6]"},{"why":"LoRA is used to fine-tune the 7B teacher efficiently.","marker":"[8]"}],"fun_headline_variants":["110M student model beats 7B LLM teacher in search relevance","Tiny distilled model outperforms 7B teacher in e-commerce search","Distillation makes a 110M student more accurate than its 7B teacher","Small model, big win: distilled student outperforms 7B teacher","Teacher knowledge fits into 110M model that beats 7B LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the golden test queries and the teacher-labeled training pairs come from disjoint search traffic; the paper does not state that the 2,354 test queries were excluded from the llm-judged datasets, so if they overlap, part of the student's apparent parity with the teacher could be memorization of teacher scores rather than learned generalization.","fun_headline_variants_meta":{"raw":{"variants":["110M student model beats 7B LLM teacher in search relevance","Tiny distilled model outperforms 7B teacher in e-commerce search","Distillation makes a 110M student more accurate than its 7B teacher","Small model, big win: distilled student outperforms 7B teacher","Teacher knowledge fits into 110M model that beats 7B LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001301,"raw_usage":{"total_tokens":5304,"prompt_tokens":936,"completion_tokens":4368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":4269}},"tokens_in":552,"tokens_out":4368,"duration_ms":29916,"temperature":1.0,"reasoning_tokens":4269,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:24:17.786487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the 2,354 golden test queries against the 5.2M unique queries in the llm-judged datasets. If a meaningful fraction of test queries appear in the teacher-labeled training data, retrain the student on data with those queries removed and re-measure the NDCG@5 lift over the baseline; the central claim predicts the lift will remain near +1.5%, while a large drop would indicate leakage-driven inflation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the cross-encoder knowledge-distillation setup and soft-target formulation the student training builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LoRA is used to fine-tune the 7B teacher efficiently."}],"review_version":1}