{"id":"c977642c-d5fd-4087-a457-be08582767ed","arxiv_id":"2502.09089","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A category-pretrained DistilBERT two-tower Siamese model with progressive human-in-the-loop domain fusion improves Walmart ad retrieval relevance and revenue over a DSSM baseline.","lead":"Walmart engineers describe a production ads-retrieval system that pre-trains a DistilBERT model on product categories, then fine-tunes it as a two-tower Siamese encoder with progressive, human-weighted training on multiple datasets. Offline and online tests report up to 16% better relevance and 5.16% higher search-page ad revenue than the existing DSSM-based production model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No deduplication between training and offline evaluation is described; overlapping query-item pairs could inflate the reported NDCG gains.","rationale":"I read the paper as an industrial systems report whose central claim is that the described two-stage progressive training pipeline, when deployed, improves ad relevance and revenue over a DSSM baseline. The most load-bearing premise is that the offline relevance numbers in Table 2 are trustworthy, since they are the primary evidence for the 'up to 16%' claim. The reader's weakest assumption was the click-based pseudo-labeling of ad logs; that is a legitimate validity threat, but it bears on the training signal, not directly on the offline evaluation, which uses third-party human labels. The more serious unexamined thread is the absence of any stated de-duplication between the large training corpora (§2.2) and the 2,000-query offline evaluation (§4.2). Both come from Walmart search traffic, and the 8M human-evaluated search-log training set labels query-item pairs from that same traffic. Without a stated separation, near-duplicate queries are almost certain, and the model could score artificially high by memorization. This would invalidate the central claim in a way that click-bias does not, because the offline test labels are independent of clicks. I do not claim the authors intentionally leaked data; I claim the paper has not provided evidence that leakage did not occur, and the check is straightforward. I also note the paper's own limitation statement (Section 5) about pre-computed query embeddings and the confidentiality-based decision to reveal only relative improvements, both of which reduce falsifiability but are not fatal by themselves. Since the reader's verdict is already CONDITIONAL, and this concern is one more condition to verify rather than a proven flaw, I would leave the verdict unchanged while strengthening the requested evidence: add a de-duplication or temporal-split analysis. Therefore, agreement_with_reader is 'partial'—I agree the evaluation could be unreliable, but I locate the main risk in train/evaluation overlap rather than click pseudo-labels.","tokens_in":8725,"tokens_out":4108,"duration_ms":41247,"concrete_test":"Ask the authors to report the overlap between the 2,000 offline evaluation queries and the queries/pairs in the 8M human-evaluated search-log set, the 20M Google site:walmart.com set, and the 12M ad-log set, including near-duplicate normalization (lowercasing, whitespace stripping, title/query token overlap). Then recompute Table 2 NDCG gains on the subset of evaluation queries that have zero overlap with any training query-item pair, or split training/evaluation by time (e.g., hold out human labels collected after the training data cutoff). If the NDCG@5 gain on the disjoint/time-held-out subset drops materially below +16%, the reported relevance improvement is inflated by contamination and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim of up to +16% search relevance rests on the offline simulation in §4.2, which uses 2,000 queries randomly drawn from Walmart traffic and human-labeled relevance. However, §2.2 lists three large training corpora derived from the same ecosystem: 20M Google site:walmart.com query-title pairs, 8M human-evaluated Walmart search logs, and 12M Walmart ad-log pairs. The 2,000 evaluation queries are sampled from search history 'according to Walmart traffic segment' (§4.2), which is the same distribution the 8M human-evaluated search-log training set is drawn from. The paper never states whether overlapping queries, query-item pairs, or near-duplicate titles were removed between training and offline evaluation. If even a few percent of evaluation queries also appear in the 8M human-labeled training pairs (or in the 20M Google pairs built from site:walmart.com queries), the model can memorize the exact relevance labels, inflating NDCG@5 by the reported +16.10%. No de-duplication protocol, no temporal split, and no 'queries seen during training' analysis is reported. This is more load-bearing than the reader's click-pseudo-label concern: even if click labels are a biased training signal, the offline human evaluation still measures true relevance unless test labels leak through overlapping training pairs. A second, related omission is that only relative improvements are disclosed (Section 4), so if the DSSM baseline is low or uncalibrated, the relative gain is not interpretable without absolute NDCG values.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes an end-to-end sponsored-product retrieval system for Walmart.com built on a DistilBERT-based two-tower Siamese network. The training pipeline has three main components: (i) language-model pretraining on Walmart product-category labels, (ii) progressive fusion training over multiple knowledge domains (SNLI/MultiNLI, Google site-restricted query-title pairs, human-evaluated search logs, and click-based ad-log pseudo-labels), and (iii) a human-in-the-loop weighting mechanism that adjusts per-domain sampling weights from nDCG scores. The authors report offline retrieval gains over a DSSM baseline (up to +16.10% NDCG@5, with IAR reductions) and online A/B improvements in NDCG, CTR, CPMV, and ad revenue, and state that the system was deployed in production in June 2023.","tokens_in":9045,"tokens_out":5733,"duration_ms":56323,"significance":"If the reported results are reliable, this is a valuable industrial case study: it combines public NLI data, Google site-restricted query-title pairs, human-labeled search logs, and click-based ad-log pseudo-labels into a two-tower embedding model that was deployed at scale, with consistent positive movements across relevance and business metrics. The hard-negative labeling method and the progressive fusion sampling mechanism are concrete, reusable ideas, and the deployment/monitoring narrative is a strength. However, the empirical core is currently under-supported: the offline evaluation may overlap with training data, no uncertainty quantification is reported for the offline numbers, and the online A/B testing lacks statistical detail. Because the central claims rest on these measurements, the paper needs substantial additional evidence before the results can be accepted at face value.","major_comments":[{"comment":"The central offline claim of +16.10% NDCG@5 rests on 2,000 queries sampled from Walmart traffic, while Section 2.2 describes training corpora drawn from the same ecosystem, including 8M human-evaluated search-log pairs and 20M Google site:walmart.com query-title pairs. The paper never states whether overlapping queries, query-item pairs, or near-duplicate titles were removed between training and this evaluation. Without a deduplication protocol or a temporal split, the reported gains can be inflated by the model memorizing labels from training pairs that reappear in the evaluation set. Please report overlap statistics and either exclude overlapping examples or use a split that guarantees disjoint queries.","section":"§4.2 / §2.2 / Table 2"},{"comment":"The offline results are single point estimates with no variance, confidence intervals, or significance tests. With 2,000 evaluation queries, the differences of +16.10% (NDCG@5), -4.07% (IAR@5), and the other rows in Table 2 may not be statistically reliable. Please provide bootstrap confidence intervals, error bars, and ideally the absolute NDCG/IAR values for both models, not only relative improvements.","section":"§4.2 / Table 2"},{"comment":"The online A/B testing claims of +5.16% ad revenue and relevance improvements up to +11.87% are reported without sample sizes, experiment duration, confidence intervals, or guardrail metrics. The phrase 'large-scale' is not supported by the reported evidence. Please include the statistical details and specify whether the differences are significant at conventional levels.","section":"§4.3 / Tables 3–5"},{"comment":"The pseudo-labeling rule treats an item as relevant if it attracts a high number of clicks within a period, and hard negatives are items that are displayed often but rarely clicked. These heuristics confound relevance with position bias, brand awareness, and price attractiveness; the 'hard negative' definition in particular may label relevant but poorly placed items as negatives. Please validate the pseudo-labels against a human-labeled sample and report agreement or a bias analysis.","section":"§2.2, 'Data Labeling for Ad Logs'"},{"comment":"The human-in-the-loop weight adjustment uses nDCG scores computed from human-labeled data, and Section 4.2's offline evaluation also uses third-party human relevance labels. The paper does not state whether these two label sets are disjoint. If the same labels or queries are used both to set the domain sampling weights and to evaluate the final model, the evaluation is not independent. Please clarify the provenance of the two label sets and confirm that no evaluation label was used for model selection or weight tuning.","section":"§2.1 / Eq. (1)"}],"minor_comments":[{"comment":"In the first sentence of Section 4.1, 'matrices' should be 'metrics', and Eq. (6) should explicitly define the indicator function I(i).","section":"§4.1"},{"comment":"'Ground-true relevance labels' should be 'ground-truth relevance labels'.","section":"§4.2"},{"comment":"The caption says 'on search page' but the rows describe item page modules; the caption should be corrected.","section":"Table 5 caption"},{"comment":"The sigmoid constants 10 and 5 in Eq. (1) and the margin alpha in Eq. (3) are free parameters with no stated values; please report the chosen values and provide a brief sensitivity check.","section":"Eq. (1) and Eq. (3)"},{"comment":"The statement that the model in [13] 'did not significantly contribute to the improvement of the advertising system' is presented as observed fact without a citation; either cite the source or rephrase as the authors' own unpublished experience.","section":"Introduction / Reference [13]"},{"comment":"The monitoring protocol is described verbally but no post-deployment numbers are given; either add a compact monitoring table or state explicitly that these numbers are omitted.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The main risk is data leakage between the training corpora and the offline evaluation set; the authors should be asked directly for deduplication and temporal-split evidence. If they cannot provide it, the offline claims should be substantially downgraded. The missing statistical details for both offline and online experiments are also necessary before any acceptance decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Mike, quick take on this one. It's an industrial systems paper from Walmart's ads retrieval team, and the substance is a two-stage pipeline: DistilBERT pre-trained on product category labels, then a two-tower Siamese encoder trained with a progressive fusion of five datasets, with weights adjusted by human-in-the-loop nDCG feedback. The engineering is real—it runs in production, at Walmart scale, and the online A/B results (search page CTR +7.2%, ads revenue +5.16%) are the kind of numbers that matter in e-commerce. The human-in-the-loop weighting in Eq. (1), where strong domains get down-weighted so the model spends capacity on weak ones, is the most interesting piece. It's not a big scientific leap, but it's sensible and not something I've seen reported in exactly this form.\n\nWhat's good: the paper is honest about being a systems paper. It describes the architecture, the data sources, the pseudo-labeling shortcut, and the deployment. The hard-negative mining from click logs and the ablation showing a 3.6% NDCG drop without hard negatives is useful evidence that the component does something.\n\nNow the soft spots, and one of them is load-bearing. The offline evaluation uses 2,000 queries sampled from Walmart search traffic, and the 8M human-evaluated search-log dataset is drawn from the same traffic. The paper never states whether overlapping queries or query-item pairs were removed between training and offline eval. If a few percent of eval queries appear in the 8M training pairs, the model can memorize labels and the +16.10% NDCG@5 is inflated. That is not a nitpick; it is a direct threat to the central claim. The click-based pseudo-labels are a separate bias, but the leakage issue is worse because it corrupts the ground-truth evaluation itself. Related: only relative improvements are disclosed, no absolute NDCG, no confidence intervals, no A/B test duration or sample size. For a WWW paper, that is thin but typical for industry.\n\nThe paper does not claim to be a formal contribution; it's a well-written case study. The data and code are proprietary, so reproducibility is limited to the described methodology. I would send it to review, but the review would need to demand a dedup protocol or a temporal split before the offline numbers can be trusted. The online A/B, if taken at face value, is the stronger evidence.\n\nBottom line: worth reading for the fusion-training idea and the deployment details. Needs a major revision to address the leakage concern. Deserves a serious referee.","headline":"A real deployed system with honest engineering claims, but the offline numbers have a possible train/eval leakage problem that the paper does not address.","tokens_in":9599,"tokens_out":1903,"would_cite":false,"duration_ms":18272,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-tower BERT model with category pretraining and human-in-the-loop progressive training improves ads-retrieval relevance by up to 16 percent and ad revenue by 5.16 percent in production on Walmart.com.","keywords":["sponsored search","ads retrieval","BERT","two-tower Siamese network","human-in-the-loop","progressive training","hard negative mining","e-commerce search"],"falsifier":"Train the same pipeline with the 12 million ad-log pseudo-labels replaced by human relevance labels on a matched subsample, and run the offline approximate-nearest-neighbor retrieval evaluation: if the NDCG@5 improvement over DSSM remains around 16 percent, then click-count pseudo-labeling is not what produces the reported relevance gains.","tokens_in":8552,"feed_emoji":"🛒","tokens_out":11685,"duration_ms":89371,"temperature":0.7,"pith_summary":"Sponsored ads on Walmart.com are matched to search queries by a retrieval model that must pick candidate products for later ranking. This paper claims that a two-stage training pipeline produces a retrieval model that improves query-ad matching over the company's previous DSSM-based model. The pipeline first teaches a BERT-style language model Walmart's product categories, then trains a two-tower Siamese encoder on several datasets at once, with a human-in-the-loop rule that shifts training weight toward domains where the model is still weak. Offline evaluation shows NDCG gains of up to 16.10 percent and a reduction in the irrelevant-ads rate by about 4 percent; online A/B tests show ad revenue up 5.16 percent. The paper's point is that the specific combination of category pretraining, hard-negative mining, and progressive fusion training is what produces these gains.","feed_headline":"Walmart's BERT ad retriever lifts relevance 16%, ad revenue 5.16%","feed_subtitle":"Category pretraining plus progressive multi-domain training beat the DSSM baseline in live A/B tests.","key_machinery":"The load-bearing mechanism is the two-stage progressive training architecture. Stage one runs multi-class classification of queries and items into Walmart 'Department' and 'Product Type' categories with DistilBERT and cross-entropy loss. Stage two adds a pooling layer to produce 384-dimensional embeddings and trains a two-tower Siamese network on multiple datasets through a dynamic fusion sampling rule: each domain's sampling weight is $w_i = 1/(1+\\exp(10x_i - 5))$, where $x_i$ is the normalized discounted cumulative gain (nDCG) from human-labeled data, so well-handled domains are down-weighted and weak domains get more training. The encoder is trained with mean-squared-error loss on cosine-similarity labels plus a triplet loss with in-batch hard negatives, where hard negatives are items often displayed for a query but seldom clicked.","core_discovery":"The central claim is that the proposed pipeline — DistilBERT pretrained on Walmart department and product-type labels, then trained in a two-tower Siamese network with cosine-similarity and triplet losses over natural-language-inference, search-engine, human-labeled, and ad-click datasets — improves both relevance and business metrics for sponsored-search retrieval on Walmart.com relative to the DSSM baseline. The authors report that the model outperforms DSSM by up to 16.10 percent in NDCG@5 offline, reduces the irrelevant-ads rate by up to 4.07 percent, and in online A/B tests increases search-page ad revenue by 5.16 percent, CTR by 7.2 percent, and CPMV by 4.9 percent. They also report that removing hard negatives degrades NDCG by 3.6 percent, evidence that the hard-negative labeling scheme is load-bearing.","pith_inferences":["The click-count pseudo-labeling is a candidate for position-bias correction; the paper provides no analysis of display position, brand, or price effects, so a testable extension is to re-label the 12 million pairs with position-adjusted click weights and compare NDCG.","The sigmoid weighting equation $w_i = 1/(1+\\exp(10x_i - 5))$ is a specific parameterization; the steepness 10 and offset 5 are not experimentally justified, so the same framework might perform differently with other coefficients.","The category-pretraining-plus-two-tower recipe likely transfers to other large e-commerce marketplaces with hierarchical taxonomies, but the paper only demonstrates it on Walmart data, so a cross-platform replication would clarify how much of the gain is Walmart-specific."],"forward_implications":["The two-tower BERT model was deployed as the production ads-retrieval model on Walmart.com in June 2023, so the reported gains reflect live-traffic behavior, not just offline simulations.","Retrieval-stage relevance gains partially propagate through the funnel: online NDCG on final ad placements improved by roughly 9-12 percent, smaller than the offline retrieval gains because ranking and re-ranking follow retrieval.","Hard-negative mining from search logs (items displayed but rarely clicked) contributes a measurable 3.6 percent NDCG gain, so retrieval pipelines should construct such negatives explicitly.","The dynamic fusion weighting rule offers a reusable way to balance multiple training domains without manual curriculum design, by down-weighting domains where human-labeled nDCG is already high."],"supporting_citations":[{"why":"Supplies the BERT architecture whose embeddings the whole pipeline builds on.","marker":"[3]"},{"why":"Defines the DSSM baseline model and the clickthrough learning paradigm that the paper's approach replaces and outperforms.","marker":"[6]"},{"why":"Documents Walmart's prior organic-search semantic retrieval model, which the paper says did not improve the advertising system and thus motivated this work.","marker":"[13]"},{"why":"Provides DistilBERT, the compact BERT variant used as the encoder backbone for efficiency.","marker":"[15]"},{"why":"Supplies the first natural-language-inference dataset (570k pairs) used in stage-two progressive fusion training.","marker":"[1]"},{"why":"Supplies the MultiNLI corpus that adds genre diversity to the NLI training domain.","marker":"[19]"},{"why":"Provides the approximate nearest-neighbor library used to simulate retrieval in offline evaluation.","marker":"[9]"},{"why":"Defines NDCG, the primary relevance metric used in both offline and online evaluation.","marker":"[8]"}],"fun_headline_variants":["Walmart ads retrieval: BERT pipeline boosts relevance 16%, revenue 5.16%","Progressive BERT training lifts Walmart ad relevance by 16%","Walmart's semantic ad retriever: 16% relevance lift, 5.16% revenue gain","Multi-domain BERT beats DSSM in Walmart ad retrieval tests","Walmart's progressive multi-domain BERT lifts ad revenue 5.16%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ad-log training data are labeled by treating an item as relevant if it attracts many clicks in a designated period, so the entire 12-million-pair training signal assumes clicks are a truthful proxy for topical relevance; position bias, brand effects, and price sensitivity could distort that signal.","fun_headline_variants_meta":{"raw":{"variants":["Walmart ads retrieval: BERT pipeline boosts relevance 16%, revenue 5.16%","Progressive BERT training lifts Walmart ad relevance by 16%","Walmart's semantic ad retriever: 16% relevance lift, 5.16% revenue gain","Multi-domain BERT beats DSSM in Walmart ad retrieval tests","Walmart's progressive multi-domain BERT lifts ad revenue 5.16%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000829,"raw_usage":{"total_tokens":3613,"prompt_tokens":931,"completion_tokens":2682,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":2574}},"tokens_in":547,"tokens_out":2682,"duration_ms":16688,"temperature":1.0,"reasoning_tokens":2574,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T22:40:29.160553+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same pipeline with the 12 million ad-log pseudo-labels replaced by human relevance labels on a matched subsample, and run the offline approximate-nearest-neighbor retrieval evaluation: if the NDCG@5 improvement over DSSM remains around 16 percent, then click-count pseudo-labeling is not what produces the reported relevance gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DSSM baseline model and the clickthrough learning paradigm that the paper's approach replaces and outperforms."}],"review_version":1}