{"id":"9fe5c1a0-c0ab-48b2-863b-cfd294197dc4","arxiv_id":"2505.08182","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Demoting semantically duplicate queries in typeahead suggestions, using embedding similarity, improved add-to-cart rates and reduced null page views in Walmart A/B tests.","lead":"This paper describes a method for e-commerce query autocomplete that pushes semantically similar suggestions lower in the list using language-model embeddings. In a Walmart A/B test, the change increased add-to-cart rates and reduced null page views, though experiment details and statistics are not given.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5.3's O(n) sequential demotion is under-specified: the only exact reading is O(n^2), and the O(n) reading leaves non-adjacent semantic duplicates, so the A/B result may not correspond to the described algorithm.","rationale":"The reader's stated weakest assumption is the manual threshold delta and its generalizability. That is a legitimate external-validity concern, but the A/B test itself is within the setting where the threshold was tuned, so it does not directly undermine the central empirical claim. A more load-bearing internal issue is the algorithm identity in Section 5.3: the paper's unique contribution is runtime demotion under latency constraints, and the claimed O(n) optimization is neither precisely specified nor proven. The two natural formalizations have opposite problems: exact deduplication is O(n^2), while the O(n) variant is not deduplication by the paper's own pairwise criterion. Either way, the deployed system's behavior relative to the described method is unclear, making reproduction difficult. The paper also contains a likely sign error in the cosine-similarity inequality in Section 4 (semantically similar queries should have high, not low, cosine similarity) and an unsupported statement that a non-transitive similarity relation partitions queries into disjoint sets; these are additional clarity issues, but the complexity/dedup mismatch is the most consequential for the central claim. I therefore keep the reader's CONDITIONAL verdict: the positive A/B metrics directionally support the claim, but without a precise algorithmic specification or statistical details the claim cannot be fully verified.","tokens_in":5420,"tokens_out":5951,"duration_ms":67966,"concrete_test":"Implement the Section 5.3 sequential strategy on logged ranked suggestion lists (n=50) with the paper's BERT embeddings and threshold delta, in two variants: (A) compare each new query to every retained suggestion; (B) compare only to the immediately preceding retained suggestion. Report the worst-case and average number of cosine computations per prefix, the fraction of prefixes where a pair with cosine similarity above delta survives in the final list under variant (B), and the diversity/MRR difference between variants (A) and (B). If variant (A) exceeds O(n) comparisons, or variant (B) leaves any surviving similar pairs, the paper's complexity and deduplication claims fail, and the reported A/B effect cannot be attributed to the algorithm as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on runtime demotion, and the feasibility of this runtime step depends entirely on the 'O(n) computation' optimization in Section 5.3. The text says to 'fix the query at position 1, and comparing the next query to it', then 'move on to query 3 and repeat the same process', but it never specifies whether each incoming query is compared only to the immediately preceding retained suggestion or to every higher-ranked retained suggestion. If every higher-ranked retained suggestion is compared, the loop is O(n^2) in the worst case (for n=50, up to 1,225 pairwise cosine computations per prefix), contradicting the claimed O(n) complexity and the latency motivation. If only the immediately preceding retained suggestion is compared, the algorithm is linear but does not implement the paper's own definition of semantic similarity, namely that any pair of queries with cosine similarity above the threshold is considered equivalent; non-adjacent semantically similar queries can survive, so the final list is not actually deduplicated. The paper gives no proof, approximation guarantee, or measurement of 'almost the same amount of query diversity'. Thus the link between the deployed A/B system and the described algorithm is under-specified, and the central reproducibility claim is compromised.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the problem of redundant semantically similar queries in e-commerce query autocomplete. It proposes a runtime de-boosting strategy: precompute BERT embeddings for candidate queries, use a cosine-similarity threshold to detect semantically equivalent suggestions, and demote lower-ranked duplicates instead of removing them from the query index. The authors present a sequential version of the demotion loop that is claimed to reduce complexity from O(n^2) to O(n), and they report an online A/B test in which the feature produced statistically significant increases in search Add-to-Cart rate (+0.34% iOS, +0.69% web), decreases in clicks-to-ATC and null page views, and neutral GMV, orders, and units. The paper also sketches an extension based on maximal marginal relevance.","tokens_in":5713,"tokens_out":5177,"duration_ms":57185,"significance":"If the A/B results are reliable and the algorithm is deployable, this is a practically valuable system contribution to e-commerce query autocomplete. The external, randomized A/B experiment against a control group is a genuine strength, as is the decision to preserve index coverage by demoting rather than deleting queries. The use of precomputed, quantized embeddings with O(1) retrieval is a sensible engineering choice. However, the central evidence is under-reported statistically, the threshold selection is opaque, and the O(n) sequential algorithm is under-specified, so the reproducibility of both the method and its measured impact is not yet established.","major_comments":[{"comment":"The manuscript repeatedly states that the A/B metrics are 'statistically significant' but provides no p-values, confidence intervals, sample sizes, experiment duration, or description of the significance test used. Since the paper's central claim rests on these online metrics, the reader cannot assess whether the reported +0.34% iOS and +0.69% web ATC increases are robust or an artifact of multiple comparisons. Please add a results table with estimates, standard errors, confidence intervals, and the statistical test used, and report the denominator (e.g., number of search sessions) for each metric.","section":"Section 7, Experimental Results"},{"comment":"The sequential comparison strategy is under-specified. If 'comparing the next query to it' means comparing each incoming query only to the immediately preceding retained suggestion, the loop is linear but does not implement the paper's own definition of semantic similarity (any pair above threshold): non-adjacent semantically similar queries can survive in the final list. If 'semantically similar to another query ranked higher' means comparing to every retained higher-ranked suggestion, the worst case is O(n^2) (for n=50, up to 1,225 pairwise cosine computations per prefix), contradicting the claimed O(n) runtime. No approximation guarantee, empirical complexity measurement, or diversity metric is provided for the assertion that the sequential procedure 'achieve[s] almost the same amount of query diversity', so the relationship between the described algorithm and the deployed system is unclear.","section":"Section 5.3, Optimizing the query demotion algorithm"},{"comment":"The threshold δ is 'chosen by manually analyzing query pairs and their cosine similarity scores', but the manuscript does not describe the annotation procedure, the labeled data, the chosen threshold value, the distribution of similarity scores, or any validation on unseen prefixes. Because this single scalar threshold controls every demotion decision, the reported A/B performance cannot be assessed for generalization. Please describe how δ was selected, report a sensitivity analysis, and validate the threshold on held-out query pairs, ideally with inter-annotator agreement if human judgments are involved.","section":"Section 4, Identification of semantically similar queries"}],"minor_comments":[{"comment":"The inequality in the semantic-similarity definition is written as a cosine similarity 'e_i · e_j / (|e_i||e_j|) ≤ δ' while the text refers to 'cosine distance'. If δ is a similarity threshold, the direction of the inequality and the mapping between similarity and distance should be clarified; if δ is a distance threshold, the formula should use 1 - cosine similarity.","section":"Section 4, Identification of semantically similar queries"},{"comment":"The claim that the similarity relation 'makes it possible to partition a set of queries into disjoint sets of semantically similar queries' is inconsistent with the immediately preceding statement that the relation is not transitive. Non-transitive relations do not induce partitions; for example, q_i ~ q_j and q_j ~ q_k without q_i ~ q_k violates the equivalence-class property. This terminology is used again in Section 5.1 when defining the sets M_i, so the definition should be corrected.","section":"Section 4, Identification of semantically similar queries"},{"comment":"The demotion step places a duplicate at 'a very low position in the list, such as position 20' and 'substituted with a query suggestion from a lower position', but the mechanics of this substitution are not specified: what happens when multiple demotions occur, and how is the ordering of the substituted suggestions determined?","section":"Section 5.2, Demoting semantically similar queries from the rerank set"},{"comment":"The manuscript says the authors 'relied upon extensive human evaluation' to balance query diversity and expected engagement, but gives no details on the evaluation protocol, number of annotators, number of query-prefix samples, or how the human judgments were reconciled with the threshold and the A/B settings.","section":"Section 6, Evaluation Strategy"},{"comment":"Equation (1) defines the score using ATC, item clicks, and impressions, but the text says the weights a, b, c are learned from 'ATC, orders, clicks and impressions of the past 50 week data'; the role of orders is not reflected in the equation and should be clarified.","section":"Section 2, Background"},{"comment":"The paper refers to 'Figure 2' but no figures appear in the submitted text, and the reference to Figure 2 in Section 5.2 cannot be checked.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper would be more credible as a systems/industry contribution if the authors supply the missing statistical details and resolve the O(n) ambiguity. The external A/B evidence is a point in its favor, but the current manuscript does not support the central runtime and threshold claims as written. I would recommend major revision rather than rejection because the issues appear addressable within the scope of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here’s my take. The paper is a legit applied-industry contribution, not a conceptual breakthrough. What’s genuinely new is the choice to demote semantically similar typeahead suggestions at runtime instead of removing them from the index, which preserves prefix coverage and avoids the cold-prefix failure mode they illustrate nicely. The precomputed quantized BERT embeddings and the measured online A/B results give the work real empirical weight. The reported numbers move in a consistent direction (ATC up, null page views down, GMV/orders neutral) and they honestly report that MRR goes down, which is the expected cost of diversification. That transparency earns credit.\n\nThe soft spots are real but mostly fixable. Section 5.3’s O(n) sequential demotion is under-specified. As written, “fixing the query at position 1 and comparing the next query to it” and then “repeat the same process” doesn’t say whether each new query is compared only to the immediately preceding retained suggestion or to every higher-ranked retained suggestion. If it’s the former, it’s linear but you don’t actually deduplicate non-adjacent similar queries. If it’s the latter, you’re back to O(n^2). The paper gives no proof, no pseudocode, and no measurement of “almost the same amount of query diversity.” So the link between the described algorithm and the A/B result is loose. That’s a load-bearing ambiguity.\n\nTwo smaller things: the similarity definition has the inequality backwards (they write ≤ δ where the surrounding text clearly implies ≥ δ), and the sentence about partitioning into disjoint sets conflicts with their own refusal to use transitivity. Also missing: any statistical detail for the A/B metrics, a description of the human evaluation, and obviously code/data. The threshold δ is hand-picked, which is fine for an industry paper but should be acknowledged more directly as a limitation.\n\nNet: the central empirical claim—that demoting semantically similar suggestions helps a couple of engagement and efficiency metrics without hurting business outcomes—probably holds up. The paper deserves refereeing, but the O(n) algorithm and the statistical reporting need to be tightened before publication. I wouldn’t cite it in my own work, but I’d bring it to a reading group as a case study in how industry papers can be empirically honest while sloppy about algorithmic detail.","headline":"A modest, honest industry paper with a real A/B result, but the O(n) demotion algorithm is under-specified and the similarity definition has a sign error.","tokens_in":6182,"tokens_out":3479,"would_cite":false,"duration_ms":33943,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Semantic look-alikes in query autocomplete should be demoted, not deleted: runtime embedding similarity lifts add-to-cart rates and cuts null page views.","keywords":["semantic deduplication","typeahead","query autocomplete","embedding similarity","cosine similarity","A/B testing","e-commerce search","query diversity"],"falsifier":"Re-run the online A/B test after choosing the threshold $\\delta$ on a held-out validation set of labelled query pairs rather than by hand; if the add-to-cart lift and null-page-view reduction disappear or invert, the manual threshold, not the demotion mechanism, carried the reported effect.","tokens_in":5252,"feed_emoji":"🛒","tokens_out":6701,"duration_ms":63535,"temperature":0.7,"pith_summary":"The paper tries to establish that query autocomplete can be made more useful by demoting semantically redundant suggestions at runtime instead of removing them from the suggestion index. The demotion decision uses cosine similarity between precomputed language-model embeddings of queries; when two suggestions fall on the same side of a hand-chosen threshold, the lower-ranked one is pushed down and replaced by a more distinct query. An online A/B test in e-commerce search supports the claim: statistically significant increases in search add-to-cart rate (+0.34% on iOS, +0.69% on web), a decrease in clicks to add-to-cart on web, and fewer null page views (-1.00% iOS, -4.35% web), with GMV, orders, and units neutral. The paper also argues that the observed offline drop in mean reciprocal rank is expected, because historical logs under-record engagement with newly surfaced, less popular suggestions.","feed_headline":"Demoting redundant autocomplete queries lifts add-to-cart rates","feed_subtitle":"A/B test: demoting look-alike suggestions cut null page views by up to 4.35 percent on web.","key_machinery":"The load-bearing object is the runtime de-boosting step: a third ranking phase after QAC's matching and reranking. For a user's prefix, the system takes the top $n = 50$ suggestions, compares each suggestion's precomputed BERT embedding with the embeddings of higher-ranked suggestions that survived, and if their cosine similarity crosses a threshold $\\delta$, demotes the later suggestion to position 20. The $O(n)$ optimization fixes the suggestion at rank 1, scans downward, and promotes the next candidate into any vacated slot, while embeddings are quantized to 8 bits and stored as base-64 strings so retrieval is $O(1)$. This preserves full query coverage because no suggestion is removed from the index.","core_discovery":"The central claim is that semantically similar query suggestions should be demoted, not deleted. Given a ranked list of autocomplete matches, the system keeps the highest-ranked member of each cluster of equivalent queries, moves the other members to a very low position, and promotes alternative queries from below. Clusters are detected at runtime by comparing 768-dimensional BERT embeddings of queries with a manually chosen cosine-similarity threshold, and the sequential comparison loop is arranged to run in $O(n)$ instead of $O(n^2)$. The online A/B test found a statistically significant rise in search add-to-cart rate (+0.34% iOS, +0.69% web), a decrease in clicks to add-to-cart on web, and a reduction in null page views (-1.00% iOS, -4.35% web), while GMV, orders, and units stayed neutral; the drop in offline MRR is presented as consistent with the intended behavior, since users previously engaged with different paraphrases of the same query on aggregate.","pith_inferences":["Because the threshold is global and hand-picked, one direct extension is to learn $\\delta$ per prefix or category; categories with many paraphrases should tolerate or benefit from different cutoffs.","The proposed maximal-marginal-relevance variant would interpolate the ranking score with embedding distance; a testable prediction is that conversion gains first rise, then fall, as the diversity weight increases.","This pattern suggests reranking for diversity can substitute for expensive index curation in any suggestion system whose underlying retrieval is already relevance-optimized."],"forward_implications":["No query is ever removed from the index, so prefixes that match a demoted query still return suggestions; the loss of coverage that plagues index-side deduplication is avoided.","The demotion step adds diversity at runtime under strict latency budgets, since the comparison loop is $O(n)$ with $O(1)$ embedding lookup.","MRR measured on historical logs will systematically understate the value of the feature, because the newly promoted suggestions lack logged engagement.","If the threshold transfers across markets or categories, the same precomputed embeddings can be reused without retraining the deduplication rule."],"supporting_citations":[{"why":"Supplies the ranking objective that QAC suggestions are optimized for, placing the demotion step after it.","marker":"[1]"},{"why":"Provides the exact-duplicate trie deduplication baseline that the semantic approach extends.","marker":"[2]"},{"why":"Shows precomputed interpretations used for semantic deduplication in another autocomplete domain, which motivates the runtime embedding approach.","marker":"[3]"}],"fun_headline_variants":["Demote look-alike autocomplete queries, lift add-to-cart","Demoting duplicate query suggestions lifts add-to-cart and cuts null views","De-boost similar autocomplete queries to lift conversion and diversity","Semantic de-boosting: fewer duplicate suggestions, more conversions","Demote similar queries in autocomplete to boost ATC and cut null views"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one hand-picked threshold on BERT-embedding cosine similarity cleanly separates queries with the same meaning from queries with different meanings for every prefix and user.","fun_headline_variants_meta":{"raw":{"variants":["Demote look-alike autocomplete queries, lift add-to-cart","Demoting duplicate query suggestions lifts add-to-cart and cuts null views","De-boost similar autocomplete queries to lift conversion and diversity","Semantic de-boosting: fewer duplicate suggestions, more conversions","Demote similar queries in autocomplete to boost ATC and cut null views"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000299,"raw_usage":{"total_tokens":1775,"prompt_tokens":1040,"completion_tokens":735,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":642}},"tokens_in":656,"tokens_out":735,"duration_ms":6137,"temperature":1.0,"reasoning_tokens":642,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:00:55.899740+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the online A/B test after choosing the threshold $\\delta$ on a held-out validation set of labelled query pairs rather than by hand; if the add-to-cart lift and null-page-view reduction disappear or invert, the manual threshold, not the demotion mechanism, carried the reported effect.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ranking objective that QAC suggestions are optimized for, placing the demotion step after it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the exact-duplicate trie deduplication baseline that the semantic approach extends."},{"cited_title":"Semantically Driven Auto-completion","cited_arxiv_id":null,"evidence_quote":"Shows precomputed interpretations used for semantic deduplication in another autocomplete domain, which motivates the runtime embedding approach."}],"review_version":1}