{"id":"662f99ee-d070-40d6-aef3-48142b174a0b","arxiv_id":"2411.14466","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":16,"one_line_summary":"A joint embedding model for conversational product search that learns user, query, item, and conversation representations in one objective and selects clarifying questions with greedy or explore-exploit strategies.","lead":"This paper presents ConvPS, a conversational product search model that learns user, query, item, and conversation representations in one generative framework. It also tests four question-selection strategies (greedy splitting, a bandit, and two Gaussian process variants) and reports large ranking gains over earlier conversational search baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Offline evaluation discards out-of-vocabulary answers as 'invalid', so the reported gains may reflect an idealized user whose answers always lie in the training vocabulary.","rationale":"For the central claim to hold, the reported ranking gains must come from the model's representations and question-selection strategies rather than from the evaluation protocol silently discarding difficult user answers. The paper's own Section 4.1.4 states that OOV answers are ignored, and Section 5 acknowledges this as a limitation. Table 5 quantifies the asymmetry: the random strategy gets 94.1% invalid questions versus 3.0% for LinRel, so the Table 4 comparison is not at equal information. This is more directly load-bearing than the absence of error bars, because even with error bars the protocol would still be biased; and it is more central than the overclaim about ConvPS_random on Movies & TV, because the primary comparison is among the four proposed strategies. The online user study is a genuine mitigating check, but it only evaluates LinRel, reports no OOV/invalid rate, and the reported MAP of 0.249 does not clearly map to any single Table 4 category, leaving the systematic question open. The proposed re-run would settle whether the central claim holds when the invalid-answer filter is removed. If the gains persist under nearest-neighbor OOV mapping, the claim is robust; if not, the paper should be revised to claim effectiveness only for in-vocabulary oracle responses.","tokens_in":27757,"tokens_out":8668,"duration_ms":91266,"concrete_test":"Re-run the Cell Phones and Health experiments of Table 4 and Figure 2 under two alternative answer-handling rules: (i) map every OOV value to its nearest in-vocabulary value embedding by cosine similarity and use it in Eq. 24; (ii) treat an OOV answer as negative feedback q−. Record, for each strategy, the effective number of non-ignored updates over the 5-question horizon. Compute MAP/MRR/NDCG over 5 random seeds with paired permutation tests against AVLEM and HEM. If the gains of ConvPS_LinRel, ConvPS_GBS, or the GP variants over AVLEM and HEM become non-significant or shrink substantially under either rule, the reported superiority is an artifact of the invalid-answer filter.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the offline protocol of Section 4.1.4, where a simulated user answers each question with the exact slot value from the target item, and any answer whose value is outside the training vocabulary is marked 'invalid' and ignored. Since ranking updates in Eq. 24 are applied only for valid (q,a) pairs, a strategy that happens to ask slots with in-vocabulary values receives more effective feedback than one that does not. Table 5 shows this asymmetry: ConvPS_random has 94.1% invalid questions, while ConvPS_LinRel has 3.0%; hence the 5-question comparison in Table 4 is not a comparison at equal effective feedback, and the random control is not a clean null model of question selection. In a real deployment the user's answer still exists even if the value is out of vocabulary; discarding it inflates the apparent value of the learned strategies, which are optimized to select slots whose target-item values are in the training vocabulary. The small online study in Section 4.6 is real evidence but tests only LinRel, and its OOV handling is not reported, so it does not by itself close the gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ConvPS, a conversational product search model in which user, query, item, and conversation (slot-value pair) embeddings are learned jointly by maximizing a factorized generative likelihood with negative sampling. At test time, item ranking is a softmax over a linear combination of user, query, and conversation embeddings (Eq. 24), and clarifying questions are selected by one of four strategies: GBS, LinRel, GP+EI, and GP+UCB. Experiments on three Amazon categories report that the learned strategies substantially outperform static and conversational baselines after five questions, with ablations and a small online user study supporting the main claims.","tokens_in":28121,"tokens_out":9835,"duration_ms":95471,"significance":"If the reported comparisons were clean, the paper would make a solid contribution: it provides a unified generative training objective for conversational product search, a tractable negative-sampling approximation, and a systematic comparison of question-selection strategies, including bandit/GP approaches, with an online sanity check. The strengths are the explicit likelihood factorization, the inclusion of negative feedback through separate slot embeddings, and the attention to explore-exploit trade-offs. However, the headline empirical claims rest on an offline simulation whose invalid-answer rule creates unequal effective feedback across strategies, and this currently tempers the significance of the gains.","major_comments":[{"comment":"In the offline protocol, answers whose values are outside the training vocabulary are marked 'invalid' and ignored for ranking updates. Table 5 shows the asymmetry this creates: ConvPS_random receives only 5.0% positive and 0.9% negative feedback, with 94.1% invalid, whereas ConvPS_LinRel receives 71.0% positive and 26.0% negative, with 3.0% invalid. After five questions, random therefore gets about 0.3 effective feedback rounds and LinRel about 4.85. The comparison in Table 4 and Figure 2 is thus not at equal effective feedback, and the random control is not a clean null model of question selection; the learned strategies may be rewarded for selecting slots whose target-item values are in-vocabulary rather than for asking more informative questions. Please report the comparison conditioned on equal effective feedback (e.g., as a function of number of valid answers), or handle OOV answers with a fallback embedding, and report the OOV rate in the online study (Section 4.6).","section":"Section 4.1.4 and Table 5"},{"comment":"The LinRel update is dimensionally inconsistent as written. With x_q a 1xN row and X an FxN matrix, h_q = x_q (X^T X + lambda I)^{-1} X^T is a 1xF vector, while r is defined as an (l-1)-dimensional vector of received feedback; the inner product h_q . r in Eq. 18 is undefined unless l-1 = F. Moreover, the term c||h_q||/2 is not the LinRel confidence bound, which should be based on x_q^T (X^T X + lambda I)^{-1} x_q. Since ConvPS_LinRel is the best or second-best strategy in Table 4, please correct the equations or state the exact computation used for the reported results.","section":"Section 3.4.2, Eqs. (17)-(18)"},{"comment":"No variance estimates are reported anywhere in the main comparison, and the Fisher random test mentioned in Table 4's footnote is not described (randomization unit, number of permutations, whether it is over test user-query pairs or over re-training runs). The gains on Movies & TV are numerically small (e.g., ConvPS_LinRel MAP 0.044 vs best baseline 0.029), so without error bars the reader cannot assess stability. Please add standard deviations or confidence intervals over multiple runs or bootstrap over test pairs, and describe the significance test.","section":"Table 4 and Figure 2"}],"minor_comments":[{"comment":"The footnote refers to 'significant differences between ConvNF and ConvPS_init', but no model named ConvNF appears in the paper; the intended variant should be identified.","section":"Table 4 footnote"},{"comment":"The sentence claiming that ConvPS_random outperforms the baselines on all three metrics is contradicted by Table 4: on Movies & TV, ConvPS_random achieves 0.022 MAP and 0.023 MRR, below every baseline on those metrics, and its NDCG 0.024 exceeds only PMMN's 0.023. The claim should be restricted to the learned question-selection strategies.","section":"Section 4.2, first paragraph"},{"comment":"The y-axis label shown for panels (b) and (c) is 'MAP' even though the captions identify those panels as MRR and NDCG; the axis labels should be corrected.","section":"Figure 3"},{"comment":"The batch-size tuning list is given as '[64, 128, 256, 612]', while Figure 5 shows 512 as the largest value; one of these is a typo and should be fixed.","section":"Section 4.1.6"},{"comment":"The claim that the online user study results are 'in agreement' with Table 4 is hard to verify because Table 6 aggregates over categories and does not report the distribution of target categories or the OOV rate for user answers; please provide these details.","section":"Section 4.6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a resubmission of an older TOIS-style paper, and the central model idea is reasonable. The main risk is not the model design but the evaluation protocol: the invalid-answer rule and the oracle construction of answers should be addressed head-on, ideally by rerunning all strategies with a fallback for OOV values and reporting valid-feedback-normalized curves. I would also ask the authors to provide the exact LinRel implementation, given the equation inconsistency in Section 3.4.2."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the contribution is real: a unified generative objective (Eq. 13) that ties user, query, item, and conversation embeddings together, plus a systematic comparison of four question-selection strategies—GBS, LinRel, GP-UCB, and GP-EI. Prior conversational product search either modeled conversations separately or relied on lexical overlap, so this is a useful step for the subfield. Second, the empirical claims are plausible but the offline evaluation is less clean than the abstract suggests. The simulated user answers from the target item's slot values, and any answer outside the training vocabulary is dropped as invalid. That asymmetry inflates the gap between the learned strategies and the random control.\n\nWhat the paper does well: the derivation in Eqs. 10–15 is coherent, negative sampling is standard, and the four strategies beat the baselines on all three categories with large margins. The ablations show the slot-value language model and the non-conversational term both matter. The small online user study is genuine evidence that real users will answer the questions, and LinRel's offline behavior roughly transfers, though it only tests one strategy.\n\nSoft spots: the OOV handling is the load-bearing issue. Table 5 shows ConvPS_random has 94.1% invalid questions versus 3.0% for LinRel. Since only valid answers update the ranking, the 5-question comparison in Table 4 is not a comparison at equal effective feedback. A real user's OOV answer still carries information; discarding it punishes strategies that ask rare slots. The paper explicitly acknowledges this limitation in Section 5, so it is not hidden, but the headline claim depends on the protocol. Also, Table 4 has no variance estimates, and the text overclaims for ConvPS_random on Movies & TV—Table 4 shows it below LSE and HEM on all three metrics there. These are fixable with a re-analysis that reports valid-question counts and equal-feedback curves.\n\nThe citation pattern is fine: prior work in this exact line is cited, and the self-citations are to relevant earlier models, not padding. This is a solid incremental advance in an active subfield, not a breakthrough. Researchers in conversational product search and interactive IR will get value from it. A serious referee should engage with it, and the right outcome is likely major revision rather than rejection.","headline":"A genuinely unified generative model for conversational product search with a sensible question-strategy comparison, but the offline simulator discards out-of-vocabulary answers, which inflates the reported gains.","tokens_in":28615,"tokens_out":2601,"would_cite":true,"duration_ms":25099,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ConvPS jointly learns user, query, item, and conversation embeddings in one generative model, and with five clarifying questions it ranks target products far above four baselines on three e-commerce categories.","keywords":["conversational product search","representation learning","learning to ask","slot-value pairs","question selection","explore-exploit","personalized product search","e-commerce retrieval"],"falsifier":"Re-run the Cell Phones & Accessories evaluation with human-like answers that are correct only 77 percent of the time, the accuracy the paper's own user study observed, and map off-vocabulary values to a random training value instead of discarding them; if ConvPS no longer beats AVLEM at five questions, the central claim is falsified.","tokens_in":27605,"feed_emoji":"🛒","tokens_out":4844,"duration_ms":45957,"temperature":0.7,"pith_summary":"This paper tries to show that a shopping assistant can zero in on a desired product by asking a sequence of clarifying questions, provided the system jointly learns what users, queries, products, and conversations mean. The proposed model, ConvPS, embeds all four in the same latent space through a single generative objective, so each answer moves the target item up the ranking. With several question-selection strategies, one greedy and three explore-exploit, the paper reports that five questions roughly double retrieval quality on three product categories and beat both static product-search baselines and earlier conversational models. The underlying claim is that conversation itself carries retrieval value that static personalized search leaves on the table.","feed_headline":"Five clarifying questions roughly double retrieval quality","feed_subtitle":"Joint user-query-item-conversation embeddings beat static and conversational baselines on three product categories.","key_machinery":"The load-bearing object is the joint training objective of Equation 13, a log-likelihood that sums six terms: the initial non-conversational item probability, the conversational item probability given user, query, and current slot-value feedback, item and user language models over words, and item and user language models over slot-value pairs. Conversation is represented as slot-value embeddings; positive answers add (q+a)/2 to the ranking sum, while negative answers add a separately learned slot embedding q−. On top of this, four question selectors decide which slot to ask next: GBS splits the estimated preference mass, and LinRel, GP+UCB, and GP+EI use upper-confidence or expected-improvement criteria to balance exploring new slots against exploiting known ones. The machinery works by making every piece of evidence, who is asking, the initial query, and each answer, comparable in the same vector space, so updating the ranking is a simple addition in that space.","core_discovery":"On the paper's own terms, the central discovery is that treating a user's answers as another embedding term in a unified generative model, rather than modeling conversations separately or relying on lexical overlap, lets retrieval improve sharply as questions accumulate. The probability of an item given the user, the initial query, and accumulated slot-value feedback is a softmax over the sum of their embeddings, and all embeddings are trained together from item descriptions, reviews, and historical slot-value feedback. The reported numbers show the LinRel question-selection variant, which balances exploration and exploitation, reaching the best overall results, and the gains grow as more questions are asked. The same pattern appears in a small online user study with real crowd workers, where users answered an average of nine questions and most said they were willing to answer at least five.","pith_inferences":["The oracle-answer simulator likely inflates the reported gains, because answers outside the training vocabulary are dropped instead of interpreted; with noisy or open-vocabulary human answers, the advantage of learned question selection may shrink.","The representation-learning recipe could be reused for any structured item metadata, such as facets, categories, or price ranges, not only aspect-value pairs extracted from reviews, a possibility the paper itself notes.","A testable extension is to replace the additive slot-value composition in Equation 7 with a learned combiner or nonlinear projection and measure whether retrieval gains persist.","The paper's own user study, where users were correct 77 percent of the time, suggests that modeling answer uncertainty rather than only binary positive or negative feedback is the next bottleneck."],"forward_implications":["If the central claim is correct, conversational product search can be built by extending a static embedding retriever with a trainable conversation term, without needing a separate dialogue policy or heavy natural-language understanding.","Asking questions selected by explore-exploit strategies should outperform random or purely greedy question orderings, and the gap should widen as conversation length grows.","Negative feedback expressed as 'not relevant' is not wasted signal: learning a dedicated embedding for it contributes to ranking quality.","The same jointly learned embeddings can serve both retrieval and question choice, so the two modules can be trained once and used together.","The reported gains across large and small catalogs suggest the method transfers to product collections of different scale."],"supporting_citations":[{"why":"Supplies the hierarchical embedding model HEM, a static personalized product-search baseline that must be beaten, and the embedding-based generative modeling approach that ConvPS extends.","marker":"[3]"},{"why":"Provides AVLEM, a state-of-the-art conversational product search baseline using negative feedback, and the simulation approach for constructing slot-value conversations.","marker":"[9]"},{"why":"Provides the LSE latent vector space model for product search, one of the static baselines, and the query-embedding construction idea.","marker":"[63]"},{"why":"Provides PMMN, the first conversational product search model asking questions over aspect-value pairs, a baseline the paper must outperform.","marker":"[78]"},{"why":"Supplies a question-based sequential Bayesian product search baseline and prior work on learning to ask in product search.","marker":"[85]"},{"why":"Defines Generalized Binary Search, the greedy question-selection strategy adapted and compared in this paper.","marker":"[48]"},{"why":"Defines the LinRel bandit algorithm used for the explore-exploit question-selection strategy.","marker":"[7]"},{"why":"Supplies the Amazon product dataset from which the three experimental categories are drawn.","marker":"[45]"}],"fun_headline_variants":["Five clarifying questions roughly double retrieval quality","Asking users questions improves conversational product search","Joint user-query-item-conversation embeddings lift retrieval","Learn to ask: conversational search with unified representation","Conversational search via a single learned representation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes an oracle user who has a specific target item in mind, answers each question with the exact slot value taken from that item, and says 'not relevant' when the slot is missing, while answers outside the training vocabulary are discarded rather than interpreted.","fun_headline_variants_meta":{"raw":{"variants":["Five clarifying questions roughly double retrieval quality","Asking users questions improves conversational product search","Joint user-query-item-conversation embeddings lift retrieval","Learn to ask: conversational search with unified representation","Conversational search via a single learned representation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000706,"raw_usage":{"total_tokens":3179,"prompt_tokens":936,"completion_tokens":2243,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":2174}},"tokens_in":552,"tokens_out":2243,"duration_ms":16745,"temperature":1.0,"reasoning_tokens":2174,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:20:43.843491+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Cell Phones & Accessories evaluation with human-like answers that are correct only 77 percent of the time, the accuracy the paper's own user study observed, and map off-vocabulary values to a random training value instead of discarding them; if ConvPS no longer beats AVLEM at five questions, the central claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LSE latent vector space model for product search, one of the static baselines, and the query-embedding construction idea."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides PMMN, the first conversational product search model asking questions over aspect-value pairs, a baseline the paper must outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a question-based sequential Bayesian product search baseline and prior work on learning to ask in product search."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Generalized Binary Search, the greedy question-selection strategy adapted and compared in this paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Amazon product dataset from which the three experimental categories are drawn."}],"review_version":1}