Wizard of Shopping: Target-Oriented E-commerce Dialogue Generation with Decision Tree Branching

Besnik Fetahu; Jason Ingyu Choi; Nikhita Vedula; Oleg Rokhlenko; Shervin Malmasi; Xiangci Li; Zhiyu Chen

REVIEW 2 major objections 4 minor 13 references

Wizard of Shopping: Target-Oriented E-commerce Dialogue Generation with Decision Tree Branching

T0 review · 2 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TRACER generates realistic e-commerce dialogues by grounding LLM verbalization in decision-tree search trajectories, producing the 3,600-conversation WoS dataset and downstream gains in query generation and ranking.

desk verdict Useful new dataset and a sensible generation pipeline, but the optimality guarantee and closed-loop downstream evaluation need work. read the letter →

arxiv 2502.00969 v1 pith:VNPOGMVD submitted 2025-02-03 cs.CL

Xiangci Li , Zhiyu Chen , Jason Ingyu Choi , Nikhita Vedula , Besnik Fetahu , Oleg Rokhlenko , Shervin Malmasi This is my paper · ORCID

classification cs.CL

keywords conversationalproductsearche-commercedialoguegenerationdecisiontreeplanninglargelanguagemodelssyntheticdatasetWizardofShoppingqueryranking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Conversational product search needs many realistic shopper–assistant dialogues, but human-annotated data are tiny and older simulated sets use rigid templates. The paper argues that the gap can be filled by having large language models verbalize a dialogue plan produced by a decision tree: at each step the tree selects the product attribute that best splits the remaining candidate products, and the sampled customer preference determines which branch to follow. The resulting Wizard of Shopping (WoS) dataset contains 3,600 conversations across three e-commerce domains, scored by human raters as natural and coherent. Training a query generator and a product ranker on WoS improves both tasks over baselines that feed the raw conversation history. The decision tree is what turns open-ended dialogue generation into a controllable, target-oriented planning problem.

What carries the argument

The load-bearing object is the decision-tree dialogue plan: a sequence of (aspect, value, interest) tuples obtained by fitting a decision tree at each search step and traversing it along the customer's sampled preferences. Because the tree's splits maximize purity over the currently retrieved product set, the plan asks about attributes that most reduce the candidate space, which the paper claims reaches a relevant product family in the fewest search conditions. The LLM handles only verbalization: converting the plan into fluent dialogue, locally reordering attributes for naturalness, adding up to three typical values as hints, and explaining unfamiliar aspects.

What would settle it

Run the WoS-trained query generator and product ranker on a human-written e-commerce dialogue set with known target products; if their query F1 and ranking MRR on those human conversations are no better than the untrained baselines, the claim that WoS teaches real conversational product search is falsified.

Watch

Extended reading notes

Core claim

TRACER samples a product from a real catalog, labels each of its aspect-value pairs as wanted, unwanted, or optional, and then runs a search-and-plan loop. In each loop a rule-based searcher retrieves the products consistent with the preferences revealed so far; a decision tree is fit to that candidate set using the products' aspect-value vectors as features, and traversing the tree along the sampled preference yields the next product aspect to bring up in dialogue. The customer agent and seller agent (or, in the simpler single-pass mode, one scriptwriter LLM) verbalize this plan into a natural conversation, with the seller offering frequent values as hints and the customer allowed to ask what an aspect means. Single-pass generation with the stronger LLM receives human scores of 4.3/5 for realism and 4.7/5 for coherence, and the released WoS dataset of 3,600 conversations supports downstream systems: a fine-tuned Longformer encoder-decoder reaches 0.834 exact-F1 and word-overlap ranking with generated queries reaches 0.838 MRR, versus 0.162 for the raw-history baseline.

Load-bearing premise

The reported downstream gains are measured on held-out conversations produced by the same pipeline, so the load-bearing assumption is that query generators and rankers trained on this synthetic dialogue will transfer to real human shopping conversations.

Editorial extensions

If this is right

If TRACER works, any indexed product catalog is enough to generate large volumes of target-oriented shopping dialogue with no human annotation, removing the main bottleneck for training conversational product search assistants.
A WoS-trained query generator can compress verbose dialogue into a compact query holding the product category and the wanted, unwanted, and optional features, which is what makes downstream word-overlap ranking strong.
The same pipeline extends to new shopping domains simply by indexing a product catalog, unlike human-annotated sets that require fresh curation and annotator training.
The paper's comparison supports single-pass generation from a global plan over turn-by-turn interactive generation, since the single-pass setting yields better human scores on realism, conciseness, coherence, and naturalness.
Conversations converge to a product family rather than an exact item, so the dataset fits tasks that need a target-oriented search trajectory rather than open-ended shopping chit-chat.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: The shortest-path guarantee is relative to the attributes present in the product catalog and assumes the simulated customer answers every question accurately; noisy or missing attributes, or shoppers who skip answers, would weaken the guarantee in deployment.
Editorial inference: The downstream improvements may partly reflect internal consistency, since the query generator and ranker are trained and evaluated on conversations produced by the same preference-to-dialogue pipeline; measuring on human-human dialogues with gold product targets would show how much transfers.
Editorial inference: The decision-tree planning idea transfers to other diagnostic settings in which an agent must identify an item in a large catalog by asking the fewest questions, such as technical troubleshooting or product support.
Editorial inference: A testable extension would be to use WoS as few-shot seed material for a much smaller generation run and measure whether human naturalness scores remain high, since the current release uses 3,600 conversations.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper has a genuinely useful new artifact and a clean generation method. TRACER's idea of grounding LLM verbalization in decision-tree search trajectories, especially with single-pass generation, is a new combination and worth taking seriously. The WoS dataset of 3,600 conversations across three e-commerce domains fills a real gap in conversational product search, and the human evaluations support that the conversations are natural (4.2/5.0 naturalness, 4.7/5.0 coherence for GPT-4 single-pass). The authors also do honest error analysis on their own generation, which I credit.

The soft spots are real but not fatal. First, the abstract's claim that the decision tree 'guarantees relevant product discovery in the shortest number of search conditions' is too strong. The tree is fit greedily, the search space changes as preferences are revealed, and no optimality proof is offered. That should be toned down to a greedy heuristic. Second, the downstream evaluation is circular in a way the paper does not address. CQG literally extracts the preference tuple that was fed to the generator, and the test set comes from the same TRACER pipeline. The high F1 and MRR are expected if the extractor learns the generator's verbalization patterns; they do not demonstrate transfer to human-human dialogues. Given the paper's own comparison shows WoS differs sharply from MG-ShopDial in intent distribution and utterance style, an external transfer test is needed before claiming effectiveness. To their credit, the ethical statement does flag the risk of biased simulation, and the limitations section mentions catalog noise, but the downstream claims still overreach. The promise to release code and data only upon acceptance also makes the resource contribution conditional.

This is a serious paper, not a slick one. The thinking is clear, the method is described in enough detail to reproduce, and the dataset, if released, will be useful. I would send it to peer review with the expectation of substantial revision: temper the guarantee, add at least one experiment on MG-ShopDial for CQG and CPR, and commit to artifact release. I'd cite it if the artifacts ship.

My recommendation: engage with it seriously, but do not let the current claims pass without those changes.

Referee Report

2 major / 4 minor

Summary. The paper presents TRACER, a method for generating target-oriented e-commerce shopping dialogues by combining (i) sampling of customer preferences from real product metadata, (ii) decision-tree-based dialogue planning that selects product aspects to discuss, and (iii) LLM verbalization of the resulting plan into a conversation between a customer and a seller. The authors release Wizard-of-Shopping (WoS), a dataset of 3,600 GPT-4 single-pass generated conversations across three product domains. Human evaluations indicate that GPT-4 single-pass dialogues are rated highly on realism, conciseness, coherence, and naturalness. The paper further reports that fine-tuning models on WoS improves conversational query generation (CQG) and conversational product ranking (CPR) on held-out WoS conversations, and it compares WoS with the human-generated MG-ShopDial dataset. The central claims are that the decision-tree plan guarantees product discovery in the shortest number of search conditions and that WoS is effective for real-world conversational product search.

Significance. If the dataset and method hold up, WoS would be a practical resource for a field where public e-commerce dialogue data is extremely scarce; the reported human evaluation shows that GPT-4 single-pass generation produces coherent and natural-looking conversations, and the authors are transparent about error types and about differences from MG-ShopDial. The strongest parts of the paper are the careful two-level human evaluation protocol with inter-annotator agreement, the explicit error analysis, and the reproducible-sounding generation pipeline that ties LLM verbalization to structured plans. However, the two headline results are currently in question: the optimality guarantee for the decision-tree plan is not proven, and the downstream CQG/CPR evaluation is a closed loop in which training and test dialogues are produced by the same generator from the same preference tuples that the downstream task is asked to recover. The significance of the work would be considerably higher if the decision-tree claim were weakened or proved and if at least one external transfer evaluation on human-human data were provided.

major comments (2)

[§3.2 / Abstract] The claim that the decision-tree plan 'guarantees relevant product discovery in the shortest number of search conditions' is not supported. Algorithm 1 greedily selects the attribute with maximal split of the current candidate set and refits a tree at every turn; greedy split maximization is not globally optimal for minimizing the number of conditions needed to reach a product family, and no proof or counter-analysis is provided. Because this guarantee is stated in the abstract and Introduction as a central advantage over prior work, it should either be proved, replaced with a weaker 'heuristic/effective' claim, or tested against an exact or global baseline.
[§5.1, Eq. (2), Tables 6-7] The downstream evaluation is circular and does not, by itself, support the abstract's effectiveness claim. CQG is explicitly described as 'a reverse task of the LLM verbalization,' and both CQG and CPR are trained and tested on held-out WoS conversations produced by the same TRACER/GPT-4 pipeline from the same preference tuples that CQG must recover. The high F1 and MRR values are therefore largely expected if the extractor learns the generator's verbalization style; they do not establish that WoS improves real conversational product search. No transfer experiment is reported on MG-ShopDial or any human-human dialogue set, despite the large distributional differences documented in §4.6. I request an external transfer evaluation or a substantial reframing of the downstream contribution as an internal-consistency check.

minor comments (4)

[§4.2] Please clarify the exact count of evaluated dialogues; the text moves from 60 conversations for inter-annotator agreement to 108 total, and the arithmetic (5 groups × 3 judges × 12 examples, plus 2 judges × 24 examples) is not explained.
[Table 7] Hit@10 and Hit@100 are both reported as 0.903 for D2Q2P (GPT-4); if this is not a typo, please explain why the two metrics coincide.
[Algorithm 2] The stopping condition 'until |Po| has not converged' is undefined; specify the convergence criterion so Table 5's average of 2.2 searches per conversation can be reproduced.
[§3.1] The text uses 'aspects' and 'features' nearly interchangeably in places; pick one consistent term and define the relation between aspect-value pairs and product features.

Circularity Check

2 steps flagged · score 6.0 of 10

Downstream 'effectiveness' experiments are a closed loop: CQG is the explicit inverse of TRACER's LLM verbalization and is evaluated only on WoS, so the reported gains measure recovery of generation inputs rather than transfer to real user behavior.

self definitional [Section 5.1, Conversational Query Generation (CQG), around Eq. (2)]
"And in fact, this is a reverse task of the LLM verbalization where we extract user preferences from the shopping dialogues."

The CQG gold output is exactly the preference tuple [PC, Wanted, Unwanted, Optional] that TRACER samples from a product and feeds into the GPT-4 prompt to produce each WoS dialogue (§3.1, §3.3). Training and testing on held-out WoS therefore evaluates how well the extractor inverts the generator's prompt patterns; high F1/ROUGE in Table 6 is a round-trip consistency result, not evidence that the model would work on human-human e-commerce dialogues. The paper's 'effectiveness' claim for WoS is thus demonstrated on a task whose target is the generation input.
self definitional [Section 5.2.2, CPR results, discussion of Table 7]
"This is because the product representations include feature names that are lexically similar to the gold queries. Consequently, BM25 exhibits a strong performance in this ranking task."

The gold queries are the preference tuples sampled from the target product's own aspect-value pairs (§3.1: 'The wanted aspect-value pairs come directly from p'), and the product index is built by concatenating each product's title and aspect-value pairs (§5.2). Therefore the target product's indexed text contains the query terms by construction, so ranking it first with BM25 is a lexical match on the generation input, not a learned conversational ranking ability. The D2Q2P pipeline is a closed loop: preference → dialogue → extracted preference → product whose features defined the preference.

full rationale

The circularity is confined to the downstream-task evidence. TRACER's dialogue generation itself is grounded in a decision-tree search trajectory and is evaluated by human judges on naturalness and coherence, which is independent of the circular loop; however, the paper's third contribution—'demonstrate the utility of WoS by showing improvements in conversational query generation and product ranking'—rests on WoS-only evaluation where the CQG targets are the generative inputs. The paper explicitly calls CQG 'a reverse task of the LLM verbalization,' and the CPR index is lexically aligned with those same inputs, so the reported MRR/F1 gains reduce to internal consistency. No transfer experiment on MG-ShopDial or any human dialogue set is reported, and the Section 4.6 comparison shows large distributional differences (e.g., 34.2% vs. 9.3% Elicit preferences intent, 22.8 vs. 8.5 words per utterance), so external validity is unverified. Because the dataset-release and human-evaluation contributions remain independent, the overall circularity is partial rather than total.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on two main assumptions: that greedy decision-tree splits give an optimal questioning order, and that LLM verbalization faithfully expresses the plan. Both are only partially validated. The method has three hand-set hyperparameters (hint count, post-convergence turns, preference sampling distribution) and inherits noise from the product catalog. No new entities are introduced.

free parameters (3)

Number of typical values per aspect hint = 3
In §3.4, the seller is instructed to provide up to 3 most frequent values as hints. This is a hand-chosen hyperparameter that affects conversation naturalness but is not derived from data or theory.
Maximum post-convergence turns = 3
In §3.4, after search converges, the conversation continues for up to three more turns to conclude. This is an arbitrary design choice that shapes the dataset.
Preference sampling distribution = not specified
In §3.1, customer interest values (wanted, unwanted, optional) are randomly assigned, but the probabilities are not stated. This controls the balance of the dataset and is not justified.

assumptions (4)

domain assumption Minimizing user effort in web search is a valid objective for conversational product search, and customers prefer fewer questions.
Invoked in §3.2 citing Al-Maskari and Sanderson (2010); this transfers an IR finding to CPS without direct evidence.
ad hoc to paper A greedy decision tree that picks the attribute with maximal split of the current candidate set yields the shortest sequence of search conditions.
Assumed in §3.2 and the abstract; no proof or counterexample analysis is given, and greedy tree induction is not guaranteed to minimize the expected number of questions.
domain assumption LLM agents will verbalize the planned aspect-value-interest tuples without inventing unassigned features or switching preference types.
The entire pipeline depends on this, but the paper's own error analysis (§4.5) documents cases where LLMs invent features or fail to follow scripts.
domain assumption The cleaned product catalog attributes are accurate enough that dialogues grounded in them are realistic.
Stated in §4.1 and in Limitations (quality of conversations relies on quality of catalog); noisy attributes propagate into dialogues.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Wizard of Shopping: Target-Oriented E-commerce Dialogue Generation with Decision Tree Branching." pith.science (2026). https://pith.science/paper/VNPOGMVD

@misc{pith2026250200969,
  author       = {Pith},
  title        = {Pith review of: Wizard of Shopping: Target-Oriented E-commerce Dialogue Generation with Decision Tree Branching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VNPOGMVD}},
  note         = {Machine review of arXiv:2502.00969}
}

read the original abstract

The goal of conversational product search (CPS) is to develop an intelligent, chat-based shopping assistant that can directly interact with customers to understand shopping intents, ask clarification questions, and find relevant products. However, training such assistants is hindered mainly due to the lack of reliable and large-scale datasets. Prior human-annotated CPS datasets are extremely small in size and lack integration with real-world product search systems. We propose a novel approach, TRACER, which leverages large language models (LLMs) to generate realistic and natural conversations for different shopping domains. TRACER's novelty lies in grounding the generation to dialogue plans, which are product search trajectories predicted from a decision tree model, that guarantees relevant product discovery in the shortest number of search conditions. We also release the first target-oriented CPS dataset Wizard of Shopping (WoS), containing highly natural and coherent conversations (3.6k) from three shopping domains. Finally, we demonstrate the quality and effectiveness of WoS via human evaluations and downstream tasks.

Figures

Figures reproduced from arXiv: 2502.00969 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages

[1]

It’s also generally resistant to dirt and easy to clean

**Nylon**: This is a type of synthetic fabric that’s known for its strength and durability. It’s also generally resistant to dirt and easy to clean

work page
[2]

These types are lightweight, and plastic ones are particularly good for water resistance

**Polycarbonate or Plastic**: Some organizers are designed with a hard exterior for added protection. These types are lightweight, and plastic ones are particularly good for water resistance

work page
[3]

They offer a sleek, stylish look while also providing a good level of protection

**Leather**: There are also organizers made from leather or PU leather. They offer a sleek, stylish look while also providing a good level of protection. Each material comes with its own advantages. Nylon is usually longer-lasting, polycarbonate or plastic is lighter and waterproof, and leather gives a classy touch. Do any of these materials align with yo...

work page
[5]

Springer

Proceedings 31, pages 796–800. Springer. Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu, and Raviteja Anantha. 2021. Question rewriting for conversational question answering. In Proceedings of the 14th ACM international conference on web search and data mining, pages 355–363. Nikhita Vedula, Marcus Collins, Eugene Agichtein, and Oleg Rokhlenko. 2022. Wha...

arXiv 2023
[6]

model number

The meaning of each of the scores is interpreted in Table 8. Realism. To distinguish counter-intuitive conver- sations, realism examines whether the conversa- tion is likely to happen in the real world from a logical perspective. We also consider Factuality, which captures the commonsense-level factual er- rors, since they essentially reflect similar crit...

work page 2023
[11]

It’s a great choice with several compartments to store your items such as a 9.7 inch iPad, Kindle, and Power Adapter

The first is the [BAGSMART Universal Travel Cable Organizer](https://www.amazon.com/BAGSMART- Universal-Organizer-Electronics-Accessories/dp/B017SKRWL4). It’s a great choice with several compartments to store your items such as a 9.7 inch iPad, Kindle, and Power Adapter. It’s black with a hint of red color that gives it an edgy look

work page
[12]

It’s also a black, splashproof and waterproof travel cable organizer that should serve your needs well

The second option is [Inateck Electronics Organizer](https://www.amazon.com/Inateck-Electronics-Organizer- Splashproof-Waterproof/dp/B07ZV2CZSR). It’s also a black, splashproof and waterproof travel cable organizer that should serve your needs well

work page
[13]

microfiber

The third one is [Luxtude Electronics Organizer](https://www.amazon.com/Luxtude-Electronics-Organizer- Portable-Essentials/dp/B086M63FXV). This Travel Cable Case is a portable tech bag perfect for cable storage, cord storage and storing electronic accessories/USB/SD/Charger. It’s also black, aligning with your color preference. Please let me know if you n...

work page

Show all 13 references

[1996]

In Advances in Database Technology—EDBT’96: 5th International Conference on Extending Database Technology Avignon, France, March 25–29, 1996 Proceedings 5, pages 18–32

Sliq: A fast scalable classifier for data mining. In Advances in Database Technology—EDBT’96: 5th International Conference on Extending Database Technology Avignon, France, March 25–29, 1996 Proceedings 5, pages 18–32. Springer. Duangmanee Putthividhya and Junling Hu. 2011. Bo...

1996
[2009]

In Advances in Information Retrieval: 31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9,

An unsupervised approach to product at- tribute extraction. In Advances in Information Retrieval: 31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9,

2009
[2018]

Advances in neural information processing systems, 31

Towards deep conversational recommenda- tions. Advances in neural information processing systems, 31. Siheng Li, Cheng Yang, Yichun Yin, Xinyu Zhu, Zesen Cheng, Lifeng Shang, Xin Jiang, Qun Liu, and Yujiu Yang. 2023. AutoConv: Auto- matically generating information-seeking con...

2023 arXiv
[2020]

update_dialogue_state

training approach 11. For training both D2P 10https://huggingface.co/castorini/ ance-msmarco-passage 11https://haystack.deepset.ai/ tutorials/09_dpr_training Approach QG Feat. F1 R-1 R-2 R-L Baseline - + 0 0.056 0.020 0.048 Baseline - +/-/? 0.008 0.137 0.047 0.087 Utterance BA...
[2023]

arXiv e-prints, pages arXiv–2307

Challenges and applications of large language models. arXiv e-prints, pages arXiv–2307. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proce...

2020

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

It’s also generally resistant to dirt and easy to clean

**Nylon**: This is a type of synthetic fabric that’s known for its strength and durability. It’s also generally resistant to dirt and easy to clean

work page

[2] [2]

These types are lightweight, and plastic ones are particularly good for water resistance

**Polycarbonate or Plastic**: Some organizers are designed with a hard exterior for added protection. These types are lightweight, and plastic ones are particularly good for water resistance

work page

[3] [3]

They offer a sleek, stylish look while also providing a good level of protection

**Leather**: There are also organizers made from leather or PU leather. They offer a sleek, stylish look while also providing a good level of protection. Each material comes with its own advantages. Nylon is usually longer-lasting, polycarbonate or plastic is lighter and waterproof, and leather gives a classy touch. Do any of these materials align with yo...

work page

[4] [5]

Springer

Proceedings 31, pages 796–800. Springer. Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu, and Raviteja Anantha. 2021. Question rewriting for conversational question answering. In Proceedings of the 14th ACM international conference on web search and data mining, pages 355–363. Nikhita Vedula, Marcus Collins, Eugene Agichtein, and Oleg Rokhlenko. 2022. Wha...

arXiv 2023

[5] [6]

model number

The meaning of each of the scores is interpreted in Table 8. Realism. To distinguish counter-intuitive conver- sations, realism examines whether the conversa- tion is likely to happen in the real world from a logical perspective. We also consider Factuality, which captures the commonsense-level factual er- rors, since they essentially reflect similar crit...

work page 2023

[6] [11]

It’s a great choice with several compartments to store your items such as a 9.7 inch iPad, Kindle, and Power Adapter

The first is the [BAGSMART Universal Travel Cable Organizer](https://www.amazon.com/BAGSMART- Universal-Organizer-Electronics-Accessories/dp/B017SKRWL4). It’s a great choice with several compartments to store your items such as a 9.7 inch iPad, Kindle, and Power Adapter. It’s black with a hint of red color that gives it an edgy look

work page

[7] [12]

It’s also a black, splashproof and waterproof travel cable organizer that should serve your needs well

The second option is [Inateck Electronics Organizer](https://www.amazon.com/Inateck-Electronics-Organizer- Splashproof-Waterproof/dp/B07ZV2CZSR). It’s also a black, splashproof and waterproof travel cable organizer that should serve your needs well

work page

[8] [13]

microfiber

The third one is [Luxtude Electronics Organizer](https://www.amazon.com/Luxtude-Electronics-Organizer- Portable-Essentials/dp/B086M63FXV). This Travel Cable Case is a portable tech bag perfect for cable storage, cord storage and storing electronic accessories/USB/SD/Charger. It’s also black, aligning with your color preference. Please let me know if you n...

work page