Pith. sign in

REVIEW 3 major objections 7 minor 58 references

Fashion-AlterEval: A Dataset for Improved Evaluation of Conversational Recommendation Systems with Alternative Relevant Items

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that evaluating conversational fashion recommenders against a single target item understates their effectiveness, and that letting simulated users switch to visually similar alternatives lets systems satisfy users sooner.

desk verdict The dataset is the real contribution; the headline 'rapid satisfaction' result is an artifact of the simulator's own target-selection rule and should not be taken at face value. read the letter →

arxiv 2507.18017 v1 pith:2YXPLKIN submitted 2025-07-24 cs.IR cs.AI

Maria Vlachou This is my paper · ORCID
classification cs.IRcs.AI
keywords conversationalrecommendationusersimulationrelevancejudgmentsalternativeitemsfashionretrievalevaluationmethodologymeta-simulatortestcollections
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that offline evaluation of conversational recommendation systems is too pessimistic because simulated users are assumed to want exactly one target item and to be infinitely patient. To fix this, it contributes Fashion-AlterEval, human relevance judgments marking which candidate items are acceptable alternatives for 200 target shoes and 200 target dresses from two existing fashion datasets. It wraps the standard single-target simulator with two meta-simulators: one that switches to the closest alternative once patience runs out, and one that switches probabilistically after a perceived loss. On three CRS models, using alternatives raises MRR@10 by up to 149.92% and often raises nDCG@10, leading the paper to conclude that previous evaluations underestimated how quickly systems can satisfy users who accept substitutes.

What carries the argument

The central object is Fashion-AlterEval, a set of relevance judgments in which human annotators marked a pool of candidate images as sufficient alternatives for 200 target items in each of two fashion categories; on average 3.5 alternatives per target were judged relevant, with high annotator agreement. The two meta-simulators are the operational mechanism: they take the existing target-based critique function $Usersim.critique(turn, top\_ranked, target)$ and re-target it, selecting a new target $alts[\arg\max(sims(alts, top1k))]$ once the turn count exceeds a tolerance, or, in the probabilistic variant, only when the similarity difference $\delta = sim(top1k, target) - sim(top1k-1, target)$ is negative and a random draw falls below $P_{switch}$. This machinery converts a single-target evaluation into a multi-target one by reusing the same learned critique model, so the observed gains come from expanding what counts as relevant rather than from retraining the recommender.

What would settle it

Run the same three CRS models with meta-simulators that select the new target randomly, or from human-elicited alternative preferences, instead of by maximum image similarity to the top-ranked item; if the MRR@10 gains drop far below the reported values, the effect is an artifact of the similarity-based selection rule rather than a property of alternative-aware evaluation.

Watch

Extended reading notes

Core claim

The central claim is that existing single-target user simulators underestimate the effectiveness of conversational fashion recommenders. The paper proposes that relevance judgments for alternative items, collected from human assessors, can extend the target space of simulated users, and it introduces two meta-user simulators that wrap an existing relative-captioning simulator. MetaSimTol switches to the alternative most visually similar to the currently top-ranked image after a tolerance threshold of turns, and MetaSimProb adds a gain-loss heuristic so that a user who perceives a drop in similarity from the previous turn switches to an alternative with probability $P_{switch}$. Across GRU-SL, GRU-RL, and EGE on Shoes and Dresses, the alternative-aware simulators produce large gains at turn 10, up to +149.92% in MRR@10 for EGE on Dresses, which the paper interprets as evidence that single-target evaluation systematically understates how well these systems can respond to a user who is willing to change their mind.

Load-bearing premise

The load-bearing premise is that a user who decides to accept an alternative will pick the item that looks most like the one the system just displayed; if real users switch on other grounds, the reported gains may not transfer.

Editorial extensions

If this is right

  • If the central claim is right, previously reported CRS numbers on the Shoes and FashionIQ Dresses datasets understate how well these systems serve users who accept substitute items.
  • The meta-simulators plug into any existing critique-based user simulator without retraining the recommender, because they only change which target image the critique call is asked about.
  • Switching earlier in the dialogue produces a larger difference between old and new evaluation estimates, while a more patient simulated user is already receiving better rankings before the switch.
  • The probabilistic gain-loss variant is consistently at least as good as the fixed-tolerance variant at tolerances 2 and 3, especially for models that attend mainly to the previous turn.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the same alternative judgments could support a single-turn retrieval evaluation, in which any judged alternative counts as a hit, making offline CRS numbers directly comparable to standard image retrieval test collections.
  • A natural stress test is to replace the similarity-based alternative selection with alternatives drawn from human preference judgments or at random; if the gains mostly disappear, the reported improvement reflects the selection rule rather than the value of alternative-aware evaluation.
  • Because the new target is chosen to look almost identical to the last displayed image, the next critique describes only small visual differences, which makes rapid retrieval of that new target nearly automatic; a fairer reading of +149.92% is that it measures estimation of a broader user need, not an improvement in the recommender itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces Fashion-AlterEval, a dataset of human judgments of alternative relevant items for a sample of target items from the Shoes and FashionIQ Dresses conversational-fashion datasets. The authors also propose two meta-user simulators, MetaSimTol and MetaSimProb, that wrap an existing single-target relative-captioning simulator and allow the simulated user to switch to an alternative target after a fixed tolerance or probabilistically after a perceived loss. They evaluate three CRS models (GRU-SL, GRU-RL, EGE) with and without these simulators and report large improvements, e.g., up to +149.92% MRR@10 for EGE on Dresses, concluding that single-target evaluation underestimates CRS effectiveness and that alternative-aware simulators allow systems to satisfy users more quickly.

Significance. The dataset construction is a genuine contribution: it uses a power analysis, pools candidates from multiple systems, reports an assessor agreement of Cohen's kappa 0.87, and is publicly released with code. If the central claim were supported, the paper would provide a practical test-collection resource for evaluating conversational recommendation with alternative relevance. However, the main quantitative claim depends heavily on a simulator design choice that is not validated, and the reported effect sizes are likely inflated. The paper deserves credit for making the data and code available, but the experimental evidence as currently presented is not sufficient to establish that single-target evaluation underestimates real effectiveness.

major comments (3)
  1. [4.1, Algorithm 1 line 3; 4.2, Algorithm 2 line 5] In both meta-simulators, the simulated user selects as the new target the alternative with maximum image similarity to the currently displayed top-1 item (target = alts[arg max(sims(alts, top1k))]). This operationalizes intuition I2 but is not grounded in observed user behavior. Because the new target is near-identical to the item the system just ranked first, the next critique describes only small differences, and the ranker is highly likely to retrieve that near-identical alternative quickly. The reported improvements in Table 3 (up to +149.92% MRR@10) therefore conflate "the CRS satisfies an alternative-accepting user" with "the simulator chose a target adjacent to the system's current guess." To support the central claim, the paper should include a robustness ablation that selects the new target differently: e.g., uniformly at random among judged alternatives, least similar to the displayed item, or by using human judgments of similarity rather than image-embedding similarity. Without such an ablation, the direction and size of the reported gains are not identifiable.
  2. [5.3, Table 3] For each tolerance level of MetaSimProb, the paper reports the best metric over the threshold grid [0.55, 0.65, 0.75, 0.85, 0.95] (Section 5.1). This amounts to selecting a free parameter per cell to maximize the reported improvement, and no variance or confidence intervals are given. Consequently, the comparison between MetaSimProb and SimBase (and between MetaSimProb and MetaSimTol) is optimistic by construction. Please report results for a single a priori threshold (e.g., 0.75, as derived from Klein et al.) or average over thresholds, and provide standard errors over the 200 target items or over multiple simulation seeds.
  3. [5.3, Table 3] The headline claim that "single-target evaluation underestimates their effectiveness" is contradicted by the GRU-RL Shoes row, where NDCG@10 drops by 13.75% while MRR@10 improves by 64.80%. The paper does not explain this divergence. The evaluation also counts a conversation as successful if any alternative is retrieved (Section 5.2), which combined with the near-identical target selection amplifies the already high MRR numbers. Please discuss this discrepancy and reconcile the metric-level conclusions with the abstract's blanket statement.
minor comments (7)
  1. [1, paragraph 2] "the realisticity of an interaction" should be "the realism of an interaction."
  2. [3.2.2] "making a total cost of the study was $305" is ungrammatical; revise to "the total cost of the study was $305."
  3. [4.1, 4.2] The notation sims(alts, top1k) is not formally defined; specify which image embedding and similarity function are used, and whether the same representation is used for the CRS ranker's top-1 item.
  4. [5.1] The sentence "We convert Cohen's d to probability, which gives a value of 0.75" needs a formula or reference; the conversion depends on the assumed distribution and is not self-evident.
  5. [Table 3] Check the SimBase row for GRU-RL on Dresses; MRR@10 (0.269) being higher than NDCG@10 (0.099) is unexpected and may be a typo or misalignment.
  6. [5.5, Figure 8] The description of training with "all target items instead of a sample of 200" is underspecified; clarify which items are used, how the models are trained, and why this comparison is appropriate.
  7. [Conclusion] The stated limitation "our simulators do not consider multiple alternatives" is slightly misleading; the simulators select a single alternative at a time, but the dataset provides multiple judged alternatives per target. The paper should clarify that the current implementation does not use the full alternative set as a relevance set during simulation.

Circularity Check

2 steps flagged · score 6.0 of 10

Rapid-satisfaction claim is substantially built into the alternative-selection rule and best-threshold reporting

  1. self definitional [Section 4.1 Algorithm 1 line 3; Section 4.2 Algorithm 2 line 5; Section 5.2 evaluation criterion]
    "Among all of the alternatives for a given target, we select the alternative that is closest in image similarity to the current top-ranked image as the target (line 3, addressing I2). Algorithm 1 line 3: target = alts[arg max(sims(alts, top1k))]."

    The simulated user's new target is, by construction, the alternative with maximum image similarity to the item the system already ranked first. The relative-captioning simulator then critiques that top-1 item against this near-visual-twin target, and the CRS models are trained to retrieve exactly such critique-matched images. Since Section 5.2 defines success as retrieving any alternative or the original target, the 'system can rapidly respond to more quickly satisfy the user' result in the abstract is largely an entailment of the choice rule: the next target is deliberately aligned with the system's current guess. The reported MRR@10 improvements therefore measure the similarity heuristic in Algorithm 1, not an independent property of CRS effectiveness for alternative-accepting users.

  2. fitted input called prediction [Section 5.2 and Table 3]
    "For each tolerance level of MetaSimProb, we report the metric value of the highest performing threshold level, as described in Section 5.2."

    The Pswitch threshold is a free parameter of the probabilistic simulator, and the paper scans five values and reports, per tolerance level, the threshold that maximizes the reported metric. The headline '+149.92%' and the other percentage improvements in Table 3 are therefore maxima over a tuned parameter rather than the performance of a fixed, pre-specified simulator. Presenting grid-selected values as evidence that alternative-aware evaluation improves measured effectiveness is a fitted input renamed as a prediction; no variance or held-out threshold choice is reported.

full rationale

The dataset collection itself is not circular: the alternative judgments come from a human MTurk study, are independently collected, and are reported with high inter-assessor agreement (Cohen's kappa = 0.87). The self-citation [46] is used only to supply effect sizes for the power analysis and is not load-bearing for the central evaluation claim. However, the central claim that single-target evaluation 'underestimates their effectiveness' and that systems 'rapidly respond' is only demonstrated under a simulator whose alternative-selection rule (Algorithm 1 line 3 and Algorithm 2 line 5) chooses the new target as the alternative most similar to the currently top-ranked item. Combined with the decision to count any alternative as a successful retrieval, this makes rapid success substantially a consequence of the construction. The extra choice to report the best Pswitch per cell further means the headline improvements are optimized, not predicted. Accordingly, the paper's evaluation conclusion partially reduces to its own inputs, even though the dataset and simulators are transparently described.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The quantitative conclusions depend on a user model (tolerance, Pswitch, similarity-based target switching) that is not validated against observed switching behavior. The dataset itself rests on pooling completeness and MTurk reliability assumptions.

free parameters (4)
  • tolerance = 1, 2, 3, 4
    Hand-chosen grid for the number of turns before the meta-simulator may switch to an alternative; no user data support the patience levels.
  • Pswitch = 0.55, 0.65, 0.75, 0.85, 0.95 (base 0.75 from Cohen's d)
    Probability of switching after a perceived loss; base value from Klein et al., then grid searched. Table 3 reports the best threshold per cell, which effectively fits the threshold to the outcome.
  • Pooling mixture = 60% nearest neighbours, 40% top-retrieved
    Design choice in Section 3.2.1; affects which candidate alternatives are available for relevance assessment.
  • Candidate rank cutoffs = top-4 neighbours + top-3 retrieved per model = 14 candidates
    Design choice in Section 3.2.1; defines the completeness of alternative judgments.
assumptions (5)
  • domain assumption A relative-captioning user simulator can be invoked with any alternative target image (intuition I3).
    Section 4.1 states the base simulator is called with a new target; this assumes the learned captioner generalizes across targets, which is untested in the alternative setting.
  • ad hoc to paper Visual similarity between the current top-1 item and candidate alternatives is a valid proxy for which alternative a real user would choose (intuition I2).
    Algorithm 1 line 3 and Algorithm 2 line 5 select the alternative by arg max image similarity; no behavioral data support this.
  • ad hoc to paper Users lose patience after a fixed number of turns (I1) and switch strategy with probability Pswitch after a perceived loss (I4/I5).
    Algorithms 1 and 2 encode these intuitions; tolerance values 1 to 4 are hand-chosen and Pswitch is anchored to a prior psychology effect size rather than to observed shopping behavior.
  • domain assumption The 14 pooled candidates per target give a sufficiently complete set of relevant alternatives for evaluation.
    Section 3.2.1 uses top-4 nearest neighbors and top-3 retrieved from two models; incomplete pooling could miss relevant alternatives and bias measured effectiveness.
  • domain assumption MTurk workers' judgments are reliable enough to serve as ground truth, supported by kappa 0.87 on a 40-target subset.
    Section 3.2.2 reports one replication round; workers are US-based and screened for familiarity, but only 3 HITs rejected out of the study.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fashion-AlterEval: A Dataset for Improved Evaluation of Conversational Recommendation Systems with Alternative Relevant Items." pith.science (2026). https://pith.science/paper/2YXPLKIN

@misc{pith2026250718017,
  author       = {Pith},
  title        = {Pith review of: Fashion-AlterEval: A Dataset for Improved Evaluation of Conversational Recommendation Systems with Alternative Relevant Items},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2YXPLKIN}},
  note         = {Machine review of arXiv:2507.18017}
}
read the original abstract

In Conversational Recommendation Systems (CRS), a user provides feedback on recommended items at each turn, leading the CRS towards improved recommendations. Due to the need for a large amount of data, a user simulator is employed for both training and evaluation. Such user simulators critique the current retrieved item based on knowledge of a single target item. However, system evaluation in offline settings with simulators is limited by the focus on a single target item and their unlimited patience over a large number of turns. To overcome these limitations of existing simulators, we propose Fashion-AlterEval, a new dataset that contains human judgments for a selection of alternative items by adding new annotations in common fashion CRS datasets. Consequently, we propose two novel meta-user simulators that use the collected judgments and allow simulated users not only to express their preferences about alternative items to their original target, but also to change their mind and level of patience. In our experiments using the Shoes and Fashion IQ as the original datasets and three CRS models, we find that using the knowledge of alternatives by the simulator can have a considerable impact on the evaluation of existing CRS models, specifically that the existing single-target evaluation underestimates their effectiveness, and when simulatedusers are allowed to instead consider alternative relevant items, the system can rapidly respond to more quickly satisfy the user.

Figures

Figures reproduced from arXiv: 2507.18017 by the authors.

Figure 1
Figure 1. Example of a Conversational Image Recommenda [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Schematic representation of our target pooling strat [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Example HIT (Amazon Mechanical Turk task) from our user study for the Shoes dataset. The target item appears at [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Examples of identified alternative relevant items from our user study for Shoes (top) and Dresses (bottom). The [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Schematic representation of our 𝑀𝑒𝑡𝑎𝑆𝑖𝑚𝑃𝑟𝑜𝑏 meta￾simulator. If the difference of target-candidate similarity between turns is negative, the meta-user opts for a switch to an alternative with probability 𝑃𝑠𝑤𝑖𝑡𝑐ℎ. Algorithm 2 MetaSimProb 1: procedure MetaUserSim.critiqe(…
Figure 6
Figure 6. Figure 6: NDCG@10 performance for the GRU-RL model for [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: SR for Shoes before and after selecting an alterna [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Inspection of the different evaluation settings (and [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 40 canonical work pages

  1. [1]

    Conversational AI from an information retrieval perspective: Remaining challenges and a case for user simulation

    Balog, K. Conversational AI from an information retrieval perspective: Remaining challenges and a case for user simulation. DESIRES. (2021)

    work page 2021

  2. [2]

    & Shih, J

    Berg, T., Berg, A. & Shih, J. Automatic attribute discovery and characterization from noisy web data. Proc. ECCV. pp. 663-676 (2010)

    2010

  3. [3]

    A taxonomy of web search

    Broder, A. A taxonomy of web search. ACM Sigir Forum. 36, 3-10 (2002)

    work page 2002

  4. [4]

    & Voorhees, E

    Buckley, C., Dimmick, D., Soboroff, I. & Voorhees, E. Bias and the limits of pooling for large collections. Information Retrieval. 10, 491-508 (2007)

    work page 2007

  5. [5]

    & Voorhees, E

    Buckley, C. & Voorhees, E. Retrieval evaluation with incomplete information. Proceedings Of The 27th Annual International ACM SIGIR Conference On Research And Development In Information Retrieval . pp. 25-32 (2004)

    work page 2004

  6. [6]

    & Yom-Tov, E

    Carmel, D. & Yom-Tov, E. Estimating the query difficulty for information retrieval. (Morgan & Claypool Publishers,2010)

    2010

  7. [7]

    & Engelhardt, B

    Chaney, A., Stewart, B. & Engelhardt, B. How algorithmic confounding in recom- mendation systems increases homogeneity and decreases utility. Proceedings Of The 12th ACM Conference On Recommender Systems . pp. 224-232 (2018)

    work page 2018

  8. [8]

    & Corbin, J

    Chick, C., Reyna, V. & Corbin, J. Framing effects are robust to linguistic disambigua- tion: A critical test of contemporary theory.. Journal Of Experimental Psychology: Learning, Memory, And Cognition. 42, 238 (2016)

    work page 2016

Show all 58 references
  1. [9]

    & Bengio, Y

    Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H. & Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. ArXiv Preprint ArXiv:1406.1078. (2014)

    2014 arXiv

  2. [10]

    Developing a flexible spoken dialog system using simulation

    Chung, G. Developing a flexible spoken dialog system using simulation. Proc. ACL. (2004)

    2004

  3. [11]

    & Voorhees, E

    Craswell, N., Mitra, B., Yilmaz, E., Campos, D. & Voorhees, E. Overview of the TREC 2019 deep learning track. ArXiv Preprint ArXiv:2003.07820. (2020)

    2020 arXiv

  4. [12]

    & Croft, W

    Cronen-Townsend, S., Zhou, Y. & Croft, W. Predicting query performance.Proc. SIGIR. (2002)

    2002

  5. [13]

    & Callan, J

    Dalton, J., Xiong, C., Kumar, V. & Callan, J. Cast-19: A dataset for conversational information seeking. Proc. SIGIR. (2020)

    2020

  6. [14]

    & Callan, J

    Dalton, J., Xiong, C. & Callan, J. TREC CAsT 2019: The conversational assistance track overview. ArXiv Preprint ArXiv:2003.13624. (2020)

    2020 arXiv

  7. [15]

    & Molina, J

    Griol, D., Carbó, J. & Molina, J. An automatic dialog simulation technique to de- velop and evaluate interactive conversational agents. Applied Artificial Intelligence. 27, 759-780 (2013)

    2013

  8. [16]

    & Feris, R

    Guo, X., Wu, H., Cheng, Y., Rennie, S., Tesauro, G. & Feris, R. Dialog-based interactive image retrieval. Proc. NeurIPS. pp. 678-688 (2018)

    2018

  9. [17]

    & Davis, L

    Han, X., Wu, Z., Jiang, Y. & Davis, L. Learning fashion compatibility with bidirec- tional LSTMs. Proc. ACM Multimedia. (2017)

    2017

  10. [18]

    & McAuley, J

    He, R. & McAuley, J. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. Proc. WWW. (2016)

    2016

  11. [19]

    & Tikk, D

    Hidasi, B., Karatzoglou, A., Baltrunas, L. & Tikk, D. Session-based recommenda- tions with recurrent neural networks. ArXiv Preprint ArXiv:1511.06939. (2015)

    2015 arXiv

  12. [20]

    & Ounis, I

    Jadidinejad, A., Macdonald, C. & Ounis, I. Using exploration to alleviate closed loop effects in recommender systems. Proceedings Of The 43rd International ACM SIGIR Conference On Research And Development In Information Retrieval . pp. 2025- 2028 (2020)

    2020

  13. [21]

    & Ounis, I

    Jadidinejad, A., Macdonald, C. & Ounis, I. The simpson’s paradox in the offline evaluation of recommendation systems. ACM Transactions On Information Systems (TOIS). 40, 1-22 (2021)

    2021

  14. [22]

    & Young, S

    Jurcıcek, F., Keizer, S., Gašic, M., Mairesse, F., Thomson, B., Yu, K. & Young, S. Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk. Proc. INTERSPEECH. 11 (2011)

    2011

  15. [23]

    many labs

    Klein, R., Ratliff, K., Vianello, M., Adams Jr, R., Bahník, S., Bernstein, M. & Others. Investigating variation in replicability: a “many labs” replication project. Open Science Framework. (2014)

    2014

  16. [24]

    & Grauman, K

    Kovashka, A. & Grauman, K. Attribute pivots for guiding relevance feedback in image search. Proc. ICCV. (2013)

    2013

  17. [25]

    & Grauman, K

    Kovashka, A. & Grauman, K. Attributes for image retrieval. Visual Attributes. pp. 89-117 (2017)

    2017

  18. [26]

    & Chen, Y

    Li, X., Lipton, Z., Dhingra, B., Li, L., Gao, J. & Chen, Y. A user simulator for task-completion dialogues. ArXiv Preprint ArXiv:1612.05688. (2016)

    2016 arXiv

  19. [27]

    & Tang, X

    Liu, Z., Luo, P., Qiu, S., Wang, X. & Tang, X. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. Proc. CVPR. (2016)

    2016

  20. [28]

    & Liu, T

    Liu, Z., Wang, H., Niu, Z., Wu, H., Che, W. & Liu, T. Towards Conversational Recommendation over Multi-Type Dialogs. Proc. ACL. (2020)

    2020

  21. [29]

    & Soldaini, L

    MacAvaney, S. & Soldaini, L. One-Shot Labeling for Automatic Relevance Esti- mation. ArXiv Preprint ArXiv:2302.11266. (2023)

    2023 arXiv

  22. [30]

    & Van Den Hengel, A

    McAuley, J., Targett, C., Shi, Q. & Van Den Hengel, A. Image-based recommenda- tions on styles and substitutes. Proc. SIGIR. (2015)

    2015

  23. [31]

    & Crestani, F

    Owoicho, P., Sekulic, I., Aliannejadi, M., Dalton, J. & Crestani, F. Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond. Proc. SIGIR. pp. 632-642 (2023)

    2023

  24. [32]

    The race to create a ‘smart’google

    O’Brien, J. The race to create a ‘smart’google. Fortune Magazine. (2006)

    2006

  25. [33]

    & Gambara, H

    Piñon, A. & Gambara, H. A meta-analytic review of framing effect: risky, attribute and goal framing. Psicothema. 17, 325-331 (2005)

    2005

  26. [34]

    & Weiner, B

    Roitman, H., Erera, S., Sar-Shalom, O. & Weiner, B. Enhanced mean retrieval score estimation for query performance prediction. Proc. ICTIR. (2017)

    2017

  27. [35]

    Product search in e-shopping: a review and research propositions

    Rowley, J. Product search in e-shopping: a review and research propositions. Journal Of Consumer Marketing . 17, 20-35 (2000)

    2000

  28. [36]

    & Others Test collection based evaluation of information retrieval systems

    Sanderson, M. & Others Test collection based evaluation of information retrieval systems. Foundations And Trends® In Information Retrieval. 4, 247-375 (2010)

    2010

  29. [37]

    & Young, S

    Schatzmann, J., Thomson, B., Weilhammer, K., Ye, H. & Young, S. Agenda-based user simulation for bootstrapping a POMDP dialogue system. NAACL-Short. pp. 149-152 (2007)

    2007

  30. [38]

    Shi, W., Qian, K., Wang, X. & Yu, Z. How to build user simulators to train rl-based dialog systems. ArXiv Preprint ArXiv:1909.01388. (2019)

    2019 arXiv

  31. [39]

    & Markovits, G

    Shtok, A., Kurland, O., Carmel, D., Raiber, F. & Markovits, G. Predicting query performance by query-drift estimation. ACM Transactions On Information Systems (TOIS). 30, 1-35 (2012)

    2012

  32. [40]

    & Ren, Z

    Sun, W., Guo, S., Zhang, S., Ren, P., Chen, Z., Rijke, M. & Ren, Z. Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems.ACM Transactions On Information Systems. (2023)

    2023

  33. [41]

    & Rijke, M

    Sun, W., Zhang, S., Balog, K., Ren, Z., Ren, P., Chen, Z. & Rijke, M. Simulating user satisfaction for the evaluation of task-oriented dialogue systems. Proc. SIGIR. pp. 2499-2506 (2021)

    2021

  34. [42]

    & Zhang, Y

    Sun, Y. & Zhang, Y. Conversational recommender system. Proc. SIGIR. (2018)

    2018

  35. [43]

    & Kahneman, D

    Tversky, A. & Kahneman, D. The framing of decisions and the psychology of choice. Science. 211, 453-458 (1981)

    1981

  36. [44]

    & Rijke, M

    Vakulenko, S., Revoredo, K., Di Ciccio, C. & Rijke, M. QRFA: A data-driven model of information-seeking dialogues. Proc. ECIR. pp. 541-557 (2019)

    2019

  37. [45]

    & Kraaij, W

    Verberne, S., Sappelli, M., Järvelin, K. & Kraaij, W. User simulations for interactive search: Evaluating personalized query suggestion. Proc. ECIR. pp. 678-690 (2015)

    2015

  38. [46]

    & Macdonald, C

    Vlachou, M. & Macdonald, C. Performance Predictors for Conversational Fashion Recommendation. Proc. KaRS Workshop At RecSys. (2022)

    2022

  39. [47]

    & Ounis, I

    Wang, X., Macdonald, C., Tonellotto, N. & Ounis, I. ColBERT-PRF: Semantic pseudo-relevance feedback for dense passage and document retrieval. ACM Trans- actions On The Web. 17, 1-39 (2023)

    2023

  40. [48]

    & Feris, R

    Wu, H., Gao, Y., Guo, X., Al-Halah, Z., Rennie, S., Grauman, K. & Feris, R. Fashion iq: A new dataset towards retrieving images by natural language feedback. Proc. CVPR. pp. 11307-11317 (2021)

    2021

  41. [49]

    & Feris, R

    Wu, H., Gao, Y., Guo, X., Al-Halah, Z., Rennie, S., Grauman, K. & Feris, R. Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback. Proc. CVPR. (2021)

    2021

  42. [50]

    & Ounis, I

    Wu, Y., Macdonald, C. & Ounis, I. Partially Observable Reinforcement Learning for Dialog-based Interactive Recommendation. Proc. RecSys. (2021)

    2021

  43. [51]

    & Ounis, I

    Wu, Y., Macdonald, C. & Ounis, I. Multi-Modal Dialog State Tracking for Interac- tive Fashion Recommendation. Proc. RecSys. (2022)

    2022

  44. [52]

    & Grauman, K

    Yu, A. & Grauman, K. Fine-grained comparisons with attributes.Visual Attributes. pp. 119-154 (2017)

    2017

  45. [53]

    & Jin, H

    Yu, T., Shen, Y. & Jin, H. A visual dialog augmented interactive recommender system. Proc. KDD. (2019)

    2019

  46. [54]

    & Jin, H

    Yu, T., Shen, Y. & Jin, H. Towards hands-free visual dialog interactive recommen- dation. Proc. AAAI. (2020)

    2020

  47. [55]

    & Balog, K

    Zhang, S. & Balog, K. Evaluating conversational recommender systems via user simulation. Proc. KDD. pp. 1512-1520 (2020)

    2020

  48. [56]

    & Balog, K

    Zhang, S., Wang, M. & Balog, K. Analyzing and simulating user utterance refor- mulation in conversational recommender systems. Proc. SIGIR. (2022)

    2022

  49. [57]

    & Wen, J

    Zhou, K., Zhou, Y., Zhao, W., Wang, X. & Wen, J. Towards topic-guided conversa- tional recommender system. ArXiv Preprint ArXiv:2010.04125. (2020)

    2020 arXiv

  50. [58]

    & Kanoulas, E

    Zou, J. & Kanoulas, E. Learning to ask: Question-based sequential Bayesian product search. Proc. CIKM. (2019)

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.