{"id":"8870d1c1-86aa-41d7-b2c4-07b6ed8e777e","arxiv_id":"2505.02560","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Providing simulated search users with summaries of both relevant and irrelevant documents improves retrieval effectiveness on TREC Core17 but not on Core18, and no significance tests are reported.","lead":"This paper tests whether showing a simulated search user both good and bad example documents helps it find better results in interactive search sessions. The contrastive feedback helps on one test collection but not on a second, so the benefit is real but inconsistent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim overstates the evidence: CRF beats positive-only feedback on Core17 but merely ties it on Core18, and no significance testing or multiple seeds are reported.","rationale":"The central claim is a general benefit of contrastive examples across simulated sessions. For that claim to hold, CRF should dominate PRF in the experiments. It does not: Section 4 reports CRF tied with PRF on Core18 and PRF as the leading performer, and the paper itself states there is no benefit to irrelevant summaries on that collection. Since query generation is stochastic (temperature 1.0) and only one seed is used, the Core17 advantage could be a single-trajectory artifact. This concern is more load-bearing than the prompt-template issue: even if all configurations differ only in feedback modality, the reported evidence supports contrastive benefit on one of two datasets, not 'throughout interactive retrieval simulations.' The reader's weakest_assumption points to the prompt-template confound; the reader's rationale does note the mixed results and lack of significance, which is why I mark agreement as partial. The remedy is statistical: per-topic paired tests and multiple seeds. This supports the existing CONDITIONAL verdict; no verdict change is needed.","tokens_in":8401,"tokens_out":4653,"duration_ms":52396,"concrete_test":"Using the released repository logs, reconstruct per-topic IG and sDCG for CRF and PRF on Core17 and Core18, then run paired permutation tests (or Wilcoxon signed-rank, 10,000 resamples) on the topic-level differences. Additionally, rerun query generation with at least five different seeds to estimate trajectory variance. If CRF is not significantly better than PRF on Core18, or if the Core17 difference is not robust to resampling and seed variation, the conclusion must be narrowed to a collection-specific effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section 5) is that 'LLMs benefit from providing contrastive examples in the prompt throughout interactive retrieval simulations.' The load-bearing condition for this claim is that the contrastive user (CRF) is reliably more effective than the positive-only user (PRF), since PRF is the strongest non-contrastive feedback condition. The paper's own Section 4 results contradict that condition: on Core18, CRF is 'on par with the PRF user' and 'there is obviously no benefit to including summaries of irrelevant documents'; PRF remains the leading performer. Only on Core17 does CRF strictly outperform PRF. No significance tests, confidence intervals, or variance estimates are reported; Section 3.3 fixes the random seed and uses temperature 1.0 for query generation, so each configuration is a single stochastic trajectory. Consequently, the observed Core17 advantage could be sampling noise, and the conclusion as stated is not supported by the evidence. The prompt-template confound flagged by the reader is real but secondary: templates are in the public repository and can be inspected, whereas the Core18 tie is an internal inconsistency in the paper's own reported results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates whether providing contrastive relevance feedback — summaries of both judged-relevant and judged-irrelevant documents — in the prompt improves the effectiveness of LLM-based simulated users in interactive information retrieval. Using the SimIIR 3 framework, BM25 retrieval, and a Llama3.3 model, the authors compare eight user configurations on TREC Core17 and Core18, measuring Effort-Effect information gain and sDCG. They report that the contrastive user (CRF) is the top performer on Core17 but is on par with the positive-only user (PRF) on Core18, yet the conclusion states that LLMs benefit from contrastive examples in the prompt throughout interactive retrieval simulations.","tokens_in":8542,"tokens_out":1857,"duration_ms":24312,"significance":"If the central claim were robust, the paper would provide a useful prompt-engineering recipe for LLM-based user simulation and would connect contrastive training intuitions to in-context interaction. The work has clear strengths: the experimental setup is described in reasonable detail, the code and results are publicly available, the authors compare against random and topic-only baselines, and they are explicitly honest about the mixed Core18 result. However, the current evidence is not sufficient to support the stated conclusion. The contrastive advantage is observed on only one of two collections, and the experimental design fixes a single random seed with no repeated runs or significance testing, so the Core17 advantage could be sampling noise. The paper is therefore a useful preliminary study rather than a demonstrated result, and the conclusion needs to be substantially softened or the experiments strengthened.","major_comments":[{"comment":"The central claim that \"LLMs benefit from providing contrastive examples in the prompt throughout interactive retrieval simulations\" is not supported by the paper's own reported results. In Section 4, the authors state that on Core18, \"CRF is on par with the PRF user, which remains the leading performer overall\" and that \"there is obviously no benefit to including summaries of irrelevant documents in the prompt.\" Since PRF is the strongest non-contrastive feedback condition, the contrastive advantage is observed only on Core17. The conclusion in Section 5 overstates this evidence and should be revised to reflect the collection-dependent nature of the finding.","section":"Section 4 and Section 5"},{"comment":"The experimental design fixes a single random seed (default 0.0) and uses temperature 1.0 for query generation, so each user configuration produces one stochastic trajectory per test collection. No variance estimates, confidence intervals, or significance tests are reported. Consequently, the observed Core17 advantage of CRF over PRF could be due to sampling noise, and the Core18 tie offers no support for the contrastive mechanism. The authors should run multiple seeds or repeated sessions and report variance, or at minimum frame the results as descriptive without claiming a reliable benefit.","section":"Section 3.3"},{"comment":"The prompt templates and summary-generation logic are only available in the GitHub repository and are not described in the paper. Since the entire comparison is between prompting strategies, differences in wording, summary formatting, or instruction phrasing between conditions could confound the observed effectiveness differences with the feedback-modality variable. The paper should include the actual templates or a detailed description of their structure in an appendix, even in compressed form, so that readers can verify that the conditions differ only in whether relevant, irrelevant, or both summaries are included.","section":"Section 3.2"}],"minor_comments":[{"comment":"The phrase \"the realism of the user simulation when proving the LLM with the explicitly formulated topic descriptions\" appears to be a typo for \"providing the LLM.\"","section":"Introduction"},{"comment":"\"determinstic rules\" should be spelled \"deterministic rules.\"","section":"Related Work"},{"comment":"\"theIR community\" is missing a space and should read \"the IR community.\"","section":"Related Work"},{"comment":"\"While the CRF use can access the entire topic's contents\" should read \"While the CRF user can access\".","section":"Section 3.1"},{"comment":"The notation in Eq. (1) is unclear: \"rel_d\" should be explicitly defined as the relevance value of a document d, and it would help to state the units or scale of the information gain contribution.","section":"Section 3.4, Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clean, honest 5-page study comparing prompt variants for LLM-based user simulators, and the specific comparison (positive vs negative vs contrastive summaries) is new to the best of my knowledge. But the central claim in the conclusion runs ahead of the data: on Core18 the contrastive user ties positive-only, and there are no significance tests or multiple seeds, so the Core17 advantage could be a fluke.\n\nWhat I like: the setup is clear, the baselines are sensible, the choice to include random-judgment users helps interpret the LLM results, and the paper says plainly in Section 4 that on Core18 there is no benefit to adding irrelevant summaries. The code and prompts are in a public repo, which is the right way to ship this line of work. The authors also flag the unjudged-document limitation themselves, which is honest.\n\nWhere it's soft: the conclusion doesn't match the results. Section 5 says 'LLMs benefit from providing contrastive examples' — but the paper's own results show that benefit only on Core17, not Core18. With one seed and no error bars, 'suggest' is doing too much work. I'd also want the missing CRF' condition on Core18 explained; it appears in the Core17 plot but not the Core18 plot, which makes the conclusion's secondary claim about CRF' hard to verify. And the prompts live in the repository rather than the paper: acceptable for a short paper, but the evaluation would be stronger with the templates or at least the essential formatting choices described.\n\nThe citation pattern is fine; SimIIR 3 is their own prior work but they use it as a real framework, not a citation accessory. The mixed result does not sink the paper — a well-run null comparison is still a useful result — but the framing needs to change. Who is this for? People building LLM-based user simulators and anyone comparing prompting strategies in interactive evaluation. It's a reasonable SIGIR short paper; a serious referee should engage with it, and my recommendation would be major revision: temper the conclusion, add variance estimates or multiple runs, and resolve the CRF' omission.","headline":"Clean but overreaching short study of contrastive prompt feedback for LLM user simulators; worth refereeing if the conclusion is scaled back.","tokens_in":9099,"tokens_out":2610,"would_cite":true,"duration_ms":30106,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-based simulated search users become more effective when the prompt contains summaries of both relevant and irrelevant documents they have already judged, compared with positive-only, negative-only, or no feedback.","keywords":["interactive information retrieval","user simulation","contrastive feedback","prompt engineering","large language models","relevance feedback","session evaluation"],"falsifier":"A replication run that fixes a single prompt template across all conditions—varying only whether the injected summary block lists relevant documents, irrelevant documents, both, or neither—would settle the claim; if the contrastive advantage disappears or reverses, the reported effect comes from template wording rather than from contrastive feedback.","tokens_in":8151,"feed_emoji":"🔍","tokens_out":8047,"duration_ms":81287,"temperature":0.7,"pith_summary":"This paper tests whether showing an LLM-based simulated user—an agent that mimics a human searcher—summaries of both relevant and irrelevant documents during an interactive search session makes it find more relevant documents than simpler feedback schemes. Eight user configurations are compared on two TREC newswire collections, varying whether the prompt carries full topic information, only a title, positive feedback, negative feedback, contrastive feedback, or random relevance decisions. On the first collection the contrastive user is the top performer, and on the second it ties with the positive-only user; all feedback variants outperform topic-only baselines after the first few queries. The paper reads this as evidence that contrastive examples help the LLM internalize the distinction between desired and undesired documents, in line with contrastive-training results from fine-tuning. If correct, the finding gives interactive-retrieval evaluation a way to build more realistic simulated users without relying on explicit topic narratives.","feed_headline":"Simulated searchers improve when prompts show hits and misses","feed_subtitle":"Adding judged hits and misses to the prompt yields more relevant documents per search effort","key_machinery":"The carrying mechanism is the contrastive relevance feedback (CRF) prompt: a prompt that appends two summary blocks—one compressing previously judged relevant documents and one compressing previously judged irrelevant documents—to the topic context before asking the LLM to generate the next query or judge the next document. The summaries are refreshed after every relevance judgment, so the model's knowledge state evolves over the session. This mechanism is what lets the contrastive-training idea act at the prompting stage rather than the weight-update stage, and the experimental design isolates it by keeping the feedback type as the main difference between user configurations.","core_discovery":"The central hypothesis is that the contrastive-learning principle—training a model on both positive and negative examples—carries over from fine-tuning to prompt engineering for user simulation. In the strongest configuration, every query-generation and relevance-judgment step receives a running summary of documents the simulated user has judged relevant plus a separate summary of documents judged irrelevant, updated after each judgment, alongside the full TREC topic context. On the Core17 collection this contrastive relevance feedback user is the most effective by both effort-effect information gain and session-discounted cumulative gain, while on Core18 it matches the positive-only user, which stays marginally ahead. The authors conclude that LLMs benefit from providing contrastive examples in the prompt throughout interactive retrieval simulations, and they note that a title-only contrastive variant performs comparably well to full-topic, no-feedback users, suggesting session context can partly replace detailed topic descriptions.","pith_inferences":["A testable extension the paper does not run: vary summary granularity (single sentence vs. paragraph, extractive vs. abstractive) to confirm the contrast set itself, rather than extra text, drives the gain.","The cross-collection difference suggests a hypothesis the authors leave open: negative examples matter most when judged-relevant documents are sparse or when many retrieved documents resemble the target without being relevant; comparing relevance-label density across collections would test it.","Because the prompt templates are only available in a code repository, a replication that fixes all template wording except the feedback block is needed before the effect can be attributed to contrastivity rather than incidental phrasing."],"forward_implications":["Simulated users should include session feedback summaries, not just static topic descriptions, when the goal is to emulate an evolving information need.","Contrastive feedback's advantage emerges only after several queries, so simulation evaluations need sessions long enough to capture the uptake phase.","A title-only contrastive variant outperforms full-topic, no-feedback users, so detailed topic narratives may not be necessary when the simulator accumulates session context.","The Core18 result, where positive-only feedback matches contrastive feedback, shows the benefit of negative examples is dataset-dependent and should not be assumed universal."],"supporting_citations":[{"why":"supplies the interactive-simulation framework whose session logic the experiments build on","marker":"[3]"},{"why":"the context-driven query simulation approach showing LLMs benefit from TREC topic context, which this study extends with feedback summaries","marker":"[8]"},{"why":"supervised contrastive learning for fine-tuning, a result behind the hypothesis that contrastive examples help LLMs","marker":"[9]"},{"why":"defines the sDCG measure used to evaluate session-level effectiveness","marker":"[10]"},{"why":"the rule-based Common Interaction Model baseline that motivates the move to more realistic simulated users","marker":"[14]"},{"why":"contrastive learning reduces hallucination in conversations, another contrastive-training result feeding the hypothesis","marker":"[17]"},{"why":"provides the Core17 test collection used in the first experiment","marker":"[2]"},{"why":"provides the Core18 test collection used in the second experiment","marker":"[19]"}],"fun_headline_variants":["Contrastive prompts make simulated searchers smarter","Shows hits and misses in prompts boost user sims","Contrastive feedback sharpens simulated users","User simulations improve with contrastive examples","Relevant plus irrelevant in prompts aids sims"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the feedback type is the only difference between user configurations, since the exact prompt templates and summary formatting are not shown in the paper but left in a code repository.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive prompts make simulated searchers smarter","Shows hits and misses in prompts boost user sims","Contrastive feedback sharpens simulated users","User simulations improve with contrastive examples","Relevant plus irrelevant in prompts aids sims"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1527,"prompt_tokens":915,"completion_tokens":612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":541}},"tokens_in":531,"tokens_out":612,"duration_ms":6366,"temperature":1.0,"reasoning_tokens":541,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:47:16.749724+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication run that fixes a single prompt template across all conditions—varying only whether the injected summary block lists relevant documents, irrelevant documents, both, or neither—would settle the claim; if the contrastive advantage disappears or reverses, the reported effect comes from template wording rather than from contrastive feedback.","supporting_citations":[{"cited_title":"Contrastive Learning Reduces Hallucination in Conversations","cited_arxiv_id":"2212.10400","evidence_quote":"contrastive learning reduces hallucination in conversations, another contrastive-training result feeding the hypothesis"},{"cited_title":"Voorhees","cited_arxiv_id":null,"evidence_quote":"provides the Core17 test collection used in the first experiment"},{"cited_title":"Voorhees and Angela Ellis (Eds.)","cited_arxiv_id":null,"evidence_quote":"provides the Core18 test collection used in the second experiment"}],"review_version":1}