{"id":"f7e595d0-8909-4306-a660-f6ce63c2a221","arxiv_id":"2501.03468","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MTRAG is a human-generated multi-turn RAG benchmark (110 conversations, 842 tasks, four domains) on which state-of-the-art LLM RAG systems perform poorly.","lead":"Researchers built MTRAG, a new benchmark of 110 human-written multi-turn conversations for testing AI systems that answer questions using retrieved documents. It shows that even top AI models struggle with follow-up questions, unanswerable questions, and later turns in a conversation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrieval comparison is potentially circular: reference passages were selected inside an ELSER-based annotation interface, so the reported Elser advantage in Table 3 may partly reflect construction bias rather than intrinsic retrieval quality.","rationale":"I evaluated whether the paper's central claim—that MTRAG is a useful first-of-its-kind benchmark and that state-of-the-art RAG systems struggle on it—survives scrutiny. The human-generated conversations, the FANC-based reference answers, the human evaluation showing reference preference, and the public release are real strengths. The most fragile link is the retrieval comparison in Section 5.3, because the gold reference passages were created with ELSER in the loop. The paper acknowledges this in Section 5.1 but does not quantify the degree of overlap, so the Table 3 conclusion that ELSER beats BM25 and BGE could be in part a home-field advantage. This is a genuine correctness risk for the retrieval-results claim, though it does not invalidate the benchmark or the generation findings. A targeted re-annotation or overlap measurement would settle it. I therefore keep the reader's CONDITIONAL verdict: the benchmark is valuable, but the retrieval-superiority claim needs a control before being taken at face value.","tokens_in":25617,"tokens_out":4764,"duration_ms":47361,"concrete_test":"Re-annotate a random sample of 30 MTRAG conversations (about 230 tasks) with independent annotators who are shown the full corpus and asked to mark every passage relevant to each question, without seeing any retriever output or the existing gold passages. Recompute the Elser vs. BM25 vs. BGE comparison of Table 3 on this independently pooled gold set. If Elser's Recall@5 advantage over BM25 shrinks by more than, say, 0.05, or if a large fraction of the original gold passages are found to be ELSER top-10 candidates presented during creation, the reported retrieval superiority is construction-biased and should be reported as such.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.3 uses MTRAG's reference passages as the gold set for retrieval. Those passages were produced in Section 4.3 through a chat application whose default candidate passages came from an ELSER index; annotators were allowed to edit or add passages via custom search, but the initial retrieval was ELSER's. The paper itself states in Section 5.1 that 'since we use Elser for retrieval during data creation, there may be some biases towards Elser,' yet no analysis quantifies the overlap between reference passages and ELSER's initial candidate list. If most gold passages are drawn from ELSER's top results, then BM25 and BGE are being evaluated against a label set that ELSER had direct access to, making the conclusion that ELSER outperforms these systems (Table 3) partially circular. This does not undermine the benchmark's generation difficulty or human-evaluation evidence, but it does weaken the retrieval-focused part of the central claim ('the need for strong retrieval systems'). The concern is not that the authors hid the issue; it is that the paper reports a comparative retrieval result without the control needed to separate construction bias from intrinsic retriever quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MTRAG, a human-generated multi-turn RAG benchmark containing 110 conversations (842 tasks) over four domains, annotated along question type, multi-turn type, and answerability. Conversations are created with an interactive chat application in which annotators write questions, adjust retrieved passages, and repair LLM responses; the resulting reference answers and passage sets are the benchmark labels. The paper evaluates retrieval with BM25, BGE, and ELSER under last-turn and query-rewrite strategies, and evaluates nine LLM generators under reference, reference+RAG, and full-RAG settings, supplemented by a human evaluation of two frontier models and by a synthetic companion benchmark MTRAG-S. The central findings are that all evaluated systems struggle on MTRAG, especially on unanswerable questions and later turns, and that several automatic metrics correlate only weakly with human judgments.","tokens_in":25860,"tokens_out":9038,"duration_ms":88809,"significance":"If the benchmark stands up to scrutiny, MTRAG fills a real gap: multi-turn RAG evaluation that includes active retrieval, long-form answers, unanswerable questions, and multiple domains. The release of the benchmark, the transparent reporting of annotation details and agreement, the inclusion of a human evaluation, and the honest discussion of weak automatic-metric correlations are concrete strengths. The headline finding that frontier LLM RAG systems degrade on later turns and unanswerable inputs is credible, falsifiable, and useful to the community. The main risk is that the comparative retrieval result (ELSER outperforming BM25 and BGE in Table 3) is confounded by the use of ELSER as the retrieval engine during data creation; the paper acknowledges this possibility in Section 5.1 but does not bound it quantitatively.","major_comments":[{"comment":"The retrieval comparison is potentially circular in a way that is load-bearing for the retrieval conclusions. The reference passages used as gold labels in Section 5.3 were selected inside an annotation interface whose initial candidate list was produced by the ELSER index (Section 4.3), and Section 5.1 explicitly says \"there may be some biases towards Elser.\" Because Table 3 then shows that ELSER outperforms BM25 and BGE, the result may partly reflect construction bias rather than intrinsic retrieval quality. The paper does not quantify the fraction of gold passages that came from ELSER's initial candidate list versus passages added by annotators through custom search. Please add: (i) a measurement of the overlap between ELSER's top-k retrieval results and the reference passage sets per turn; (ii) retrieval results restricted to the subset of turns or passages that were not present in ELSER's initial candidate list; and (iii) if feasible, an independent human relevance annotation on a small subset of turns to re-score the retrievers without the construction bias. This does not undermine the generation-difficulty results or the human-evaluation evidence, but it is required before the comparative retrieval conclusions can be taken at face value.","section":"5.1, 5.3, Table 3"}],"minor_comments":[{"comment":"The claim that MTRAG is \"the first end-to-end human-generated multi-turn RAG benchmark\" is stated too absolutely, since iKAT and RAD-Bench also contain human-generated multi-turn RAG dialogs; the paper's differentiator is the specific combination of active retrieval, long-form answers, unanswerable questions, and multiple domains, and the wording should be qualified accordingly.","section":"1, Table 1"},{"comment":"The decision to use only the last user turn as the retrieval query is motivated by the statement that full-conversation queries \"consistently under-performed,\" but no supporting numbers or table are provided; please include the comparison in the main text or in an appendix so the methodological choice is verifiable.","section":"5.2"},{"comment":"Please report the numerical per-property Spearman correlations behind Figure 4, particularly for RLF versus human faithfulness, since the aggregate win-rate correlation of 0.01 in Table 7 makes the headline usefulness of RLF otherwise difficult to assess and the paper's decision to use RLF as a main metric currently depends on unpublished per-property values.","section":"8, Table 7, Figure 4"},{"comment":"The text should state explicitly that Table 5 reports all three retrieval settings on the same 426-task subset, not only the Reference+RAG setting; the caption says n=426, but the main text currently introduces the restriction only in the context of Reference+RAG.","section":"6.1, Table 5"},{"comment":"The paper should state explicitly that MTRAG-S has no human validation, so the consistently higher faithfulness scores on synthetic data (Table 14) could reflect biases in the automatic metrics rather than genuine quality differences; the current discussion mentions possible explanations but does not foreground this absence of human ground truth.","section":"9, Appendix G"}],"recommendation":"major_revision","confidential_remarks":"The principal scientific risk is the ELSER circularity in the retrieval evaluation. If the requested overlap analysis shows that most gold passages were not drawn from ELSER's initial candidate lists, the paper should be acceptable after the minor revisions; if the overlap is high, the retrieval comparisons will need to be substantially reframed or downweighted. The benchmark, human evaluation, and generation findings are strong enough that the paper warrants revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: MTRAG is a genuinely useful benchmark and the paper deserves a serious referee. It is not the first multi-turn RAG dataset, but it is the first to combine active retrieval, long-form answers, unanswerable questions, and multi-domain coverage in one human-generated resource, and the construction process is the real story: annotators interacted with a live RAG agent and repaired both passages and answers in real time. That is a meaningful departure from static dataset collection, and the paper documents the process with unusual honesty — repair rates, edit scores, annotator agreement, and even the weak correlations between automatic metrics and human judgments are all reported.\n\nThe strongest part is the evaluation. The human evaluation on 159 tasks shows the reference answers are preferred over GPT-4o and Llama 3.1 405B by a wide margin, and models indeed struggle on unanswerable questions and later turns. The metric correlation analysis is also careful; the authors don't oversell LLM-as-a-judge. Released code and data are a plus.\n\nThe main soft spot is the retrieval comparison. The same ELSER retriever used during data creation is evaluated against reference passages that were selected inside an ELSER-powered annotation interface. The paper discloses this and says there 'may be some biases towards Elser,' but it never quantifies the overlap between reference passages and ELSER's initial candidate lists. That makes Table 3 partially circular: BM25 and BGE are being scored against labels that ELSER had direct access to. This weakens the retrieval-focused part of the central claim, but it does not touch the generation results, which rest on human evaluation. A control analysis (e.g., reporting retrieval results only on passages added by annotators via custom search, or computing how often the reference passages came from ELSER's top-10) would fix it.\n\nA second, minor point: the benchmark's difficulty is partly engineered — annotators were told to aim for ~25% unanswerable questions and to repair 92% of model responses. So 'state-of-the-art systems struggle on MTRAG' should be read as 'struggle on this deliberately constructed test,' which is fine for a benchmark, but the paper could say it more plainly. The size (110 conversations) is small, though acceptable for a human-generated resource of this density.\n\nWho this is for: anyone evaluating RAG systems, especially multi-turn or conversational RAG. I'd bring it to a reading group and would cite it if I were working on RAG evaluation. Recommendation: accept the paper for peer review; ask for a revision that addresses the retrieval bias quantification. The benchmark itself should be released and used.","headline":"A useful, honestly reported multi-turn RAG benchmark; the retrieval comparison needs a bias control before the ELSER advantage can be trusted.","tokens_in":26401,"tokens_out":2383,"would_cite":true,"duration_ms":21573,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new human-built benchmark shows RAG systems stumble on multi-turn chats.","keywords":["retrieval-augmented generation","multi-turn benchmark","conversational question answering","RAG evaluation","unanswerable questions","query rewriting","human evaluation","synthetic data"],"falsifier":"Re-annotate a random subset of MTRAG conversations using a different retriever/generator combination (for example, a dense retriever with GPT-4o) under the same instructions, then compare retrieval rankings and model scores on the re-annotated subset. If ELSER's advantage over BM25 and BGE shrinks or disappears, or if the frontier models' performance gaps versus the reference answers narrow substantially, the reported difficulty and retriever rankings are largely artifacts of the construction pipeline rather than intrinsic properties of multi-turn RAG.","tokens_in":25447,"feed_emoji":"💬","tokens_out":6086,"duration_ms":49157,"temperature":0.7,"pith_summary":"Retrieval-augmented generation (RAG) is usually tested one question at a time, but real assistants face multi-turn conversations with follow-ups, references to earlier answers, and questions that cannot be answered from the documents. This paper presents MTRAG, a human-generated benchmark of 110 conversations (842 tasks) across four document corpora, built by having annotators converse with a live RAG agent and edit both the retrieved passages and the agent's responses. The paper claims MTRAG is the first end-to-end human-generated multi-turn RAG benchmark that reflects real-world conversation properties, and that even state-of-the-art LLM RAG systems struggle on it. The central evidence is that scores drop sharply on later turns and on unanswerable questions, and that human annotators consistently prefer the reference answers over frontier model responses. If the benchmark is widely adopted, it would change how RAG systems are measured, pushing evaluation beyond single-turn accuracy toward conversation-level retrieval, refusal, and faithfulness.","feed_headline":"RAG models fail late turns and unanswerable questions in new benchmark","feed_subtitle":"MTRAG's 842 human-written tasks test retrieval and generation together; even GPT-4o and Llama 405B lag on later turns.","key_machinery":"The load-bearing object is the benchmark itself, produced by a creation protocol in which human annotators interact with a live RAG agent — an ELSER sparse retriever with a Mixtral 8x7B generator — and at each turn modify the retrieved passage set and repair the generated response before accepting it. This protocol is what makes the data 'end-to-end human-generated': the conversations inherit real multi-turn phenomena like co-reference, clarification, and unanswerability, while the 92% repair rate marks turns that the generator could not handle alone. The evaluation machinery rests on answerability-conditioned metrics (RBalg, RBllm, and the faithfulness judge RLF), gated by an IDK detector that decides whether a response admits ignorance, without which unanswerable questions would be scored misleadingly.","core_discovery":"On its own terms, the paper establishes that multi-turn RAG is substantially harder than single-turn RAG for current systems, and that the difficulty concentrates in specific, measurable places: later turns, non-standalone questions, and unanswerable questions. Retrieval with query rewriting still drops from 0.89 recall@5 on first turns to 0.47 on later turns, and generation scores fall as the tasks become less answerable, with models frequently hallucinating instead of saying they do not know. The paper further claims that no existing multi-turn benchmark combines active retrieval, long-form answers, unanswerable questions, and multiple domains, and that MTRAG fills that gap with human-generated conversations whose reference answers are preferred by human judges over responses from GPT-4o and Llama 3.1 405B.","pith_inferences":["Because the reference data were created with ELSER as the retriever, the measured ELSER advantage may be partly an artifact of construction; re-running the annotation protocol with a different retriever (e.g., a dense model) on a subset would quantify this bias.","The 92% repair rate suggests that an LLM-judge or synthetic-data shortcut trained on this benchmark would inherit the editing decisions of the annotators; a useful extension is to release the pre-repair agent outputs alongside the benchmark so the community can measure the 'repair gap' directly.","If MTRAG becomes a standard evaluation, downstream leaderboards should adopt IDK-conditioned scoring, since unanswerable questions are where the benchmark separates models most sharply.","The paper's finding that most automated metrics correlate poorly with human win-rate implies that multi-turn RAG needs its own evaluation research, not just reuse of single-turn or chat metrics."],"forward_implications":["Retrievers must be judged on later turns and non-standalone questions, not just first turns: the paper's best retriever drops from 0.89 to 0.47 recall@5 from the first to later turns.","Generators need explicit refusal behavior: on unanswerable questions, models that say 'I don't know' are scored as better than those that hallucinate, and answerability-conditioned metrics are necessary to see this.","Adding noisy retrieved passages (Reference+RAG versus Reference) consistently degrades generation, so RAG evaluation should report both an upper-bound and a full-pipeline setting.","Query rewriting is a necessary component for non-standalone questions; without it retrieval on later turns is worse, and with it non-standalone questions still lag standalone ones (0.42 vs 0.48 recall@5).","Synthetic conversations are not a drop-in substitute: models score higher faithfulness on MTRAG-S than on human MTRAG, suggesting synthetic data under-represents the difficulty of real multi-turn dialogue."],"supporting_citations":[{"why":"Supplies the CLAPNQ Wikipedia corpus and its long-form answer style, one of MTRAG's four domains and a source of seed questions.","marker":"(Rosenthal et al., 2024)"},{"why":"Supplies the FiQA finance corpus of StackExchange posts, a second domain and seed-question source.","marker":"(Maia et al., 2018)"},{"why":"Describes the Mixtral model used as the generator in the live annotation agent and as one of the evaluated systems.","marker":"(Jiang et al., 2024)"},{"why":"RAD-Bench, the prior multi-turn RAG benchmark whose reference-based judge is adapted into the RBllm metric.","marker":"(Kuo et al., 2024)"},{"why":"RAGAS, the source of the faithfulness metric RLF and the answer-relevance metric RLR used in automated evaluation.","marker":"(Es et al., 2024)"},{"why":"The synthetic multi-document dialog framework extended to generate the companion benchmark MTRAG-S.","marker":"(Lee et al., 2024)"},{"why":"TREC iKAT, a conversational and interactive knowledge assistant test collection used as a comparison point in Table 1.","marker":"(Aliannejadi et al., 2024)"},{"why":"FaithDial, the only prior multi-turn RAG benchmark with active retrieval, used as a comparison point in Table 1.","marker":"(Dziri et al., 2022)"},{"why":"Establishes the contextual query rewriting strategy that the paper shows improves retrieval on non-standalone questions.","marker":"(Zhou et al., 2023)"}],"fun_headline_variants":["MTRAG benchmark: Multi-turn RAG stumps even GPT-4o and Llama 405B","New benchmark shows multi-turn RAG breaks down on later turns","MTRAG: Multi-turn RAG benchmark exposes LLM failures beyond single-turn","Even top models hallucinate on multi-turn RAG, new benchmark shows"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the annotation process did not bias the benchmark: specifically, that having annotators use an ELSER retriever and Mixtral generator, and edit their outputs, produced reference passages and answers that reflect genuine multi-turn conversation difficulty rather than the quirks of that particular retrieval and generation pair.","fun_headline_variants_meta":{"raw":{"variants":["MTRAG benchmark: Multi-turn RAG stumps even GPT-4o and Llama 405B","New benchmark shows multi-turn RAG breaks down on later turns","MTRAG: Multi-turn RAG benchmark exposes LLM failures beyond single-turn","Even top models hallucinate on multi-turn RAG, new benchmark shows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000715,"raw_usage":{"total_tokens":3200,"prompt_tokens":913,"completion_tokens":2287,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":2212}},"tokens_in":529,"tokens_out":2287,"duration_ms":14840,"temperature":1.0,"reasoning_tokens":2212,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:52:26.199575+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random subset of MTRAG conversations using a different retriever/generator combination (for example, a dense retriever with GPT-4o) under the same instructions, then compare retrieval rankings and model scores on the re-annotated subset. If ELSER's advantage over BM25 and BGE shrinks or disappears, or if the frontier models' performance gaps versus the reference answers narrow substantially, the reported difficulty and retriever rankings are largely artifacts of the construction pipeline rather than intrinsic properties of multi-turn RAG.","supporting_citations":[],"review_version":1}