Pith. sign in

REVIEW 1 major objections 5 minor 6 cited by

MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems

T0 review · 1 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new human-built benchmark shows RAG systems stumble on multi-turn chats.

desk verdict A useful, honestly reported multi-turn RAG benchmark; the retrieval comparison needs a bias control before the ELSER advantage can be trusted. read the letter →

arxiv 2501.03468 v1 pith:OPQDGLA5 submitted 2025-01-07 cs.CL cs.AI

Yannis Katsis , Sara Rosenthal , Kshitij Fadnis , Chulaka Gunasekara , Young-Suk Lee , Lucian Popa , Vraj Shah , Huaiyu Zhu
show 2 more authors
Danish Contractor Marina Danilevsky
This is my paper · ORCID
classification cs.CLcs.AI
keywords retrieval-augmentedgenerationmulti-turnbenchmarkconversationalquestionansweringRAGevaluationunanswerablequestionsqueryrewritinghumansyntheticdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Retrieval-augmented generation (RAG) is usually tested one question at a time, but real assistants face multi-turn conversations with follow-ups, references to earlier answers, and questions that cannot be answered from the documents. This paper presents MTRAG, a human-generated benchmark of 110 conversations (842 tasks) across four document corpora, built by having annotators converse with a live RAG agent and edit both the retrieved passages and the agent's responses. The paper claims MTRAG is the first end-to-end human-generated multi-turn RAG benchmark that reflects real-world conversation properties, and that even state-of-the-art LLM RAG systems struggle on it. The central evidence is that scores drop sharply on later turns and on unanswerable questions, and that human annotators consistently prefer the reference answers over frontier model responses. If the benchmark is widely adopted, it would change how RAG systems are measured, pushing evaluation beyond single-turn accuracy toward conversation-level retrieval, refusal, and faithfulness.

What carries the argument

The load-bearing object is the benchmark itself, produced by a creation protocol in which human annotators interact with a live RAG agent — an ELSER sparse retriever with a Mixtral 8x7B generator — and at each turn modify the retrieved passage set and repair the generated response before accepting it. This protocol is what makes the data 'end-to-end human-generated': the conversations inherit real multi-turn phenomena like co-reference, clarification, and unanswerability, while the 92% repair rate marks turns that the generator could not handle alone. The evaluation machinery rests on answerability-conditioned metrics (RBalg, RBllm, and the faithfulness judge RLF), gated by an IDK detector that decides whether a response admits ignorance, without which unanswerable questions would be scored misleadingly.

What would settle it

Re-annotate a random subset of MTRAG conversations using a different retriever/generator combination (for example, a dense retriever with GPT-4o) under the same instructions, then compare retrieval rankings and model scores on the re-annotated subset. If ELSER's advantage over BM25 and BGE shrinks or disappears, or if the frontier models' performance gaps versus the reference answers narrow substantially, the reported difficulty and retriever rankings are largely artifacts of the construction pipeline rather than intrinsic properties of multi-turn RAG.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that multi-turn RAG is substantially harder than single-turn RAG for current systems, and that the difficulty concentrates in specific, measurable places: later turns, non-standalone questions, and unanswerable questions. Retrieval with query rewriting still drops from 0.89 recall@5 on first turns to 0.47 on later turns, and generation scores fall as the tasks become less answerable, with models frequently hallucinating instead of saying they do not know. The paper further claims that no existing multi-turn benchmark combines active retrieval, long-form answers, unanswerable questions, and multiple domains, and that MTRAG fills that gap with human-generated conversations whose reference answers are preferred by human judges over responses from GPT-4o and Llama 3.1 405B.

Load-bearing premise

The load-bearing premise is that the annotation process did not bias the benchmark: specifically, that having annotators use an ELSER retriever and Mixtral generator, and edit their outputs, produced reference passages and answers that reflect genuine multi-turn conversation difficulty rather than the quirks of that particular retrieval and generation pair.

Editorial extensions

If this is right

  • Retrievers must be judged on later turns and non-standalone questions, not just first turns: the paper's best retriever drops from 0.89 to 0.47 recall@5 from the first to later turns.
  • Generators need explicit refusal behavior: on unanswerable questions, models that say 'I don't know' are scored as better than those that hallucinate, and answerability-conditioned metrics are necessary to see this.
  • Adding noisy retrieved passages (Reference+RAG versus Reference) consistently degrades generation, so RAG evaluation should report both an upper-bound and a full-pipeline setting.
  • Query rewriting is a necessary component for non-standalone questions; without it retrieval on later turns is worse, and with it non-standalone questions still lag standalone ones (0.42 vs 0.48 recall@5).
  • Synthetic conversations are not a drop-in substitute: models score higher faithfulness on MTRAG-S than on human MTRAG, suggesting synthetic data under-represents the difficulty of real multi-turn dialogue.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reference data were created with ELSER as the retriever, the measured ELSER advantage may be partly an artifact of construction; re-running the annotation protocol with a different retriever (e.g., a dense model) on a subset would quantify this bias.
  • The 92% repair rate suggests that an LLM-judge or synthetic-data shortcut trained on this benchmark would inherit the editing decisions of the annotators; a useful extension is to release the pre-repair agent outputs alongside the benchmark so the community can measure the 'repair gap' directly.
  • If MTRAG becomes a standard evaluation, downstream leaderboards should adopt IDK-conditioned scoring, since unanswerable questions are where the benchmark separates models most sharply.
  • The paper's finding that most automated metrics correlate poorly with human win-rate implies that multi-turn RAG needs its own evaluation research, not just reuse of single-turn or chat metrics.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper introduces MTRAG, a human-generated multi-turn RAG benchmark containing 110 conversations (842 tasks) over four domains, annotated along question type, multi-turn type, and answerability. Conversations are created with an interactive chat application in which annotators write questions, adjust retrieved passages, and repair LLM responses; the resulting reference answers and passage sets are the benchmark labels. The paper evaluates retrieval with BM25, BGE, and ELSER under last-turn and query-rewrite strategies, and evaluates nine LLM generators under reference, reference+RAG, and full-RAG settings, supplemented by a human evaluation of two frontier models and by a synthetic companion benchmark MTRAG-S. The central findings are that all evaluated systems struggle on MTRAG, especially on unanswerable questions and later turns, and that several automatic metrics correlate only weakly with human judgments.

Significance. If the benchmark stands up to scrutiny, MTRAG fills a real gap: multi-turn RAG evaluation that includes active retrieval, long-form answers, unanswerable questions, and multiple domains. The release of the benchmark, the transparent reporting of annotation details and agreement, the inclusion of a human evaluation, and the honest discussion of weak automatic-metric correlations are concrete strengths. The headline finding that frontier LLM RAG systems degrade on later turns and unanswerable inputs is credible, falsifiable, and useful to the community. The main risk is that the comparative retrieval result (ELSER outperforming BM25 and BGE in Table 3) is confounded by the use of ELSER as the retrieval engine during data creation; the paper acknowledges this possibility in Section 5.1 but does not bound it quantitatively.

major comments (1)
  1. [5.1, 5.3, Table 3] The retrieval comparison is potentially circular in a way that is load-bearing for the retrieval conclusions. The reference passages used as gold labels in Section 5.3 were selected inside an annotation interface whose initial candidate list was produced by the ELSER index (Section 4.3), and Section 5.1 explicitly says "there may be some biases towards Elser." Because Table 3 then shows that ELSER outperforms BM25 and BGE, the result may partly reflect construction bias rather than intrinsic retrieval quality. The paper does not quantify the fraction of gold passages that came from ELSER's initial candidate list versus passages added by annotators through custom search. Please add: (i) a measurement of the overlap between ELSER's top-k retrieval results and the reference passage sets per turn; (ii) retrieval results restricted to the subset of turns or passages that were not present in ELSER's initial candidate list; and (iii) if feasible, an independent human relevance annotation on a small subset of turns to re-score the retrievers without the construction bias. This does not undermine the generation-difficulty results or the human-evaluation evidence, but it is required before the comparative retrieval conclusions can be taken at face value.
minor comments (5)
  1. [1, Table 1] The claim that MTRAG is "the first end-to-end human-generated multi-turn RAG benchmark" is stated too absolutely, since iKAT and RAD-Bench also contain human-generated multi-turn RAG dialogs; the paper's differentiator is the specific combination of active retrieval, long-form answers, unanswerable questions, and multiple domains, and the wording should be qualified accordingly.
  2. [5.2] The decision to use only the last user turn as the retrieval query is motivated by the statement that full-conversation queries "consistently under-performed," but no supporting numbers or table are provided; please include the comparison in the main text or in an appendix so the methodological choice is verifiable.
  3. [8, Table 7, Figure 4] Please report the numerical per-property Spearman correlations behind Figure 4, particularly for RLF versus human faithfulness, since the aggregate win-rate correlation of 0.01 in Table 7 makes the headline usefulness of RLF otherwise difficult to assess and the paper's decision to use RLF as a main metric currently depends on unpublished per-property values.
  4. [6.1, Table 5] The text should state explicitly that Table 5 reports all three retrieval settings on the same 426-task subset, not only the Reference+RAG setting; the caption says n=426, but the main text currently introduces the restriction only in the context of Reference+RAG.
  5. [9, Appendix G] The paper should state explicitly that MTRAG-S has no human validation, so the consistently higher faithfulness scores on synthetic data (Table 14) could reflect biases in the automatic metrics rather than genuine quality differences; the current discussion mentions possible explanations but does not foreground this absence of human ground truth.

Circularity Check

1 steps flagged · score 4.0 of 10

Retrieval comparison is partially circular: ELSER created the candidate passages that became MTRAG's gold passages, and is then evaluated against those same gold passages.

  1. other [Sections 4.3, 5.1, and 5.3 (Table 3)]
    "Once annotators wrote a question, the chat application queried the retriever for potentially relevant passages, ran the generator to produce an agent response based on the retrieved passages, and presented everything to the user for repair. ... Since we use Elser for retrieval during data creation, there may be some biases towards Elser."

    The retrieval evaluation's gold set is the reference passages marked during creation (Section 5.1), and those passages were initially surfaced to annotators by the ELSER-based chat application; annotators could edit or add passages, but ELSER's index was the default candidate source. Evaluating ELSER against these reference passages therefore measures, at least in part, how well ELSER reproduces candidate lists it itself helped generate during annotation. The paper acknowledges the possible bias in a single sentence but provides no overlap analysis between reference passages and ELSER's initial candidates, so ELSER's reported advantage in Table 3 is not an independent comparison.

full rationale

Most of the paper's claims are self-contained. MTRAG is a new human-annotated benchmark with publicly released data; the annotation process uses a live RAG agent, but the gold responses are repaired by annotators and independently preferred in a human evaluation (Section 7), so the 'models struggle' finding does not reduce to the construction machinery. The generation comparison is also largely insulated: Mixtral 8x7B, the generator used during creation, is excluded from evaluation. No load-bearing self-citation or uniqueness-import pattern appears: prior author citations (CLAPNQ, Lee et al. 2024) are used as corpora or as a synthetic-generation framework, not to force the benchmark's conclusions. The main circular liability is confined to the retrieval comparison: reference passages were seeded and edited inside an ELSER-based chat interface, and the same reference passages are then used as the gold set to compare ELSER against BM25 and BGE. The paper itself states 'there may be some biases towards Elser' but does not quantify how much of ELSER's Table 3 advantage is due to this construction bias. That makes the retrieval-ranking result partially endogenous, though the generation and human-evaluation evidence for benchmark difficulty remains independent.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The benchmark's validity rests on the quality and neutrality of the human annotation process, on the FANC properties as the right criteria, and on the reliability of the automatic judges used in the experiments. The most consequential assumption is that using ELSER during creation did not bias the evaluation toward ELSER, which the paper acknowledges but does not test.

free parameters (3)
  • target proportion of unanswerable questions = ~25% (max 2 per conversation)
    Annotator instructions in Appendix B.2 explicitly aim for about 25% unanswerable questions per conversation, with no more than two unanswerable turns. This shapes the benchmark's difficulty and the 'models struggle on unanswerables' finding.
  • passage chunk size = 512 tokens with 100-token overlap
    Section 4.2: documents are split into passages of 512 tokens with an overlap stride of 100 tokens. This chunking affects retrieval difficulty and passage diversity.
  • number of retrieved passages for generation (N) = 5
    Section 6: N=5 was selected because it achieves considerable improvement compared to top-3 while remaining manageable. This affects the Reference+RAG and Full RAG generation results.
assumptions (4)
  • domain assumption The reference answers and relevant passage annotations produced by the annotators are correct and unbiased gold standards.
    The entire evaluation treats the human-repaired answers and annotator-selected passages as ground truth (Sections 4.3, 7). If the repair process introduced a specific style or completeness bias, measured model deficits would be inflated.
  • domain assumption ELSER retrieval during creation does not systematically bias the benchmark toward ELSER in the final evaluation.
    Section 5.1 notes there may be biases toward Elser, but the paper does not quantify or correct for them. The retrieval comparison (ELSER best) depends on this assumption.
  • domain assumption The FANC response properties (faithfulness, appropriateness, naturalness, completeness) are the correct and sufficient criteria for RAG quality.
    Section 3.2 defines reference responses via FANC; all metrics and human evaluation are anchored on these properties. If completeness is overweighted, models that are concise but correct would be unfairly penalized.
  • domain assumption The IDK judge and LLM-based metrics are reliable enough to support the reported comparisons.
    The IDK judge achieves >97% accuracy (Section 6.3.3), but the RBllm and RLF judges are not validated beyond correlation with the paper's own human eval, and some correlations are weak (Table 7).
invented entities (2)
  • MTRAG benchmark independent evidence
    purpose: Human-generated multi-turn RAG evaluation benchmark with 110 conversations, 842 tasks, four domains, covering active retrieval, long answers, unanswerable questions, and multi-domain.
    The benchmark is released on GitHub, so external users can test models on it, providing an independent falsifiable handle on its utility.
  • MTRAG-S synthetic benchmark independent evidence
    purpose: Companion synthetic benchmark of 200 conversations over the same corpora for studying automation and comparing human vs synthetic data.
    Also released; models can be evaluated on it. Its usefulness is testable, though the paper shows faithfulness scores are higher on synthetic data, which may indicate lower difficulty.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems." pith.science (2026). https://pith.science/paper/OPQDGLA5

@misc{pith2026250103468,
  author       = {Pith},
  title        = {Pith review of: MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OPQDGLA5}},
  note         = {Machine review of arXiv:2501.03468}
}
read the original abstract

Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on multi-turn RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present MTRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. MTRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judge evaluation. Our human and automatic evaluations show that even state-of-the-art LLM RAG systems struggle on MTRAG. We demonstrate the need for strong retrieval and generation systems that can handle later turns, unanswerable questions, non-standalone questions, and multiple domains. MTRAG is available at https://github.com/ibm/mt-rag-benchmark.

Figures

Figures reproduced from arXiv: 2501.03468 by the authors.

Figure 1
Figure 1. 5/8 turns of a conversation from the CLAPNQ domain. The conversation is enriched with question di￾mensions, passage diversity, and repair. The answerabil￾ity is shown using the agent response color: answerable, unanswerable, and partial. The multi-turn type is shown using the question circle: follow-up and clarification. The different relevant passages highlight diversity and the original text shows a repair from th… view at source ↗
Figure 2
Figure 2. Distribution of tasks in MTRAG based on each of the benchmark’s dimensions. changes could negatively affect the conversation flow. During this phase, most conversations were kept (see review process details in Appendix B.3). Our human evaluation (Section 7) shows that the reference responses are preferred by humans. 4.4 Data Statistics This process yielded a benchmark of 110 conver￾sations (29 ClapNQ, 27 FiQA, 28 Go… view at source ↗
Figure 3
Figure 3. Generation results in the Reference (•) retrieval setting using a single metric, RBalg, on three different dimensions: (a) answerability, (b) turns, and (c) domains dicting A. Finally, conversational questions also require special handling for evaluation. Since only 10 tasks fall in this category, we exclude them from the experiments, leaving their study as future work. 6.4 Generation: Evaluation Results The overall… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Weighted Spearman correlation of human evaluation with the automated metrics on the answerable subset for the GPT-4o and Llama 3.1 405B Inst. models. 8 Automatic Evaluation Human evaluation is not feasible as a long term solution for evaluating models as it does not sc…
Figure 5
Figure 5. Figure 5: Query rewrite prompt For the retrieval experiments, we implemented query rewriting by sending the prompt of [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Prompt used for the RBllm judge F.2 RLMTB implementation The MT-Bench judge was originally designed for evaluating conversations containing exactly two turns (i.e., user question/agent response pairs), with a focus on properties such as helpfulness, depth, and creativi…
Figure 7
Figure 7. Figure 7: Prompt used for the RLMTB judge F.3 IDK correction to the metrics To determine whether a model response corre￾sponds to an IDK (I-Don’t-know) answer, we use an LLM judge that we developed for this purpose. The judge sends the prompt of [PITH_FULL_IMAGE:figures/full_fi…
Figure 9
Figure 9. Figure 9: Generation results in the Reference (•) retrieval setting by question type based on three metrics: RBalg, RBllm, and RLF (a) Based on RBalg (b) Based on RBllm (c) Based on RLF [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Generation results in the Reference (•) retrieval setting by multi-turn type based on three metrics: RBalg, RBllm, and RLF [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RAGAPHENE: A RAG Annotation Platform with Human Enhancements and Edits

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A platform that helps annotators create, review, and run small experiments on multi-turn retrieval-augmented generation conversations for benchmarking.

  2. FlexRAG: A Flexible and Comprehensive Framework for Retrieval-Augmented Generation

    cs.CL 2025-06 conditional novelty 5.0 of 10

    FlexRAG is a modular, open-source RAG framework with text, multimodal, and web retrieval, plus evaluation tools and efficient memory-mapped indexing.

  3. Benchmarking Poisoning Attacks against Retrieval-Augmented Generation

    cs.CR 2025-05 conditional novelty 5.0 of 10

    A unified benchmark evaluation finds that existing RAG poisoning attacks remain effective on standard QA datasets, drop on expanded knowledge bases, and are only partially mitigated by current defenses.

  4. Granite Embedding R2 Models

    cs.CL 2025-08 conditional novelty 4.0 of 10

    Granite Embedding R2 is an Apache-2.0 family of ModernBERT-based retrieval and reranking models that posts high average scores on several benchmarks but falls short of top code-retrieval and reranking baselines.

  5. Conversational Search: From Fundamentals to Frontiers in the LLM Era

    cs.IR 2025-06 unverdicted novelty 3.0 of 10

    A four-page proposal for a SIGIR 2025 half-day tutorial connecting conversational search fundamentals with LLM-era techniques; it contains no new experiments, data, or results.

  6. Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey

    cs.CL 2025-04 conditional novelty 3.0 of 10

    A review that organizes RAG evaluation into internal and external categories, catalogs dozens of benchmarks, and analyzes evaluation practices in 582 conference papers.

Reference graph

Works this paper leans on

44 extracted references · 5 canonical work pages · cited by 6 Pith papers

  1. [1]

    Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2024. https://doi.org/10.1162/tacl_a_00667 Evaluating correctness and faithfulness of instruction-following models for question answering . Transactions of the Association for Computational Linguistics, 12:681--699

    doi:10.1162/tacl_a_00667 2024

  2. [2]

    Mohammad Aliannejadi, Zahra Abbasiantaeb, Shubham Chatterjee, Jeffrey Dalton, and Leif Azzopardi. 2024. https://doi.org/10.1145/3626772.3657860 TREC iKAT 2023: A test collection for evaluating conversational and interactive knowledge assistants . In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retri...

    arXiv 2024

  3. [3]

    Yang Bai and Daisy Zhe Wang. 2021. https://arxiv.org/abs/2109.12264 More than reading comprehension: A survey on datasets and metrics of textual question answering . CoRR, abs/2109.12264

    arXiv 2021

  4. [4]

    Barla Cambazoglu, Mark Sanderson, Falk Scholer, and Bruce Croft

    B. Barla Cambazoglu, Mark Sanderson, Falk Scholer, and Bruce Croft. 2021. https://doi.org/10.1145/3483382.3483389 A review of public datasets in question answering research . SIGIR Forum, 54(2)

    work page arXiv 2021

  5. [5]

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. https://doi.org/10.1609/aaai.v38i16.29728 Benchmarking large language models in retrieval-augmented generation . Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):17754--17762

    doi:10.1609/aaai.v38i16.29728 2024

  6. [6]

    Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/D18-1241 Q u AC : Question answering in context . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174--2184, Brussels, Belgium. Association for Computational Linguistics

    doi:10.18653/v1/d18-1241 2018

  7. [7]

    Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. https://openreview.net/forum?id=r1l73iRqKm Wizard of wikipedia: Knowledge-powered conversational agents . In International Conference on Learning Representations

    2019

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany...

    arXiv 2024

Show all 44 references
  1. [9]

    Ponti, and Siva Reddy

    Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M. Ponti, and Siva Reddy. 2022. https://doi.org/10.1162/tacl_a_00529 F aith D ial: A faithful benchmark for information-seeking dialogue . Transactions of the Association for Computational Linguistics, 10:...

    2022 doi

  2. [10]

    Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. https://aclanthology.org/2024.eacl-demo.16 RAGA s: Automated evaluation of retrieval augmented generation . In Proceedings of the 18th Conference of the European Chapter of the Association for Computatio...

    2024

  3. [11]

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. https://doi.org/10.18653/v1/P19-1346 ELI 5: Long form question answering . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558--356...

    2019 doi

  4. [12]

    Song Feng, Siva Sankalp Patel, Hui Wan, and Sachindra Joshi. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.498 M ulti D oc2 D ial: Modeling dialogues grounded in multiple documents . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,...

    2021 doi

  5. [13]

    Robert Friel, Masha Belyi, and Atindriyo Sanyal. 2024. https://arxiv.org/abs/2407.11005 RAGBench : Explainable benchmark for retrieval-augmented generation systems . Preprint, arXiv:2407.11005

    2024 arXiv

  6. [14]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  7. [15]

    Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.495 Active retrieval augmented generation . In Proceedings of the 2023 Conference on Empirical Methods in...

    2023 doi

  8. [16]

    Tzu-Lin Kuo, Feng-Ting Liao, Mu-Wei Hsieh, Fu-Chieh Chang, Po-Chun Hsu, and Da-Shan Shiu. 2024. https://arxiv.org/abs/2409.12558 RAD-Bench : Evaluating large language models capabilities in retrieval augmented dialogues . Preprint, arXiv:2409.12558

    2024 arXiv

  9. [17]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  10. [18]

    Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/P19-1612 Latent retrieval for weakly supervised open domain question answering . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086--6096, ...

    2019 doi

  11. [19]

    Young-Suk Lee, Chulaka Gunasekara, Danish Contractor, Ramón Fernandez Astudillo, and Radu Florian. 2024. https://arxiv.org/abs/arXiv:2409.11500 Multi-document grounded multi-turn synthetic dialog generation

    2024 arXiv

  12. [20]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf...

    2020

  13. [21]

    Chien-Chang Lin, Anna Y. Q. Huang, and Stephen J. H. Yang. 2023. https://doi.org/10.3390/su15054012 A review of AI -driven conversational chatbots implementation methodologies and challenges (1999–2022) . Sustainability, 15(5)

    2023 doi

  14. [22]

    Macedo Maia, Siegfried Handschuh, Andr\' e Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. https://doi.org/10.1145/3184558.3192301 WWW'18 open challenge: Financial opinion mining and question answering . In Companion Proceedings of the The Web...

    2018

  15. [23]

    Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. https://doi.org/10.18653/v1/2024.acl-long.585 RAGT ruth: A hallucination corpus for developing trustworthy retrieval-augmented language models . In Proceedings of the 62n...

    2024 doi

  16. [24]

    Bruce Croft, and Mohit Iyyer

    Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W. Bruce Croft, and Mohit Iyyer. 2020. https://doi.org/10.1145/3397271.3401110 Open-retrieval conversational question answering . In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information ...

    2020

  17. [25]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know what you don ' t know: Unanswerable questions for SQ u AD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages ...

    2018 doi

  18. [26]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  19. [27]

    Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. https://doi.org/10.1162/tacl_a_00266 C o QA : A conversational question answering challenge . Transactions of the Association for Computational Linguistics, 7:249--266

    2019 doi

  20. [28]

    Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023. https://doi.org/10.1145/3560260 QA dataset explosion: A taxonomy of NLP resources for question answering and reading comprehension . ACM Comput. Surv., 55(10)

    2023 doi

  21. [29]

    Sara Rosenthal, Avirup Sil, Radu Florian, and Salim Roukos. 2024. https://arxiv.org/abs/2404.02103 CLAPNQ : Cohesive long-form answers from passages in natural questions for RAG systems . Preprint, arXiv:2404.02103

    2024 arXiv

  22. [30]

    Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rockt \"a schel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. https://doi.org/10.18653/v1/D18-1233 Interpretation of natural language rules in conversational machine reading . In Proceedings of the ...

    2018 doi

  23. [31]

    Maojia Song, Shang Hong Sim, Rishabh Bhardwaj, Hai Leong Chieu, Navonil Majumder, and Soujanya Poria. 2024. https://arxiv.org/abs/2409.11242 Measuring and enhancing trustworthiness of LLMs in RAG through grounded attributions and learning to refuse . Preprint, arXiv:2409.11242

    2024 arXiv

  24. [32]

    Heydar Soudani, Roxana Petcu, Evangelos Kanoulas, and Faegheh Hasibi. 2024. https://arxiv.org/abs/2405.13003 A survey on recent advances in conversational data generation . Preprint, arXiv:2405.13003

    2024 arXiv

  25. [33]

    Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.566 ASQA : Factoid questions meet long-form answers . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8273--8288, Abu ...

    2022 doi

  26. [34]

    Yuhong Sun, Zhangyue Yin, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Hui Zhao. 2024. https://arxiv.org/abs/2403.03558 Benchmarking hallucination in large language models based on unanswerable math word problem . Preprint, arXiv:2403.03558

    2024 arXiv

  27. [35]

    Zhongkai Sun, Yingxue Zhou, Jie Hao, Xing Fan, Yanbin Lu, Chengyuan Ma, Wei (Sawyer) Shen, and Chenlei (Edward) Guo. 2023. https://www.amazon.science/publications/improving-contextual-query-rewrite-for-conversational-ai-agents-through-user-preference-feedback-learning Improvin...

    2023

  28. [36]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  29. [37]

    Nandan Thakur, Nils Reimers, Andreas R \"u ckl \'e , Abhishek Srivastava, and Iryna Gurevych. 2021. https://openreview.net/forum?id=wCu6T5xFjeJ BEIR : A heterogeneous benchmark for zero-shot evaluation of information retrieval models . In Thirty-fifth Conference on Neural Info...

    2021

  30. [38]

    Zhen Wang. 2022. https://arxiv.org/abs/2206.15030 Modern question answering datasets and benchmarks: A survey . Preprint, arXiv:2206.15030

    2022 arXiv

  31. [39]

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. https://arxiv.org/abs/2309.07597 C - P ack: Packaged resources to advance general chinese embedding . Preprint, arXiv:2309.07597

    2023 arXiv

  32. [40]

    Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nicolas Scheffer, Yue...

    2024 arXiv

  33. [41]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Pap...

    2023

  34. [42]

    Yingxue Zhou, Jie Hao, Mukund Rungta, Yang Liu, Eunah Cho, Xing Fan, Yanbin Lu, Vishal Vasudevan, Kellen Gillespie, Zeynab Raeesy, Wei (Sawyer) Shen, Chenlei (Edward) Guo, and Gokhan Tur. 2023. https://www.amazon.science/publications/unified-contextual-query-rewriting Unified ...

    2023

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.