Pith. sign in

REVIEW 3 major objections 7 minor 2 cited by

DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces CONFLICTS, the first RAG benchmark with expert-annotated conflict types, and shows that supplying the conflict type to an LLM substantially improves whether its response adopts the right style.

desk verdict A useful new conflict-annotated RAG benchmark, but the headline oracle gain partly measures adherence to the authors' own rubric rather than externally validated user preferences. read the letter →

arxiv 2506.08500 v2 pith:QVWQLNQN submitted 2025-06-10 cs.CL cs.AIcs.LG

Arie Cattan , Alon Jacovi , Ori Ram , Jonathan Herzig , Roee Aharoni , Sasha Goldshtein , Eran Ofek , Idan Szpektor
show 1 more author
Avi Caciularu
This is my paper
classification cs.CLcs.AIcs.LG
keywords retrieval-augmentedgenerationknowledgeconflictsconflicttaxonomyRAGbenchmarkexpectedbehaviortypepredictionLLMevaluationsearch-augmentedLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that conflicts among retrieved sources in search-augmented generation are not one problem but several, and that each kind of conflict calls for a different response style. It proposes a five-category taxonomy—no conflict, complementary information, conflicting opinions or research outcomes, outdated information, and misinformation—and defines an expected behavior for each category. To make this testable, it introduces CONFLICTS, a benchmark of 458 expert-annotated queries paired with real search results, the first such resource to label conflict types explicitly. Experiments show that models often answer accurately yet in the wrong style, and that telling the model the gold conflict type lifts adherence to the expected behavior by roughly 24 points on average, with smaller gains from prompting the model to reason about the conflict itself.

What carries the argument

The load-bearing machinery is the conflict-type taxonomy mapped to expected behaviors, plus the benchmark that instantiates it. Each category carries a distinctive response style: answer directly for no conflict, consolidate complementary answers, neutrally summarize opposing views, prioritize the newest sources for freshness conflicts, and disregard unreliable sources for misinformation. The benchmark contributes gold labels and correct answers, and the experiments compare vanilla, pipeline, taxonomy-aware, and oracle prompts to isolate how much of the improvement comes from knowing the type rather than from extra reasoning effort.

What would settle it

A user study presenting the same conflicting queries with different response styles and asking people which they prefer would settle the mapping; if preferences diverge from the rubric, expected-behavior accuracy measures conformity to an arbitrary standard. A larger validation of the LLM adherence judge against expert raters would also test whether the headline gains are measurement artifacts.

Watch

Extended reading notes

Core claim

The central claim is that the appropriate response to conflicting search results is determined by the type of conflict, and that this type is a usable input signal for LLMs. On CONFLICTS, standard RAG prompts make models follow the type-specific expected behavior only 59 to 68 percent of the time; giving the model the annotated conflict type raises that to roughly 88 to 91 percent while preserving answer recall and factual grounding. The paper also reports that conflict type prediction is still hard, with the best model reaching only 65.3 percent accuracy, and that both a two-step pipeline and a single-pass taxonomy-aware prompt improve expected behavior over a vanilla prompt without degrading factual measures.

Load-bearing premise

The assumption that the expected behavior assigned to each conflict type is what humans actually prefer, since no user study validates that mapping and the automatic adherence evaluator was checked on only 100 examples.

Editorial extensions

If this is right

  • Search-augmented models can be improved substantially by adding conflict-type awareness: pipeline and taxonomy-aware prompts beat vanilla by 9 and 5.5 points on average.
  • Gold conflict type approaches an upper bound of roughly 24 points over vanilla, so future work should concentrate on accurate conflict-type detection to realize that gain.
  • CONFLICTS gives a shared benchmark for tracking progress across the full range of conflicts rather than one type per dataset.
  • Answer recall and factual grounding stay high even when response style changes, showing that style adherence is a separable quality dimension worth measuring.
  • The evaluation methodology adds style adherence alongside factual grounding, enabling judgments of how models resolve conflicts, not just whether they answer correctly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmark's category distribution is curated, its mix should not be read as the real-world frequency of conflicts; systems tuned to it would need rechecking on naturally occurring search results.
  • A cheap testable extension is an end-to-end system that predicts conflict type with a lightweight classifier and feeds that prediction into the oracle-style prompt; the reported pipeline gains suggest this could close much of the gap.
  • The same type signal could steer non-generative decisions, such as whether to re-retrieve newer sources for freshness conflicts or to flag a source as misinformation, which goes beyond what the paper tests.
  • The five misinformation instances suggest that genuinely false top-ranked results are rare; an adversarially constructed set may be needed to measure that category reliably.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes a five-category taxonomy of knowledge conflicts in retrieval-augmented generation (RAG), defines an expected response style for each category, and introduces CONFLICTS, a 458-instance benchmark with expert annotations of conflict types over real Google Search results. It evaluates six LLMs on conflict-type classification and response generation under four prompting strategies (vanilla, pipeline, taxonomy-aware, oracle). The main findings are that models classify conflict types with at most 65.3% accuracy, vanilla RAG responses adhere to the expected behavior only moderately (59.4-68.3 across models), and providing the gold conflict type improves expected-behavior adherence by roughly 24 points on average. The paper also reports answer recall and factual grounding, which remain high across conditions.

Significance. If the expected-behavior rubric is accepted, the benchmark is a valuable and reusable resource: it is the first RAG benchmark with expert-annotated conflict types, it uses realistic retrieval results, and it ships public code/data and full prompts. The multi-metric evaluation (factual grounding, answer recall, expected-behavior adherence) is a reasonable design, and the model sweep is broad. However, the headline expected-behavior metric rests on a normative mapping that is not validated externally; the central quantitative claim needs an independent human-preference check before it can be taken as evidence about response quality rather than instruction-following. The paper also has important coverage and reliability gaps in the Misinformation category and the annotation process.

major comments (3)
  1. [§2/§4/§5.1, Table 5] Section 2 defines the expected behavior s=f(t) as the style that 'emulates how a human would typically address' that conflict type, but the paper provides no user study or external preference data validating this mapping. Section 4's third evaluation metric prompts an LLM with the same conflict-type description and expected-behavior definition, and the Oracle condition in Section 5.1 gives the generator the gold conflict type. The 100-example validation (accuracy 0.89) checks the LLM rater against human judgments that use the authors' rubric, so it does not validate the rubric itself. The ~24-point Oracle gain in Table 5 therefore may reflect the model's ability to follow an explicitly provided style instruction rather than a genuine improvement in response quality. I recommend adding an independent human-preference evaluation (e.g., pairwise preference between Vanilla and Oracle responses, or a rating task using non-author annotators) and reporting agreement between the automatic metric and those judgments. Additionally, the evaluator is Gemini 2.5 Flash, which is also one of the six evaluated generators; the paper should discuss or control for possible self-evaluation bias.
  2. [§3, Table 2, §5.3, Table 6] Table 2 shows that only 5 of 458 instances are labeled Misinformation, yet the abstract and Section 3 describe CONFLICTS as covering a 'wide range' of knowledge conflict types. Section 5.3 and Table 6 pool 'Freshness and Misinformation,' so no category-level result is reported for Misinformation. With n=5, the standard error of a binary accuracy estimate is roughly 22 percentage points, making the Misinformation results uninformative. To support the coverage claim, the authors should either add substantially more Misinformation instances (e.g., from naturally occurring low-quality sources or independently verified cases) or explicitly state that Misinformation is a pilot category with 5 instances and exclude it from the headline 'wide range' claims.
  3. [§3] Section 3 describes a two-annotator labeling process with reconciliation and a third-expert review, but no inter-annotator agreement statistics are reported. Because the labels define both Task 1 and the oracle condition for Task 2, the absence of initial agreement and disagreement-resolution statistics makes it difficult to assess the reliability of the taxonomy and annotations. Please report initial per-category agreement (e.g., Cohen's kappa or percent agreement) and summarize how disagreements were resolved.
minor comments (7)
  1. [Table 3] In the Misinformation row, the evaluation comment says 'April 7, 2020,' but the response and Table 1 give 'April 4, 2017' as the correct date; this inconsistency should be fixed.
  2. [§3, Table 7, References] The QACC dataset is cited as Liu et al. (2024b) in the text and Table 7, while the reference list contains Liu et al. (2025) for the same title; please harmonize the citation.
  3. [§5.2] The sentence 'using Gemini 2.5 Flash (§4)' is ambiguous about whether all three metrics (expected behavior, answer recall, factual grounding) use that evaluator; please clarify for each metric.
  4. [§5.3, Table 6] The pooling of Freshness and Misinformation is justified only by 'both categories require selecting the correct response,' but Section 2 defines different expected behaviors for the two categories; please justify the pooling or report the categories separately.
  5. [§3] The absence of 'Other' annotations is presented as evidence that the taxonomy is comprehensive, but annotators chose from a fixed taxonomy; this claim should be softened or supported with additional analysis.
  6. [Table 5] Several Pipeline improvements over Vanilla are 5-9 points; reporting confidence intervals or significance tests would help distinguish genuine gains from noise.
  7. [§1] The Introduction contains a duplicated word: 'assuming that the the type of conflict is known in advance' should read 'assuming that the type of conflict is known in advance.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark, taxonomy, and evaluation are constructed artifacts rather than derived predictions.

full rationale

The paper's central claims are that CONFLICTS is a human-annotated benchmark with conflict-type labels made from existing query sets, and that supplying the gold conflict type to a generator improves adherence to an explicitly stated expected-behavior rubric. Neither claim is obtained by fitting a parameter to the target quantity, nor by invoking a self-citation as the proof of the result. The taxonomy and expected behaviors in Section 2 are author-defined proposals presented as design choices, not as derivations from the experiments. The expected-behavior metric in Section 4 uses an LLM judge that is few-shot prompted with the rubric and validated against human judgments on 100 examples; while the Oracle condition shares the rubric with the judge, this tests whether models can follow a given style instruction, so the measured variable is not constructed by the evaluation itself. The factual-grounding and answer-recall metrics rely on established external protocols. Self-citations, such as adapting the FACTS grounding prompt from Jacovi et al. (2025), are methodological precedents rather than load-bearing evidence for the paper's headline results. A possible weakness is that the expected-behavior rubric itself is not validated against independent user-preference data, but that is a construct-validity concern, not a circularity concern.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. The central claims rest on the author-defined taxonomy, the quality of expert annotations, and the reliability of LLM-as-judge evaluation, which are domain assumptions rather than formal axioms. The most fragile assumption is that the expected-behavior rubric reflects genuine human preferences, because the entire evaluation is structured around that rubric.

assumptions (4)
  • domain assumption The five conflict types in the taxonomy are comprehensive and mutually exclusive.
    The paper states that no instance was annotated as 'Other' (Section 3), but the taxonomy is defined by the authors and is not independently validated as a complete or natural partition of knowledge conflicts.
  • domain assumption Expert annotations of conflict type are accurate despite the absence of reported inter-annotator agreement statistics.
    The annotation process uses two annotators, reconciliation, and a third reviewer (Section 3), but the paper does not report agreement numbers, so the reader cannot assess annotation reliability.
  • domain assumption The LLM-based evaluator for expected-behavior adherence generalizes beyond the 100 examples used for validation.
    The automatic judge is few-shot prompted and achieves 0.89 accuracy on a 100-example human validation subset (Section 4), but the paper does not analyze per-category errors or the evaluator's failure modes on the full test set.
  • domain assumption The expected behavior mapping f(t) reflects how humans would actually prefer models to respond to each conflict type.
    The expected behaviors are defined by the authors in Section 2 as emulating human behavior, but no user study or preference elicitation is conducted to confirm that these response styles are what users want.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs." pith.science (2026). https://pith.science/paper/QVWQLNQN

@misc{pith2026250608500,
  author       = {Pith},
  title        = {Pith review of: DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QVWQLNQN}},
  note         = {Machine review of arXiv:2506.08500}
}
read the original abstract

Retrieval Augmented Generation (RAG) is a commonly used approach for enhancing large language models (LLMs) with relevant and up-to-date information. However, the retrieved sources can often contain conflicting information and it remains unclear how models should address such discrepancies. In this work, we first propose a novel taxonomy of knowledge conflict types in RAG, along with the desired model behavior for each type. We then introduce CONFLICTS, a high-quality benchmark with expert annotations of conflict types in a realistic RAG setting. CONFLICTS is the first benchmark that enables tracking progress on how models address a wide range of knowledge conflicts. We conduct extensive experiments on this benchmark, showing that LLMs often struggle to appropriately resolve conflicts between sources. While prompting LLMs to explicitly reason about the potential conflict in the retrieved documents significantly improves the quality and appropriateness of their responses, substantial room for improvement in future research remains.

Figures

Figures reproduced from arXiv: 2506.08500 by the authors.

Figure 1
Figure 1. Examples of user queries with knowl￾edge conflicts in the search results. The left panel illustrates a freshness conflict arising from outdated information, while the right panel shows conflict￾ing opinions on a controversial query. Identifying the conflict type is essential for generating the ap￾propriate response (e.g., prioritizing recent data or presenting diverse perspectives). should optimally address these di… view at source ↗
Figure 2
Figure 2. Prompt for predicting the conflict type. [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Template prompt for Vanilla [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Detecting Knowledge Inconsistencies Across Text, Tables, and Knowledge Graphs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Introduces a taxonomy and an automatic pipeline (Kontrast) that compares Wikipedia table answers with Wikidata answers to detect and label cross-modal knowledge inconsistencies.

  2. DeepStress: Stress-Testing Deep Search Agents

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A controlled synthetic-retrieval stress test shows that AI search agents differ greatly in handling untrustworthy, irrelevant, or false documents, and that HotpotQA is partly answerable from parametric knowledge alone.

Reference graph

Works this paper leans on

62 extracted references · 19 canonical work pages · cited by 2 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Crestani, and W

    Mohammad Aliannejadi, Hamed Zamani, Fabio A. Crestani, and W. Bruce Croft. 2019. https://api.semanticscholar.org/CorpusID:196623463 Asking clarifying questions in open-domain information-seeking conversations . Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

    work page 2019

  4. [4]

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, T. W. Hennigan, Saffron Huang, Lorenzo Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving,...

    work page 2021

  5. [5]

    Arie Cattan, Paul Roit, Shiyue Zhang, David Wan, Roee Aharoni, Idan Szpektor, Mohit Bansal, and Ido Dagan. 2024. https://api.semanticscholar.org/CorpusID:273233152 Localizing factual inconsistencies in attributable text generation . ArXiv, abs/2410.07473

    work page Pith review arXiv 2024

  6. [6]

    Cheng-Han Chiang and Hung-yi Lee. 2023. https://doi.org/10.18653/v1/2023.acl-long.870 Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607--15631, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.870 2023

  7. [7]

    Du, Antoine Bosselut, and Christopher D

    Y. Du, Antoine Bosselut, and Christopher D. Manning. 2022. https://api.semanticscholar.org/CorpusID:247011536 Synthetic disinformation attacks on automated fact verification systems . In AAAI Conference on Artificial Intelligence

    work page 2022

  8. [8]

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.398 Enabling large language models to generate text with citations . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465--6488, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.398 2023

Show all 62 references
  1. [9]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023 b . Retrieval-augmented generation for large language models: A survey. ArXiv, abs/2312.10997

    2023 arXiv

  2. [10]

    Team Gemini. 2024. http://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    2024 arXiv

  3. [11]

    Carlos G \'o mez-Rodr \'i guez and Paul Williams. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.966 A confederacy of models: a comprehensive evaluation of LLM s on creative writing . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14504-...

    2023 doi

  4. [12]

    Google. 2024. https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024 Gemini 2.0

    2024

  5. [13]

    Google. 2025. https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/ Gemini 2.5 Thinking

    2025

  6. [14]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: retrieval-augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org

    2020

  7. [15]

    Sebastian Hofst\" a tter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. https://doi.org/10.1145/3404835.3462891 Efficiently teaching an effective dense retriever with balanced topic aware sampling . In Proceedings of the 44th International ACM SIGIR Con...

    2021

  8. [16]

    Tchrakian, Radu Marinescu, Elizabeth M

    Yufang Hou, Alessandra Pascale, Javier Carnerero-Cano, Tigran T. Tchrakian, Radu Marinescu, Elizabeth M. Daly, Inkit Padhi, and Prasanna Sattigeri. 2024. https://openreview.net/forum?id=5t7DtLwTVC Wikicontradict: A benchmark for evaluating LLM s on real-world knowledge conflic...

    2024

  9. [17]

    OpenAI Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mkadry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alexander Kirillov, Alex Nichol, Alex P...

    2024 arXiv

  10. [18]

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: few-shot learning with retrieval augmented language models. J. Mach. Learn. Res., 24(1)

    2023

  11. [19]

    Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, Carl Saroufim, Corey Fry, Doron Kukliansky, Gaurav, Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, Michael Aaron, Mo...

    2025 arXiv

  12. [20]

    Cheng Jiayang, Chunkit Chan, Qianqian Zhuang, Lin Qiu, Tianhang Zhang, Tengxiao Liu, Yangqiu Song, Yue Zhang, Pengfei Liu, and Zheng Zhang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.447 ECON : On the detection and resolution of evidence conflicts . In Proceedings of th...

    2024 doi

  13. [21]

    Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. 2023. https://doi.org/10.18653/v1/2023.acl-long.307 Evaluating open-domain question answering in the era of large language models . In Proceedings of the 61st Annual Meeting of the Association for Computational Li...

    2023 doi

  14. [22]

    Gemma Team Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram'e, Morgane Rivi \`e re, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casb...

    2025 arXiv

  15. [23]

    Smith, Yejin Choi, and Kentaro Inui

    Jungo Kasai, Keisuke Sakaguchi, yoichi takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2023. https://openreview.net/forum?id=HfKOIPCvsv Realtime QA : What's the answer right now? In Thirty-seventh Conferenc...

    2023

  16. [24]

    Tom Kocmi and Christian Federmann. 2023. https://aclanthology.org/2023.eamt-1.19/ Large language models are state-of-the-art evaluators of translation quality . In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193--203, Ta...

    2023

  17. [25]

    Evgenii Kortukov, Alexander Rubinstein, Elisa Nguyen, and Seong Joon Oh. 2024. https://openreview.net/forum?id=xm8zYRfrqE Studying large language model behaviors under context-memory conflicts with real documents . In First Conference on Language Modeling

    2024

  18. [26]

    Yoonsang Lee, Xi Ye, and Eunsol Choi. 2024. https://openreview.net/forum?id=mkYCfO822n Ambigdocs: Reasoning across documents on different entities under the same name . In First Conference on Language Modeling

    2024

  19. [27]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proc...

    2020

  20. [28]

    Jiarui Li, Ye Yuan, and Zehua Zhang. 2024. Enhancing llm factual accuracy with rag to counter hallucinations: A case study on domain-specific queries in private knowledge-bases. ArXiv, abs/2403.10446

    2024 arXiv

  21. [29]

    Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien De Masson D'Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-Mcmahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. https://proceedings.mlr.press/v...

    2022

  22. [30]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024 a . https://doi.org/10.1162/tacl_a_00638 Lost in the middle: How language models use long contexts . Transactions of the Association for Computational Linguistics,...

    2024 doi

  23. [31]

    Siyi Liu, Qiang Ning, Kishaloy Halder, Zheng Qi, Wei Xiao, Phu Mon Htut, Yi Zhang, Neha Anna John, Bonan Min, Yassine Benajiba, and Dan Roth. 2025. https://aclanthology.org/2025.findings-naacl.99/ Open domain question answering with conflicting contexts . In Findings of the As...

    2025

  24. [32]

    Siyi Liu, Qiang Ning, Kishaloy Halder, Wei Xiao, Zheng Qi, Phu Mon Htut, Yi Zhang, Neha Ann John, Bonan Min, Yassine Benajiba, and Dan Roth. 2024 b . https://api.semanticscholar.org/CorpusID:273375445 Open domain question answering with conflicting contexts . ArXiv, abs/2410.12311

    2024 arXiv

  25. [33]

    Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Andrew Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Xiaotao Gu, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024 c . https://doi.org/10.18653/v1/2024.acl-long.62...

    2024 doi

  26. [34]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  27. [35]

    Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.565 Entity-based knowledge conflicts in question answering . In Proceedings of the 2021 Conference on Empirical Methods in Natural L...

    2021 doi

  28. [36]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.546 When not to trust language models: Investigating effectiveness of parametric and non-parametric memories . In Proceedings of the 6...

    2023 doi

  29. [37]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.741 FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generatio...

    2023 doi

  30. [38]

    Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.466 A mbig QA : Answering ambiguous open-domain questions . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)...

    2020 doi

  31. [39]

    Yifei Ming, Senthil Purushwalkam, Shrey Pandit, Zixuan Ke, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. 2025. https://openreview.net/forum?id=UeVx6L59fg Faitheval: Can your language model stay faithful to context, even if ''the moon is made of marshmallows'' . In The Thirt...

    2025

  32. [40]

    Liangming Pan, Wenhu Chen, Min-Yen Kan, and William Yang Wang. 2023. https://doi.org/10.18653/v1/2023.ijcnlp-main.35 Attacking open-domain question answering by injecting misinformation . In Proceedings of the 13th International Joint Conference on Natural Language Processing ...

    2023 doi

  33. [41]

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. https://doi.org/10.1162/tacl_a_00605 In-context retrieval-augmented language models . Transactions of the Association for Computational Linguistics, 11:1316--1331

    2023 doi

  34. [42]

    Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. https://doi.org/10.1162/coli_a_00486 Measuring attribution in natural language generation models . Computational Ling...

    2023 doi

  35. [43]

    Abhilasha Ravichander, Shrusti Ghela, David Wadden, and Yejin Choi. 2025. https://api.semanticscholar.org/CorpusID:275516060 Halogen: Fantastic llm hallucinations and where to find them

    2025

  36. [44]

    Ondrej Skopek, Rahul Aralikatte, Sian Gooding, and Victor Carbune. 2023. https://doi.org/10.18653/v1/2023.conll-1.16 Towards better evaluation of instruction-following: A case-study in summarization . In Proceedings of the 27th Conference on Computational Natural Language Lear...

    2023 doi

  37. [45]

    Aviv Slobodkin, Eran Hirsch, Arie Cattan, Tal Schuster, and Ido Dagan. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.182 Attribute first, then generate: Locally-attributable grounded text generation . In Proceedings of the 62nd Annual Meeting of the Association for Comput...

    2024 doi

  38. [46]

    Aviv Slobodkin, Ori Shapira, Ran Levy, and Ido Dagan. 2024 b . https://doi.org/10.18653/v1/2024.findings-naacl.189 Multi-review fusion-in-context . In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3003--3021, Mexico City, Mexico. Association for ...

    2024 doi

  39. [47]

    Yixiao Song, Yekyung Kim, and Mohit Iyyer. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.552 V eri S core: Evaluating the factuality of verifiable claims in long-form text generation . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9447...

    2024 doi

  40. [48]

    Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.566 ASQA : Factoid questions meet long-form answers . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8273--8288, Abu ...

    2022 doi

  41. [49]

    Hexiang Tan, Fei Sun, Wanli Yang, Yuanzhuo Wang, Qi Cao, and Xueqi Cheng. 2024. https://doi.org/10.18653/v1/2024.acl-long.337 Blinded by generated contexts: How language models merge generated and retrieved contexts when knowledge conflicts? In Proceedings of the 62nd Annual M...

    2024 doi

  42. [50]

    Liyan Tang, Philippe Laban, and Greg Durrett. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.499 M ini C heck: Efficient fact-checking of LLM s on grounding documents . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8818--88...

    2024 doi

  43. [51]

    Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2024. https://doi.org/10.18653/v1/2024.findings-acl.813 F resh LLM s: Refreshing large language models with search engine augmentation . In Fi...

    2024 doi

  44. [52]

    Alexander Wan, Eric Wallace, and Dan Klein. 2024. https://doi.org/10.18653/v1/2024.acl-long.403 What evidence do language models find convincing? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7468--748...

    2024 doi

  45. [53]

    Han Wang, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal. 2025. Retrieval-augmented generation with conflicting evidence. arXiv preprint arXiv:2504.13079

    2025 arXiv

  46. [54]

    Yike Wang, Shangbin Feng, Heng Wang, Weijia Shi, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2024. https://openreview.net/forum?id=ptvV5HGTNN Resolving knowledge conflicts in large language models . In First Conference on Language Modeling

    2024

  47. [55]

    Rongwu Xu, Xuan Qi, Zehan Qi, Wei Xu, and Zhijiang Guo. 2024 a . https://api.semanticscholar.org/CorpusID:271693518 Debateqa: Evaluating question answering on debatable knowledge . ArXiv, abs/2408.01419

    2024 arXiv

  48. [56]

    Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-main.486 Knowledge conflicts for LLM s: A survey . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p...

    2024 doi

  49. [57]

    Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, ...

    2024 arXiv

  50. [58]

    Gal Yona, Roee Aharoni, and Mor Geva. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.443 Can large language models faithfully express their intrinsic uncertainty in words? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7752-...

    2024 doi

  51. [59]

    Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024. https://openreview.net/forum?id=ZS4m74kZpH Making retrieval-augmented language models robust to irrelevant context . In The Twelfth International Conference on Learning Representations

    2024

  52. [60]

    Michael Zhang and Eunsol Choi. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.586 S ituated QA : Incorporating extra-linguistic contexts into QA . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371--7387, Online and Punta C...

    2021 doi

  53. [61]

    Michael JQ Zhang and Eunsol Choi. 2025. https://aclanthology.org/2025.findings-naacl.306/ Clarify when necessary: Resolving ambiguity through interaction with LM s . In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5526--5543, Albuquerque, New Me...

    2025

  54. [62]

    Xing, Haotong Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://api.semanticscholar.org/CorpusID:259129398 Judging llm-as-a-judge with mt-bench...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.