Pith. sign in

REVIEW 3 major objections 4 minor 93 references

Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RAG answers collapse under adversarial evidence, recover when helpful docs dominate.

desk verdict Solid, reproducible measurement of adversarial RAG vulnerability in health QA, but the abstract overclaims that mere presence of helpful evidence rescues alignment—domination, not presence, is what the data show. read the letter →

arxiv 2509.03787 v1 pith:E7ONGMUO submitted 2025-09-04 cs.IR cs.CL

Shakiba Amirshahi , Amin Bigdeli , Charles L. A. Clarke , Amira Ghenai This is my paper
classification cs.IRcs.CL
keywords RAGrobustnessadversarialevidencehealthmisinformationground-truthalignmentqueryframingretrievalpoolcompositionLLMevaluationTRECTrack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper systematically tests how much the evidence fed into a retrieval-augmented generation (RAG) system controls whether a language model gives the correct answer to a health question. The authors find that a single helpful document reliably lifts accuracy above the no-context baseline, while a single adversarial document—especially one generated from scratch to push a false stance—can drive ground-truth alignment below 5 percent. When the retrieved pool is skewed eight-to-two toward helpful material, alignment returns to near-perfect levels, even under misleading question phrasing. The study shows that the composition and balance of the retrieval pool, more than the order of documents or the size of the model, determines whether a RAG system stays trustworthy in high-stakes medical settings.

What carries the argument

The controlled experimental grid: four document types (helpful, harmful, six adversarial variants, and no-context baseline), three query framings (consistent, neutral, inconsistent), and four retrieval-pool constructions (single-document, paired-document, passage-based reranking with MonoT5, and bias-controlled 8:2 pools). The adversarial documents produced by Bigdeli et al. [6]—Rewriter, Paraphraser, Fact-Inversion, FSAP-InterQ, FSAP-IntraQ, and Liar—are the mechanism that pushes the system to its failure point, and the bias-controlled pools are the mechanism that restores it.

What would settle it

A direct check: take the same TREC queries and pool compositions, but re-score a sample of the generated responses with human medical expert annotators instead of the two LLM stance classifiers. If the human experts find that the adversarial documents do not actually push the model into reproducing the false stance (or that the models resist Liar documents at scale), the headline degradation numbers would not survive. A second check: run the bias-controlled pools at helpful ratios of 6:4, 4:6, and 2:8; if alignment does not fall off monotonically with the helpful fraction, the claim that compo

Watch

Extended reading notes

Core claim

The central claim is that the reliability of a RAG system answering health questions is governed primarily by the mix of document types in the retrieved context. Helpful documents raise ground-truth alignment above the no-context baseline, harmful documents lower it, and adversarially generated documents—particularly 'Liar' documents built from scratch to assert an incorrect stance—can collapse alignment to near zero across all tested models, from GPT-4.1 and GPT-5 to LLaMA-3 8B. However, when the retrieval pool is deliberately biased toward helpful evidence at an eight-to-two ratio, alignment recovers to near-perfect levels, and even a single helpful document placed alongside an adversarial

Load-bearing premise

The adversarial documents, imported from a prior study, are genuinely evasive misinformation for the six language models tested here, even though the paper does not re-validate their effectiveness against those specific models.

Editorial extensions

If this is right

  • Adding one helpful document to a RAG context reliably improves ground-truth alignment beyond the no-context baseline, across model families.
  • Adversarial documents of the Liar type can drive alignment below 5 percent, showing that the generation stage rarely recovers once malicious content is ranked into the top-k.
  • An 8:2 helpful-biased retrieval pool restores alignment to near-perfect levels, effectively neutralizing the adversarial evidence and the misleading query framing.
  • Document order (helpful first vs. helpful second) has minimal influence; what matters is the presence and proportion of helpful content.
  • LLMs are markedly more resistant to COVID-19-related misinformation (TREC 2020) than to general health misinformation (TREC 2021), suggesting post-training alignment is uneven across medical topics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: measuring alignment while varying the helpful-to-adversarial ratio between 0:10 and 10:0 would identify the minimal helpful fraction needed for protection, potentially a design target for retrieval safeguards.
  • The results imply that retrieval-stage defenses—blocking or reranking out adversarial documents before they reach the prompt—matter more than prompt-centered or generation-stage defenses, since the model follows whatever dominates the context.
  • The same bias-controlled methodology could be applied to other high-stakes domains (legal, financial, public-safety) where ground-truth labels exist, to see whether the 'helpful majority restores robustness' effect generalizes.
  • A user-facing implication the authors stop short of drawing: an end user who manually injects a single trusted source into a RAG prompt may substantially shield themselves from misinformation embedded in the retrieved context.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper empirically evaluates RAG robustness in the health domain. Using TREC 2020 and TREC 2021 Health Misinformation Track collections alongside six types of adversarial documents from Bigdeli et al., it measures ground-truth alignment of six LLMs (GPT-4.1, GPT-5, Claude-3.5-Haiku, DeepSeek-R1-Distill-Qwen-32B, Phi-4, LLaMA-3 8B Instruct) under four retrieval-pool conditions: single-document, paired-document, passage-based pooling, and bias-controlled pooling. Queries are framed as consistent, neutral, or inconsistent relative to ground truth. The main findings are that helpful documents improve alignment relative to the Non-RAG baseline, harmful and adversarial documents degrade it, strong adversarial strategies such as Liar nearly eliminate alignment, and helpful-dominated pools restore near-perfect alignment. The paper also reports that COVID-19 queries (TREC 2020) are more robust than general health queries (TREC 2021), and that query framing consistently orders alignment as consistent > neutral > inconsistent. All experimental results are released on GitHub.

Significance. If the results hold, this is a useful empirical contribution: it provides the first systematic comparison of multiple adversarial document generation strategies against multiple LLM families in a high-stakes health domain. The study is well-designed in several respects: large per-condition sample sizes, bootstrap confidence intervals, explicit replication across model families and two datasets, and public release of responses and evaluation prompts. The paper does not introduce a new attack method, but the quantitative ranking of attack effectiveness (Rewriter/Paraphraser being milder than Fact-Inversion and FSAP, Liar being most damaging) and the demonstration that helpful-document dominance matters more than mere presence are actionable for retrieval-safeguard design. The central claims rest entirely on measured alignment rates, so the validity of the automated stance classifier and the representativeness of the imported adversarial documents are the key correctness risks.

major comments (3)
  1. [Abstract and §5] The headline claim that 'robustness can be preserved when helpful evidence is also present in the retrieval pool' is only supported by the bias-controlled 8:2 helpful-dominant condition (§3.3.4). The other pooling experiments contradict the unqualified wording. In the paired-document setup (§4.3.1), a single helpful document paired with a Liar document raises TREC 2021 alignment from 1.7% to only 39.1–45.0%, far below the ~89% Non-RAG baseline and ~98% helpful-only rate. In passage-based pooling (§4.3.2), helpful passages are present (~5% of the pool) but alignment remains 4.3–43.7% depending on query framing. The conclusion should state that helpful evidence must dominate the pool, not merely be present. This is an internal-consistency issue between the abstract/conclusion and the reported results, and it changes the practical takeaway about retrieval safeguards.
  2. [§3.4 and §6] All quantitative claims depend on automated stance classification by gemini-2.0-flash, validated only against gpt-4o-mini (Cohen's κ = 0.90 for TREC 2020, 0.82 for TREC 2021). No human-label validation is reported, and the Limitations section itself concedes that automated classifiers may miss subtle inaccuracies. Inter-model agreement does not rule out systematic bias; if the classifier is differentially sensitive to document style, adversarial tone, or response length, the reported alignment rates and the Liar-vs-Rewriter hierarchy could shift. I request a human-annotated spot-check stratified by document type and query framing, or an explicit error analysis, before the headline numbers are taken at face value.
  3. [§3.1.2] The six adversarial document types are imported from Bigdeli et al. [6] without re-validating their effectiveness for the six generation models studied here. The cited paper demonstrates ranking-based effectiveness; in this study the documents are supplied directly to the generator, so the generation-stage 'adversarialness' is assumed rather than verified. The measured degradation is internally consistent, but external validity as 'realistic misinformation' is not established. A concrete test would be to have health-information experts rate a sample of the adversarial documents for credibility and misleadingness, or to include naturally occurring misinformation passages, to confirm the attack-type hierarchy is not an artifact of these specific generated documents.
minor comments (4)
  1. [§4.2] The hierarchy 'Liar≫ FSAP-IntraQ > FSAP-InterQ > ...' mixes the 'much greater' symbol with the ordinary chain '>'. Clarify whether Liar is intended to be categorically worse than the rest or simply the top of a total order.
  2. [Figure 3] The x-axis labels 'Passage-Based', 'Harmful', 'Rewriter', 'Paraphraser', etc., are not self-explanatory in grayscale or print. Add a legend or axis key that clearly distinguishes the passage-based baseline from the biased-to-helpful and biased-to-harmful conditions.
  3. [§3.3.4] The composition of biased pools is described for one attack type at a time. State explicitly whether the 8:2 helpful/harmful skew is applied per attack type or across all adversarial segments combined, and whether the two non-dominant slots are selected independently per query.
  4. [§3.3.2] Paired-document sampling is capped at ten pairs per query per condition. Report the distribution of sampled pairs after filtering and note whether any query had fewer than ten; unequal pair counts across queries can weight the aggregate alignment rate unevenly.

Circularity Check

0 steps flagged · score 2.0 of 10

Measured empirical study; no derivation-level circularity; minor same-author citation for adversarial stimulus corpus keeps score at 2.

full rationale

This paper reports measured alignment rates of LLMs under different retrieved-context conditions; it does not fit parameters and then 'predict' a related quantity, nor does it derive a result from the definitions of its variables. The core experiments (Section 3.3.1) pair fixed TREC helpful/harmful documents and Bigdeli et al. adversarial documents with fixed query templates, and the outcome is the observed proportion of responses whose stance matches the TREC ground-truth stance (Tables 2-5). No fitted parameter is renamed as a prediction. The only notable self-citation is reference [6], used in Section 3.1.2 to supply the adversarial documents and to assert that they 'are shown to be effective at bypassing filters and can routinely outrank reliable, helpful documents in re-ranking pipelines'. Bigdeli and Clarke are also authors of the present paper, so this is a same-author citation. However, the present paper independently measures what happens when those documents are given to six LLMs, and Section 4.3.2 re-measures the reranking outcome with MonoT5 (adversarial passages compose ~92-94% of the pools). Thus the quantitative alignment findings do not reduce to the self-citation. The RQ4 query-framing results are also empirical rather than self-definitional: although 'consistent' and 'inconsistent' are defined relative to the ground-truth stance, the observed rates (e.g., Liar documents on TREC 2021 giving 1.7-4.4% alignment) show the models are not simply echoing the query's presupposition. Section 6 explicitly acknowledges the main limitations (no full retrieval pipeline, reliance on automated stance classification), which further supports the absence of a hidden derivation-level circularity. Overall, the measured results are self-contained against external TREC ground-truth benchmarks; the score of 2 reflects only the minor, non-load-bearing same-author citation used for stimulus provenance.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper contributes no fitted parameters or new theoretical entities; its numbers are measurements. The load-bearing assumptions are the validity of TREC helpful/harmful labels, the quality of the imported adversarial documents, and the accuracy of LLM-based stance scoring. The headline claim also depends on hand-picked experimental parameters, pool size 10 and the 8:2 skew ratio, which were not swept.

free parameters (4)
  • Helpful-to-harmful pool skew ratio = 8 helpful : 2 harmful/adversarial segments
    Chosen by hand in Section 3.3.4. The abstract claim that helpful evidence preserves robustness is only demonstrated at this strong skew; less skewed conditions show much lower alignment.
  • Retrieval pool size k = 10 segments
    Fixed at 10 for passage-based and bias-controlled pooling setups. Results likely depend on this choice.
  • Paired-document sampling cap = Up to 10 pairs per condition per query
    Chosen for computational feasibility in Section 3.3.2; estimates for pairing effects depend on this sample.
  • Per-category document cap = Up to 10 documents per category per query
    Inherited from Bigdeli et al. [6] and applied in Section 3.1.2; random subset selection when more than ten are available can alter the difficulty mix.
assumptions (4)
  • domain assumption TREC Health Misinformation Track preference codes are a valid ground truth for helpful vs harmful health information.
    Section 3.1.1 selects documents based on these labels without re-validating the medical correctness of the labels.
  • domain assumption The adversarial documents from Bigdeli et al. [6] are realistic, evasive, and effective against the six LLMs tested here.
    Section 3.1.2 imports their effectiveness from the prior paper and does not independently test them against the current models.
  • domain assumption Gemini-2.0-flash stance classification accurately measures ground-truth alignment.
    Section 3.4 validates only against a second LLM, gpt-4o-mini, with no human-labeled stance judgments.
  • domain assumption Fixed template queries for consistent and inconsistent framings represent natural user presuppositions.
    Section 3.2 constructs queries from templates rather than collecting real user questions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain." pith.science (2026). https://pith.science/paper/E7ONGMUO

@misc{pith2026250903787,
  author       = {Pith},
  title        = {Pith review of: Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E7ONGMUO}},
  note         = {Machine review of arXiv:2509.03787}
}
read the original abstract

Retrieval augmented generation (RAG) systems provide a method for factually grounding the responses of a Large Language Model (LLM) by providing retrieved evidence, or context, as support. Guided by this context, RAG systems can reduce hallucinations and expand the ability of LLMs to accurately answer questions outside the scope of their training data. Unfortunately, this design introduces a critical vulnerability: LLMs may absorb and reproduce misinformation present in retrieved evidence. This problem is magnified if retrieved evidence contains adversarial material explicitly intended to promulgate misinformation. This paper presents a systematic evaluation of RAG robustness in the health domain and examines alignment between model outputs and ground-truth answers. We focus on the health domain due to the potential for harm caused by incorrect responses, as well as the availability of evidence-based ground truth for many common health-related questions. We conduct controlled experiments using common health questions, varying both the type and composition of the retrieved documents (helpful, harmful, and adversarial) as well as the framing of the question by the user (consistent, neutral, and inconsistent). Our findings reveal that adversarial documents substantially degrade alignment, but robustness can be preserved when helpful evidence is also present in the retrieval pool. These findings offer actionable insights for designing safer RAG systems in high-stakes domains by highlighting the need for retrieval safeguards. To enable reproducibility and facilitate future research, all experimental results are publicly available in our github repository. https://github.com/shakibaam/RAG_ROBUSTNESS_EVAL

Figures

Figures reproduced from arXiv: 2509.03787 by the authors.

Figure 1
Figure 1. Results of the single-document setup with GPT-4.1 on TREC 2020 and TREC 2021. The x-axis shows the different document types provided to the LLMs, as well as the Non-RAG baseline, and the y-axis reports ground-truth alignment rate (%). Error bars indicate 95% confidence intervals estimated via bootstrapping. Across both datasets, providing helpful documents leads to the highest alignment rate, while presenting Liar d… view at source ↗
Figure 2
Figure 2. Results of paired-document setup for GPT-4.1 on TREC 2020 and TREC 2021 with consistent, neutral, and inconsistent user queries. The x-axis shows the document type with which the helpful document is paired (either first or second), and the y-axis reports ground-truth alignment rate (%). Error bars indicate 95% confidence intervals estimated via bootstrapping. Bars compare helpful-first, helpful-second, and single-do… view at source ↗
Figure 3
Figure 3. Results of passage-based pooling and biased-controlled pooling setups for GPT-4.1 on TREC 2020 and TREC 2021 across consistent, neutral, and inconsistent user queries. The x-axis includes the passage-based pooling baseline as well as the different document types used in biased conditions, where helpful documents are paired with harmful or adversarial ones. The y-axis reports ground-truth alignment rate (%). Error ba… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

93 extracted references · 31 canonical work pages

  1. [6]

    Amin Bigdeli, Negar Arabzadeh, Ebrahim Bagheri, and Charles L. A. Clarke. 2025. Adversarial Attacks against Neural Ranking Models via In-Context Learning. arXiv:2508.15283 [cs.IR] https://arxiv.org/abs/2508.15283

    arXiv 2025

  2. [1]

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905 (2024)

    arXiv 2024

  3. [2]

    Md Adnan Arefeen, Biplob Debnath, and Srimat Chakradhar. 2024. Leancontext: Cost-efficient domain-specific question answering using llms. Natural Language Processing Journal 7 (2024), 100065

    2024

  4. [3]

    Md Ahsan Ayub and Subhabrata Majumdar. 2024. Embedding-based classifiers can detect prompt injection attacks. arXiv preprint arXiv:2410.22284 (2024)

    arXiv 2024

  5. [4]

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency . 610–623

    2021

  6. [5]

    Amin Bigdeli, Negar Arabzadeh, Ebrahim Bagheri, and Charles LA Clarke. 2024. EMPRA: Embedding Perturbation Rank Attack against Neural Ranking Models. arXiv preprint arXiv:2412.16382 (2024)

    arXiv 2024

  7. [7]

    Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Hadfield-Menell. 2024. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint arXiv:2403.05030 (2024)

    arXiv 2024

  8. [8]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology 15, 3 (2024), 1–45

    2024

Show all 93 references
  1. [9]

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2025. Jailbreaking black box large language models in twenty queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) . IEEE, 23–42

    2025

  2. [10]

    Harsh Chaudhari, Giorgio Severi, John Abascal, Matthew Jagielski, Christopher A Choquette-Choo, Milad Nasr, Cristina Nita-Rotaru, and Alina Oprea. 2024. Phantom: General trigger attacks on retrieval augmented language generation. arXiv preprint arXiv:2405.20485 (2024)

    2024

  3. [11]

    Sizhe Chen, Yizhu Wang, Nicholas Carlini, Chawin Sitawarin, and David Wagner. 2025. Defending Against Prompt Injection With a Few DefensiveTokens. arXiv preprint arXiv:2507.07974 (2025)

    2025 arXiv

  4. [12]

    Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, David Wagner, and Chuan Guo. 2024. Secalign: Defending against prompt injection with preference optimization. arXiv preprint arXiv:2410.05451 (2024)

    2024 arXiv

  5. [13]

    Sizhe Chen, Arman Zharmagambetov, David Wagner, and Chuan Guo. 2025. Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attacks. arXiv preprint arXiv:2507.02735 (2025). Manuscript submitted to ACM Evaluating the Robustness of Retrieval-Augmented Generation to Adve...

    2025

  6. [14]

    Xuanang Chen, Ben He, Zheng Ye, Le Sun, and Yingfei Sun. 2023. Towards imperceptible document manipulations against neural ranking models. arXiv preprint arXiv:2305.01860 (2023)

    2023 arXiv

  7. [15]

    Pengzhou Cheng, Yidong Ding, Tianjie Ju, Zongru Wu, Wei Du, Ping Yi, Zhuosheng Zhang, and Gongshen Liu. 2024. Trojanrag: Retrieval-augmented generation can be backdoor driver in large language models. arXiv preprint arXiv:2405.13401 (2024)

    2024 arXiv

  8. [16]

    Cheng-Han Chiang and Hung-yi Lee. 2023. Can Large Language Models Be an Alternative to Human Evaluations?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Toronto, Canada, 15607–15631. http...

    2023

  9. [17]

    Sukmin Cho, Soyeong Jeong, Jeongyeon Seo, Taeho Hwang, and Jong C Park. 2024. Typos that Broke the RAG’s Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations. arXiv preprint arXiv:2404.13948 (2024)

    2024 arXiv

  10. [18]

    Charles L. A. Clarke, Maria Maistro, and Mark D. Smucker. 2021. Overview of the TREC 2021 Health Misinformation Track. In Proceedings of the Thirtieth Text REtrieval Conference, TREC 2021, online, November 15-19, 2021 (NIST Special Publication, Vol. 500-335) , Ian Soboroff and...

    2021

  11. [19]

    Charles L. A. Clarke, Saira Rizvi, Mark D. Smucker, Maria Maistro, and Guido Zuccon. 2020. Overview of the TREC 2020 Health Misinformation Track. In Proceedings of the Twenty-Ninth Text REtrieval Conference, TREC 2020, Virtual Event [Gaithersburg, Maryland, USA], November 16-2...

    2020

  12. [20]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the TREC 2020 deep learning track. CoRR abs/2102.07662 (2021). arXiv:2102.07662 https://arxiv.org/abs/2102.07662

    2021 arXiv

  13. [21]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M Voorhees, and Ian Soboroff. 2025. Overview of the TREC 2022 deep learning track. arXiv preprint arXiv:2507.10865 (2025)

    2025 arXiv

  14. [22]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the TREC 2019 deep learning track. arXiv preprint arXiv:2003.07820 (2020)

    2020 arXiv

  15. [23]

    Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri

  16. [24]

    Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023. Masterkey: Automated jailbreak across multiple large language model chatbots. arXiv preprint arXiv:2307.08715 (2023)

    2023 arXiv

  17. [25]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv e-prints (2024), arXiv–2407

    2024

  18. [26]

    Joseph L Fleiss, Bruce Levin, Myunghee Cho Paik, et al. 1981. The measurement of interrater agreement. Statistical methods for rates and proportions 2, 212-236 (1981), 22–23

    1981

  19. [27]

    Jingsheng Gao, Linxu Li, Weiyuan Li, Yuzhuo Fu, and Bin Dai. 2024. Smartrag: Jointly learn rag-related tasks from the environment feedback. arXiv preprint arXiv:2410.18141 (2024)

    2024 arXiv

  20. [28]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv preprint arXiv:2312.10997 (2024)

    2024 arXiv

  21. [29]

    Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms crowd workers for text-annotation tasks.Proceedings of the National Academy of Sciences 120, 30 (2023), e2305016120

    2023

  22. [30]

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM workshop on artificial inte...

    2023

  23. [31]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. InInternational conference on machine learning . PMLR, 3929–3938

    2020

  24. [32]

    Zhibo Hu, Chen Wang, Yanfeng Shu, Hye-Young Paik, and Liming Zhu. 2024. Prompt perturbation in retrieval-augmented generation based large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 1119–1130

    2024

  25. [33]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Info...

    2025

  26. [34]

    Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282 (2020)

    2020 arXiv

  27. [35]

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299 1, 2 (2022)

    2022 arXiv

  28. [36]

    Dennis Jacob, Hend Alzahrani, Zhanhao Hu, Basel Alomair, and David Wagner. 2024. Promptshield: Deployable detection for prompt injection attacks. In Proceedings of the Fifteenth ACM Conference on Data and Application Security and Privacy . 341–352

    2024

  29. [37]

    Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614 (...

    2023 arXiv

  30. [38]

    Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. 2023. Certifying llm safety against adversarial prompting. arXiv preprint arXiv:2309.02705 (2023)

    2023 arXiv

  31. [39]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.134...

    2019 arXiv

  32. [40]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing s...

    2020

  33. [41]

    Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. 2023. Multi-step jailbreaking privacy attacks on chatgpt. arXiv preprint arXiv:2304.05197 (2023)

    2023 arXiv

  34. [42]

    Siran Li, Linus Stenzel, Carsten Eickhoff, and Seyed Ali Bahrainian. 2025. Enhancing retrieval-augmented generation: a study of best practices. arXiv preprint arXiv:2501.07391 (2025)

    2025 arXiv

  35. [43]

    Jiawei Liu, Yangyang Kang, Di Tang, Kaisong Song, Changlong Sun, Xiaofeng Wang, Wei Lu, and Xiaozhong Liu. 2022. Order-disorder: Imitation adversarial attacks for black-box neural ranking models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications S...

    2022

  36. [44]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172 (2023)

    2023 arXiv

  37. [45]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Comput. Surv. (2023)

    2023

  38. [46]

    Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24) . 1831–1847

    2024

  39. [47]

    Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, and Xueqi Cheng. 2023. Black-box adversarial attacks against dense retrieval models: A multi-view contrastive learning method. In Proceedings of the 32nd ACM International Conference on Information a...

    2023

  40. [48]

    Craig Macdonald, Jinyuan Fang, Andrew Parry, and Zaiqiao Meng. 2025. Constructing and Evaluating Declarative RAG Pipelines in PyTerrier. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval . 4035–4040

    2025

  41. [49]

    Junyu Mao, Stuart E Middleton, and Mahesan Niranjan. 2023. Do prompt positions really matter? arXiv preprint arXiv:2305.14493 (2023)

    2023 arXiv

  42. [50]

    Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics 12 (2024), 933–949

    2024

  43. [51]

    Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian

  44. [52]

    Fatemeh Nazary, Yashar Deldjoo, and Tommaso di Noia. 2025. Poison-rag: Adversarial data poisoning attacks on retrieval-augmented generation in recommender systems. In European Conference on Information Retrieval . Springer, 239–251

    2025

  45. [53]

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human-generated machine reading comprehension dataset. (2016)

    2016

  46. [54]

    Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713 (2020)

    2020 arXiv

  47. [55]

    Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. 2022. Entity cloze by date: What LMs know about unseen entities.arXiv preprint arXiv:2205.02832 (2022)

    2022 arXiv

  48. [56]

    Rodrigo Pedro, Daniel Castro, Paulo Carreira, and Nuno Santos. 2023. From prompt injections to sql injection attacks: How protected is your llm-integrated web application? arXiv preprint arXiv:2308.01990 (2023)

    2023 arXiv

  49. [57]

    Sezen Perçin, Xin Su, Qutub Sha Syed, Phillip Howard, Aleksei Kuvshinov, Leo Schwinn, and Kay-Ulrich Scholl. 2025. Investigating the Robustness of Retrieval-Augmented Generation at the Query Level. arXiv preprint arXiv:2507.06956 (2025)

    2025 arXiv

  50. [58]

    Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527 (2022)

    2022 arXiv

  51. [59]

    Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. 2023. Llm self defense: By self examination, llms know they are being tricked. arXiv preprint arXiv:2308.07308 (2023)

    2023 arXiv

  52. [60]

    Julien Piet, Maha Alrashed, Chawin Sitawarin, Sizhe Chen, Zeming Wei, Elizabeth Sun, Basel Alomair, and David Wagner. 2024. Jatmo: Prompt injection defense by task-specific finetuning. In European Symposium on Research in Computer Security . Springer, 105–124

    2024

  53. [61]

    Frances A Pogacar, Amira Ghenai, Mark D Smucker, and Charles LA Clarke. 2017. The positive and negative influence of search results on people’s decisions about the efficacy of medical treatments. In Proceedings of the ACM SIGIR International Conference on Theory of Information...

    2017

  54. [62]

    Ronak Pradeep, Nandan Thakur, Sahel Sharifymoghaddam, Eric Zhang, Ryan Nguyen, Daniel Campos, Nick Craswell, and Jimmy Lin. 2025. Ragnarök: A reusable RAG framework and baselines for TREC 2024 retrieval-augmented generation track. In European Conference on Information Retrieva...

    2025

  55. [63]

    Yao Qiang, Subhrangshu Nandi, Ninareh Mehrabi, Greg Ver Steeg, Anoop Kumar, Anna Rumshisky, and Aram Galstyan. 2024. Prompt perturbation consistency learning for robust language models. arXiv preprint arXiv:2402.15833 (2024). Manuscript submitted to ACM Evaluating the Robustne...

    2024 arXiv

  56. [64]

    Md Abdur Rahman, Hossain Shahriar, Guillermo Francia, Fan Wu, Alfredo Cuzzocrea, Muhammad Rahman, Md Jobair Faruk, and Sheikh Ahamed

  57. [65]

    Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, and Ebrahim Bagheri. 2025. Benchmarking prompt sensitivity in large language models. In European Conference on Information Retrieval . Springer, 303–313

    2025

  58. [66]

    Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684 (2023)

    2023 arXiv

  59. [67]

    Fine-tuned large language models (llms): Improved prompt injection attacks detection. (2025)

    2025

  60. [68]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security . 1671–1685

    2024

  61. [69]

    Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. 2024. Optimization-based prompt injection attack to llm-as-a-judge. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security . 660–674

    2024

  62. [70]

    Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, et al. 2025. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming. arXiv preprint arXiv:2501.18...

    2025 arXiv

  63. [71]

    Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. 2023. Improving the domain adaptation of retrieval augmented generation (RAG) models for open domain question answering. Transactions of the Association for Comp...

    2023

  64. [72]

    Viju Sudhi, Sinchana Ramakanth Bhat, Max Rudat, and Roman Teucher. 2024. Rag-ex: A generic framework for explaining retrieval augmented generation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . 2776–2780

    2024

  65. [73]

    Tianneng Shi, Kaijie Zhu, Zhun Wang, Yuqi Jia, Will Cai, Weida Liang, Haonan Wang, Hend Alzahrani, Joshua Lu, Kenji Kawaguchi, et al. 2025. PromptArmor: Simple yet Effective Prompt Injection Defenses. arXiv preprint arXiv:2507.15219 (2025)

    2025 arXiv

  66. [74]

    Shuai Wang, Ekaterina Khramtsova, Shengyao Zhuang, and Guido Zuccon. 2024. Feb4rag: Evaluating federated search in the context of retrieval augmented generation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval ...

    2024

  67. [75]

    Weixuan Wang, Barry Haddow, Alexandra Birch, and Wei Peng. 2024. Assessing factual reliability of large language model knowledge. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . 805–819

    2024

  68. [76]

    Jerry Wang and Fang Yu. 2025. DeRAG: Black-box Adversarial Attacks on Multiple Retrieval-Augmented Generation Applications via Prompt Injection. arXiv preprint arXiv:2507.15042 (2025)

    2025 arXiv

  69. [77]

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems 36 (2023), 80079–80110

    2023

  70. [78]

    Chen Wu, Ruqing Zhang, Jiafeng Guo, Maarten De Rijke, Yixing Fan, and Xueqi Cheng. 2023. Prada: Practical black-box adversarial attacks against neural ranking models. ACM Transactions on Information Systems 41, 4 (2023), 1–27

    2023

  71. [79]

    Yumeng Wang, Lijun Lyu, and Avishek Anand. 2022. Bert rankers are brittle: a study using adversarial document perturbations. InProceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval . 115–120

    2022

  72. [80]

    Jiaqi Xue, Mengxin Zheng, Yebowen Hu, Fei Liu, Xun Chen, and Qian Lou. 2024. BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models. arXiv preprint arXiv:2406.00083 (2024)

    2024 arXiv

  73. [81]

    Zhuowen Yuan, Zidi Xiong, Yi Zeng, Ning Yu, Ruoxi Jia, Dawn Song, and Bo Li. 2024. Rigorllm: Resilient guardrails for large language models against undesired content. arXiv preprint arXiv:2403.13031 (2024)

    2024 arXiv

  74. [82]

    Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. Benchmarking Retrieval-Augmented Generation for Medicine. In Findings of the Association for Computational Linguistics: ACL 2024

    2024

  75. [83]

    Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023. How language model hallucinations can snowball. arXiv preprint arXiv:2305.13534 (2023)

    2023 arXiv

  76. [84]

    Peitian Zhang, Zheng Liu, Shitao Xiao, Zhicheng Dou, and Jian-Yun Nie. 2024. A multi-task embedder for retrieval augmented LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics . 3537–3553

    2024

  77. [85]

    Yifan Zeng, Yiran Wu, Xiao Zhang, Huazheng Wang, and Qingyun Wu. 2024. Autodefense: Multi-agent llm defense against jailbreak attacks. arXiv preprint arXiv:2403.04783 (2024)

    2024 arXiv

  78. [86]

    Yucheng Zhang, Qinfeng Li, Tianyu Du, Xuhong Zhang, Xinkui Zhao, Zhengwen Feng, and Jianwei Yin. 2024. Hijackrag: Hijacking attacks against retrieval-augmented large language models. arXiv preprint arXiv:2410.22832 (2024)

    2024 arXiv

  79. [87]

    Shuyan Zhou, Uri Alon, Frank F Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2022. Docprompting: Generating code by retrieving the docs. arXiv preprint arXiv: 2207.05987 (2022)

    2022 arXiv

  80. [88]

    Shenyi Zhang, Yuchen Zhai, Keyan Guo, Hongxin Hu, Shengnan Guo, Zheng Fang, Lingchen Zhao, Chao Shen, Cong Wang, and Qian Wang

  81. [89]

    arXiv preprint arXiv:2502.07557 (2025)

    Jbshield: Defending large language models from jailbreak attacks through activated concept analysis and manipulation. arXiv preprint arXiv:2502.07557 (2025)

    2025 arXiv

  82. [92]

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 (2023)

    2023 arXiv

  83. [93]

    Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. 2025. {PoisonedRAG}: Knowledge Corruption Attacks to {Retrieval-Augmented} Generation of Large Language Models. In 34th USENIX Security Symposium (USENIX Security 25) . 3827–3844. Manuscript submitted to ACM

    2025

  84. [2024]

    In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

    The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . 719–729

  85. [2025]

    ACM Transactions on Intelligent Systems and Technology 16, 5 (2025), 1–72

    A comprehensive overview of large language models. ACM Transactions on Intelligent Systems and Technology 16, 5 (2025), 1–72

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.