REVIEW 14 cited by
Making Retrieval-Augmented Language Models Robust to Irrelevant Context
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Retrieval-augmented language models (RALMs) hold promise to produce language understanding systems that are are factual, efficient, and up-to-date. An important desideratum of RALMs, is that retrieved information helps model performance when it is relevant, and does not harm performance when it is not. This is particularly important in multi-hop reasoning scenarios, where misuse of irrelevant evidence can lead to cascading errors. However, recent work has shown that retrieval augmentation can sometimes have a negative effect on performance. In this work, we present a thorough analysis on five open-domain question answering benchmarks, characterizing cases when retrieval reduces accuracy. We then propose two methods to mitigate this issue. First, a simple baseline that filters out retrieved passages that do not entail question-answer pairs according to a natural language inference (NLI) model. This is effective in preventing performance reduction, but at a cost of also discarding relevant passages. Thus, we propose a method for automatically generating data to fine-tune the language model to properly leverage retrieved passages, using a mix of relevant and irrelevant contexts at training time. We empirically show that even 1,000 examples suffice to train the model to be robust to irrelevant contexts while maintaining high performance on examples with relevant ones.
Forward citations
Cited by 14 Pith papers
-
AR-RAG: Autoregressive Retrieval Augmentation for Image Generation
Autoregressive patch-level retrieval augmentation improves text-to-image generation on GenEval, DPG-Bench, and Midjourney-30K, with a training-free decoding variant and a fine-tuned variant.
-
PURPOSE: Poisoning Conflict Resolution in RAG via Proxy-Fact-Grounded Updates
A black-box poisoning attack that frames false information as a fact-compatible update defeats conflict-resolution safeguards in RAG on most tested settings.
-
White Box Evidence Packages for Policy Audit Reports
In a 60-case controlled audit study, adding white-box model evidence to an LLM auditor increased citation volume but weakened passage grounding and raised evidence misuse, while a shuffled control showed reports can s...
-
Implicit Reasoning Steering via Concept Chaining
Reinforcement-learning-optimized concept-chain paragraphs covertly steer language-model multiple-choice preferences after continued pretraining, with far lower detectability than direct paraphrases.
-
Rethinking LLM Parametric Knowledge as Post-retrieval Confidence for Dynamic Retrieval and Reranking
Shifts in an LLM's hidden-state confidence, before and after a retrieved context, are used as a preference signal to fine-tune a reranker and to trigger retrieval only when initial confidence is low.
-
Med-R$^3$: Enhancing Medical Retrieval-Augmented Reasoning of LLMs via Progressive Reinforcement Learning
A progressive three-stage RL framework with medical-specific rewards improves retrieval-augmented reasoning on medical QA benchmarks, reportedly surpassing GPT-4o-mini with an 8B model.
-
Weak-to-Strong GraphRAG: Aligning Weak Retrievers with Large Language Models for Graph-based Retrieval Augmented Generation
ReG refines weak graph-retriever supervision with LLM-selected reasoning chains and reorganizes retrieved triples into coherent evidence chains, improving KGQA accuracy, data efficiency, and reasoning token efficiency.
-
Lessons from Training Grounded LLMs with Verifiable Rewards
A two-stage GRPO reward scheme improves citation-grounded answering and refusal in RAG models, with reasoning models benefiting more than instruction-tuned ones.
-
DeRAGEC: Denoising Named Entity Candidates with Synthetic Rationale for ASR Error Correction
DeRAGEC explicitly denoises retrieved named-entity candidates with phonetic scores, definitions, and synthetic rationales, improving ASR error-correction WER and NE hit ratio without additional training.
-
Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
Retrieval-augmented generation performance drops noticeably under minor query perturbations, with end-to-end results often tracking retriever behavior.
-
Beyond Independent Passages: Adaptive Passage Combination Retrieval for Retrieval Augmented Open-Domain Question Answering
AdaPCR jointly retrieves and reranks passage pairs for open-domain QA, showing small EM/F1 gains over an in-context retrieval baseline, mostly on multi-hop HotpotQA.
-
Enterprise Large Language Model Evaluation Benchmark
A 14-task enterprise LLM benchmark built mostly from GPT-4o-generated labels and scored by GPT-4o-as-judge shows open-source models closing the reasoning gap, but the dataset is not public and the evaluation is partly...
-
AdaDocVQA: Adaptive Framework for Long Document Visual Question Answering in Low-Resource Settings
A hybrid retrieval, data augmentation, and ensemble inference framework fine-tunes Qwen2.5-VL-72B to 59% on the Japanese LAVA document VQA benchmark.
-
Small Encoders Can Rival Large Decoders in Detecting Groundedness
Task-specific encoders (e.g., RoBERTa-large) rival large decoders such as Llama-3-8B and GPT-4o on binary groundedness detection, within 5 to 10 accuracy points while requiring one to three orders of magnitude fewer FLOPs.
Discussion (0). Sign in to comment.