Pith. sign in

REVIEW 9 cited by

CRAG -- Comprehensive RAG Benchmark

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.04744 v2 pith:P7K7CM6T submitted 2024-06-07 cs.CL

Xiao Yang , Kai Sun , Hao Xin , Yushi Sun , Nikita Bhalla , Xiangsen Chen , Sajal Choudhary , Rongze Daniel Gui
show 19 more authors
Ziran Will Jiang Ziyu Jiang Lingkun Kong Brian Moran Jiaqi Wang Yifan Ethan Xu An Yan Chenyu Yang Eting Yuan Hanwen Zha Nan Tang Lei Chen Nicolas Scheffer Yue Liu Nirav Shah Rakesh Wanga Anuj Kumar Wen-tau Yih Xin Luna Dong
This is my paper · ORCID
classification cs.CL
keywords cragbenchmarkaccuracyansweringquestionquestionssolutionscomprehensive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution to alleviate Large Language Model (LLM)'s deficiency in lack of knowledge. Existing RAG datasets, however, do not adequately represent the diverse and dynamic nature of real-world Question Answering (QA) tasks. To bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual question answering benchmark of 4,409 question-answer pairs and mock APIs to simulate web and Knowledge Graph (KG) search. CRAG is designed to encapsulate a diverse array of questions across five domains and eight question categories, reflecting varied entity popularity from popular to long-tail, and temporal dynamisms ranging from years to seconds. Our evaluation of this benchmark highlights the gap to fully trustworthy QA. Whereas most advanced LLMs achieve <=34% accuracy on CRAG, adding RAG in a straightforward manner improves the accuracy only to 44%. State-of-the-art industry RAG solutions only answer 63% of questions without any hallucination. CRAG also reveals much lower accuracy in answering questions regarding facts with higher dynamism, lower popularity, or higher complexity, suggesting future research directions. The CRAG benchmark laid the groundwork for a KDD Cup 2024 challenge and attracted thousands of participants and submissions. We commit to maintaining CRAG to serve research communities in advancing RAG solutions and general QA solutions. CRAG is available at https://github.com/facebookresearch/CRAG/.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

    cs.CL 2026-07 conditional novelty 7.0 of 10

    On an enterprise corpus scaled from 1.7M to 601M tokens, BM25 beats raw-file agentic search, dense retrieval, and graph RAG at large sizes, crossing near 10M tokens.

  2. GABench: A Comprehensive Benchmark for Evaluating LLM Agents on Graph Analysis Tasks

    cs.AI 2026-08 conditional novelty 6.0 of 10

    GABench provides the first agentic graph-analysis benchmark with 10,400 executable tasks, and finds existing LLM agents succeed on under 40% of complex graph tasks.

  3. EvidentialRAG: Quantifying and Mitigating Information Conflict in Multi-Source Retrieval-Augmented Generation via Evidential Deep Learning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Mapping multi-source RAG chunks to Dirichlet evidence and fusing them with conflict-preserving Dempster-Shafer mass reduces hallucination and raises conflict resolution without large standard-QA loss.

  4. Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook

    cs.CL 2026-03 unverdicted novelty 6.0 of 10

    DOVE measures LLM cultural value alignment via a rate-distortion value codebook and unbalanced optimal transport between human and model open-ended text distributions.

  5. KERAG: Knowledge-Enhanced Retrieval-Augmented Generation for Advanced Question Answering

    cs.CL 2025-09 conditional novelty 6.0 of 10

    KERAG improves knowledge-graph question answering by retrieving broad entity neighborhoods instead of exact query paths and using a fine-tuned chain-of-thought summarizer.

  6. syftr: Pareto-Optimal Generative AI

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Syftr applies multi-objective Bayesian optimization to RAG pipeline search, reporting Pareto frontiers on six benchmarks with 6% accuracy gain at equal cost or 37% cost reduction at equal accuracy versus a LlamaIndex default.

  7. APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encoding

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Adaptive Parallel Encoding restores sequential-style attention in parallel KV-cache RAG/ICL with three training-free tweaks, enabling fast pre-cached long-context generation.

  8. Predict the Retrieval! Test time adaptation for Retrieval Augmented Generation

    cs.CL 2026-01 conditional novelty 5.0 of 10

    Training the language model for a few gradient steps to predict the end of retrieved passages improves specialized-domain RAG accuracy in the paper's tests, by up to 25 points on medical QA.

  9. SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Adding a KL penalty between fine-tuned and original model logits on input tokens during RAG fine-tuning reduces catastrophic forgetting while preserving downstream performance.

Pith tools