The reviewed record of science sign in
Pith

arxiv: 2406.04744 · v2 · pith:P7K7CM6T · submitted 2024-06-07 · cs.CL

CRAG -- Comprehensive RAG Benchmark

Xiao Yang , Kai Sun , Hao Xin , Yushi Sun , Nikita Bhalla , Xiangsen Chen , Sajal Choudhary , Rongze Daniel Gui
show 19 more authors
Ziran Will Jiang Ziyu Jiang Lingkun Kong Brian Moran Jiaqi Wang Yifan Ethan Xu An Yan Chenyu Yang Eting Yuan Hanwen Zha Nan Tang Lei Chen Nicolas Scheffer Yue Liu Nirav Shah Rakesh Wanga Anuj Kumar Wen-tau Yih Xin Luna Dong
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:P7K7CM6Trecord.jsonopen to challenge →

classification cs.CL
keywords cragbenchmarkaccuracyansweringquestionquestionssolutionscomprehensive
0
0 comments X
read the original abstract

Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution to alleviate Large Language Model (LLM)'s deficiency in lack of knowledge. Existing RAG datasets, however, do not adequately represent the diverse and dynamic nature of real-world Question Answering (QA) tasks. To bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG), a factual question answering benchmark of 4,409 question-answer pairs and mock APIs to simulate web and Knowledge Graph (KG) search. CRAG is designed to encapsulate a diverse array of questions across five domains and eight question categories, reflecting varied entity popularity from popular to long-tail, and temporal dynamisms ranging from years to seconds. Our evaluation of this benchmark highlights the gap to fully trustworthy QA. Whereas most advanced LLMs achieve <=34% accuracy on CRAG, adding RAG in a straightforward manner improves the accuracy only to 44%. State-of-the-art industry RAG solutions only answer 63% of questions without any hallucination. CRAG also reveals much lower accuracy in answering questions regarding facts with higher dynamism, lower popularity, or higher complexity, suggesting future research directions. The CRAG benchmark laid the groundwork for a KDD Cup 2024 challenge and attracted thousands of participants and submissions. We commit to maintaining CRAG to serve research communities in advancing RAG solutions and general QA solutions. CRAG is available at https://github.com/facebookresearch/CRAG/.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Illocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Models

    cs.CL 2026-03 conditional novelty 7.0

    Chain-of-illocution prompting improves source adherence in RAG explanations for programming education by up to 63% over baselines.

  2. CacheRAG: A Semantic Caching System for Retrieval-Augmented Generation in Knowledge Graph Question Answering

    cs.DB 2026-04 unverdicted novelty 6.0

    CacheRAG turns stateless LLM planners for KGQA into continual learners via schema-agnostic parsing, diversity-optimized hierarchical caching, and bounded subgraph expansion, yielding up to 13.2% accuracy gains on benchmarks.

  3. CacheRAG: A Semantic Caching System for Retrieval-Augmented Generation in Knowledge Graph Question Answering

    cs.DB 2026-04 unverdicted novelty 6.0

    CacheRAG is a cache-augmented architecture for LLM KGQA using ISR parsing, hierarchical MMR-based retrieval, and bounded subgraph expansion, claiming +13.2% accuracy gains on CRAG.

  4. Association Is Not Similarity: Learning Corpus-Specific Associations for Multi-Hop Retrieval

    cs.IR 2026-02 conditional novelty 6.0

    AAR trains a small MLP with contrastive learning on co-occurrence annotations to rerank dense retrieval results using bi-directional association scores, improving multi-hop passage recall on HotpotQA and MuSiQue.

  5. Always-OnAgents:A Survey of Persistent Memory, State, and Governance in LLMAgents

    cs.MA 2026-06 unverdicted novelty 5.0

    Survey mapping persistent state in LLM agents along six axes and proposing the AOEP-v0 protocol to evaluate governance and recovery obligations.

  6. Executable Schema Contracts: From Automatic Ingestion to Multi-Source Retrieval

    cs.CL 2026-06 unverdicted novelty 5.0

    A system for automatic schema discovery from multi-source data builds provenance-aware knowledge graphs and conditions multi-tool agents for improved QA retrieval over baselines.

  7. RAG-DIVE: A Dynamic Approach for Multi-Turn Dialogue Evaluation in Retrieval-Augmented Generation

    cs.IR 2026-01 unverdicted novelty 5.0

    RAG-DIVE uses an LLM to dynamically generate, validate, and evaluate multi-turn dialogues for assessing RAG system performance in interactive settings.

  8. From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review

    cs.AI 2025-04 accept novelty 4.0

    A survey consolidating benchmarks, agent frameworks, real-world applications, and protocols for LLM-based autonomous agents into a proposed taxonomy with recommendations for future research.

  9. Little Brains, Big Feats: Exploring Compact Language Models

    cs.CL 2026-06 unverdicted novelty 3.0

    Small language models can run RAG generation on-device without GPUs in reasonable time.