Pith. sign in

REVIEW 8 cited by

CodeRAG-Bench: Can Retrieval Augment Code Generation?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.14497 v2 pith:YJJX2R2P submitted 2024-06-20 cs.SE cs.CL

Zora Zhiruo Wang , Akari Asai , Xinyan Velocity Yu , Frank F. Xu , Yiqing Xie , Graham Neubig , Daniel Fried This is my paper
classification cs.SEcs.CL
keywords codecontextsgenerationcoderag-benchmodelsanalysisdocumentationgenerating
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

While language models (LMs) have proven remarkably adept at generating code, many programs are challenging for LMs to generate using their parametric knowledge alone. Providing external contexts such as library documentation can facilitate generating accurate and functional code. Despite the success of retrieval-augmented generation (RAG) in various text-oriented tasks, its potential for improving code generation remains under-explored. In this work, we conduct a systematic, large-scale analysis by asking: in what scenarios can retrieval benefit code generation models? and what challenges remain? We first curate a comprehensive evaluation benchmark, CodeRAG-Bench, encompassing three categories of code generation tasks, including basic programming, open-domain, and repository-level problems. We aggregate documents from five sources for models to retrieve contexts: competition solutions, online tutorials, library documentation, StackOverflow posts, and GitHub repositories. We examine top-performing models on CodeRAG-Bench by providing contexts retrieved from one or multiple sources. While notable gains are made in final code generation by retrieving high-quality contexts across various settings, our analysis reveals room for improvement -- current retrievers still struggle to fetch useful contexts especially with limited lexical overlap, and generators fail to improve with limited context lengths or abilities to integrate additional contexts. We hope CodeRAG-Bench serves as an effective testbed to encourage further development of advanced code-oriented RAG methods.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SERA: Soft-Verified Efficient Repository Agents

    cs.CL 2026-01 conditional novelty 6.0 of 10

    SERA trains competitive coding agents via supervised finetuning on synthetic trajectories verified only by patch-overlap recall, cutting data-generation cost by roughly two orders of magnitude.

  2. ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation

    cs.CR 2025-09 conditional novelty 6.0 of 10

    Documentation poisoning with hidden ranking and suggestion sequences can make RAG-based code generators confidently recommend malicious dependencies, even at 0.01% poisoning ratios.

  3. CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval

    cs.SE 2025-05 conditional novelty 6.0 of 10

    A code retrieval benchmark with quality labels for correctness, efficiency, security, and maintainability shows most retrievers fail to prefer high-quality code, and quality-aware fine-tuning improves this without mea...

  4. CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution

    cs.CR 2025-05 conditional novelty 6.0 of 10

    CRAKEN, an LLM agent combining Self-RAG and Graph-RAG over a CTF writeup database, solves 22% of NYU CTF Bench challenges, three percentage points above the prior D-CIPHER baseline.

  5. SACL: Understanding and Combating Textual Bias in Code Retrieval with Semantic-Augmented Reranking and Localization

    cs.CL 2025-06 reject novelty 5.0 of 10

    Code retrievers rely on surface text like identifiers and docstrings rather than code semantics, and a description-based reranking pipeline, SACL, compensates for that bias but its headline gains come mostly from arti...

  6. CoRet: Improved Retriever for Code Editing

    cs.LG 2025-05 conditional novelty 5.0 of 10

    CoRet fine-tunes a code encoder with a repository-level likelihood loss and call-graph context, improving retrieval recall on SWE-bench and Long Code Arena.

  7. Never Come Up Empty: Adaptive HyDE Retrieval for Improving LLM Developer Support

    cs.SE 2025-07 conditional novelty 4.0 of 10

    A HyDE retrieval pipeline with full-answer context and adaptive similarity thresholding improves LLM answers to Stack Overflow questions over zero-shot prompting for three of four open-source models.

  8. Knowledge Graph Based Repository-Level Code Generation

    cs.AI 2025-05 reject novelty 4.0 of 10

    A knowledge graph code retrieval pipeline is described, but its headline results come from an evaluation that skips the retrieval step and anchors context on the known target function.

Pith tools