REVIEW 28 cited by
CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) have shown promise in automated code generation but typically excel only in simpler tasks such as generating standalone code units. Real-world software development, however, often involves complex code repositories (named repo) with complex dependencies and extensive documentation. To fill this gap, our research pivots towards evaluating LLMs in a more realistic setting -- real-world repo-level code generation. We introduce CodeAgentBench, a manually curated benchmark for repo-level code generation. This benchmark comprises five high-quality Python projects, encompassing a total of 101 samples. We assess nine leading LLMs on repo-level tasks and observe a decline in their performance. To tackle this, we present CodeAgent, a novel LLM-based agent framework that employs external tools for effective repo-level code generation. CodeAgent integrates five programming tools, enabling interaction with software artifacts for information retrieval, code symbol navigation, and code testing. We implement four agent strategies to optimize these tools' usage. Our experiments on CodeAgentBench show that CodeAgent enhances LLM performance significantly, with improvements ranging from 18.1\% to 250\%. Further tests on the HumanEval benchmark confirm CodeAgent's adaptability and efficacy across various code generation tasks. Notably, CodeAgent outperforms commercial products like Github Copilot, showcasing superior accuracy and efficiency. These results demonstrate CodeAgent's robust capabilities in code generation, highlighting its potential for real-world repo-level coding challenges.
Forward citations
Cited by 28 Pith papers
-
RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models
RepoReasoner is a repository-level code-reasoning benchmark with output-prediction and call-chain tasks; the best LLM reaches only 69.1% Pass@1 even with oracle context, with low recall in dependency tracing.
-
Sakura: An Approach for Generating Complex Tests from Natural Language Test Descriptions
Sakura is a multi-agent system that generates structurally complex tests from NL descriptions, achieving 50-78% higher compilability and 38-66% higher coverage overlap than baselines on 1,464 scenarios from 20 Apache ...
-
SkillJect: Effectively Automating Skill-Based Prompt Injection for Skill-Enabled Agents
SkillJect automatically generates poisoned agent skills — payload hidden in a helper script, inducement front-loaded in SKILL.md — achieving ~95% attack success across four LLM backends.
-
Tracing Errors, Constructing Fixes: Repository-Level Memory Error Repair via Typestate-Guided Context Retrieval
LTFix uses typestate-guided context retrieval to repair 37 of 49 real-world C memory errors with an LLM, outperforming SAVER, ProveNFix, and SWE-agent.
-
Leveraging Trajectory Graphs for Pre-Execution Error Diagnosis in Agentic LLM Systems
Modeling agent trajectories as action-centric probabilistic graphs lets a GNN warn LLM agents of likely step-level errors before execution, improving pass ratio ~14.7% across four benchmarks.
-
SGAgent: Suggestion-Guided LLM-Based Multi-Agent Framework for Repository-Level Software Repair
A three-agent locate-suggest-fix framework with a knowledge-graph toolkit resolves 154/300 SWE-Bench-Lite issues with Claude-3.5, outperforming same-model baselines by 5-10 points.
-
ValueFlow: Measuring the Propagation of Value Perturbations in Multi-Agent LLM Systems
A perturbation-based framework measures how value opinions propagate through multi-agent LLM systems, revealing that susceptibility varies by value, model, and topology.
-
Agent-Based Software Artifact Evaluation
ArtifactCopilot automates artifact evaluation end-to-end with graph-based execution, claiming 85.42% agreement with human badge outcomes on 48 software engineering artifacts.
-
Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations
A new hypothesis test for binary LLM responses treats semantically equivalent query perturbations as an unknown null set and gives asymptotic validity and consistency guarantees under a uniformity assumption.
-
MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation
MRG-Bench is a multi-language, executable repository-level code generation benchmark on which current LLMs and RAG methods perform poorly, with the dominant failure mode being poor understanding of user requirements.
-
CoCoTen: Detecting Adversarial Inputs to Large Language Models through Latent Space Features of Contextual Co-occurrence Tensors
MRG-Bench is a multi-language, runnable-test benchmark showing that LLMs' main failure in repository-level code generation is understanding user requirements.
-
Large Language Model-Based Agents for Automated Research Reproducibility: An Exploratory Study in Alzheimer's Disease
LLM agents given abstracts, methods, and data dictionary entries approximately reproduced about 53% of key abstract findings across five Alzheimer's disease studies, with frequent mismatches in statistical methods.
-
RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving
RepoMaster, a repository-aware code agent, lifts the task pass rate from 40.7% to 62.9% and cuts token use by about 95% versus OpenHands on the new GitTaskBench benchmark.
-
Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selection
Route-To-Reason learns to predict the accuracy and token usage of each model-strategy pair and routes every query to the pair that maximizes a tunable accuracy-cost score.
-
From EduVisBench to EduVisAgent: A Benchmark and Multi-Agent Framework for Reasoning-Driven Pedagogical Visualization
EduVisAgent, a five-agent framework, outperforms all baseline AI models at generating pedagogically effective interactive visualizations for STEM problems, according to the new EduVisBench benchmark and its GPT-4o-bas...
-
JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generation
A multi-agent LLM framework with rule enforcement, compiler feedback, and retrieval achieves 92/93/81% pass@1 on three self-built EDA benchmarks, up from 67/62/43% for the best single model.
-
Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details
For Other-Play in Yokai, agents trained with different implementation details coordinate across implementations about as well as across seeds, supporting inter-seed cross-play as a proxy for cross-implementation evaluation.
-
TAPO: Transition-Aware Policy Optimization for LLM Agents
Interleaving action-conditioned next-observation supervision with group RL on a shared LLM backbone consistently lifts long-horizon agent success over pure policy optimization.
-
Bridging Inference-Time Scaling and Episodic Memory with Action-Centric Graphs
Graph-based action memory with dual-stream TD learning improves best-of-N inference scaling for LLM agents, reporting +20.81% success / +6.17% progress over vanilla baselines.
-
MRCoder: An Efficient Context Selecting Approach for Repository-Level Code Generation
Draft-guided, API-and-similarity context selection plus parallel draft verification improves repository-level code generation quality while cutting tokens and latency versus RAG and prior compressors.
-
Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents
EMPG re-weights policy-gradient updates by step-level token entropy, amplifying confident correct actions and muting uncertain ones, and adds a future-clarity bonus.
-
Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs
The abstract claims a new local search framework for code generation, but the manuscript body is a different mathematics paper.
-
TripTailor: A Real-World Benchmark for Personalized Travel Planning
A travel-planning benchmark is claimed in the abstract, but the full text is an unrelated supernova spectroscopy paper, leaving the central claim completely unsupported.
-
Self-Evolving Coding Agents
A systematic survey that defines self-evolving coding agents, organizes systems into a five-category object-centered taxonomy, and analyzes when and on what evidence these agents evolve.
-
An Overview of Algorithms for Contactless Cardiac Feature Extraction from Radar Signals: Advances and Challenges
The paper claims to be the first review of signal-processing algorithms for radar-based cardiac monitoring, with a new taxonomy and public dataset listings.
-
CodeEdu: A Multi-Agent Collaborative Platform for Personalized Coding Education
A multi-agent LLM platform for coding education reports large pass-rate improvements on LeetCode problems, evaluated only with simulated students.
-
Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI
A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.
-
LLM-Based Agentic Systems for Software Engineering: Challenges and Opportunities
A survey of LLM-based multi-agent systems across the software development life cycle, plus a research agenda for orchestration, human coordination, cost, and data.
Discussion (0). Continue with ORCID to comment.