Pith. sign in

REVIEW 12 cited by

DebugBench: Evaluating Debugging Capability of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.04621 v3 pith:KXKXSW6Q submitted 2024-01-09 cs.SE cs.AIcs.CL

Runchu Tian , Yining Ye , Yujia Qin , Xin Cong , Yankai Lin , Yinxu Pan , Yesai Wu , Haotian Hui
show 3 more authors
Weichuan Liu Zhiyuan Liu Maosong Sun
This is my paper
classification cs.SEcs.AIcs.CL
keywords debuggingmodelsllmscapabilitydebugbenchbugsclosed-sourcecode
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) have demonstrated exceptional coding capability. However, as another critical component of programming proficiency, the debugging capability of LLMs remains relatively unexplored. Previous evaluations of LLMs' debugging ability are significantly limited by the risk of data leakage, the scale of the dataset, and the variety of tested bugs. To overcome these deficiencies, we introduce `DebugBench', an LLM debugging benchmark consisting of 4,253 instances. It covers four major bug categories and 18 minor types in C++, Java, and Python. To construct DebugBench, we collect code snippets from the LeetCode community, implant bugs into source data with GPT-4, and assure rigorous quality checks. We evaluate two commercial and four open-source models in a zero-shot scenario. We find that (1) while closed-source models exhibit inferior debugging performance compared to humans, open-source models relatively lower pass rate scores; (2) the complexity of debugging notably fluctuates depending on the bug category; (3) incorporating runtime feedback has a clear impact on debugging performance which is not always helpful. As an extension, we also compare LLM debugging and code generation, revealing a strong correlation between them for closed-source models. These findings will benefit the development of LLMs in debugging.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Anchored Self-Play for Code Repair

    cs.SE 2026-07 accept novelty 6.0 of 10

    Anchored Self-Play uses a small reference bug set and code-embedding similarity to stop generator–fixer self-play from drifting into unrealistic bugs, raising average fix rate by 7.0 points on BugSourceBench.

  2. Towards Trustworthy AI: Characterizing User-Reported Risks across LLMs "In the Wild"

    cs.CY 2025-09 conditional novelty 6.0 of 10

    Across seven AI chatbots, Reddit users report mostly reliability failures, with each chatbot showing a distinct pattern of safety, privacy, and security complaints.

  3. RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models

    cs.SE 2025-09 conditional novelty 6.0 of 10

    RepoDebug is a new multi-language, multi-task benchmark for repository-level code debugging on which current LLMs, including the best model Claude 3.5 Sonnet, perform poorly.

  4. IFEvalCode: Controlled Code Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A 1,620-sample, 8-language, Chinese/English benchmark separates code correctness from instruction-following and shows instruction compliance is far lower than correctness across 40+ LLMs.

  5. SimStep: Chain-of-Abstractions for Incremental Specification and Debugging of AI-Generated Interactive Simulations

    cs.HC 2025-07 conditional novelty 6.0 of 10

    SimStep guides teachers through four editable graph abstractions (concepts, scenario, learning goals, UI interactions) to generate and debug AI-built simulations without writing code.

  6. Multilingual Multimodal Software Developer for Code Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A 7B vision-language model trained on synthetic diagram-to-code data outperforms several larger open-weight models on a new 10-language UML/flowchart code-generation benchmark.

  7. The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Study

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A systematic review of 39 studies shows LLM coding assistants generally improve perceived speed and reduce online search, while code quality outcomes remain inconsistent and long-term team effects are understudied.

  8. MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios

    cs.SE 2025-06 conditional novelty 6.0 of 10

    MLDebugging: a new benchmark of 1,175 multi-library Python debugging tasks on which the best tested LLM, Llama-3.1-72B, passes only 58.7% of test cases.

  9. RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving

    cs.SE 2025-05 conditional novelty 6.0 of 10

    RepoMaster, a repository-aware code agent, lifts the task pass rate from 40.7% to 62.9% and cuts token use by about 95% versus OpenHands on the new GitTaskBench benchmark.

  10. MARCO: Meta-Reflection with Cross-Referencing for Code Reasoning

    cs.CL 2025-05 conditional novelty 6.0 of 10

    MARCO combines cross-problem knowledge accumulation with cross-agent lesson sharing to improve LLM code reasoning at inference time.

  11. Cognitive Agents Powered by Large Language Models for Agile Software Project Management

    cs.SE 2025-08 reject novelty 4.0 of 10

    LLM agents acting as Agile roles produced plausible project artifacts in simulation, but the claimed improvements over human teams are unsupported because no comparison or validated metrics are provided.

  12. Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI

    cs.SE 2025-05 conditional novelty 3.0 of 10

    A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.

Pith tools