Pith. sign in

REVIEW 5 major objections 5 minor 46 references

MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces a 1,175-case benchmark for debugging Python code that uses multiple libraries, and reports that the best LLM passes only 58.7 percent of cases.

desk verdict A credible first multi-library debugging benchmark, but the main evaluation prompt contains a bizarre 'hidden info' instruction that likely distorts the headline pass rates and must be ablated before the numbers can be trusted. read the letter →

arxiv 2506.13824 v1 pith:Z4XGXNV2 submitted 2025-06-15 cs.SE cs.AI

Jinyang Huang , Xiachong Feng , Qiguang Chen , Hanjie Zhao , Zihui Cheng , Jiesong Bai , Jingxuan Zhou , Min Li
show 1 more author
Libo Qin
This is my paper
classification cs.SEcs.AI
keywords multi-librarydebuggingcodebenchmarkLLMevaluationPythonlibrariesbugtaxonomyprogramrepairruntimefeedbackchain-of-thoughtreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Real-world Python code usually imports several libraries at once, but debugging benchmarks for large language models have mostly tested code with no libraries or exactly one library. This paper argues that multi-library debugging is a distinct skill and introduces MLDebugging, a benchmark of 1,175 buggy programs spanning 126 libraries and seven bug categories. Across a range of open- and closed-source models, the strongest result is a 58.7 percent pass rate, so the paper concludes that current LLMs do not yet handle multi-library debugging reliably. The hardest categories are requirement misunderstanding and import errors, which require library-level rather than parameter-level reasoning, and simply scaling models from 32B to 72B parameters does not close the gap. The authors note in their Limitations section that most of the benchmark data is model-generated, so the exact distance from real-world bugs remains uncertain.

What carries the argument

The central object is MLDebugging, a dataset of 1,175 buggy Python functions that each combine two or more libraries, together with the task description, test cases, reference solution, and a seven-category bug label whose categories fall into three viewpoints: variable transfer between libraries, library function parameters, and functionality comprehension. Its construction pipeline is what gives the benchmark its character: tasks come from BigCodeBench; GPT-4o generates code from those tasks; executing that code against the provided test cases isolates 609 failing snippets; an AST-guided injection step adds 566 more bugs to balance the seven categories; and four human programmers review, reclassify, and hand-fix samples. The measuring instrument is the test-case pass rate: a repair counts as correct only if the repaired code passes every provided test. A secondary instrument is an embedding-space comparison that places MLDebugging's bug descriptions closer to Stack Overflow's than DebugBench's are, used to argue the benchmark reflects real-world bugs.

What would settle it

Extract real multi-library bug fixes from open-source version control histories, run the same LLMs on those real bugs with the same prompts and test-case checks, and compare per-category pass rates and the overall difficulty ordering with MLDebugging; if the ordering or pass rates diverge, the synthetic generation is not a faithful proxy for real-world debugging.

Watch

Extended reading notes

Core claim

The paper's central claim is that real debugging happens across multiple libraries and that the no-library and single-library benchmarks used so far do not measure this ability. To support the claim it constructs MLDebugging: 1,175 buggy Python functions, each using two to six libraries from a pool of 126, labeled with seven bug types—type mismatch, data transfer issues, function parameter errors, parameter configuration errors, function misuse, requirement misunderstanding, and import errors—and paired with test cases and a canonical fix. On this benchmark the best tested model, DeepSeek-V3, fixes 58.7 percent of samples and GPT-4 fixes 55.6 percent; every model leaves a large fraction of multi-library bugs unfixed. Performance improves between 7B and 32B parameters but then plateaus or falls at 72B, and library prevalence in training data is the factor most correlated with success. The paper further reports that supplying both test cases and runtime error messages improves debugging, that chain-of-thought prompting helps, and that distilled long-chain reasoning models do not.

Load-bearing premise

The benchmark's buggy programs are synthetic—written by GPT-4o and then filtered by human reviewers—so the entire difficulty and category picture rests on those synthetic bugs standing in for the multi-library bugs programmers actually encounter; the paper's own Limitations section concedes this transfer is imperfect.

Editorial extensions

If this is right

  • A model's accuracy on single-library debugging should not be read as its real-world debugging accuracy; multi-library cases are a separate, harder evaluation axis.
  • Improvement efforts should target library-level reasoning—requirement misunderstanding and import errors—where pass rates lag by roughly 20 points behind parameter-level and function-level bug categories.
  • Merely scaling parameters will not solve the task; the paper's data show diminishing returns beyond 32B, so new training data or training methods are needed.
  • Debugging prompts should include both test cases and runtime error messages; the paper's experiments show either alone helps, but the combination is the most stable.
  • Distilled long chain-of-thought without direct optimization is not a shortcut for debugging; the paper's results point to methods that use execution feedback instead.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test this, but a companion benchmark mined from real pull requests that modify imports from two or more libraries would directly test whether GPT-4o-generated bugs produce the same pass-rate ordering and category gaps.
  • The library-prevalence correlation implies a concrete prediction the paper leaves untested: debugging accuracy on a rare library pair should be predictable from the frequency of that pair in training corpora, which could be checked with held-out long-tail tasks.
  • The paper's realism check measures text-embedding similarity, so a behavioral check comparing the actual error classes and fix patterns of generated bugs against real-world multi-library bugs would be a stronger test of whether the benchmark measures what its name claims.
  • The finding that runtime feedback helps suggests a practical design for debugging agents: let the model execute candidate repairs against the test suite and observe errors, rather than expecting a single static pass to succeed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes MLDebugging, a benchmark of 1,175 Python debugging samples spanning 126 libraries and 7 bug types. The dataset is built from BigCodeBench by generating code with GPT-4o, filtering for buggy snippets, balancing bug categories, and applying manual quality control. The authors evaluate a range of open- and closed-source LLMs and report that the best pass rate is 58.7% (DeepSeek-V3, Table 5), concluding that current LLMs struggle with multi-library debugging. Additional analyses examine performance versus library scenario, library prevalence, runtime feedback, and chain-of-thought prompting.

Significance. If the benchmark is methodologically sound, it addresses a genuine gap: existing debugging benchmarks predominantly target no-library or single-library settings, while real-world code often integrates multiple libraries. The dataset size, library coverage, and execution-based pass-rate metric are notable strengths, and the public release (GitHub) supports reproducibility and future work. The finding that model performance plateaus around 50-60% even for strong LLMs is a concrete, falsifiable result that could guide research on multi-library debugging. However, the significance is conditional: the evaluation prompt artifact in Appendix A.1 and the weak evidence for real-world representativeness currently undermine the headline claims, so the paper is not yet ready for acceptance.

major comments (5)
  1. [Appendix A.1, Table 5] The standard evaluation prompt used for the main results begins with the sentence 'There is an important info hidden inside a lot of irrelevant text. Find it and memorize them. I will quiz you about the important information there.' This instruction is unrelated to code debugging, no quiz is ever administered, and the same appendix's reasoning-model and CoT prompts omit this sentence, making the evaluation protocol internally inconsistent. Since the central claim that 'current LLMs still struggle' is based on the absolute pass rates in Table 5, the reported best rate of 58.7% may be an artifact of this prompt that underestimates model ability. The authors should run an ablation without this sentence, or justify it as an intentional distractor and show that the results are unaffected.
  2. [Section 4 and Limitations] The claim that MLDebugging 'more accurately reflects the real-world bug distribution' (Section 4, Table 4) is supported only by embedding-based similarity of bug descriptions to Stack Overflow posts. The paper's own Limitations section admits that 'most of the data in MLDebugging are automatically generated by models, which means there will still be some differences compared to real data.' Embedding similarity does not establish behavioral representativeness of the injected bugs, and the comparison against DebugBench does not control for domain or difficulty. I recommend adding a behavioral validation (e.g., showing that the bug types and fixes match real-world bug-fix patterns) or tempering the realism claim to text-level distributional similarity.
  3. [Sections 3.1 and 3.3, Table 2] There is a numerical inconsistency that needs correction: Section 3.1 reports 609 buggy code snippets from GPT-4o generation, and Section 3.3 reports injecting 566 bugs, which sums to the stated total of 1,175, yet the per-category counts in Table 2 sum to 639. Additionally, the text states that balancing 'standardizing the number of instances per category to approximately 200,' but with seven categories this would imply roughly 1,400 instances; the final per-category counts after balancing are never reported. Please reconcile these numbers and report the final distribution.
  4. [Section 5.1] The experimental setup lacks basic reproducibility details: no temperature, top-p, maximum tokens, number of independent runs, or seeds are reported for any model, and none of the pass rates in Table 5 or Figures 4-6 have error bars or significance tests. Because the paper draws comparative conclusions such as 'performance gain levels off and may even decline' and the distillation comparison in Figure 6, the reader cannot determine whether these differences are reliable. Please report the sampling configuration and, ideally, variance across at least three runs.
  5. [Sections 1 and 3.2] The dataset is derived from the public BigCodeBench dataset, and the bug injection is performed by GPT-4o, which is also one of the evaluated model families. This creates a potential data-contamination and generator-bias problem that is not discussed. If models have seen BigCodeBench code or if the injected bugs reflect GPT-4o's particular error patterns, the comparative results in Table 5 could be skewed. The authors should add a contamination analysis or at least a discussion of the risk and its directional effect, and ideally include a non-GPT-generated sample for robustness.
minor comments (5)
  1. [Table 2] Table 2 uses 'FPE' for 'Function Parameter Errors' but the text and Table 5 use 'PFE'; please make the abbreviation consistent throughout the paper.
  2. [Equation (2)] Equation (2) writes 'D(C|C R,l,R,L)'; the underscore notation is garbled and should be typeset cleanly, for instance as D(C_R,l, R, L).
  3. [Section 3.1] Section 3.1 describes the GPT-4o-generated snippets as 'ensuring a broad range of real and diverse code errors'; since these errors are synthetic, the word 'real' is misleading and should be rephrased.
  4. [Figures 5 and 6] Figures 5 and 6 do not report sample sizes or error bars, and the captions do not define what 'Base', 'Test_case', 'Feedback', and 'Test+Feedback' mean; please clarify.
  5. [Abstract and Section 3.1] The abstract states 126 libraries, while Section 3.1 mentions a pool of 179 libraries; please clarify the relationship between these numbers and how the 126-library subset was selected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline pass rates are execution-based against external test cases, and no claimed result reduces to its input by construction.

full rationale

The central claim (Section 5.2, Table 5) is an execution-based pass rate computed by Equation 5 against test cases that are external to the model outputs; no quantity is fitted from those outputs and then reported as a prediction. The buggy code is produced by GPT-4o from BigCodeBench tasks and filtered by execution and human review (Sections 3.1-3.4), so any generator/evaluator overlap is a data-construction limitation rather than an equation-level reduction; the paper itself acknowledges this in the Limitations section ('most of the data in MLDebugging are automatically generated by models'). The self-citations in the references are background citations and are not load-bearing, and no uniqueness theorem or ansatz is imported from the authors' prior work. The Appendix A.1 'standard evaluation prompt' contains a task-irrelevant 'hidden info' sentence that may affect the absolute pass rates, but that is a prompt-validity confound, not circularity, because it does not make the reported result equal to the input by definition. Overall, the benchmark's evaluation is measured against independently specified test cases, so the derivation chain is self-contained and no circular step is present.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the dataset being a valid proxy for real multi-library debugging. That validity depends on several domain assumptions about test-case oracle completeness, representativeness of BigCodeBench, and faithfulness of LLM-injected bugs. There are no fitted free parameters or invented entities.

assumptions (5)
  • domain assumption Pass/fail on the provided test cases is a sufficient oracle for debugging correctness (Eqs. 1-3).
    The task formulation defines error and correctness entirely by execution against test case set T; any bug not exposed by T is invisible to the benchmark.
  • domain assumption BigCodeBench tasks are representative of real-world multi-library Python programming.
    Section 3.1 sources data by adapting BigCodeBench; the realism claim of MLDebugging inherits this assumption.
  • ad hoc to paper GPT-4o-injected bugs, after manual filtering, are representative of real-world multi-library bugs.
    Sections 3.1 and 3.3 generate bugs via GPT-4o; the paper itself flags in Limitations that automatically generated data differs from real data.
  • domain assumption Embedding-based distribution similarity to Stack Overflow indicates practical relevance.
    Section 4 validates realism via cosine similarity, Euclidean distance, and distribution distance on text embeddings; textual similarity is treated as evidence of behavioral realism.
  • domain assumption Manual rechecking by 4 programmers is sufficient to ensure label and code quality.
    Section 3.4 relies on human review with cross-checks; annotator agreement statistics are not reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios." pith.science (2026). https://pith.science/paper/Z4XGXNV2

@misc{pith2026250613824,
  author       = {Pith},
  title        = {Pith review of: MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z4XGXNV2}},
  note         = {Machine review of arXiv:2506.13824}
}
read the original abstract

Code debugging is a crucial task in software engineering, which attracts increasing attention. While remarkable success has been made in the era of large language models (LLMs), current research still focuses on the simple no-library or single-library setting, ignoring the complex multi-library scenario in real-world applications. To address this limitation, we make the first attempt to introduce MLDebugging (Multi-Library Debugging), a comprehensive benchmark designed to assess debugging challenges within multi-library Python code. Specifically, MLDebugging encompasses 126 distinct Python libraries, covering a wide range of multi-library code issues, categorized into seven distinct types. Furthermore, we conduct a thorough evaluation of MLDebugging using both mainstream open-source and closed-source LLMs and highlight that current LLMs still struggle to correctly perform code debugging across multi-library scenarios. We hope this work can uncover the potential of LLMs in multi-library debugging scenario and offer insights for future research.

Figures

Figures reproduced from arXiv: 2506.13824 by the authors.

Figure 1
Figure 1. A representative static bug example (a) along [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A pipeline diagram illustrating the dataset construction process. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The t-SNE visualization of document embed [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The usage scenarios of Python libraries are [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The debugging effect achieved by incorporat [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The results of various models under base [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 15 canonical work pages

  1. [1]

    Meta AI. 2024. https://ai.meta.com/blog/meta-llama-3-1/ Llama3.1

    work page 2024

  2. [2]

    Anthropic. 2024. Claude 3.5 sonnet. Available at: https://www.anthropic.com/news/claude-3-5-sonnet

    work page 2024

  3. [3]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  4. [4]

    Berkay Berabi, Jingxuan He, Veselin Raychev, and Martin Vechev. 2021. Tfix: Learning to fix coding errors with a text-to-text transformer. In International Conference on Machine Learning, pages 780--791. PMLR

    work page 2021

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  6. [6]

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. 2025 a . Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567

    arXiv 2025

  7. [7]

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiaqi Wang, Mengkang Hu, Zhi Chen, Wanxiang Che, and Ting Liu. 2025 b . Ecm: A unified electronic circuit model for explaining the emergence of in-context learning and chain-of-thought in large language model. arXiv preprint arXiv:2502.03325

    arXiv 2025

  8. [8]

    Qiguang Chen, Libo Qin, Jiaqi WANG, Jingxuan Zhou, and Wanxiang Che. 2024. https://openreview.net/forum?id=pC44UMwy2v Unlocking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of-thought . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

Show all 46 references
  1. [9]

    DeepSeek. 2024. https://www.deepseek.com/ deepseek

    2024

  2. [10]

    Enjun Du, Xunkai Li, Tian Jin, Zhihan Zhang, Rong-Hua Li, and Guoren Wang. 2025 a . https://arxiv.org/abs/2504.00711 Graphmaster: Automated graph synthesis via llm agents in data-limited environments . A preprint

    2025 arXiv

  3. [11]

    Enjun Du, Liu Siyi, and Zhang Yongqi. 2025 b . https://arxiv.org/abs/2505.11125 Graphoracle: A foundation model for knowledge graph reasoning . A preprint

    2025

  4. [12]

    Jia Feng, Jiachen Liu, Cuiyun Gao, Chun Yong Chong, Chaozheng Wang, Shan Gao, and Xin Xia. 2024. Complexcodeeval: A benchmark for evaluating large code models on more complex code. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ...

    2024

  5. [13]

    Timur Galimzyanov, Sergey Titov, Yaroslav Golubev, and Egor Bogomolov. 2024. Drawing pandas: A benchmark for llms in generating plotting code. arXiv preprint arXiv:2412.02764

    2024 arXiv

  6. [14]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  7. [15]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2024. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Info...

    2024

  8. [16]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    2024 arXiv

  9. [17]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    2024 arXiv

  10. [18]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  11. [19]

    Ren \'e Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4j: A database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 international symposium on software testing and analysis, pages 437--440

    2014

  12. [20]

    Mohammad Abdullah Matin Khan, M Saiful Bari, Do Long, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2024. Xcodeeval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. In Proceedings of the 62nd Annu...

    2024

  13. [21]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  14. [22]

    LeetCode . 2025. Leetcode: The world's leading online programming learning platform. https://leetcode.com

    2025

  15. [23]

    Jierui Li, Hung Le, Yinbo Zhou, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. 2024. Codetree: Agent-guided tree search for code generation with large language models. arXiv preprint arXiv:2411.04329

    2024 arXiv

  16. [24]

    Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. 2017. Quixbugs: A multi-lingual program repair benchmark set based on the quixey challenge. In Proceedings Companion of the 2017 ACM SIGPLAN international conference on systems, programming, languages, and appli...

    2017

  17. [25]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  18. [26]

    Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, et al. 2024 b . Mdeval: Massively multilingual code debugging. arXiv preprint arXiv:2411.02310

    2024 arXiv

  19. [27]

    Openai. 2024 a . https://openai.com/index/hello-gpt-4o/ Hello gpt-4o

    2024

  20. [28]

    Openai. 2024 b . https://openai.com/index/introducing-chatgpt-pro/ Introducing chatgpt

    2024

  21. [29]

    Julian Aron Prenner and Romain Robbes. 2023. Runbugrun--an executable dataset for automated program repair. arXiv preprint arXiv:2304.01102

    2023 arXiv

  22. [30]

    Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S. Yu. 2024. https://arxiv.org/abs/2405.12819 Large language models meet nlp: A survey . Preprint, arXiv:2405.12819

    2024 arXiv

  23. [31]

    Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. https://arxiv.org/abs/2310.14799 Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages . Preprint, arXiv:2310.14799

    2023 arXiv

  24. [32]

    Yuling Shi, Songsong Wang, Chengcheng Wan, and Xiaodong Gu. 2024. From code to correctness: Closing the last mile of code generation with hierarchical debugging. arXiv preprint arXiv:2410.01215

    2024

  25. [33]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

    2024 arXiv

  26. [34]

    Stack Overflow . 2025. Stack overflow: Newest questions. https://stackoverflow.com/questions

    2025

  27. [35]

    Qwen Team. 2024. https://qwenlm.github.io/zh/blog/qwq-32b-preview/ Qwq

    2024

  28. [36]

    Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Haotian Hui, Weichuan Liu, Zhiyuan Liu, et al. 2024. Debugbench: Evaluating debugging capability of large language models. arXiv preprint arXiv:2401.04621

    2024 arXiv

  29. [37]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  30. [38]

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2024. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724

    2024 arXiv

  31. [39]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024 a . Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  32. [40]

    Jian Yang, Jiajun Zhang, Jiaxi Yang, Ke Jin, Lei Zhang, Qiyao Peng, Ken Deng, Yibo Miao, Tianyu Liu, Zeyu Cui, et al. 2024 b . Execrepobench: Multi-level executable code completion evaluation. arXiv preprint arXiv:2412.11990

    2024 arXiv

  33. [41]

    Yongheng Zhang, Qiguang Chen, Min Li, Wanxiang Che, and Libo Qin. 2024. Autocap: Towards automatic cross-lingual alignment planning for zero-shot chain-of-thought. arXiv preprint arXiv:2406.13940

    2024 arXiv

  34. [42]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

    2025 arXiv

  35. [43]

    Li Zhong, Zilong Wang, and Jingbo Shang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.49 Debug like a human: A large language model debugger via verifying runtime execution step by step . In Findings of the Association for Computational Linguistics: ACL 2024, pages 851-...

    2024 doi

  36. [44]

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877

    2024 arXiv

  37. [45]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.