Pith. sign in

REVIEW 5 major objections 6 minor 46 references

A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedback

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MultiCodeIF, a 2,021-task benchmark across 14 languages, shows code LLMs satisfy 63% of fine-grained constraints on average and only 18.8% when multiple hierarchical constraints stack.

desk verdict Useful benchmark, but the headline multi-level HSR drop is Data Structure-only and the LLM judge is unvalidated; worth citing after revision. read the letter →

arxiv 2507.00699 v1 pith:HFDPO3EH submitted 2025-07-01 cs.SE

Guoliang Duan , Mingwei Liu , Yanlin Wang , Chong Wang , Xin Peng , Zibin Zheng This is my paper
classification cs.SE
keywords codegenerationinstructionfollowingconstrainttaxonomyLLMevaluationmulti-turnfeedbackself-repairmulti-levelconstraintsbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MultiCodeIF is a benchmark that asks whether code-generating LLMs actually follow detailed instructions, not just whether their output passes tests. It organizes program requirements into 9 constraint categories and 27 fine-grained types, then builds 2,021 code tasks, with 1,526 single-constraint and 495 multi-constraint tasks, spanning 14 programming languages. Evaluated on six LLMs, the paper reports that the best model satisfies only 63.0% of constraints on average, that satisfaction drops from 54.5% to 18.8% as constraints stack from one level to multiple levels, and that structured feedback lifts satisfaction to 83.4% after four repair rounds. If the measurement is right, MultiCodeIF gives the field a reusable instrument for tracking constraint-aware generation and self-repair.

What carries the argument

The load-bearing mechanism is the constraint taxonomy plus the ConstraGen generation pipeline, with an evaluation pipeline of rule-based scripts and an LLM judge. The taxonomy provides 9 categories and 27 fine-grained constraint types, each paired with a validation strategy, such as Tree-sitter for syntax and data structures and GPT-4-Turbo for algorithm and code-quality judgments. ConstraGen starts from real-world seed code, abstracts it into programming concepts, generates prompt-constraint pairs, filters near-duplicates with ROUGE-L similarity, and manually validates; then it expands single-level tasks into L2-L4 ladders by adding one constraint per level. The benchmark's evaluative core is the pair of metrics SSR (soft satisfaction rate, the fraction of individual constraints met) and HSR (hard satisfaction rate, whether all constraints in a task are met), which together convert instruction adherence into a number that can be compared across models and across levels. The multi-turn mechanism is the self-repair loop, where a model receives structured diagnostics for violated constraints and regenerates, with IFRepair@k tracking the hard satisfaction rate after each round.

What would settle it

Sample roughly 200 outputs from the single-level and multi-level sets, have two human programmers independently judge each constraint as satisfied or not, then compare their verdicts to the pipeline's. If agreement is below, say, 90%, or if the pipeline systematically errs toward leniency on code-quality constraints or strictness on interface-specification constraints, the reported 63.0%, 54.5%, 18.8%, and 83.4% figures would need recalibration.

Watch

Extended reading notes

Core claim

The paper's central claim is that instruction-following in code generation can be decomposed into a hierarchy of verifiable constraints, and that current LLMs fail this test in a systematic, measurable way. On its own terms, the discovery is: with a fine-grained taxonomy of 27 constraint types, top models like Claude-3-7-Sonnet satisfy roughly six in ten constraints (63.0% average) on single-level tasks; explicit constraints such as environment and code context are handled well (above 70%), while abstract constraints like code quality fall below 40%; and multi-level compositional tasks cut the hard satisfaction rate from 54.5% to 18.8%. The paper further claims that targeted feedback works, with Claude-3-7-Sonnet improving from 63.0% to 83.4% and GPT-4o from 62.1% to 78.6% after four rounds of structured self-repair. The overall assertion is that a constraint-centric, evolvable, multi-language benchmark reveals capabilities that functional-correctness benchmarks do not.

Load-bearing premise

The whole measurement rests on the automated constraint checks, rule scripts plus a GPT-4-Turbo judge, being correct, yet no human agreement study or false-positive and false-negative analysis is reported, so any bias in the judge shifts every satisfaction rate.

Editorial extensions

If this is right

  • MultiCodeIF can be used as a drop-in evaluation protocol for code LLMs, allowing any new model to be scored on the same 27 constraint types and compared against the six models reported.
  • The L1-to-L4 ladder provides a controlled way to measure compositional instruction-following, so future work can study which constraint combinations cause failures rather than blaming overall model quality.
  • The self-repair results imply that feedback loops, not just bigger models, are a viable path to increasing constraint adherence, with most gains coming in the first two repair rounds.
  • The taxonomy's distinction between explicit and implicit constraints gives a diagnostic: high performance on environment and code-context constraints but low performance on code quality suggests models have a pattern-matching strength and a semantic-reasoning weakness.
  • Because tasks are evolvable, with new seeds, domains, and constraint types addable, the benchmark supports longitudinal tracking without the usual saturation and leakage problem, as long as new seeds are introduced.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the authors leave implicit is to use the feedback rounds and constraint diagnostics as training data, such as instruction tuning or reinforcement learning, rather than only as an evaluation loop; if the 63% to 83% gain transfers to training, self-generated repair trajectories could be a cheap supervision signal.
  • A testable follow-up would run the same 495 multi-level tasks with constraints permuted in different orders to see whether the drop from 54.5% to 18.8% reflects constraint count, constraint type, or the order in which they are added.
  • The benchmark's reliance on an automated judge suggests an obvious companion study: an inter-annotator agreement check between the rule-based and LLM judge and human programmers on a random sample of outputs, which would calibrate every reported number.
  • Neighboring natural-language instruction-following benchmarks could reuse the same L-ladder methodology to test whether the hierarchical collapse is specific to code or a general property of LLM instruction adherence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces MultiCodeIF, a benchmark for evaluating instruction-following in code generation, with a taxonomy of 9 categories and 27 constraint types. Tasks are automatically generated by an LLM pipeline (ConstraGen) from real-world seed code, with manual validation, and include 1,526 single-level and 495 multi-level tasks across 14 (or 15) programming languages. The authors evaluate six LLMs and report findings on constraint satisfaction across categories, the drop in hard satisfaction with multi-level constraints, and improvement via iterative self-repair. The paper claims that MultiCodeIF is a reusable, evolvable, feedback-sensitive benchmark, and it releases the dataset and evaluation pipeline publicly.

Significance. If the evaluation pipeline is valid, MultiCodeIF is a valuable resource: it operationalizes a fine-grained constraint taxonomy, provides a large multi-language dataset, and enables multi-turn self-repair evaluation. The public release of tasks, evaluation scripts, and prompts is a strength, as is the attempt to combine rule-based and LLM-based checks. The headline behavioral findings—performance gaps across models, difficulty of implicit constraints, and self-repair gains—are plausible and would be of interest to the LLM-coding community. However, the paper's central measurements depend on an unvalidated evaluation instrument, and several headline claims are overgeneralized; these issues are load-bearing and need to be addressed before the benchmark's reported numbers can be taken at face value.

major comments (5)
  1. [Section IV-B.1, Table II, Section VI] The accuracy labels used throughout the paper depend on the rule-based and GPT-4-Turbo-based evaluation pipeline described in Section IV-B.1, but no validation of this instrument is reported: there is no human agreement study, no per-constraint-type precision/recall, and no false-positive/false-negative analysis. This is especially concerning for the constraint types judged by LLM (e.g., Code Quality, Algorithm Type, Scenario), and for the multi-level RQ3 where HSR is a conjunction of several binary decisions, so per-constraint judge error compounds. The Threats section discusses generation noise but does not address evaluation-instrument validity. I request an inter-annotator study on a stratified sample of outputs (per constraint type and per model) using the same evaluation rubrics, plus a report of agreement and error analysis; until then, the headline numbers (63.0%, 18.8%, 83.4%) are not supported.
  2. [Abstract, Section I, Section IV-D (RQ3)] The abstract and introduction claim that 'tasks with multiple hierarchical constraints significantly reduce model success rates, from 54.5% in single-level to just 18.8% in multi-level scenarios.' Table V shows that these numbers are GPT-4o's HSR on the Data Structure category only; the multi-level experiment evaluates only Data Structure constraints, even though Section III-C Step 7 says multi-level variants were constructed for Interface Specification, Data Structure, and Code Quality. The claim as stated is not representative of the benchmark as a whole. Please report multi-level SSR/HSR for the other two constraint categories and restrict the abstract claim to the evaluated data.
  3. [Abstract, Section IV-E (RQ4)] The abstract reports that 'average constraint satisfaction rises from 63.0% to 83.4% over four iterative refinement rounds.' Table VI shows this is Claude-3-7-Sonnet's IFRepair@4; GPT-4o reaches 78.6%, and the other four models were not evaluated in RQ4. Also, IFRepair@k is defined as a hard satisfaction rate (all constraints satisfied), so calling it 'average constraint satisfaction' is misleading because the RQ1 metric is per-constraint accuracy. Please specify the model, the metric, and the dataset used for RQ4 (the paper does not state which task set is used) when making this claim.
  4. [Table II, Section IV-B.1] Data Structure constraints are labeled 'Rule' evaluation in Table II, and Section IV-B.1 states that rule-based static tools (Tree-sitter, Guesslang) verify constraints such as 'the list cannot contain more than 100 elements' and 'must use the pop() and top() operation in stack.' Tree-sitter alone cannot establish Data Structure Scale or Operation semantics without runtime analysis or additional interpretation. Please clarify the actual implementation of these checks, and if any of them fall back on the LLM judge, include them in the validation study requested in the first major comment.
  5. [Section IV-E (RQ4)] Section IV-E does not specify which tasks are used in the self-repair experiment. The initial scores (63.0% for Claude, 62.1% for GPT-4o) match the RQ1 single-level averages, but it is not stated whether all 1,526 single-level tasks, all 2,021 tasks, or a subset is used. Also, the feedback is constructed from the same evaluation pipeline, so any evaluator bias propagates into the repair loop. Please specify the task set and provide the evaluation prompts used for feedback.
minor comments (6)
  1. [Section III-D] The paper claims 14 programming languages in the abstract and Table I, but Section III-D lists 9 popular languages and 6 less popular languages (15 total). Please correct the count.
  2. [Section IV-B.2] Category names change between the table (e.g., Environment, Code Quality, Scenario) and the text (e.g., Syntax, Non-Functional Requirements, Situation). Please use a single naming convention throughout.
  3. [Section I] The sentence 'the code generation domain remains less underexplored' should be 'remains underexplored' or 'remains less explored'; the current phrasing is a double negative.
  4. [Section III-B, Step 1] The citation for GPT-4-Turbo is [8], which is the MBPP paper; the intended reference is likely [23] (GPT-4 technical report).
  5. [Section IV-E] The notation in the IFRepair@k definition, I[V_{c∈C} sat(c, y_i^(k))], is awkward; please define sat(c,·) and use consistent symbols with RQ3 (where I[sat(c)] is used).
  6. [Section III-D, Figure 5] The text refers to Figure 5 as showing 'the distribution of constraint counts,' but the figure appears to be a general overview diagram; please align the caption and the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical measurements on a constructed benchmark, not derivations from its own definitions or self-citations.

full rationale

The paper's load-bearing claims are reported measurements: model satisfaction rates, HSR/SSR across levels, and IFRepair@k gains. These are obtained by running six models on MultiCodeIF and applying the paper's evaluation pipeline, so the numbers are not equivalent to the benchmark's inputs by construction. The use of GPT-4-Turbo both as task generator and as LLM judge for abstract constraints is a legitimate measurement-validity concern, because the judge and generator share a model family and no human agreement study is reported; however, that is a threat to the accuracy of the instrument, not a circular derivation in which a fitted parameter is renamed as a prediction. The taxonomy is assembled from prior work, including the authors' own style-violation study [18], but that citation is a design starting point rather than a load-bearing justification of the empirical findings; even if removed, the comparisons among six models would still stand as measurements. HSR is a conjunctive metric, so for the same task chain HSR cannot increase as constraints are added; the paper nevertheless reports the empirical magnitudes and also uses SSR, whose direction is not fixed by definition. No uniqueness theorem or external result is imported from the authors' prior work to force a choice, and no equation in the paper reduces to its own inputs. The Threats section in Section VI acknowledges LLM-generated noise and manual validation, but it does not assert or rely on any circular justification; the unvalidated judge is a correctness risk, not a circularity step.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The benchmark's validity relies on several unproven modeling choices rather than mathematical axioms. The main postulates are that the 27 constraint types cover the relevant space of code instructions, that the rule-based and LLM-based checkers measure compliance accurately, and that the restricted multi-level subset is representative of multi-constraint difficulty.

free parameters (2)
  • ROUGE-L similarity threshold for redundancy filtering = 0.7
    Chosen by hand in Section III-B Step 4; controls diversity of generated prompts and is not derived from data.
  • Number of repair rounds N = 4
    Chosen as the stopping point for the self-repair loop in RQ4; results for later rounds are not shown.
assumptions (3)
  • domain assumption The 27 constraint types across 9 categories capture the important constraint space for real-world code instructions.
    Used throughout to define the benchmark; based on prior style guidelines and ShareGPT analysis, but no formal justification that this set is complete or unbiased.
  • domain assumption LLM-based evaluation with GPT-4-Turbo reliably measures adherence for abstract constraints such as Code Quality and Algorithm type.
    Invoked in Section IV-B.1 for model-based evaluation; no human agreement study or calibration is reported.
  • domain assumption Manual validation of generated tasks is sufficient to ensure task correctness and constraint verifiability.
    The paper states samples are manually reviewed but gives no details on number of reviewers, agreement, or criteria (Section III-B Step 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedback." pith.science (2026). https://pith.science/paper/HFDPO3EH

@misc{pith2026250700699,
  author       = {Pith},
  title        = {Pith review of: A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HFDPO3EH}},
  note         = {Machine review of arXiv:2507.00699}
}
read the original abstract

Large language models (LLMs) have advanced significantly in code generation, yet their ability to follow complex programming instructions with layered and diverse constraints remains underexplored. Existing benchmarks often prioritize functional correctness, overlooking the nuanced requirements found in real-world development. We introduce MultiCodeIF, a comprehensive benchmark designed to evaluate instruction-following in code generation across multiple dimensions: constraint type, hierarchical levels, and iterative refinement. Built upon a structured taxonomy of 9 categories and 27 constraint types, MultiCodeIF enables granular assessment of both functional and non-functional instruction adherence. Using an automated pipeline, ConstraGen, we synthesize and evolve 2,021 code tasks sourced from 14 programming languages, supporting multi-turn evaluation through feedback-driven task variants. Empirical evaluation of six state-of-the-art LLMs uncovers substantial performance disparities. The top-performing model, Claude-3-7-Sonnet, achieves 63.0% average constraint satisfaction, while smaller models like Qwen3-1.7B fall to 44.8%. Models perform well on explicit constraints, but struggle with implicit or abstract constraints. Tasks with multiple hierarchical constraints significantly reduce model success rates, from 54.5% in single-level to just 18.8% in multi-level scenarios. However, structured feedback enables progressive improvement: average constraint satisfaction rises from 63.0% to 83.4% over four iterative refinement rounds. MultiCodeIF provides a scalable, constraint-aware, and feedback-sensitive framework to benchmark LLMs under realistic code generation scenarios, bridging the gap between synthetic evaluations and real-world instruction complexity. The full benchmark dataset, evaluation pipeline, and source code are available at https://github.com/SYSUSELab/MultiCodeIF.

Figures

Figures reproduced from arXiv: 2507.00699 by the authors.

Figure 1
Figure 1. LLM’s Failure in Multi-Constraint Code Generation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of ConstraGen Gen comprises two stages ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An Example of an L2 (Level 2) Task in MultiCodeIF [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Structured prompt template for code instruction generation. It consists [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Overview of MultiCodeIF IV. EXPERIMENTS In this section, we comprehensively evaluate state-of-the-art LLMs on instruction-following in code generation, focusing on adherence to fine-grained constraints. Guided by our MultiCodeIF benchmark, we explore: • RQ1 (Model Comp…
Figure 6
Figure 6. Figure 6: Constraint adherence heatmap showing model performance across 9 [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Average Accuracy for Each Constraint Category across All Six [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Constraint adherence distribution across 9 categories for each model. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Model Performance Across Constraint Complexity Levels. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Example of self-repair that could lead to two different results. One [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Self-repair prompt template used to automatically regenerate code [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Performance of Claude 3-7-Sonnet and GPT-4o across constraint [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 10 canonical work pages

  1. [1]

    Soen-101: Code generation by emulating software process models using large language model agents,

    F. Lin, D. J. Kim, Tse-Husn, and Chen, “Soen-101: Code generation by emulating software process models using large language model agents,”

  2. [2]

    Rocode: Integrating backtracking mechanism and program analysis in large language models for code generation,

    X. Jiang, Y . Dong, Y . Tao, H. Liu, Z. Jin, W. Jiao, and G. Li, “Rocode: Integrating backtracking mechanism and program analysis in large language models for code generation,” 2025. [Online]. Available: https://arxiv.org/abs/2411.07112

    arXiv 2025

  3. [3]

    Skcoder: A sketch-based approach for automatic code generation,

    J. Li, Y . Li, G. Li, Z. Jin, Y . Hao, and X. Hu, “Skcoder: A sketch-based approach for automatic code generation,” in Proceedings of the 45th International Conference on Software Engineering , ser. ICSE ’23. IEEE Press, 2023, p. 2124–2135. [Online]. Available: https://doi.org/10.1109/ICSE48619.2023.00179

    arXiv 2023

  4. [4]

    Enhancing code generation via bidirectional comment-level mutual grounding,

    Y . Di and T. Zhang, “Enhancing code generation via bidirectional comment-level mutual grounding,” 2025. [Online]. Available: https: //arxiv.org/abs/2505.07768

    arXiv 2025

  5. [5]

    Fixing large language models’ specification misunderstanding for better code generation,

    Z. Tian, J. Chen, and X. Zhang, “Fixing large language models’ specification misunderstanding for better code generation,” 2024. [Online]. Available: https://arxiv.org/abs/2309.16120

    arXiv 2024

  6. [7]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...

    arXiv 2021

  7. [8]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

    arXiv 2021

  8. [9]

    A survey on evaluating large language models in code generation tasks,

    L. Chen, Q. Guo, H. Jia, Z. Zeng, X. Wang, Y . Xu, J. Wu, Y . Wang, Q. Gao, J. Wang, W. Ye, and S. Zhang, “A survey on evaluating large language models in code generation tasks,” 2025. [Online]. Available: https://arxiv.org/abs/2408.16498

    arXiv 2025

Show all 46 references
  1. [12]

    Instruction-following evaluation for large language models,

    J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y . Luan, D. Zhou, and L. Hou, “Instruction-following evaluation for large language models,”

  2. [13]

    Infobench: Evaluating instruction following ability in large language models,

    Y . Qin, K. Song, Y . Hu, W. Yao, S. Cho, X. Wang, X. Wu, F. Liu, P. Liu, and D. Yu, “Infobench: Evaluating instruction following ability in large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2401.03601

    2024 arXiv

  3. [14]

    Codeif: Benchmarking the instruction-following capabilities of large language models for code generation,

    K. Yan, H. Guo, X. Shi, J. Xu, Y . Gu, and Z. Li, “Codeif: Benchmarking the instruction-following capabilities of large language models for code generation,” 2025. [Online]. Available: https://arxiv.org/abs/2502.19166

    2025 arXiv

  4. [15]

    Codeif-bench: Evaluating instruction-following capabilities of large language models in interactive code generation,

    P. Wang, L. Zhang, F. Liu, L. Shi, M. Li, B. Shen, and A. Fu, “Codeif-bench: Evaluating instruction-following capabilities of large language models in interactive code generation,” 2025. [Online]. Available: https://arxiv.org/abs/2503.22688

    2025

  5. [17]

    A hierarchical and evolvable benchmark for fine-grained code instruction following with multi-turn feedback,

    “A hierarchical and evolvable benchmark for fine-grained code instruction following with multi-turn feedback,” https://github.com/SYSUSELab/ MultiCodeIF, 2025, accessed: May 2025

    2025

  6. [18]

    Beyond functional correctness: Investigating coding style inconsistencies in large language models,

    Y . Wang, T. Jiang, M. Liu, J. Chen, and Z. Zheng, “Beyond functional correctness: Investigating coding style inconsistencies in large language models,” arXiv preprint arXiv:2407.00456 , 2024

    2024 arXiv

  7. [19]

    Followbench: A multi-level fine- grained constraints following benchmark for large language models,

    Y . Jiang, Y . Wang, X. Zeng, W. Zhong, L. Li, F. Mi, L. Shang, X. Jiang, Q. Liu, and W. Wang, “Followbench: A multi-level fine- grained constraints following benchmark for large language models,” arXiv preprint arXiv:2310.20410 , 2023

    arXiv

  8. [20]

    Sharegpt,

    “Sharegpt,” 2023. [Online]. Available: https://sharegpt.com/

    2023

  9. [21]

    Prompt-based cost-effective evaluation and operation of chatgpt as a computer programming teaching assistant,

    M. Ballestero-Ribó and D. Ortiz-Martínez, “Prompt-based cost-effective evaluation and operation of chatgpt as a computer programming teaching assistant,” 2025. [Online]. Available: https://arxiv.org/abs/2501.17176

    2025 arXiv

  10. [22]

    tree-sitter/tree-sitter: v0.25.5,

    M. Brunsfeld, A. Qureshi, A. Hlynskyi, ObserverOfTime, W. Lillis, J. Vera, dundargoc, P. Turnbull, T. Clem, D. Creager, A. Helwer, R. Rix, D. Kavolis, H. van Antwerpen, M. Davis, C. Clason, R. Bruins, Ika, T.-A. Nguy˜ên, A. Ya, S. Brunk, M. Massicotte, bfredl, N. Hasabnis, M. ...

    2025 doi

  11. [23]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  12. [24]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out . Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013/

    2004

  13. [25]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al. , “Gpt-4o system card,” arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  14. [26]

    Claude 3.7 sonnet and claude code,

    Anthropic, “Claude 3.7 sonnet and claude code,” https://www.anthropic. com/news/claude-3-7-sonnet, 2025, accessed: 2025-02-25

    2025

  15. [27]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” 2025. [Online]. Available: https: //arxiv.org/abs/2501.12948

    2025 arXiv

  16. [28]

    Deepseek-v3 technical report,

    ——, “Deepseek-v3 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2412.19437

    2024 arXiv

  17. [29]

    Qwen3 technical report,

    Q. Team, “Qwen3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09388

    2025 arXiv

  18. [30]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  19. [31]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023

    2023

  20. [32]

    Guesslang: A neural network to guess the programming language from code snippet,

    Y . Blanc, “Guesslang: A neural network to guess the programming language from code snippet,” https://github.com/yoeo/guesslang, 2019, accessed: 2025-05-30

    2019

  21. [33]

    Ds-1000: A natural and reliable benchmark for data science code generation,

    Y . Lai, C. Li, Y . Wang, T. Zhang, R. Zhong, L. Zettlemoyer, S. W. tau Yih, D. Fried, S. Wang, and T. Yu, “Ds-1000: A natural and reliable benchmark for data science code generation,” 2022. [Online]. Available: https://arxiv.org/abs/2211.11501

    2022 arXiv

  22. [34]

    Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation,

    X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y . Chen, J. Feng, C. Sha, X. Peng, and Y . Lou, “Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation,” 2023. [Online]. Available: https://arxiv.org/abs/2308.01861

    2023 arXiv

  23. [35]

    Learning-based widget matching for migrating gui test cases,

    Y . Zhang, W. Zhang, D. Ran, Q. Zhu, C. Dou, D. Hao, T. Xie, and L. Zhang, “Learning-based widget matching for migrating gui test cases,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. ACM, Feb. 2024, p. 1–13. [Online]. Available: http://...

    2024

  24. [36]

    Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories,

    J. Li, G. Li, Y . Zhao, Y . Li, H. Liu, H. Zhu, L. Wang, K. Liu, Z. Fang, L. Wang, J. Ding, X. Zhang, Y . Zhu, Y . Dong, Z. Jin, B. Li, F. Huang, and Y . Li, “Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories,” 2024. [Online]. Av...

    2024 arXiv

  25. [37]

    Rustevoˆ 2: An evolving benchmark for api evolution in llm-based rust code generation,

    L. Liang, J. Gong, M. Liu, C. Wang, G. Ou, Y . Wang, X. Peng, and Z. Zheng, “Rustevoˆ 2: An evolving benchmark for api evolution in llm-based rust code generation,” arXiv preprint arXiv:2503.16922 , 2025

    2025 arXiv

  26. [38]

    Feedbackeval: A benchmark for evaluating large language models in feedback-driven code repair tasks,

    D. Dai, M. Liu, A. Li, J. Cao, Y . Wang, C. Wang, X. Peng, and Z. Zheng, “Feedbackeval: A benchmark for evaluating large language models in feedback-driven code repair tasks,” arXiv preprint arXiv:2504.06939 , 2025

    2025

  27. [39]

    Benchmarking complex instruction-following with multiple constraints composition,

    B. Wen, P. Ke, X. Gu, L. Wu, H. Huang, J. Zhou, W. Li, B. Hu, W. Gao, J. Xu, Y . Liu, J. Tang, H. Wang, and M. Huang, “Benchmarking complex instruction-following with multiple constraints composition,”

  28. [40]

    Generating equivalent representations of code by a self-reflection approach,

    J. Li, G. Li, L. Wang, H. Zhu, and Z. Jin, “Generating equivalent representations of code by a self-reflection approach,” arXiv preprint arXiv:2410.03351, 2024

    2024 arXiv

  29. [41]

    Codescore: Evaluating code generation by learning code execution,

    Y . Dong, J. Ding, X. Jiang, G. Li, Z. Li, and Z. Jin, “Codescore: Evaluating code generation by learning code execution,” ACM Trans. Softw. Eng. Methodol. , vol. 34, no. 3, Feb. 2025. [Online]. Available: https://doi.org/10.1145/3695991

    2025 doi

  30. [42]

    Available: https://arxiv.org/abs/2407.03978

    [Online]. Available: https://arxiv.org/abs/2407.03978

    arXiv

  31. [43]

    Wizardlm: Empowering large pre-trained language models to follow complex instructions,

    C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, Q. Lin, and D. Jiang, “Wizardlm: Empowering large pre-trained language models to follow complex instructions,” 2025. [Online]. Available: https://arxiv.org/abs/2304.12244

    2025 arXiv

  32. [44]

    Magicoder: Empowering code generation with oss-instruct,

    Y . Wei, Z. Wang, J. Liu, Y . Ding, and L. Zhang, “Magicoder: Empowering code generation with oss-instruct,” 2024. [Online]. Available: https://arxiv.org/abs/2312.02120

    2024 arXiv

  33. [45]

    Self-instruct: Aligning language models with self-generated instructions,

    Y . Wang, Y . Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self-generated instructions,” 2023. [Online]. Available: https://arxiv.org/ abs/2212.10560

    2023 arXiv

  34. [46]

    Genetic instruct: Scaling up synthetic generation of coding instructions for large language models,

    S. Majumdar, V . Noroozi, M. Samadi, S. Narenthiran, A. Ficek, W. U. Ahmad, J. Huang, J. Balam, and B. Ginsburg, “Genetic instruct: Scaling up synthetic generation of coding instructions for large language models,”

  35. [48]

    Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning,

    Z. Yu, X. Zhang, N. Shang, Y . Huang, C. Xu, Y . Zhao, W. Hu, and Q. Yin, “Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning,” 2024. [Online]. Available: https://arxiv.org/abs/2312.14187

    2024 arXiv

  36. [2023]

    Available: https://arxiv.org/abs/2311.07911

    [Online]. Available: https://arxiv.org/abs/2311.07911

    arXiv

  37. [2024]

    Available: https://arxiv.org/abs/2403.15852

    [Online]. Available: https://arxiv.org/abs/2403.15852

    arXiv

  38. [2025]

    Available: https://arxiv.org/abs/2407.21077

    [Online]. Available: https://arxiv.org/abs/2407.21077

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.