Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Large Language Model Critics for Execution-Free Evaluation of Code Changes

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that per-test, reference-aware LLM critics can predict whether a candidate patch passes unseen tests without executing code, reaching 91.6% F1 on test oracle prediction and 82.1% F1 on build status prediction.

desk verdict A useful framework for execution-free patch evaluation, but the headline micro-evaluation claim collapses against a majority-class baseline; the macro results are more credible. read the letter →

arxiv 2501.16655 v1 pith:XUIX6G5Q submitted 2025-01-28 cs.CL cs.AIcs.SE

Aashish Yadavally , Hoan Nguyen , Laurent Callot , Gauthier Guinet This is my paper · ORCID
classification cs.CLcs.AIcs.SE
keywords LLMcriticsexecution-freeevaluationtestoraclepredictionbuildstatusreference-awareagenticworkflowconfidencecalibrationcodepatch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a reference-aware, execution-free evaluation of repository code patches is both feasible and useful. It proposes per-test LLM critics that, given the candidate patch and one unseen test from the gold test patch, predict whether the patch makes that test pass. Aggregating these micro-judgments predicts overall build status more accurately than holistic or reference-free evaluators, and the resulting predicted test pass rates rank different agentic coding workflows in line with their true performance. If true, this gives multi-step coding agents a fine-grained, execution-free signal for progress and comparison.

What carries the argument

The central object is an 'isolated, test-aware LLM critic': a prompt that pairs the candidate patch (expanded to whole-function context, i.e. the '± function-level' variant) with a single unseen test from the gold test patch, and asks the LLM to judge whether that test would pass, along with a verbalized confidence score between 0 and 100. The micro-evaluations are aggregated into a macro-evaluation by the rule that any predicted failure means build failure. Confidence scores are used to calibrate: low-confidence predictions on high-complexity tests are treated as failures.

What would settle it

Run the per-test critic on a held-out set of task instances for which the gold tests are known but not shown to the critic, compute the per-test pass/fail predictions, and compare them against actual test outcomes from execution; if the F1 score does not exceed the class-imbalance baseline (always predicting the majority class) on a statistically meaningful sample, the central claim is refuted.

Watch

Extended reading notes

Core claim

The central discovery is that per-test, reference-aware LLM critics can predict, without running any tests, whether a candidate patch makes a given unseen test pass, and that aggregating these per-test predictions yields an accurate build-status prediction. On a benchmark of real-world GitHub issues, the context-enhanced per-test critic reaches 91.6% F1 on test oracle prediction and 82.1% F1 on build status, outperforming reference-free critics by 65.5–75.5% and other reference-aware critics by 38.9–68.3%. The authors also show that comparing agentic workflows by LLM-predicted test pass rate aligns with true rankings in about 68% of task instances, and that the framework is LLM-agnostic across the tested models.

Load-bearing premise

The method assumes access to the gold test patch—the exact set of hidden tests that defines whether a build succeeds—so without those tests the method cannot be applied as designed, and the paper's own reference-free baselines perform much worse.

Editorial extensions

If this is right

  • Build status can be predicted before any execution, enabling step-level evaluation of agent trajectories during a run.
  • Fine-grained per-test judgments are more accurate than holistic patch evaluation, suggesting that decomposing evaluation into test-sized units is the right granularity.
  • Using the gold test patch as reference is more effective than using the gold code-change patch, because tests give a finer and more semantically grounded comparison.
  • Predicted test pass rates can rank competing agentic workflows, giving a practical tool for workflow selection without running full test suites.
  • The framework transfers across different agentic workflows and several LLM critics with small performance variation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this result holds, the reference-aware critic could be used as a reward model in agent training or selection, using benchmark-provided test patches as supervision to learn execution-free progress signals.
  • The confidence-threshold heuristic suggests a general recipe: verbalized confidence can serve as a calibration signal for LLM-based evaluators, especially for high-complexity inputs where the critic is unreliable.
  • The large gap between reference-free and reference-aware performance implies that, absent access to hidden tests, reliable execution-free evaluation of code changes will require LLMs to infer test expectations from the problem statement alone—a harder task that this paper quantifies rather than solves.
  • A natural testable extension is to apply the same critic to another language or benchmark; if the 91.6% F1 does not persist, the method's generality may be limited to the Python-based, SWE-bench-like setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an execution-free, reference-aware framework for evaluating repository-level code patches produced by LLM-based agentic workflows. Using the gold test patch as a reference, the method first uses isolated LLM critics to predict, for each unseen test, whether a candidate patch would make that test pass (micro-evaluation). These per-test predictions are then aggregated with an all-pass-or-fail rule to predict the overall build status (macro-evaluation). Experiments on SWE-bench-Lite with factory-code-droid patches report a micro F1 of 91.6% and a macro F1 of 82.1%, with claimed improvements over reference-free and other reference-aware baselines. The paper also compares four agentic workflows and three Claude models, and releases an open-source library.

Significance. If the claims were established, the framework would provide a practical, execution-free signal for step-level progress in agentic coding workflows, with an open-source implementation that could be reused. The paper addresses an important problem and includes useful ideas, particularly the micro-to-macro aggregation and the comparison of workflows via predicted test-pass rates. However, the headline micro-evaluation claim is seriously undermined by the class-imbalance issue: on the reported 85:15 pass/fail ratio, a trivial always-pass classifier achieves an F1 of about 91.9%, essentially matching the reported 91.6%. The paper never reports a majority-class baseline, so the central evidence for per-test predictive skill is not established. The macro-evaluation results may retain value, but the current presentation is not rigorous enough to support the stated conclusions.

major comments (4)
  1. [§4.1, Table 1] The micro-evaluation F1 of 91.6% is not distinguishable from a trivial always-pass classifier. With the stated pass/fail ratio of 85:15, always predicting 'pass' yields precision 85.0%, recall 100%, and F1 ≈ 91.9%. The reported precision 85.4% and recall 98.8% imply a true-negative rate of only 4.3%, meaning the critic almost never identifies failing tests. The paper compares against a random baseline (F1 85.3%) that is itself worse than the majority-class oracle, so the claimed '7.4% to 12.7%' improvement is not evidence of LLM reasoning. The paper should report the majority-class baseline and balanced metrics (e.g., specificity, Matthews correlation coefficient, balanced accuracy, or F1 on the failing-test class), and the claims about per-test executability prediction need to be revised accordingly.
  2. [§4.1, Figure 3] The confidence threshold (≤65%) and test-length threshold (>50) are selected post hoc on the same instances used to report final performance. The text states 'Based on these analyses, we chose to automatically assume failures...' without describing any held-out validation or hyperparameter-selection procedure. The reported specificity improvement is small (8.3%→10.3%), and the thresholds may be overfit to the evaluation set. The paper should either report results without thresholding, describe a proper validation split for threshold selection, or provide a sensitivity analysis over threshold choices.
  3. [Abstract and Conclusion] The numerical claims in the abstract and conclusion are inconsistent with the body. The abstract says 'we can predict the build status in 84.8% of the instances', but Table 2 reports build-status accuracy of 71.4% and F1 of 82.1%; the 84.8% value is the micro-evaluation accuracy from Table 1. The conclusion says 'predict executability of all editing locations with an accuracy of 91.5%', but Table 1 reports accuracy 84.8% and F1 91.6%. These discrepancies need to be corrected, and the abstract should not conflate micro accuracy with build-status accuracy.
  4. [§4.2.1, Table 2] The edit-distance baseline is described as performing 'a grid search on the validation set to identify an optimal threshold', but no details are provided on the validation split, the grid range, or the selection criterion. Without this information, the baseline comparison is not reproducible, and it is unclear whether the reported performance (F1 47.7, accuracy 31.3) is an artifact of threshold selection. Full details of the grid search and the resulting threshold should be reported.
minor comments (5)
  1. [Abstract and §2.2] The abstract and conclusion refer to 'predicting executability of all editing locations', but the method actually predicts per-test pass/fail outcomes, not editing locations; the terminology should be aligned with the body.
  2. [Introduction, §4.4] The rank-alignment percentage is reported as 68.8% in the introduction and 68.1% in Section 4.4; these should be reconciled.
  3. [Table 1 vs Table 5] For claude-3-opus in the same micro-evaluation setting, Table 1 reports accuracy 84.8, precision 85.4, recall 98.8, and F1 91.6, while Table 5 reports 84.6, 85.1, 98.9, and 91.5. The paper should explain whether these tables correspond to different runs or different threshold settings, and should report variance across runs.
  4. [Figure 2] Figure 2 is never explicitly described or interpreted in the text; the caption says the setting is 'further described in 4.1', but the figure is not mentioned there either. Either remove it or add a discussion.
  5. [General] There are several typographical and grammatical issues, e.g., 'Let's assume a set of coding tasks T ∈ T' in §2.1.1, 'Approches' in Table 3's caption, and 'which allow further usage' in the Introduction. A careful proofread is needed.

Circularity Check

1 steps flagged · score 2.0 of 10

No central circularity; reference-aware micro/macro predictions are evaluated against external SWE-bench labels, with only a mild post-hoc threshold fit.

  1. fitted input called prediction [Section 4.1, 'Confidence Scores v/s Test Complexity']
    "Based on these analyses, we chose to automatically assume failures when the LLM critic assesses a patch with low confidence (≤ 65%) for tests with high complexity (test length > 50). We observed that such thresholding helps improve the specificity (i.e. true negative rate) by 24.1% (8.3% − →10.3%)."

    The low-confidence/high-complexity failure rule is selected after inspecting the confidence and complexity distributions of the evaluated instances, and the same section reports the resulting specificity gain. The threshold values (65%, length > 50) are therefore fitted to the evaluation labels, so the reported 24.1% specificity improvement is produced by the fitted rule itself rather than by an independent, parameter-free LLM prediction. This is a mild evaluation-optimism/fitted-input issue: the micro-evaluation metrics can incorporate the post-hoc decision, and the 'improvement' does not test the critic's standalone predictive ability.

full rationale

The paper's core derivation chain is not circular. Section 2.1.1 explicitly frames the setting as supervised/reference-aware: 'we assume the access to a ground-truth patch p⋆' and allows the score S to depend on it. The micro-evaluation predicts each unseen test's pass/fail from the candidate patch plus the gold test text, while the ground-truth labels come from actual SWE-bench execution; thus the prediction is not defined in terms of the label. The macro-evaluation aggregates these independent per-test LLM predictions using the same all-tests-pass rule that defines build success, but because the inputs are model predictions rather than oracle results, this is the intended supervised design, not circular reasoning. No load-bearing self-citation or imported uniqueness theorem appears; the only same-author citation (Tran et al., 2019) is background on BLEU limitations. Section 4.3 openly relaxes the gold-reference assumption and acknowledges that it 'might not hold' in real-world scenarios, which is an honest scope limitation. The majority-class baseline concern raised about Table 1 is a baseline-choice and validity issue, not circularity. The only mild circularity-adjacent issue is the post-hoc confidence/complexity threshold in Section 4.1, which weakly fits the reported specificity; it does not make the central claim reduce by construction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on availability of the gold test patch, on the sufficiency of static LLM reasoning over context-enhanced patches, and on the correctness of the SWE-bench ground truth. The only fitted quantities are the confidence and complexity thresholds introduced post hoc.

free parameters (2)
  • Low-confidence failure threshold = verbalized confidence <= 65
    Chosen after inspecting confidence distributions on the same data; used to convert low-confidence LLM predictions into automatic failures for complex tests. No held-out validation is reported.
  • Test complexity threshold = test length > 50 characters
    Proxy for complexity, selected in the same post-hoc analysis; affects specificity and macro predictions.
assumptions (3)
  • domain assumption Gold test patch is available as reference
    'we assume the access to a ground-truth patch p*' in Section 2.1.1; Section 4.3 concedes this does not always hold.
  • domain assumption Static LLM reasoning over patch plus test text is an adequate proxy for execution
    The framework never runs tests; it trusts that an LLM can infer pass/fail from source context, which is the core modeling bet of the paper.
  • domain assumption Expanding patch context to enclosing functions preserves information relevant to test outcomes
    The best variant uses +/- function-level context; this choice is motivated by input-propagation intuition, not formally justified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Model Critics for Execution-Free Evaluation of Code Changes." pith.science (2026). https://pith.science/paper/XUIX6G5Q

@misc{pith2026250116655,
  author       = {Pith},
  title        = {Pith review of: Large Language Model Critics for Execution-Free Evaluation of Code Changes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XUIX6G5Q}},
  note         = {Machine review of arXiv:2501.16655}
}
read the original abstract

Large language models (LLMs) offer a promising way forward for automating software engineering tasks, such as bug fixes, feature additions, etc., via multi-step LLM-based agentic workflows. However, existing metrics for evaluating such workflows, mainly build status and occasionally log analysis, are too sparse and limited in providing the information needed to assess the quality of changes made. In this work, we designed LLM-based critics to derive well-structured and rigorous intermediate/step-level, execution-free evaluation proxies for repo-level code changes. Importantly, we assume access to the gold test patch for the problem (i.e., reference-aware) to assess both semantics and executability of generated patches. With the gold test patch as a reference, we predict executability of all editing locations with an F1 score of 91.6%, aggregating which, we can predict the build status in 84.8% of the instances in SWE-bench. In particular, such an execution-focused LLM critic outperforms other reference-free and reference-aware LLM critics by 38.9% to 72.5%. Moreover, we demonstrate the usefulness of such a reference-aware framework in comparing patches generated by different agentic workflows. Finally, we open-source the library developed for this project, which allows further usage for either other agentic workflows or other benchmarks. The source code is available at https://github.com/amazon-science/code-agent-eval.

Figures

Figures reproduced from arXiv: 2501.16655 by the authors.

Figure 1
Figure 1. A snapshot of SWE-bench leaderboard for different agentic workflows, as of July [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Heatmap graph of predicted and effective test pass rate for all in￾stances of SWE benchmark. Experi￾ment setting and results are further de￾scribed in 4.1 In this work, we assume access to a given solution for the problem (e.g. successful commit), which we refer to as a gold patch in the remainder of the paper. Among the changes made by a human developer to both source code and tests, we specifically consider the go… view at source ↗
Figure 3
Figure 3. (left) Distribution of verbalized confidence scores for both correct and incorrect isolated, test-aware LLM critic predictions. (right) Plot of these confidence scores against test character length, as a proxy to test complexity. Here, the highlighted region indicates the confidence threshold. 4.4. However, note that our evaluation framework is both agentic workflow and LLM-agnostic. In Section 4.4, we compare our t… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: (left) Task instance distribution by LLM-predicted test pass rates. (right) Density plot of the Spearman’s ranking coefficient between LLM-predicted and actual task progress rankings [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.

Reference graph

Works this paper leans on

25 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [2]

    com/news/claude-3-family

    URL https://www.anthropic. com/news/claude-3-family. Accessed: 2024-08-09. Taweesup Apiwattanapong, Alessandro Orso, and Mary Jean Harrold. A differencing algorithm for object-oriented programs. In ASE, pp. 2–13. IEEE Computer Society,

    work page 2024

  2. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond ´e de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavari...

    arXiv

  3. [7]

    2301.09043

    doi: 10.48550/ARXIV . 2301.09043. URL https://doi.org/10.48550/arXiv.2301.09043. Factory Code Droid. Code droid: A technical report,

    arXiv doi:10.48550/arxiv.2301.09043

  4. [8]

    Accessed: 2024-08-09

    URL https://www.factory.ai/ news/code-droid-technical-report . Accessed: 2024-08-09. Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Classeval: A manually-crafted benchmark for evaluat- ing llms on class-level code generation. CoRR, abs/2308.01861,

    arXiv 2024

  5. [9]

    Codebert: A pre-trained model for pro- gramming and natural languages

    11 Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for pro- gramming and natural languages. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 Novem- ber ...

    work page 2020

  6. [11]

    Accessed: 2024-09-30

    URL https: //gru.ai/blog/Gru-Rank-First/. Accessed: 2024-09-30. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . Open...

    work page 2024

  7. [12]

    Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R ´emi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cy- prien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson...

    arXiv

  8. [14]

    URL https://doi.org/10.48550/arXiv.2303.08774

    doi: 10.48550/ARXIV .2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774. Zhiyuan Pan, Xing Hu, Xin Xia, and Xiaohu Yang. Enhancing repository-level code generation with integrated contextual information. CoRR, abs/2406.03283,

    arXiv doi:10.48550/arxiv.2303.08774

Show all 25 references
  1. [17]

    Llama: Open and efficient foundation language models

    12 Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aur ´elien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundati...

    arXiv

  2. [18]

    URL https://doi.org/10.48550/arXiv.2302.13971

    doi: 10.48550/ARXIV .2302.13971. URL https://doi.org/10.48550/arXiv.2302.13971. Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. Does bleu score work for code migration? In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC), volume 10, p...

    2019 doi arXiv

  3. [20]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), A...

    2022

  4. [22]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury

    URL https://arxiv.org/abs/2408.14354. Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement. CoRR, abs/2404.05427,

    arXiv

  5. [23]

    URL https://doi.org/10.48550/arXiv.2404.05427

    doi: 10.48550/ARXIV .2404.05427. URL https://doi.org/10.48550/arXiv.2404.05427. Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. Codebertscore: Evaluating code generation with pretrained models of code. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings o...

    doi arXiv

  6. [24]

    URL https: //doi.org/10.18653/v1/2023.emnlp-main.859

    doi: 10.18653/V1/2023.EMNLP-MAIN.859. URL https: //doi.org/10.18653/v1/2023.emnlp-main.859. Terry Yue Zhuo. ICE-score: Instructing large language models to evaluate code. In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguistics: EAC...

    2023 doi

  7. [25]

    URL https://aclanthology.org/2024.findings-eacl.148

    Association for Computational Linguis- tics. URL https://aclanthology.org/2024.findings-eacl.148. 13

    2024

  8. [1992]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774,

    arXiv

  9. [2002]

    Codebleu: a method for automatic evaluation of code synthesis

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. CoRR, abs/2009.10297,

    2009 arXiv

  10. [2004]

    Nye, Maarten Bosma, Henryk Michalewski, David Do- han, Ellen Jiang, Carrie J

    Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Do- han, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V . Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732,

    arXiv

  11. [2005]

    Adaptive test generation using a large language model

    Max Sch ¨afer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. Adaptive test generation using a large language model. CoRR, abs/2302.06527,

    arXiv

  12. [2019]

    URL http: //dx.doi.org/10.1109/ICPC.2019.00034

    doi: 10.1109/icpc.2019.00034. URL http: //dx.doi.org/10.1109/ICPC.2019.00034. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Opendevin: An open platform for ai software developers as genera...

    2019

  13. [2020]

    URL https: //doi.org/10.18653/v1/2020.findings-emnlp.139

    doi: 10.18653/V1/2020.FINDINGS-EMNLP.139. URL https: //doi.org/10.18653/v1/2020.findings-emnlp.139. Michael Fu and Chakkrit Tantithamthavorn. Linevul: A transformer-based line-level vulnerability prediction. In MSR, pp. 608–620. ACM,

    2020 doi

  14. [2021]

    R2c2-coder: Enhancing and benchmarking real-world repository-level code comple- tion abilities of code large language models

    Ken Deng, Jiaheng Liu, He Zhu, Congnan Liu, Jingxin Li, Jiakai Wang, Peng Zhao, Chenchen Zhang, Yanan Wu, Xueqiao Yin, Yuanxing Zhang, Wenbo Su, Bangyu Xiang, Tiezheng Ge, and Bo Zheng. R2c2-coder: Enhancing and benchmarking real-world repository-level code comple- tion abilit...

    arXiv

  15. [2022]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi

    URL http://papers.nips.cc/paper_files/paper/2022/ hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicita...

    2022

  16. [2023]

    Codescore: Evaluating code generation by learning code execution

    Yihong Dong, Jiazheng Ding, Xue Jiang, Zhuo Li, Ge Li, and Zhi Jin. Codescore: Evaluating code generation by learning code execution. CoRR, abs/2301.09043,

    arXiv

  17. [2024]

    Accessed: 2024-09-30

    URL https://aide.dev/blog/ sota-on-swe-bench-lite . Accessed: 2024-09-30. Anthropic. Introducing the next generation of claude,

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.