Pith. sign in

REVIEW 3 major objections 5 minor 51 references

On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An eight-model study finds LLM judges unreliable for code correctness but moderately aligned with humans on summaries

desk verdict Well-executed large empirical study showing GPT-4 is a mediocre code-correctness judge and a moderate summary judge; the oracle concern is real but doesn't overturn the central finding. read the letter →

arxiv 2507.16587 v1 pith:6F2AP7XE submitted 2025-07-22 cs.SE

Giuseppe Crupi , Rosalia Tufano , Alejandro Velasco , Antonio Mastropaolo , Denys Poshyvanyk , Gabriele Bavota This is my paper · ORCID
classification cs.SE
keywords LLM-as-a-judgecodegenerationsummarizationcorrectnesshumanevaluationinter-rateragreementempiricalsoftwareengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether large language models can take over the evaluation of two generative software-engineering tasks where quantitative metrics are known to be weak proxies: judging whether generated code is correct, and judging the quality of code summaries. It asks eight LLMs to assess 2,686 code-generation candidates and five LLMs to rate 1,163 summaries, comparing their verdicts against test-suite execution and against the median rating of nine human judges. The paper finds that GPT-4-turbo is the strongest judge on both tasks, yet for code generation its agreement with the test oracle is only fair for Java and weak for Python, and it misjudges 50% of wrong Java implementations and 35% of wrong Python implementations as correct. For summarization, GPT-4-turbo reaches moderate agreement with human judges on content adequacy (Krippendorff's alpha 0.58 for Java, 0.63 for Python), while smaller models stay at chance level or fail to produce judgments. The practical point is that LLM-as-a-judge cannot yet replace tests or humans for correctness, but may already be a useful complement for summary-quality assessment.

What carries the argument

For code generation, the load-bearing mechanism is a pairwise verdict protocol: the judge sees only the natural-language problem description, the required function signature, and one candidate implementation, and must answer correct/wrong with a rationale; the paper then aligns that binary verdict with the pass/fail outcome of the benchmark's test suite using confusion matrices and Cohen's kappa. For summarization, the judge rates each summary on content adequacy, conciseness, and fluency/understandability on a 1-5 scale, and the ratings are compared to the median of three independent human ratings via Krippendorff's alpha. To test whether correct verdicts reflect real code understanding rather than guessing, the authors mutate 332 implementations with a mutation tool and also create semantically equivalent variants, then check whether the judges reject the mutants and still accept the equivalent code.

What would settle it

Re-run the code-generation protocol on a set of candidates whose correctness is independently verified by formal specification or multiple human experts, and compare GPT-4-turbo's verdicts with that verified oracle. If agreement rises well above the reported kappa values (0.21 for Java, 0.10 for Python), the bottleneck is the test-suite oracle rather than the judge; if it stays low, the judge itself lacks the asserted reliability.

Watch

Extended reading notes

Core claim

The central claim is a measurement: among the eight models tested, the best judge is GPT-4-turbo, and even it is far from reliable in the correctness task. With the best performing prompt (automated chain-of-thought), Cohen's kappa between GPT-4-turbo's pass/fail judgment and actual test results is 0.21 for Java and 0.10 for Python; its confusion matrices show 72% of correct Java implementations accepted but only 50% of wrong ones rejected, and for Python 46% of correct implementations accepted while 35% of wrong ones are accepted. That is, the model errs in opposite directions across languages. For code summarization the same model's zero-shot ratings agree with the median human rating at Krippendorff's alpha 0.58 (Java) and 0.63 (Python) for content adequacy, with fair-to-moderate agreement on conciseness and fluency; this is the strongest agreement any tested judge achieves, and it occurs on the same quality dimension on which humans agree most among themselves. The paper also reports that smaller models (DeepSeek Coder up to 33B, CodeLlama up to 34B) either fail to understand the judging task or produce near-chance agreement, and that GPT-4 shows only negligible self-bias in code generation while all judges systematically underestimate human-written code.

Load-bearing premise

The entire correctness measurement rests on treating the filtered executable test suites as the ground truth for whether a function is correct, so if those tests are incomplete or misleading the reported judge accuracy is off by an unknown amount.

Editorial extensions

If this is right

  • Automated code review and grading systems that rely on the best current LLM judge would accept about half of wrong Java functions as correct, so correctness verdicts still need tests or human inspection.
  • Code-generation evaluations that cannot run tests (e.g., no executable suite) are not yet safe to delegate to LLM judges; the reported kappa values are too low.
  • For code summarization, content adequacy scores from GPT-4-turbo align with human median scores to a moderate degree, making it a plausible complement to metrics like BLEU and ROUGE that are known to correlate weakly with human judgment.
  • Model size is a practical gate: judges in the 1B-34B range either fail to produce judgments or agree with humans at chance level, so the cost-saving rationale for small local judges does not hold for these tasks.
  • The three-human-per-summary protocol with explicit rating guidelines is what makes the summarization oracle trustworthy; replacing it with a single rating would likely lower the apparent agreement of any judge.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The asymmetry—accepting wrong Java code while rejecting correct Python code—suggests a language-dependent response bias that a per-language calibration of the judge's decision threshold could partially correct; the paper does not test such calibration.
  • Because 27% of false positives were attributed to ambiguous requirements text, some judged 'misjudgments' are really specification failures; making descriptions precise might raise judge accuracy beyond what model improvements alone would yield.
  • One could extend the mutation analysis into a prompting strategy: since GPT-4-turbo already detects injected mutants in 88-97% of cases, pairing candidate code with a mutated version in the prompt might convert an unreliable absolute judge into a more reliable relative judge.
  • Another untested extension is to apply the same protocol to bug-fixing or code review tasks, where the judge's false-positive tendency would have different consequences than in generation scoring.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This manuscript reports an empirical study of eight LLMs used as judges for two code-related tasks: code generation (judging whether a candidate implementation is correct) and code summarization (judging content adequacy, conciseness, and fluency & understandability). For code generation, the authors use 1,405 Java and 1,281 Python candidate implementations drawn from CoderEval and compare each LLM's binary correctness judgment against pass/fail on the benchmark's test suites. For code summarization, five LLMs judge 1,163 summaries (human-written and LLM-generated) for 198 Java/Python functions, and the judgments are compared to median human ratings from nine judges. The authors find that GPT-4-turbo is the best judge in both tasks, that smaller open models frequently fail to produce valid judgments or produce near-constant ratings, and that even GPT-4-turbo misjudges 50% of failing Java implementations and 35% of failing Python implementations as correct. For content adequacy in summarization, GPT-4-turbo reaches Krippendorff's alpha of 0.58 (Java) and 0.63 (Python), the strongest agreement observed. The paper also analyzes self-bias, conducts a manual qualitative analysis of misjudgments, and probes code understanding through mutation and semantically-equivalent code transformation experiments.

Significance. The study is carefully designed in several important respects: every extracted judgment was manually checked by two authors, four prompt variants were compared for both tasks, multiple LLM families and parameter scales were considered, and the self-bias analysis is statistically grounded. The mutation and semantically-equivalent code experiment is a welcome direct probe of whether true-positive judgments reflect code understanding rather than guessing. The public replication package and the newly constructed human-judgment dataset for code summaries are valuable community assets. Provided the test-suite oracle concern can be addressed, the paper delivers an important, largely negative result on LLM-as-a-judge for code correctness and a cautiously positive result for summary content adequacy, with clear implications for automated code review and evaluation practice.

major comments (3)
  1. [Section 2.2.1 and Section 3.1 (Fig. 1, Table 2)] The accuracy and Kappa figures for code generation treat the filtered CoderEval test suites as a binary correctness oracle. The quality-assurance step removes only three detectable failure modes: target functions that fail their own tests, empty bodies that pass, and dummy returns that pass. It does not establish that the remaining pass/fail labels agree with a human reading of whether the candidate satisfies the natural-language description. Section 3.1.2 itself reports "a few unreliable test outcomes" and attributes 27% of false positives to "ambiguous requirements text," a category in which the test outcome and the LLM's judgment can legitimately disagree. Because the headline claim is the false-positive rate (50% for Java, 35% for Python for GPT-4-turbo), the paper should quantify how many pass/fail labels would change under a gold human correctness judgment, and/or re-present Fig. 1 and Table 2 after excluding cases labeled as ambiguous or unreliable. Without this, the absolute misjudgment rates are not well calibrated.
  2. [Section 3.1.2] The manual analysis of false positives and false negatives is based on a sample of 15 false positives and 15 false negatives per LLM, yet the text presents the resulting category percentages (37% uncaught wrong behavior, 32% coding context, 27% ambiguous requirements) as if they were exact proportions of the full confusion matrix. The paper should clarify that these percentages come from a small qualitative sample and should provide the confusion matrices restricted to the cases whose test outcome and requirements text are both deemed reliable, so the reader can see how much the headline rates change. This is particularly important because the authors themselves state that ambiguous-requirements cases are "not really failures of the judging LLMs."
  3. [Section 3.1.2 (Fig. 2)] The mutation and semantically-equivalent code experiment samples only true-positive instances that both GPT-4 and GPT-3.5 judged correctly. This selection conditions on the property being tested: those are the instances on which both judges already agreed, and they may be systematically easier than the full set of judged implementations. The reported rates (e.g., 79% of semantically equivalent Java code still judged correct by GPT-4) are therefore not population estimates for all true positives. The paper should either sample from all true positives or explicitly present these numbers as an upper-bound consistency check, and it should avoid generalizing them into overall code-understanding capabilities.
minor comments (5)
  1. [Section 2.3.1] There is a typo: "190 Pythjon" should be "190 Python."
  2. [Section 2.3.2] There is a typo: "The fist, named zero shot" should read "The first, named zero shot."
  3. [Section 3.2 and Section 3.3] The model name "CodeLlalma" appears twice and should be "CodeLlama."
  4. [Section 2.2.2 and Table 5] For conciseness and fluency & understandability, human inter-rater agreement is only moderate (alpha between 0.56 and 0.62), so the median-human oracle is noisier for those criteria. The text could add one sentence noting that agreement with the LLM is therefore likely underestimated for those dimensions, which would contextualize the lower alpha values without changing the conclusions.
  5. [Table 2 and Table 5] Kappa and Krippendorff alpha values are reported without confidence intervals. Since the number of judgments differs across models because of invalid outputs, including confidence intervals would make the ranking of near-adjacent models (e.g., GPT-3.5 versus GPT-4 on Python code generation) easier to interpret.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all judged quantities are compared against external oracles (test execution and median human ratings), and no fitted parameter or self-citation is load-bearing.

full rationale

The paper does not derive any predicted quantity from its own inputs. For code generation, the oracle is pass/fail on the filtered CoderEval test suites, an external standard defined independently of the LLM judgments; for summarization, the oracle is the median of three human ratings, also external. The study then measures agreement (Cohen's Kappa, Krippendorff's alpha, confusion matrices) between LLM judgments and these external labels. No parameter is fitted to a subset and then 'predicted' on a related subset, and no quantity is defined in terms of the quantity it is supposed to predict. The paper's acknowledgment that 'Using tests as a proxy for code correctness is a limitation of our study' and the observation that 'a few unreliable test outcomes led to false negatives' identify a possible miscalibration of the oracle, which is a construct-validity threat, not circular reasoning: the test oracle does not depend on the LLM's judgment. The choice of best prompt based on GPT-4's agreement is an empirical selection, not a fitted input renamed as a prediction. Citations to prior work (e.g., [30], [35], [49]) are used as related work or for prompt choices, and the central results are computed from data collected in this paper rather than imported from self-citations. Therefore no circularity is found.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

Empirical study with no free parameters or invented entities. It relies on external oracles (tests, human judgments) and on the representativeness of the filtered benchmarks.

assumptions (3)
  • domain assumption CoderEval test suites, after quality filtering, correctly classify generated functions as pass/fail.
    Used as ground truth oracle in Sections 2.3 and 3.1; acknowledged as a limitation in Section 4.
  • domain assumption Median of three human raters provides a reliable ground truth for code summary quality.
    Human agreement was high (Krippendorff's alpha 0.56-0.81), so median is used as oracle in Section 2.2.2.
  • domain assumption The filtered subset of CoderEval (184 Java, 190 Python problems) is representative of code generation tasks.
    Excluded problems may have different difficulty; generalization is bounded, as noted in Section 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization." pith.science (2026). https://pith.science/paper/6F2AP7XE

@misc{pith2026250716587,
  author       = {Pith},
  title        = {Pith review of: On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6F2AP7XE}},
  note         = {Machine review of arXiv:2507.16587}
}
read the original abstract

Large Language Models have been recently exploited as judges for complex natural language processing tasks, such as Q&A. The basic idea is to delegate to an LLM the assessment of the "quality" of the output provided by an automated technique for tasks for which: (i) quantitative metrics would only tell part of the story, and; (ii) a large-scale human-based evaluation would be too expensive. LLMs-as-a-judge, if proven effective for a specific task, can also unlock new possibilities for automation, with several LLMs proposing a solution for a given instance of the task and others judging and deciding what is the best output to show the user. We study the effectiveness of LLMs-as-a-judge for two code-related tasks, namely code generation and code summarization. The rationale for choosing these tasks is two-fold. First, quantitative metrics are usually not enough for the assessment of code summarizers/generators. For example, it is well documented that metrics such as BLEU are quite weak proxies for the quality of the generated summaries. Second, even state-of-the-art techniques still struggle with handling complex instances of these tasks, making them good candidates for benefiting from more advanced solutions envisioning collaboration among LLMs. For code generation, we check whether eight LLMs are able to judge the correctness of 1,405 Java methods and 1,281 Python functions generated by the same LLMs or implemented by humans. For code summarization, we compare the judgment of five LLMs to those provided by nine humans for ~1.2k summaries, related to both Java and Python functions. Our findings show that GPT-4-turbo is the best LLM in terms of judging capabilities for both tasks, with "smaller" LLMs featuring tens of billions parameters not being able to cope with judging tasks. However, even the best-performing LLM frequently misjudges the correctness of the code and summary quality.

Figures

Figures reproduced from arXiv: 2507.16587 by the authors.

Figure 1
Figure 1. Code Generation: Confusion matrices for LLM’s judgment for Java (left) and Python (right). The [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Code generation: Results of the mutants injection and [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Code summarization (Java): Scatterplots relating human to LLM [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 37 canonical work pages

  1. [1]

    Replication package,

    “Replication package,” https://github.com/crupig/ LLMs-as-a-judge-for-SE-tse RP, [n.d.]

    work page

  2. [2]

    Few-shot training llms for project- specific code-summarization,

    T. Ahmed and P . Devanbu, “Few-shot training llms for project- specific code-summarization,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’22, 2023

    work page 2023

  3. [3]

    Open AI ChatGPT,

    O. AI, “Open AI ChatGPT,” https://chatgpt.com

    work page

  4. [4]

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization . Ann Arbor, Michigan: Association for Computational Linguistics, Jun. 2005, pp. 65–72. [Online]. Available:...

    work page 2005

  5. [5]

    An empirical study on the usage of transformer models for code completion,

    M. Ciniselli, N. Cooper, L. Pascarella, A. Mastropaolo, E. Aghajani, D. Poshyvanyk, M. D. Penta, and G. Bavota, “An empirical study on the usage of transformer models for code completion,” IEEE Trans. Software Eng., vol. 48, no. 12, pp. 4818–4837, 2022

    work page 2022

  6. [6]

    A coefficient of agreement for nominal scales,

    J. Cohen, “A coefficient of agreement for nominal scales,” Educ Psychol Meas., 1960

    work page 1960

  7. [7]

    Evaluating large language models in class-level code generation,

    X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y. Chen, J. Feng, C. Sha, X. Peng, and Y. Lou, “Evaluating large language models in class-level code generation,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24, 2024

    work page 2024

  8. [8]

    Crystalbleu: Precisely and efficiently measuring the similarity of code,

    A. Eghbali and M. Pradel, “Crystalbleu: Precisely and efficiently measuring the similarity of code,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineer- ing, ser. ASE ’22, 2023

    work page 2023

Show all 51 references
  1. [9]

    Evaluating large language models trained on code,

    C. etal, “Evaluating large language models trained on code,”

  2. [10]

    Code llama: Open foundation models for code,

    R. etal, “Code llama: Open foundation models for code,” 2024. [Online]. Available: https://arxiv.org/abs/2308.12950

    2024 arXiv

  3. [11]

    R. J. Grissom and J. J. Kim, Effect sizes for research: A broad practical approach, 2nd ed. Lawrence Earlbaum Associates, 2005

    2005

  4. [12]

    An ex- tensible, regular-expression-based tool for multi-language mutant generation,

    A. Groce, J. Holmes, D. Marinov, A. Shi, and L. Zhang, “An ex- tensible, regular-expression-based tool for multi-language mutant generation,” in Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, ser. ICSE ’18, 2018, p. 25–28

    2018

  5. [13]

    Deepseek-coder: When the large language model meets programming – the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang, “Deepseek-coder: When the large language model meets programming – the rise of code intelligence,” 2024. [Online]. Available: https://arxiv.org/abs/2401.14196

    2024 arXiv

  6. [14]

    Ex- ploring the potential of chatgpt in automated code refinement: An empirical study,

    Q. Guo, J. Cao, X. Xie, S. Liu, X. Li, B. Chen, and X. Peng, “Ex- ploring the potential of chatgpt in automated code refinement: An empirical study,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–13

    2024

  7. [15]

    Semantic similarity metrics for evaluating source code summarization,

    S. Haque, Z. Eberhart, A. Bansal, and C. McMillan, “Semantic similarity metrics for evaluating source code summarization,” in Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, ser. ICPC ’22, 2022, p. 36–47

    2022

  8. [16]

    Improved auto- matic summarization of subroutines via attention to file context,

    S. Haque, A. LeClair, L. Wu, and C. McMillan, “Improved auto- matic summarization of subroutines via attention to file context,” 2020

    2020

  9. [17]

    Measuring coding challenge competence with apps,

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt, “Measuring coding challenge competence with apps,” NeurIPS, 2021

    2021

  10. [18]

    On the Limitations of Fine-tuned Judge Models for LLM Evaluation,

    H. Huang, Y. Qu, H. Zhou, J. Liu, M. Yang, B. Xu, and T. Zhao, “On the Limitations of Fine-tuned Judge Models for LLM Evaluation,” Jun. 2024, arXiv:2403.02839 [cs]. [Online]. Available: http://arxiv.org/abs/2403.02839

    2024 arXiv

  11. [19]

    Inference endpoints,

    HuggingFace, “Inference endpoints,” https://huggingface.co/ inference-endpoints/

  12. [20]

    Inferfix: End-to-end program repair with llms,

    M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, N. Sundaresan, and A. Svyatkovskiy, “Inferfix: End-to-end program repair with llms,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC...

    2023

  13. [21]

    Simllm: Calculating semantic similarity in code summaries using a large language model-based approach,

    X. Jin and Z. Lin, “Simllm: Calculating semantic similarity in code summaries using a large language model-based approach,” Proceedings of the ACM on Software Engineering , vol. 1, no. FSE, pp. 1376–1399, 2024

    2024

  14. [22]

    Kahneman, Thinking, fast and slow

    D. Kahneman, Thinking, fast and slow . Farrar, Straus and Giroux, 2011

    2011

  15. [23]

    Large language models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  16. [24]

    Evaluating Language Models for Generating and Judging Programming Feedback,

    C. Koutcheme, N. Dainese, A. Hellas, S. Sarsa, J. Leinonen, S. Ashraf, and P . Denny, “Evaluating Language Models for Generating and Judging Programming Feedback,” Jul. 2024, arXiv:2407.04873 [cs]. [Online]. Available: http://arxiv.org/abs/ 2407.04873

    2024 arXiv

  17. [25]

    Reliability in content analysis: Some common misconceptions and recommendations,

    K. Krippendorff, “Reliability in content analysis: Some common misconceptions and recommendations,” Human communication re- search, vol. 30, no. 3, pp. 411–433, 2004

    2004

  18. [26]

    Software vulnerability prediction in low-resource languages: An empirical study of code- bert and chatgpt,

    T. H. M. Le, M. A. Babar, and T. H. Thai, “Software vulnerability prediction in low-resource languages: An empirical study of code- bert and chatgpt,” in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, 2024, pp. 679– 685

    2024

  19. [27]

    Improved code summarization via a graph neural network,

    A. LeClair, S. Haque, L. Wu, and C. McMillan, “Improved code summarization via a graph neural network,” in Proceedings of the 28th International Conference on Program Comprehension , ser. ICPC ’20, 2020, p. 184–195

    2020

  20. [28]

    Automating code review activities by large-scale pre-training,

    Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu, and N. Sundaresan, “Automating code review activities by large-scale pre-training,” in Proceedings of the 30th ACM Joint European Software Engineering Conference JOURNAL OF LATEX ...

    2022

  21. [29]

    Rouge: A package for automatic evaluation of sum- maries,

    C.-Y. Lin, “Rouge: A package for automatic evaluation of sum- maries,” in Text summarization branches out, 2004, pp. 74–81

    2004

  22. [30]

    Evaluating code summarization techniques: A new metric and an empirical characterization,

    A. Mastropaolo, M. Ciniselli, M. D. Penta, and G. Bavota, “Evaluating code summarization techniques: A new metric and an empirical characterization,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20,...

    2024

  23. [31]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguis- tics, 2002, pp. 311–318

    2002

  24. [32]

    Programtransformer: A tool for generating semantically equivalent transformed programs,

    M. R. I. Rabin and M. A. Alipour, “Programtransformer: A tool for generating semantically equivalent transformed programs,” Softw. Impacts, vol. 14, p. 100429, 2022

    2022

  25. [33]

    Reassessing automatic evaluation metrics for code summarization tasks,

    D. Roy, S. Fakhoury, and V . Arnaoudova, “Reassessing automatic evaluation metrics for code summarization tasks,” in 29th ACM Joint Meeting on European Software Engineering Conference and the ACM/SIGSOFT Symposium on the Foundations of Software Engineer- ing, ESEC-FSE, 2021, p...

    2021

  26. [34]

    Is chatgpt the ultimate programming assistant–how far is it?

    H. Tian, W. Lu, T. O. Li, X. Tang, S.-C. Cheung, J. Klein, and T. F. Bissyand´e, “Is chatgpt the ultimate programming assistant–how far is it?” arXiv preprint arXiv:2304.11938, 2023

    2023 arXiv

  27. [35]

    CodeJudge: Evaluating code generation with large language models,

    W. Tong and T. Zhang, “CodeJudge: Evaluating code generation with large language models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, Eds. Association for Computational Linguistics, Nov. 2...

    2024

  28. [36]

    Using pre-trained models to boost code review automation,

    R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshy- vanyk, and G. Bavota, “Using pre-trained models to boost code review automation,” in Proceedings of the 44th International Confer- ence on Software Engineering, ser. ICSE ’22, 2022, p. 2291–2302

    2022

  29. [37]

    A systematic literature review on the use of deep learn- ing in software engineering research,

    C. Watson, N. Cooper, D. N. Palacio, K. Moran, and D. Poshy- vanyk, “A systematic literature review on the use of deep learn- ing in software engineering research,” ACM Trans. Softw. Eng. Methodol., vol. 31, no. 2, 2022

    2022

  30. [38]

    Automatic bug fixing via deliberate problem solving with large language models,

    G. Weng and A. Andrzejak, “Automatic bug fixing via deliberate problem solving with large language models,” in 2023 IEEE 34th International Symposium on Software Reliability Engineering Work- shops (ISSREW), 2023, pp. 34–36

    2023

  31. [39]

    Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences,

    M. Weyssow, A. Kamanda, and H. Sahraoui, “Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences,” 2024

    2024

  32. [40]

    Sorting and transforming program repair ingredients via deep learning code similarities,

    M. White, M. Tufano, M. Mart ´ınez, M. Monperrus, and D. Poshy- vanyk, “Sorting and transforming program repair ingredients via deep learning code similarities,” in 2019 IEEE 26th Interna- tional Conference on Software Analysis, Evolution and Reengineering (SANER), 2019, pp. 479–490

    2019

  33. [41]

    Deep learning code fragments for code clone detection,

    M. White, M. Tufano, C. Vendome, and D. Poshyvanyk, “Deep learning code fragments for code clone detection,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’16. New York, NY, USA: Association for Computing Machinery, 20...

    2016

  34. [42]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,” Bio- metrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945

    1945

  35. [43]

    Yin, “Lizard,” https://github.com/terryyin/lizard

    T. Yin, “Lizard,” https://github.com/terryyin/lizard

  36. [44]

    Controlling the false discovery rate: A practical and powerful approach to multiple testing,

    B. Yoav and H. Yosef, “Controlling the false discovery rate: A practical and powerful approach to multiple testing,” Journal of the Royal Statistical Society. Series B (Methodological) , vol. 57, no. 1, pp. 289–300, 1995

    1995

  37. [45]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–12

    2024

  38. [46]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging llm-as-a-judge with mt-bench and chatbot arena,” in Ad- vances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Glober...

    2023

  39. [47]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P . Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” Dec. 2023, arXiv:2306.05685 [cs]. [Online]. Available: http://arxiv.org/abs/2306.05685

    2023 arXiv

  40. [48]

    Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x,

    Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, Z. Wang, L. Shen, A. Wang, Y. Li et al., “Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x,” arXiv preprint arXiv:2303.17568, 2023

    2023 arXiv

  41. [49]

    ICE-score: Instructing large language models to evaluate code,

    T. Y. Zhuo, “ICE-score: Instructing large language models to evaluate code,” in Findings of the Association for Computational Linguistics: EACL 2024 , Y. Graham and M. Purver, Eds. Association for Computational Linguistics, Mar. 2024, pp. 2232–2242. [Online]. Available: https:...

    2024

  42. [50]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,

    T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul et al. , “Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,” arXiv preprint arXiv:2406.15877, 2024

    2024 arXiv

  43. [2021]

    Available: https://arxiv.org/abs/2107.03374

    [Online]. Available: https://arxiv.org/abs/2107.03374

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.