Pith. sign in

REVIEW 5 major objections 7 minor 1 cited by

Harnessing Large Language Models for Curated Code Reviews

T0 review · 5 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that cleaning a public code-review dataset with an LLM-driven curation pipeline—filtering irrelevant comments and reformulating the rest—improves both automated comment generation and code refinement.

desk verdict Useful dataset and a sensible curation pipeline, but the headline comment-generation gain is confounded by a reference-set mismatch that needs fixing before the RQ3 claim can be trusted. read the letter →

arxiv 2502.03425 v1 pith:OU7GK52I submitted 2025-02-05 cs.SE

Oussama Ben Sghaier , Martin Weyssow , Houari Sahraoui This is my paper · ORCID
classification cs.SE
keywords codereviewdatasetcurationLLM-as-a-judgecommentgenerationrefinementautomationdataquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the quality of training data, not the size or strength of the model, is the main bottleneck in automating code review. It builds a quality-evaluation framework that classifies review comments by type, nature, and civility and scores them on relevance, clarity, and conciseness, then applies this framework to the largest public code-review dataset. Using a large language model as both judge and reformulator, the authors filter out low-relevance comments and rewrite the rest to be clearer, more concise, and fully civil, producing a curated dataset. Fine-tuning a code model on the curated comments raises comment-generation BLEU from 7.71 to 11.26 and improves code refinement, with CodeBLEU rising from 0.36 to 0.44 and exact matches rising from 408 to 445. A reader should care because this suggests a cheap, model-agnostic route to better automated code review: curate the data rather than only scale the model.

What carries the argument

The load-bearing object is the curation pipeline, made up of (1) an evaluation framework that labels each comment by type, nature, and civility and scores it from 1 to 10 on relevance, clarity, and conciseness, and (2) a two-step LLM procedure: first judge, then reformulator. The judge prompt asks the LLM to write an ideal comment for the same code change before scoring the given comment, which gives the scoring an internal reference point, and to justify each score, which steers it away from giving every comment a perfect mark. The reformulator prompt then rewrites comments to be civil, clear, and concise without changing the core message, and comments whose relevance score falls below a threshold of 4 are filtered out. The experimental design pairs each original comment with its reformulated counterpart and uses identical model configurations, so any measured difference in comment generation or code refinement is attributed to the curation step.

What would settle it

Have independent human reviewers or a differently trained LLM judge score a random sample of original and curated comments on the same criteria. If the external judge does not find the curated comments clearer, more concise, and more civil—or if the improvements are small—the central claim loses its support. A second check is to retrain the comment-generation model on curated comments produced by a different reformulation model and see whether the BLEU gain persists.

Watch

Extended reading notes

Core claim

The central claim is that reformulated review comments, produced by an LLM instructed to preserve the original message while improving clarity, conciseness, and civility, are a better learning signal for comment generation and a better instruction for code refinement. The paper supports this with a before-and-after study of the same underlying code reviews. It first characterizes the raw dataset, finding that refactoring and bugfix comments dominate, most comments are prescriptive, 1.23% are uncivil, and average scores are 8.23 for relevance, 6.89 for clarity, and 7.71 for conciseness. After removing comments with relevance below 4 (5,895 samples) and reformulating the rest, average clarity rises to 8.96, average conciseness to 8.05, the prescriptive share rises to 90.20%, and all comments are civil. On paired 20,000-sample subsets, a code model fine-tuned on curated comments reaches BLEU 11.26 versus 7.71 for the original comments, and the same model given curated comments reaches CodeBLEU 0.44 and 445 exact matches versus 0.36 and 408 for original comments. The authors present this as evidence that dataset curation is an effective and largely unexplored lever for improving automated code review.

Load-bearing premise

The load-bearing premise is that the LLM used as judge gives unbiased, valid quality scores for both the original and reformulated comments, even though that same LLM produced the reformulated comments; if the judge simply prefers its own rewording, the reported clarity and conciseness improvements are mostly self-preference.

Editorial extensions

If this is right

  • Cleaning a noisy review dataset with this pipeline is sufficient to improve a code model's ability to generate review comments, with BLEU rising 46% in the paper's paired experiment.
  • The curated comments are more actionable for code refinement: the same model produces code changes closer to the expected diff, with CodeBLEU up 22% and exact matches up from 408 to 445.
  • The evaluation framework gives researchers a reusable instrument for measuring the quality of code-review datasets and for filtering them by relevance thresholds.
  • The curated dataset can serve as a public baseline for future comment-generation and code-refinement research, since it shares the same content as the original dataset but with cleaner wording.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the same LLM wrote the reformulated comments and judged the quality scores, the reported clarity and conciseness gains may partly reflect the judge preferring its own style; an independent human or differently-trained judge study would separate genuine improvement from self-preference.
  • Editorial inference: the curation step steeply shifts the nature distribution toward prescriptive comments (62.6% to 90.2%), so part of the downstream gain may come from training on a narrower, more uniform style rather than from clarity alone.
  • Editorial inference: if the result generalizes, the same filter-and-reformulate recipe could be applied to other noisy developer-generated text, such as issue reports or API documentation, where the same self-preference caveat should be tested.
  • Editorial inference: BLEU is a surface-form metric, so the comment-generation gain of 7.71 to 11.26 may understate or misstate semantic quality; measuring with semantic similarity or human preference would clarify the true improvement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes an evaluation framework for code review comments (type, nature, civility, plus clarity/relevance/conciseness scores) and a curation pipeline that filters irrelevant comments and uses Llama-3.1-70B to reformulate the rest for clarity, conciseness, and civility. The authors apply the framework to the CodeReviewer dataset (176,613 samples), report that the curated dataset improves clarity and conciseness, and show that fine-tuning DeepSeek-Coder on curated data gives higher BLEU (11.26 vs 7.71) for comment generation and higher CodeBLEU (0.44 vs 0.36) and Exact Match (445 vs 408) for code refinement. The dataset and replication package are publicly released.

Significance. The paper addresses a real bottleneck—noisy code review datasets—and the idea of using an LLM both as judge and reformulator is timely for the code review automation community. The main strengths are the clearly specified evaluation framework, the public release of the curated dataset and replication code, and the paired design of the code refinement experiment. If the evaluation confounds were resolved, the curated dataset alone would be a useful resource. However, the current evidence for the claimed improvements is weakened by (i) the BLEU comparison using mismatched reference distributions, (ii) the same model being used as judge and reformulator without validation on curated comments, and (iii) internal inconsistencies in the reported statistics. The contribution is potentially significant but is not yet established by the evidence presented.

major comments (5)
  1. [V-A, Table X] The BLEU comparison for comment generation is confounded by reference-set mismatch. The model trained on original comments is evaluated against human-written ground-truth comments, while the model trained on curated comments is evaluated against Llama-3.1-70B reformulations. Since the curated references are produced by a single LLM, they are stylistically homogeneous and likely easier for a fine-tuned LLM to match, so the 7.71→11.26 gain may reflect reference-style alignment rather than better review quality. Please perform a cross-evaluation: score both models on the same shared reference set (for example, the original human test set and the curated test set) and report BLEU with bootstrap confidence intervals. Without this, the RQ3 claim that curated data leads to more accurate comment generation is not supported.
  2. [III-D, IV-B] The quality improvements in RQ2 are measured by Llama-3.1-70B acting as judge, which is the same model that produced the reformulations in Section IV-A. The judge may systematically prefer its own outputs, and the human sanity check in Section III-D covers only 100 original comments; it does not validate the judge's scores on curated comments. Please run a human evaluation on a random sample of paired original/curated comments (e.g., 100–200) using the same scoring criteria and report human-model agreement for the curated set. In addition, the relevance threshold of 4 used for filtering is arbitrary; a sensitivity analysis (e.g., thresholds 3 and 5) should be reported to show the robustness of the curation pipeline.
  3. [V-B] The code refinement comparison assumes that each reformulated comment preserves the semantics of the original comment, but the paper provides no verification of this. The example in Table XI shows a reformulation that adds an explanation and a concrete suggestion ('Consider removing the import...') that goes beyond the original question ('why this import is required?'). If reformulation changes the information content, the CodeBLEU/EM gains may be due to richer instructions rather than improved form. Please add a semantic-equivalence check (a human-annotated sample or an automatic similarity measurement) and, ideally, restrict the comparison to pairs that are verified to preserve the original intent.
  4. [IV-A, IV-B, Tables V, VII, VIII] There are internal inconsistencies in the reported statistics. (a) The text states that filtering removed 5,895 samples, leaving a curated dataset of size 170,718, but Table VIII reports 176,613 civil comments in the curated dataset, contradicting the filter. (b) In Table V, the overall average clarity (6.89) and conciseness (7.71) are inconsistent with the per-category means (e.g., Refactoring is 80% of the data and has clarity 7.79 and conciseness 6.99). (c) The claimed conciseness increase of 0.34 (7.71→8.05) conflicts with per-category increases of roughly 1.1–2.4. Please recompute all aggregates and clarify whether they are weighted; as presented, these errors undermine the quantitative claims of RQ2.
  5. [V] None of the downstream comparisons (Tables X and XII) include statistical significance tests, confidence intervals, or multiple runs. The Exact Match increase is only 408→445 out of 5,000 test samples (8.16%→8.9%), which is small and could be due to chance. Please report bootstrap confidence intervals and a paired significance test (e.g., paired bootstrap or Wilcoxon signed-rank) for BLEU, CodeBLEU, and Exact Match.
minor comments (7)
  1. [Section III-E and Figure 3] The text reports Refactoring 80.07% and Bugfix 18.60%, while Figure 3 shows 69.7% and 16.2%, and the abstract's 85.9% matches the figure values (69.7+16.2=85.9). Please reconcile these numbers, as they are directly contradictory.
  2. [Section III-F] The sentence 'The low clarity score (average= 4.89)' conflicts with Table V's average clarity of 6.89; please correct the typo.
  3. [Section V-A and Table X] The model is called 'DeepSeek-6.7B-Instruct' in the text and 'DeepSeek-Coder-6.7B-Instruct' in Table X; please use consistent naming throughout.
  4. [Table VII] In the 'Other' Nature row, the clarity increase is listed as ↑4.76, but the original value in Table V is 4.40, so the correct increase is 4.60; please check all delta calculations for consistency.
  5. [Section III-E] The word 'prescreptive' should be 'prescriptive'.
  6. [Section V-A] The test set size is 25% of 20,000 = 5,000; consider reporting Exact Match also as a percentage (408/5000 = 8.2%) to aid interpretability, given that the raw counts are small relative to the test set.
  7. [Abstract and Answer to RQ2] The conciseness improvement is only 0.34 points on a 1–10 scale; the abstract's phrase 'substantial improvements in clarity and conciseness' overstates the conciseness result and should be qualified accordingly.

Circularity Check

2 steps flagged · score 7.0 of 10

RQ3 BLEU gain is confounded by reference-set mismatch: each model is scored against its own training target, so the curated model may simply match Llama-style text; RQ2 quality gains are judged by the same Llama model that wrote the reformulations.

  1. fitted input called prediction [Section V-A (RQ3), 'Model and data selection' and Table X; Section IV-A 'Review comments reformulation']
    "we selected a subset of 20, 000 comments from both the original and curated datasets, such that each original review comment ri from the original dataset is paired with its reformulated counterpart r′ i in the curated dataset. ... For each dataset version (original and curated), we provided the model with code changes as input and tasked it with generating the corresponding review comment."

    The two comment-generation models are trained and then evaluated against different reference distributions. The original model is learned from and scored against human-written comments; the curated model is learned from and scored against Llama-3.1-70B reformulations of those same comments. Since BLEU measures n-gram overlap with the reference, a model fine-tuned on Llama-style text will naturally achieve higher overlap with Llama-style references than with human-written ones, without evidence that the generated comments are more accurate. The paired input split removes only the input difference, not the target difference: the references used for scoring differ across the two conditions.

  2. other [Section III-C LLM-as-a-Judge, Section III-D Sanity Check, Section IV-A reformulation, Table VII (RQ2)]
    "we utilized an LLM, Llama-3.1-70B, providing specific instructions to guide the reformulation process. ... Finally, we re-evaluated the curated review comments using the same evaluation schema as was applied to the original dataset. ... Initially, the LLM is instructed to generate what it considers an ideal review comment based on the provided code changes."

    The same Llama-3.1-70B model that rewrites the comments is also the judge that assigns the clarity and conciseness scores. The evaluation prompt first makes the judge generate its own 'ideal' comment and then grade the given comment against that self-generated reference. Consequently, the post-curation increases in clarity (6.89 to 8.96) and conciseness (7.71 to 8.05) measure how close the reformulations are to Llama's own stylistic ideal, not an independently validated quality improvement. The human sanity check covered only 100 original comments and was performed before curation, so it does not validate the judge on the reformulated comments. This makes the RQ2 quality claim self-referential and also feeds the RQ3 ground truth, which consists of the same Llama rewrites.

full rationale

The central derivation chain has two circular or confounded links. First, RQ3's claim that curated data improves comment generation is supported by a comparison in which each model is trained and evaluated against its own target distribution: the original model against human comments, and the curated model against Llama-3.1-70B rewrites. Because BLEU is a surface n-gram overlap metric and the fine-tuned model is itself an LLM, the higher BLEU for the curated condition (11.26 vs 7.71) is partly an artifact of learning and then matching Llama's reformulation style; the paired input split does not control for the differing references. Second, RQ2's quality improvements are measured by Llama-3.1-70B, the same model that produced the reformulations, using a prompt that generates an 'ideal' comment as the judging reference; the human sanity check did not cover the reformulated comments, so the clarity and conciseness gains are not independently established. The code-refinement comparison (RQ4) is less confounded because the expected code diff is shared across the original and curated conditions, so its CodeBLEU and Exact Match gains retain some independent content. No load-bearing self-citation circularity is present: the authors' prior work appears in related-work and technical references, not as the justification for the empirical claims. Overall, the paper's headline predictive claims are partially circular but not entirely forced, warranting a score of 7.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper's central claims rest on the reliability of LLM-based evaluation and reformulation, plus a hand-chosen filtering threshold. No new physical or conceptual entities are introduced.

free parameters (1)
  • Relevance threshold = 4
    Comments with LLM-judged relevance below 4 are filtered out, affecting the size and composition of the curated dataset. The threshold is chosen by the authors without a sensitivity analysis, making it a hand-set free parameter.
assumptions (5)
  • domain assumption Llama-3.1-70B provides valid and unbiased quality scores for review comments.
    Invoked in Section III-C, the paper assumes LLM-as-a-judge reliability based on prior literature and a 100-sample human sanity check, but this is assumed for the full dataset and for evaluating the model's own reformulations.
  • domain assumption Reformulating comments to improve clarity, conciseness, and civility does not alter their core meaning or intent.
    Section IV-A uses this assumption to justify treating the curated comments as valid representations of the original reviews, without human verification of semantic preservation.
  • domain assumption Relevance scores do not change with reformulation.
    Section IV-A states 'Relevance was omitted from this re-evaluation, as it does not change with reformulation', but this invariance is not empirically tested.
  • domain assumption BLEU and CodeBLEU are appropriate metrics for evaluating comment generation and code refinement quality.
    Sections V-A and V-B use these metrics without validating their correlation with human judgments in this specific context.
  • domain assumption A shift toward prescriptive comments is an improvement.
    Section IV-B interprets the increase in prescriptive comments (62.6% to 90.2%) as a positive outcome, but this reflects a value judgment about code review style rather than an established fact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Harnessing Large Language Models for Curated Code Reviews." pith.science (2026). https://pith.science/paper/OU7GK52I

@misc{pith2026250203425,
  author       = {Pith},
  title        = {Pith review of: Harnessing Large Language Models for Curated Code Reviews},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OU7GK52I}},
  note         = {Machine review of arXiv:2502.03425}
}
read the original abstract

In code review, generating structured and relevant comments is crucial for identifying code issues and facilitating accurate code changes that ensure an efficient code review process. Well-crafted comments not only streamline the code review itself but are also essential for subsequent tasks like code refinement, where the code is modified to satisfy the input review comment. Although various AI-based approaches aimed to automate comment generation, their effectiveness remains limited by the quality of the training data. Existing code review datasets are often noisy and unrefined, posing limitations to the learning potential of AI models and hindering the automation process. To address these challenges, we propose a curation pipeline designed to enhance the quality of the largest publicly available code review dataset. We begin by establishing an evaluation framework, incorporating specific criteria and categories to empirically study the initial quality of the dataset. Using a large language model (LLM)-driven approach, we then apply our curation pipeline to refine the dataset. A comparative analysis of the newly curated dataset, based on the same evaluation framework, demonstrates substantial improvements in the clarity and conciseness of the comments. Additionally, we assess the impact of the curated dataset on automating downstream tasks, specifically comment generation and code refinement. Our findings show that the curated dataset leads to enhanced model performance in generating more accurate comments. Curated comments are also more useful as they lead to more accurate code refinement.

Figures

Figures reproduced from arXiv: 2502.03425 by the authors.

Figure 1
Figure 1. Overview of our methodology. We use a large code review dataset of samples comprising pre-commit and post-commit [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of our evaluation framework. TABLE I: Dataset distribution over programming languages. Programming Language # Samples PHP 9, 984 Ruby 6, 713 C# 17, 085 C 4, 108 Java 35, 671 Python 36, 382 C++ 15, 944 Go 36, 123 JS 14, 603 Total 176, 613 in existing datasets. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Distribution of the different categories across the original dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of scoring criteria on the original dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Distribution of the clarity and conciseness scoring [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks

    cs.SE 2025-06 conditional novelty 6.0 of 10

    Across ten software engineering tasks and four LLMs, no prompting technique wins consistently; ES-KNN is best on many tasks, some techniques underperform the baseline, and USC is best for code QA and code generation.

Reference graph

Works this paper leans on

42 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    The impact of code review coverage and code review participation on software quality: A case study of the Qt, VTK, and ITK projects,

    S. McIntosh, Y . Kamei, B. Adams, and A. E. Hassan, “The impact of code review coverage and code review participation on software quality: A case study of the Qt, VTK, and ITK projects,” in 11th working conference on mining software repositories , 2014, pp. 192–201

    work page 2014

  2. [2]

    An empirical study of the impact of modern code review practices on software quality,

    ——, “An empirical study of the impact of modern code review practices on software quality,” Empirical Software Engineering, vol. 21, no. 5, pp. 2146–2189, 2016

    work page 2016

  3. [3]

    Software inspec- tions: an effective verification process,

    A. F. Ackerman, L. S. Buchwald, and F. H. Lewski, “Software inspec- tions: an effective verification process,” IEEE software, vol. 6, no. 3, pp. 31–36, 1989

    work page 1989

  4. [4]

    Software inspections and the industrial production of software,

    A. F. Ackerman, P. J. Fowler, and R. G. Ebenau, “Software inspections and the industrial production of software,” in Proc. of a symposium on Software validation: inspection-testing-verification-alternatives , 1984, pp. 13–40

    work page 1984

  5. [5]

    Do code review practices impact design quality? a case study of the Qt, VTK, and ITK projects,

    R. Morales, S. McIntosh, and F. Khomh, “Do code review practices impact design quality? a case study of the Qt, VTK, and ITK projects,” in 2015 IEEE 22nd international conference on software analysis, evolution, and reengineering (SANER) . IEEE, 2015, pp. 171–180

    work page 2015

  6. [6]

    Design and code inspections to reduce errors in program development,

    M. Fagan, “Design and code inspections to reduce errors in program development,” in Software pioneers. Springer, 2002, pp. 575–607

    work page 2002

  7. [7]

    Four eyes are better than two: On the impact of code reviews on software quality,

    G. Bavota and B. Russo, “Four eyes are better than two: On the impact of code reviews on software quality,” in 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2015, pp. 81–90

    2015

  8. [8]

    Expectations, outcomes, and challenges of modern code review,

    A. Bacchelli and C. Bird, “Expectations, outcomes, and challenges of modern code review,” in2013 35th International Conference on Software Engineering (ICSE), 2013, pp. 712–721

    work page 2013

Show all 42 references
  1. [9]

    Towards automating code review activities,

    R. Tufano, L. Pascarella, M. Tufanoy, D. Poshyvanykz, and G. Bavota, “Towards automating code review activities,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) , 2021, pp. 163–174

    2021

  2. [10]

    Using pre-trained models to boost code review automa- tion,

    R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk, and G. Bavota, “Using pre-trained models to boost code review automa- tion,” arXiv preprint arXiv:2201.06850 , 2022

    2022 arXiv

  3. [11]

    Automating code review activities by large-scale pre-training,

    Z. Li, S. Lu, and D. e. a. Guo, “Automating code review activities by large-scale pre-training,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2022, pp. 1035–1047

    2022

  4. [12]

    Auger: automatically generating review comments with pre-training models,

    L. Li, L. Yang, H. Jiang, J. Yan, T. Luo, Z. Hua, G. Liang, and C. Zuo, “Auger: automatically generating review comments with pre-training models,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engine...

    2022

  5. [13]

    Improving the learning of code review successive tasks with cross-task knowledge distillation,

    O. Ben Sghaier and H. Sahraoui, “Improving the learning of code review successive tasks with cross-task knowledge distillation,” Proceedings of the ACM on Software Engineering , vol. 1, no. FSE, pp. 1086–1106, 2024

    2024

  6. [14]

    A multi-step learning approach to assist code review,

    O. B. Sghaier and H. Sahraoui, “A multi-step learning approach to assist code review,” in 2023 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) . IEEE, 2023

    2023

  7. [15]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  8. [16]

    Deepseek-coder: When the large language model meets programming – the rise of code intelligence,

    D. Guo, Q. Zhu, and D. e. a. Yang, “Deepseek-coder: When the large language model meets programming – the rise of code intelligence,”

  9. [17]

    Unity is strength: Cross- task knowledge distillation to improve code review generation,

    O. B. Sghaier, L. Maes, and H. Sahraoui, “Unity is strength: Cross- task knowledge distillation to improve code review generation,” arXiv preprint arXiv:2309.03362, 2023

    2023 arXiv

  10. [18]

    Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,

    J. Lu, L. Yu, X. Li, L. Yang, and C. Zuo, “Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,” in 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 2023, pp. 647–658

    2023

  11. [19]

    Code review automation: strengths and weaknesses of the state of the art,

    R. Tufano, O. Dabi ´c, A. Mastropaolo, M. Ciniselli, and G. Bavota, “Code review automation: strengths and weaknesses of the state of the art,” IEEE Transactions on Software Engineering , 2024

    2024

  12. [20]

    Do words have power? understanding and fostering civility in code review discussion,

    M. S. Rahman, Z. Codabux, and C. K. Roy, “Do words have power? understanding and fostering civility in code review discussion,” Pro- ceedings of the ACM on Software Engineering , vol. 1, no. FSE, pp. 1632–1655, 2024

    2024

  13. [21]

    A decade of code comment quality assessment: A systematic literature review,

    P. Rani, A. Blasi, N. Stulova, S. Panichella, A. Gorla, and O. Nierstrasz, “A decade of code comment quality assessment: A systematic literature review,” Journal of Systems and Software , vol. 195, p. 111515, 2023

    2023

  14. [22]

    How good is your comment? a study of comments in java programs,

    D. Haouari, H. Sahraoui, and P. Langlais, “How good is your comment? a study of comments in java programs,” in2011 International symposium on empirical software engineering and measurement . IEEE, 2011, pp. 137–146

    2011

  15. [23]

    Ice-score: Instructing large language models to evaluate code,

    T. Y . Zhuo, “Ice-score: Instructing large language models to evaluate code,” arXiv preprint arXiv:2304.14317 , 2023

    2023 arXiv

  16. [24]

    A survey on evaluation of large language models,

    Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang et al. , “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology , vol. 15, no. 3, pp. 1–45, 2024

    2024

  17. [25]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  18. [26]

    Alpacaeval: An automatic evaluator of instruction-following models,

    X. Li, T. Zhang, Y . Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto, “Alpacaeval: An automatic evaluator of instruction-following models,” 2023

    2023

  19. [27]

    Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences,

    M. Weyssow, A. Kamanda, and H. Sahraoui, “Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences,” arXiv preprint arXiv:2403.09032 , 2024

    2024 arXiv

  20. [28]

    Interrater reliability: the kappa statistic,

    M. L. McHugh, “Interrater reliability: the kappa statistic,” Biochemia medica, vol. 22, no. 3, pp. 276–282, 2012

    2012

  21. [29]

    Exploratory study on the impact of english bias of generative large language models in dutch and french,

    A. R. Terryn and M. de Lhoneux, “Exploratory study on the impact of english bias of generative large language models in dutch and french,” in Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval)@ LREC-COLING 2024 , 2024, pp. 12–27

    2024

  22. [30]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

    2021 arXiv

  23. [31]

    Exploring parameter-efficient fine-tuning techniques for code generation with large language models,

    M. Weyssow, X. Zhou, K. Kim, D. Lo, and H. Sahraoui, “Exploring parameter-efficient fine-tuning techniques for code generation with large language models,” arXiv preprint arXiv:2308.10462 , 2023

    2023 arXiv

  24. [32]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology , 2023

    2023

  25. [33]

    Repairllama: Efficient repre- sentations and fine-tuned adapters for program repair,

    A. Silva, S. Fang, and M. Monperrus, “Repairllama: Efficient repre- sentations and fine-tuned adapters for program repair,” arXiv preprint arXiv:2312.15698, 2023

    2023 arXiv

  26. [34]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  27. [35]

    Codebleu: a method for automatic evaluation of code synthesis,

    S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” arXiv preprint arXiv:2009.10297 , 2020

    2009 arXiv

  28. [36]

    Neural machine translation of rare words with subword units,

    R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  29. [37]

    Com- mentfinder: a simpler, faster, more accurate code review comments recommendation,

    Y . Hong, C. Tantithamthavorn, P. Thongtanunam, and A. Aleti, “Com- mentfinder: a simpler, faster, more accurate code review comments recommendation,” in Proceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software eng...

    2022

  30. [38]

    Intelligent code reviews using deep learning,

    A. Gupta and N. Sundaresan, “Intelligent code reviews using deep learning,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’18) Deep Learning Day, 2018

    2018

  31. [39]

    Improving the learning of code review successive tasks with cross-task knowledge distillation,

    O. B. Sghaier and H. Sahraoui, “Improving the learning of code review successive tasks with cross-task knowledge distillation,” arXiv preprint arXiv:2402.02063, 2024

    2024 arXiv

  32. [40]

    Replication package,

    “Replication package,” https://github.com/OussamaSghaier/CuREV

  33. [41]

    Data and models,

    “Data and models,” https://zenodo.org/records/14812107

  34. [2024]

    Available: https://arxiv.org/abs/2401.14196

    [Online]. Available: https://arxiv.org/abs/2401.14196

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.