Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword?

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM-generated code reviews anchor reviewers to flagged lines without saving time or boosting confidence.

desk verdict A genuinely new controlled experiment on LLM-assisted code review, where the anchoring result is solid and the 'no time saved / no confidence gain' claims outrun the statistical power. read the letter →

arxiv 2411.11401 v3 pith:4ACNLO3T submitted 2024-11-18 cs.SE

Rosalia Tufano , Alberto Martin-Lopez , Ahmad Tayeb , Ozren Dabić , Sonia Haiduc , Gabriele Bavota This is my paper · ORCID
classification cs.SE
keywords codereviewlargelanguagemodelscontrolledexperimentreviewerbiasanchoringqualitytimeconfidence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

An automated code review generated by an LLM, when handed to a professional developer as a starting point, does not work as an accelerator or accuracy booster: it changes where the reviewer looks. In a controlled experiment with 29 professional developers and 72 reviews, reviewers kept 89% of the LLM-flagged issues on average, but concentrated on those flagged code locations instead of searching the rest of the code. Reviews produced with the LLM starting point contained more low-severity issues than fully manual reviews, yet did not uncover more high-severity (injected) issues. The automated support also did not reduce review time, even when the provided review was perfect, and did not increase reviewers' confidence. The paper concludes that current LLM co-reviewers are best used as a complement after the human review, not as the starting point.

What carries the argument

Three treatments form the experiment's core mechanism: MCR (no support), ACR (a ChatGPT Plus review as the starting point), and CCR (a knowingly perfect review, rephrased by ChatGPT, so issues could not be dismissed as LLM errors). Into 12 small Java and Python programs the authors injected 48 quality issues; this lets them measure hit rates on known high-severity defects and separately count additional issues found. IDE telemetry records time on reading, writing, and code inspection, and every submitted review is manually inspected by two coders to count issues, locations, and kept comments. The CCR condition is what carries the time-and-confidence argument: because even a perfect automated review did not save time or raise confidence, the cost is attributed to the cognitive work of interpreting and checking generated comments, not to the LLM's errors.

What would settle it

A pre-registered replication with a larger sample (e.g., at least 100 reviews per treatment) that observes ACR reviewers finding significantly more injected high-severity issues than MCR reviewers, or completing reviews in significantly less time, would undercut the paper's null conclusions. A cheaper check: if reviewers in the ACR condition, when their gaze is tracked, spend a substantial share of time outside the LLM-flagged lines and find new issues there, the anchoring mechanism would not explain the results.

Watch

Extended reading notes

Core claim

The paper's central claim is that an automatically generated review used as a starting point changes the reviewer's behavior without delivering the benefits claimed for automation. Reviewers accepted most LLM-generated comments (89% on average) and their final reviews reported more issues overall than manual reviews, but the extra issues were predominantly low severity; the odds of spotting the injected high-severity issues were statistically the same for the ChatGPT-assisted and fully manual treatments. Even when the provided review was perfect (the CCR treatment), reviewers did not finish faster than manual reviewers, because they spent time reading and verifying the provided comments. Confidence scores were likewise indistinguishable across treatments. The paper frames this as evidence that LLM co-reviewers, as currently used, should be positioned as a final complementary check rather than as the starting point of a review.

Load-bearing premise

The load-bearing assumption is that the study's sample (24 reviews per treatment) was large enough that the non-significant differences in high-severity issue detection, time, and confidence really mean 'no effect'; the authors themselves note that the study could be statistically underpowered and thus lead to biased conclusions.

Editorial extensions

If this is right

  • Adopting an LLM review as the first step in code review will likely raise the number of low-severity issues reported without increasing detection of high-severity defects.
  • Review time savings should not be assumed; teams may need to budget time for reading and checking generated comments, which the experiment found offset any speed gain.
  • Reviewer confidence is not a useful internal signal for whether automated support is helping, since confidence did not move across treatments.
  • Because reviewers anchor on flagged locations, showing the automated review after the human draft (or using it only as a final checklist) is a concrete way to preserve independent inspection.
  • Tool designers should target high-severity issue types and keep generated reviews concise, since verbosity adds contributor cost without expanding code coverage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The anchoring effect is likely not specific to code review; any AI tool that outputs a list of candidate locations before a human inspects a large space could narrow attention in the same way, and that generalization is worth testing in other domains.
  • A direct testable extension would compare 'before' versus 'after' presentation of the same LLM review, as well as a condition where the LLM review is hidden until after the reviewer submits, to quantify how much of the lost coverage comes from order alone.
  • The 89% acceptance rate and the severity gap between injected and LLM-found issues suggest that benchmark metrics based only on issue counts may overstate the practical value of LLM reviews; reporting severity distributions of findings is a cheap improvement.
  • The null time result, if it holds in larger samples, would shift the business case for LLM co-reviewers from productivity to thoroughness or training, which are not measured here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports a controlled experiment with 29 professional (or formerly professional) developers who performed 72 code reviews across six Java/Python projects under three treatments: fully manual review (MCR), review with a ChatGPT-generated starting review (ACR), and review with a 'comprehensive' starting review containing all injected issues (CCR). The authors measure review quality, review time, and reviewer confidence, and report three classes of findings: (i) reviewers kept, on average, 89% of the issues in the automated review; (ii) providing an automated review anchors reviewers to the code locations highlighted by the tool, leading to more low-severity issues but not more injected high-severity issues; and (iii) automated support did not save time and did not increase confidence. The paper concludes with recommendations for reviewers, tool designers, and researchers.

Significance. If the findings hold, the paper makes a timely and important empirical contribution: it is among the first controlled studies of LLM-generated reviews as starting points in human code review, and its observational evidence about anchoring and issue acceptance is valuable for both practitioners and tool designers. The study has notable strengths: 29 industrial participants, over 50 hours of IDE-monitored review activity, double-coding of the 72 submitted reviews, three treatments including a best-case 'comprehensive' reference condition, and a public replication package. The descriptive results, especially the 89% acceptance rate and the line-coverage Venn analysis, support the behavioral claim that reviewers tend to stay close to the LLM-provided comments. However, the headline negative claims about time savings and confidence are inferred from non-significant regression results in a small sample, which the authors themselves acknowledge may be underpowered (Section IV). The paper's contribution would be strengthened substantially by calibrating those claims to what the design can actually support.

major comments (3)
  1. [III-C, III-D, IV (and abstract/conclusion)] The headline negative claims rest on non-significant results in a small sample. For RQ2, the time models are only reported in the replication package; the text states that none of the independent variables is statistically significant and then concludes that automation 'did not result in saved time' and 'debunks one of the motivations for automated code review.' For RQ3, Table V gives ACR=0.099 (SE=0.211) and CCR=0.180 (SE=0.212); the statement 'does not have a significant effect' is statistically accurate, but the abstract's 'did not increase the reviewers' confidence' goes beyond it. The same issue appears in RQ1: the ACR coefficient in Table IV is -0.004 (SE=0.351), so the confidence interval is compatible with both meaningful benefit and meaningful harm for high-severity issue identification. The authors themselves write in Section IV that 'our study could be statistically underpowered, thus leading to biased conclusions.' With 24 reviews per treatment, non-significance is not evidence of absence. I ask the authors to re-express these results with effect sizes and confidence intervals (or equivalence tests) and to soften the abstract and conclusions to forms such as 'we did not detect a significant time saving' rather than 'the automated support did not result in saved time.'
  2. [III-B (severity analysis)] The claim that ACR identifies more low-severity issues but not more high-severity issues depends on the severity ratings of two external developers. The reported weighted kappa is 0.315, which is conventionally only 'fair' agreement, making the conclusion fragile. In addition, the severity comparison is made across different pools of issues (injected issues vs. additional ChatGPT issues vs. additional manual issues), which is not the same as a direct comparison of severity as a function of treatment. The logistic model with severity as a cofactor is said to leave results unchanged, but the model output is not shown in the paper. I recommend reporting the full model, detailed rater-agreement information, and a sensitivity analysis that treats the two raters' severity scores separately.
  3. [III-A (RQ0, Figs. 3 and 4)] The behavioral anchoring claim is central to the paper, but it is supported only by a descriptive Venn diagram and a single example. The statement that reviews written without automated support 'had a higher variability' is not backed by any statistical comparison of line-coverage overlap or dispersion across treatments. I suggest adding an explicit metric (for example, per-review overlap or Jaccard similarity, variance of covered locations, or a mixed-effects model on coverage) with confidence intervals, or explicitly labeling this result as a descriptive observation rather than a strong behavioral conclusion.
minor comments (5)
  1. [IV (Threats to validity)] The text reports removed data points as '4 for NCR, 5 for ACR, and 4 for CCR'; 'NCR' appears to be a typo for 'MCR' and should be corrected.
  2. [Abstract and II-A] The abstract says '29 experts,' while Section II-A describes 29 participants, including one who is not currently a professional developer; please use consistent terminology.
  3. [III-B] The text uses 'weighted k agreement' and cites Cohen [41]; please use the standard symbol kappa and, if conventional benchmarks are used, describe 0.315 as 'fair' agreement.
  4. [References] Reference [26] is given as 'M. V. et al.' with an incomplete author list; please provide the full author names.
  5. [III-A, Fig. 3] The numbers around the Venn diagram are difficult to parse in the printed figure; a legend or a small table of overlap values would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study's claims are evaluated against externally measured human reviewer behavior, not derived from fitted parameters or self-citations.

full rationale

This is a controlled empirical experiment, not a derivation chain with inputs and outputs that could coincide. The automated reviews generated by ChatGPT (ACR) and the author-crafted comprehensive reviews (CCR) are experimental stimuli, not parameters fitted to the measured outcomes; the key dependent variables—number of injected issues identified, review time, and confidence—are collected from 29 professional developers and compared across treatments using regression models whose reference level is manual review (MCR). The CCR treatment is admittedly constructed by the authors to contain all injected issues, but the paper explicitly presents it as a hypothetical upper-bound scenario rather than as a prediction, and its use does not make the observed anchoring behavior or time costs definitional: reviewers could still reject, rephrase, or fail to verify the provided comments, and the measured outcomes are behavioral facts about humans, not consequences of the construction. The authors' own caveat that the study 'could be statistically underpowered, thus leading to biased conclusions' concerns the strength of the null findings, which is a correctness/statistical-power concern, not circularity, and per the reviewing rules absence of evidence treated as evidence of absence is not a form of circular derivation. Self-citations appear in the related work and motivation (e.g., prior code-review automation work by the same group), but no load-bearing argument reduces to those citations: the paper does not invoke its own prior work to justify the treatments or the conclusions. The central claims are therefore self-contained against an external benchmark—human reviewer behavior—and no circular step can be exhibited from the paper's text.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on domain assumptions about representativeness of injected issues, statistical power for null conclusions, the validity of the deceptive CCR treatment, and post-hoc data exclusions. No mathematical or fitted parameters are introduced.

assumptions (4)
  • domain assumption The 48 injected issues are representative of issues found in real code reviews, and their identification is a valid proxy for review quality.
    The authors based issues on the taxonomy by Mäntylä and Lassenius, but acknowledge in Section IV that triviality and representativeness are threats (e.g., participants identified all injected issues in only 12 of 48 tasks).
  • domain assumption Non-significant regression coefficients can be read as evidence that automated support does not save time or increase high-severity issue detection.
    RQ2 and RQ3 rely on absence of significant effects; Section IV states the study may be underpowered, so absence of significance is weak evidence of no effect.
  • domain assumption Participants in the CCR treatment behave as they would with a genuinely perfect automated review despite knowing it is automated when it is actually hand-crafted.
    Section II-B describes the deception; if participants detected the true origin or treated the ideal review differently, the CCR-based time and confidence conclusions are undermined.
  • domain assumption The 13 excluded time measurements are genuine logging errors, and their exclusion is balanced and innocuous.
    Section IV Construct validity; exclusion is post hoc and relies on visual comparison with other timings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword?." pith.science (2026). https://pith.science/paper/4ACNLO3T

@misc{pith2026241111401,
  author       = {Pith},
  title        = {Pith review of: Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ACNLO3T}},
  note         = {Machine review of arXiv:2411.11401}
}
read the original abstract

Several techniques have been proposed to automate code review. Early support consisted in recommending the most suited reviewer for a given change or in prioritizing the review tasks. With the advent of deep learning in software engineering, the level of automation has been pushed to new heights, with approaches able to provide feedback on source code in natural language as a human reviewer would do. Also, recent work documented open source projects adopting Large Language Models (LLMs) as co-reviewers. Although the research in this field is very active, little is known about the actual impact of including automatically generated code reviews in the code review process. While there are many aspects worth investigating, in this work we focus on three of them: (i) review quality, i.e., the reviewer's ability to identify issues in the code; (ii) review cost, i.e., the time spent reviewing the code; and (iii) reviewer's confidence, i.e., how confident is the reviewer about the provided feedback. We run a controlled experiment with 29 experts who reviewed different programs with/without the support of an automatically generated code review. During the experiment we monitored the reviewers' activities, for over 50 hours of recorded code reviews. We show that reviewers consider valid most of the issues automatically identified by the LLM and that the availability of an automated review as a starting point strongly influences their behavior: Reviewers tend to focus on the code locations indicated by the LLM rather than searching for additional issues in other parts of the code. The reviewers who started from an automated review identified a higher number of low-severity issues while, however, not identifying more high-severity issues as compared to a completely manual process. Finally, the automated support did not result in saved time and did not increase the reviewers' confidence.

Figures

Figures reproduced from arXiv: 2411.11401 by the authors.

Figure 1
Figure 1. Examples of injected issues in Java and Python. The top part of each [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. RQ0: Characteristics of the final code reviews under the three treatments. TABLE III RQ0: MULTIVARIATE LINEAR REGRESSION MODELS (ESTIMATE, STD. ERROR, SIGNIFICANCE). N. of reported quality iss. Length of the code review Covered code locations Estim. S.E. Sig. Estim. S.E. Sig. Estim. S.E. Sig. Intercept 2.460 3.962 4.122 8.439 -17.611 35.131 ACR 4.269 1.331 ** 11.862 2.836 *** 5.297 11.806 CCR 1.770 1.338 2.549 2.850… view at source ↗
Figure 3
Figure 3. RQ0: Number of distinct lines covered (i.e., commented on by participants) in the final reviews of the three treatments and their overlap. 1 10 20 30 40 50 60 70 80 90 100 110 116 ACR reviews NCR reviews Code lines Legend: Review 1 Review 2 Review 3 Issue in unique code location ACR reviews MCR reviews [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Example of distinct code lines covered by different reviews. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: RQ2: Time spent in reviews across different treatments. This might be due to the fact that the introduction of an automated review, even if correct (CCR), comes with a price, namely the reading, understanding, and double-checking of the provided comments. This seems to…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking Code Review Workflows with LLM Assistance: An Empirical Study

    cs.SE 2025-05 conditional novelty 5.0 of 10

    In a real-world experiment, developers preferred an AI-led code review mode over an interactive assistant, especially for large or unfamiliar pull requests.

Reference graph

Works this paper leans on

67 extracted references · 57 canonical work pages · cited by 1 Pith paper

  1. [1]

    Modern code review: A case study at google,

    C. Sadowski, E. S ¨oderberg, L. Church, M. Sipko, and A. Bacchelli, “Modern code review: A case study at google,” in 40th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP, 2018, pp. 181–190

    work page 2018

  2. [2]

    Expectations, outcomes, and challenges of modern code review,

    A. Bacchelli and C. Bird, “Expectations, outcomes, and challenges of modern code review,” in 35th IEEE/ACM International Conference on Software Engineering, ICSE , 2013, pp. 712–721

    work page 2013

  3. [3]

    Do code review practices impact design quality? a case study of the qt, vtk, and itk projects,

    R. Morales, S. McIntosh, and F. Khomh, “Do code review practices impact design quality? a case study of the qt, vtk, and itk projects,” in 22nd IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER , 2015, pp. 171–180

    work page 2015

  4. [4]

    Four eyes are better than two: On the impact of code reviews on software quality,

    G. Bavota and B. Russo, “Four eyes are better than two: On the impact of code reviews on software quality,” in IEEE International Conference on Software Maintenance and Evolution, ICSME , 2015, pp. 81–90

    work page 2015

  5. [5]

    The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects,

    S. McIntosh, Y . Kamei, B. Adams, and A. E. Hassan, “The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects,” in 11th IEEE/ACM Working Conference on Mining Software Repositories, MSR , 2014, pp. 192–201

    work page 2014

  6. [6]

    Impact of peer code review on peer impres- sion formation: A survey,

    A. Bosu and J. C. Carver, “Impact of peer code review on peer impres- sion formation: A survey,” in 7th IEEE/ACM International Symposium on Empirical Software Engineering and Measurement, ESEM , 2013, pp. 133–142

    work page 2013

  7. [7]

    Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommenda- tion,

    V . Balachandran, “Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommenda- tion,” in 2013 35th International Conference on Software Engineering (ICSE). IEEE, 2013, pp. 931–940

    2013

  8. [8]

    Who should review my code? a file location- based code-reviewer recommendation approach for modern code re- view,

    P. Thongtanunam, C. Tantithamthavorn, R. G. Kula, N. Yoshida, H. Iida, and K.-i. Matsumoto, “Who should review my code? a file location- based code-reviewer recommendation approach for modern code re- view,” in 2015 IEEE 22nd International Conference on Software Analy- sis, Evolution, and Reengineering (SANER) . IEEE, 2015, pp. 141–150

    work page 2015

Show all 67 references
  1. [9]

    Who should review this change?: Putting text and file location analyses together for more accurate recommendations,

    X. Xia, D. Lo, X. Wang, and X. Yang, “Who should review this change?: Putting text and file location analyses together for more accurate recommendations,” in 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME) , 2015, pp. 261–270

    2015

  2. [10]

    Search-based peer reviewers recommendation in modern code review,

    A. Ouni, R. G. Kula, and K. Inoue, “Search-based peer reviewers recommendation in modern code review,” in 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2016, pp. 367–377

    2016

  3. [11]

    Predicting defectiveness of software patches,

    B. Soltanifar, A. Erdem, and A. Bener, “Predicting defectiveness of software patches,” in Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2016, pp. 1–10

    2016

  4. [12]

    Using stack overflow content to assist in code review,

    S. Sharma and B. Sodhi, “Using stack overflow content to assist in code review,” Software: Practice and Experience , vol. 49, no. 8, pp. 1255– 1277, 2019

    2019

  5. [13]

    Assess- ing mcr discussion usefulness using semantic similarity,

    T. Pangsakulyanont, P. Thongtanunam, D. Port, and H. Iida, “Assess- ing mcr discussion usefulness using semantic similarity,” in 2014 6th International Workshop on Empirical Software Engineering in Practice . IEEE, 2014, pp. 49–54

    2014

  6. [14]

    Predicting usefulness of code review comments using textual features and developer experience,

    M. M. Rahman, C. K. Roy, and R. G. Kula, “Predicting usefulness of code review comments using textual features and developer experience,” in 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 2017, pp. 215–226

    2017

  7. [15]

    Automating code review activities by large-scale pre-training,

    L. Zhiyu, L. Shuai, G. Daya, D. Nan, J. Shailesh, J. Grant, M. Deep, G. Jared, S. Alexey, F. Shengyu, and N. Sundaresan, “Automating code review activities by large-scale pre-training,” in 30th ACM Joint European Software Engineering Conference and the ACM/SIGSOFT Internationa...

    2022

  8. [16]

    Using pre-trained models to boost code review au- tomation,

    R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk, and G. Bavota, “Using pre-trained models to boost code review au- tomation,” in 44th IEEE/ACM International Conference on Software Engineering, ICSE, 2022, pp. 2291–2302

    2022

  9. [17]

    Auger: automatically generating review comments with pre-training models,

    L. Li, L. Yang, H. Jiang, J. Yan, T. Luo, Z. Hua, G. Liang, and C. Zuo, “Auger: automatically generating review comments with pre-training models,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engine...

    2022

  10. [18]

    Unveiling chatgpt’s usage in open source projects: A mining-based study,

    R. Tufano, A. Mastropaolo, F. Pepe, O. Dabic, M. Di Penta, and G. Bavota, “Unveiling chatgpt’s usage in open source projects: A mining-based study,” in Proceedings of 2024 IEEE/ACM 21th Inter- national Conference on Mining Software Repositories (MSR) . IEEE, 2024, p. To Appear

    2024

  11. [19]

    Make llm a testing expert: Bringing human-like interac- tion to mobile gui testing via functionality-aware decisions,

    Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, X. Che, D. Wang, and Q. Wang, “Make llm a testing expert: Bringing human-like interac- tion to mobile gui testing via functionality-aware decisions,” in 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE), 2024, ...

    2024

  12. [20]

    Llmparser: An exploratory study on using large language models for log parsing,

    Z. Ma, A. R. Chen, D. J. Kim, T.-H. P. Chen, and S. Wang, “Llmparser: An exploratory study on using large language models for log parsing,” in 2024 IEEE/ACM 46th International Conference on Software Engi- neering (ICSE), 2024, p. To appear

    2024

  13. [21]

    Unilog: Automatic logging via llm and in-context learning,

    J. Xu, Z. Cui, Y . Zhao, X. Zhang, S. He, P. He, L. Li, Y . Kang, Q. Lin, Y . Dang, S. Rajmohan, and D. Zhang, “Unilog: Automatic logging via llm and in-context learning,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ser. ICSE ’24, 2024

    2024

  14. [22]

    Using an llm to help with code understanding,

    D. Nam, A. Macvean, V . J. Hellendoorn, B. Vasilescu, and B. A. Myers, “Using an llm to help with code understanding,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ser. ICSE ’24, 2024

    2024

  15. [23]

    What types of defects are really dis- covered in code reviews?

    M. V . M ¨antyl¨a and C. Lassenius, “What types of defects are really dis- covered in code reviews?” IEEE Transactions on Software Engineering, vol. 35, no. 3, pp. 430–448, 2008

    2008

  16. [24]

    ChatGPT,

    “ChatGPT,” https://chat.openai.com/, accessed: 2024-02-27

    2024

  17. [25]

    Replication package,

    “Replication package,” https://github.com/CodeReviewExperiment/ code review controlled experiment, [n.d.]

  18. [26]

    Ai-assisted assessment of coding practices in modern code review,

    M. V . et al., “Ai-assisted assessment of coding practices in modern code review,” CoRR, vol. abs/2405.13565, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2405.13565

    2024 doi arXiv

  19. [27]

    Convenience sampling,

    P. Sedgwick, “Convenience sampling,” BMJ, 2013

    2013

  20. [28]

    Population research: convenience sampling strategies,

    S. J. Stratton, “Population research: convenience sampling strategies,” Prehospital and disaster Medicine , vol. 36, no. 4, pp. 373–374, 2021

    2021

  21. [29]

    Rosetta Code,

    “Rosetta Code,” https://rosettacode.org/, accessed: 2024-02-22

    2024

  22. [30]

    Apache Commons Lang,

    “Apache Commons Lang,” https://github.com/apache/commons-lang, accessed: 2024-02-22

    2024

  23. [31]

    What hap- pens in my code reviews? an investigation on automatically classifying review changes,

    E. Fregnan, F. Petrulio, L. Di Geronimo, and A. Bacchelli, “What hap- pens in my code reviews? an investigation on automatically classifying review changes,” Empirical Software Engineering, vol. 27, no. 4, p. 89, 2022

    2022

  24. [32]

    Remote Development - Visual Studio Marketplace,

    “Remote Development - Visual Studio Marketplace,” https: //marketplace.visualstudio.com/items?itemName=ms-vscode-remote. vscode-remote-extensionpack, accessed: 2024-02-28

    2024

  25. [33]

    Extension Pack for Java - Visual Studio Marketplace,

    “Extension Pack for Java - Visual Studio Marketplace,” https://marketplace.visualstudio.com/items?itemName=vscjava. vscode-java-pack, accessed: 2024-02-28

    2024

  26. [34]

    Python - Visual Studio Marketplace,

    “Python - Visual Studio Marketplace,” https://marketplace.visualstudio. com/items?itemName=ms-python.python, accessed: 2024-02-28

    2024

  27. [35]

    Code Review - Visual Studio Marketplace,

    “Code Review - Visual Studio Marketplace,” https: //marketplace.visualstudio.com/items?itemName=d-koppenhagen. vscode-code-review, accessed: 2024-02-28

    2024

  28. [36]

    Tako - Visual Studio Marketplace,

    “Tako - Visual Studio Marketplace,” https://marketplace.visualstudio. com/items?itemName=codelounge.tako, accessed: 2024-02-28

    2024

  29. [37]

    I know what you did last summer: an investigation of how developers spend their time,

    R. Minelli, A. Mocci, and M. Lanza, “I know what you did last summer: an investigation of how developers spend their time,” in 23rd IEEE International Conference on Program Comprehension, ICPC , 2015, pp. 25–35

    2015

  30. [38]

    Hughes and D

    A. Hughes and D. Grawoig, Statistics, a Foundation for Analysis , ser. Business and Economics Series. Addison-Wesley Publishing Company, 1971

    1971

  31. [39]

    Multiple comparisons among means,

    O. J. Dunn, “Multiple comparisons among means,” Journal of the American Statistical Association , vol. 56, no. 293, pp. 52–64, 1961

    1961

  32. [40]

    Controlling the false discovery rate: A practical and powerful approach to multiple testing,

    B. Yoav and H. Yosef, “Controlling the false discovery rate: A practical and powerful approach to multiple testing,” Journal of the Royal Statistical Society. Series B (Methodological) , vol. 57, no. 1, pp. 289– 300, 1995

    1995

  33. [41]

    Cohen, Statistical power analysis for the behavioral sciences

    J. Cohen, Statistical power analysis for the behavioral sciences . Lawrence Earlbaum Associates, 1988

    1988

  34. [42]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945

    1945

  35. [43]

    R. J. Grissom and J. J. Kim, Effect sizes for research: A broad practical approach, 2nd ed. Lawrence Earlbaum Associates, 2005

    2005

  36. [44]

    Few-shot training llms for project-specific code-summarization,

    T. Ahmed and P. Devanbu, “Few-shot training llms for project-specific code-summarization,” in Proceedings of the 37th IEEE/ACM Interna- tional Conference on Automated Software Engineering , ser. ASE ’22, 2023

    2023

  37. [45]

    Correct: Code reviewer recommendation in github based on cross-project and technology expe- rience,

    M. M. Rahman, C. K. Roy, and J. A. Collins, “Correct: Code reviewer recommendation in github based on cross-project and technology expe- rience,” in 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C) , 2016, pp. 222–231

    2016

  38. [46]

    Automatically recommending peer reviewers in modern code review,

    M. B. Zanjani, H. Kagdi, and C. Bird, “Automatically recommending peer reviewers in modern code review,” IEEE Transactions on Software Engineering, vol. 42, no. 6, pp. 530–543, 2016

    2016

  39. [47]

    Whodo: Automating reviewer suggestions at scale,

    S. Asthana, R. Kumar, R. Bhagwan, C. Bird, C. Bansal, C. Maddila, S. Mehta, and B. Ashok, “Whodo: Automating reviewer suggestions at scale,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softwa...

    2019

  40. [48]

    Who should make decision on this pull request? analyzing time-decaying relationships and file similarities for integrator prediction,

    J. Jiang, D. Lo, J. Zheng, X. Xia, Y . Yang, and L. Zhang, “Who should make decision on this pull request? analyzing time-decaying relationships and file similarities for integrator prediction,” Journal of Systems and Software , vol. 154, pp. 196–210, 2019

    2019

  41. [49]

    Mitigating turnover with code review recommendation: Balancing expertise, workload, and knowledge distribution,

    E. Mirsaeedi and P. C. Rigby, “Mitigating turnover with code review recommendation: Balancing expertise, workload, and knowledge distribution,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , ser. ICSE ’20. New York, NY , USA: Association...

    2020

  42. [50]

    Using a context- aware approach to recommend code reviewers: findings from an indus- trial case study,

    A. Strand, M. Gunnarson, R. Britto, and M. Usman, “Using a context- aware approach to recommend code reviewers: findings from an indus- trial case study,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice, 20...

    2020

  43. [51]

    Corms: A github and gerrit based hybrid code reviewer recommendation approach for modern code review,

    P. Pandya and S. Tiwari, “Corms: A github and gerrit based hybrid code reviewer recommendation approach for modern code review,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FS...

    2022

  44. [52]

    Senticr: a customized sentiment analysis tool for code review interactions,

    T. Ahmed, A. Bosu, A. Iqbal, and S. Rahimi, “Senticr: a customized sentiment analysis tool for code review interactions,” in 2017 32nd IEEE/ACM International Conference on Automated Software Engineer- ing (ASE). IEEE, 2017, pp. 106–111

    2017

  45. [53]

    Predicting developers’ negative feelings about code review,

    C. D. Egelman, E. Murphy-Hill, E. Kammer, M. M. Hodges, C. Green, C. Jaspan, and J. Lin, “Predicting developers’ negative feelings about code review,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , 2020, pp. 174–185

    2020

  46. [54]

    Using a balanced scorecard to identify opportunities to improve code review effectiveness: An industrial experience report,

    M. Hasan, A. Iqbal, M. R. U. Islam, A. I. Rahman, and A. Bosu, “Using a balanced scorecard to identify opportunities to improve code review effectiveness: An industrial experience report,” Empirical Software En- gineering, vol. 26, pp. 1–34, 2021

    2021

  47. [55]

    Com- mentfinder: a simpler, faster, more accurate code review comments recommendation,

    Y . Hong, C. Tantithamthavorn, P. Thongtanunam, and A. Aleti, “Com- mentfinder: a simpler, faster, more accurate code review comments recommendation,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Eng...

    2022

  48. [56]

    Review4repair: Code review aided automatic program repairing,

    F. Huq, M. Hasan, M. M. A. Haque, S. Mahbub, A. Iqbal, and T. Ahmed, “Review4repair: Code review aided automatic program repairing,” In- formation and Software Technology , vol. 143, p. 106765, 2022

    2022

  49. [57]

    Towards automating code review activities,

    R. Tufano, L. Pascarella, M. Tufano, D. Poshyvanyk, and G. Bavota, “Towards automating code review activities,” in 43rd IEEE/ACM Inter- national Conference on Software Engineering, ICSE, 2021, pp. 163–174

    2021

  50. [58]

    Code review automation: Strengths and weaknesses of the state of the art,

    R. Tufano, O. Dabic, A. Mastropaolo, M. Ciniselli, and G. Bavota, “Code review automation: Strengths and weaknesses of the state of the art,” IEEE Trans. Software Eng. , vol. 50, no. 2, pp. 338–353, 2024

    2024

  51. [59]

    An experimental evaluation of an experience-based capture-recapturemethod in software code inspec- tions,

    P. Runeson and C. Wohlin, “An experimental evaluation of an experience-based capture-recapturemethod in software code inspec- tions,” Empirical Softw. Engg., 1998

    1998

  52. [60]

    Partitioning composite code changes to facilitate code review,

    Y . Tao and S. Kim, “Partitioning composite code changes to facilitate code review,” in 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 2015, pp. 180–190

    2015

  53. [61]

    Interactive code review for systematic changes,

    T. Zhang, M. Song, J. Pinedo, and M. Kim, “Interactive code review for systematic changes,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , vol. 1, 2015, pp. 111–122

    2015

  54. [62]

    Impact of gamification on code review process: An experimental study,

    S. Khandelwal, S. K. Sripada, and Y . R. Reddy, “Impact of gamification on code review process: An experimental study,” in Proceedings of the 10th Innovations in Software Engineering Conference , ser. ISEC ’17, 2017, p. 122–126

    2017

  55. [63]

    Cldiff: generating concise linked code differences,

    K. Huang, B. Chen, X. Peng, D. Zhou, Y . Wang, Y . Liu, and W. Zhao, “Cldiff: generating concise linked code differences,” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ser. ASE ’18, 2018, p. 679–690

    2018

  56. [64]

    Salient-class location: help developers understand code change in code review,

    Y . Huang, N. Jia, X. Chen, K. Hong, and Z. Zheng, “Salient-class location: help developers understand code change in code review,” in Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engine...

    2018

  57. [65]

    Aiding code change understand- ing with semantic change impact analysis,

    Q. Hanam, A. Mesbah, and R. Holmes, “Aiding code change understand- ing with semantic change impact analysis,” in 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2019, pp. 202–212

    2019

  58. [66]

    Primers or reminders? the effects of existing review comments on code review,

    D. Spadini, G. C ¸ alikli, and A. Bacchelli, “Primers or reminders? the effects of existing review comments on code review,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ser. ICSE ’20, 2020, p. 1171–1182

    2020

  59. [67]

    First come first served: the impact of file position on code review,

    E. Fregnan, L. Braz, M. D’Ambros, G. C ¸ alıklı, and A. Bacchelli, “First come first served: the impact of file position on code review,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser...

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.