Pith. sign in

REVIEW 3 major objections 7 minor 46 references

Fuller AI explanations raise developer trust in code reviews but lower agreement with the AI’s accept-or-reject calls.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 10:50 UTC pith:RGT5AA7D

load-bearing objection Solid controlled study on XAI depth vs trust/agreement in LLM code review; the A-vs-B tradeoff is descriptive, and the admitted shuffle imbalance is a real but checkable soft spot. the 3 major comments →

arxiv 2607.24601 v1 pith:RGT5AA7D submitted 2026-07-27 cs.SE cs.AIcs.HC

Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review

Zhenhan Gao , Marvin Mu\~noz Bar\'on , Umm-e Habiba , Daniel Graziotin , Stefan Wagner This is my paper
classification cs.SE cs.AIcs.HC
keywords explainable artificial intelligencecode reviewdeveloper trustlarge language modelsuser studyAI-assisted software engineeringcalibrated trust
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether adding explainable AI to automated code review changes how much developers trust the tool and how often they follow its accept-or-reject advice. In a within-subjects study, 34 programmers reviewed real pull requests under three explanation levels: full inline justifications plus review comments, review comments alone, or a bare accept/reject label. Trust scores rose with more explanation, peaking when the system highlighted the exact code fragments that drove its decision. Agreement with the AI, however, peaked at the middle level—comments without the deep inline layer—suggesting that richer rationales invite more scrutiny and more overrides. Time to decide did not reliably change. The result matters because teams are already wiring LLMs into review pipelines; the depth of explanation they ship will shape both how trusted and how obeyed those tools become.

Core claim

The level of explanation significantly moves both perceived trust and behavioural agreement with AI code-review recommendations, but in opposite directions: full explanations produce the highest trust (mean 3.99/5) while moderate explanations produce the highest agreement (89.22 %), and no explanations produce the lowest of both. Explanation depth does not significantly change review time. The authors treat this divergence as evidence of a gap between stated trust and action.

What carries the argument

A three-level XAI pipeline for code review (Condition A: Phase-1 feedback plus Phase-2 inline highlights and justifications; B: Phase-1 feedback only; C: bare accept/reject) administered within-subjects, with trust measured by the TXAI scale and agreement measured by match to the AI’s binary decision.

Load-bearing premise

That trust scores and accept/reject choices collected in short, consequence-free online tasks on always-correct AI recommendations from one educational Python repository stand in for calibrated developer trust under real project risk.

What would settle it

Re-run the same three explanation levels but deliberately include incorrect AI recommendations; if fuller explanations no longer raise trust relative to moderate ones, or if they fail to help developers catch the flawed calls, the claimed trust–agreement trade-off collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • AI code-review tools should ship some form of explanation; bare accept/reject labels yield the lowest trust and agreement.
  • Embedding justifications directly in the diff (inline highlights) raises perceived trust more than separate review comments alone.
  • Designers face an explicit trade-off: deeper explanation can raise trust while lowering automatic agreement, which may be desirable if the goal is calibrated rather than blind acceptance.
  • Studies that evaluate AI review tools should report explanation level alongside accuracy, because format alone changes developer behaviour when recommendation quality is held fixed.
  • Concise feedback may be preferable when the goal is rapid alignment; richer feedback when the goal is deliberate, contested decisions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the trust–agreement divergence is real, product teams may need separate modes (fast-accept vs. audit) rather than a single default explanation depth.
  • Crossing explanation depth with deliberately wrong recommendations is the natural next experiment; the present design cannot speak to over-trust detection.
  • The same intermediate-detail peak for agreement may appear in other AI-assisted SE tasks (test generation, refactoring suggestions) where more text gives developers more surface to contest.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper reports a within-subjects online experiment (N = 34, Prolific) in which participants reviewed nine real Python pull requests under three conditions of AI-generated review support: (A) recommendation + review feedback + inline explanations with code highlighting, (B) recommendation + review feedback, (C) recommendation only. Trust was measured per condition with a 7-item version of Hoffman's TXAI scale; agreement with the AI's accept/reject recommendation, task completion time, and free-text reasoning were also recorded. The main findings: trust is highest in A and lowest in C (RM-ANOVA p = 0.0006, η²p = 0.2; only A–C survives Bonferroni); agreement is highest in B (89.22%), lowest in C (66.67%), with only B–C significant after correction; time does not differ across conditions; qualitative coding (α = 0.7564) shows readability and correctness dominate stated reasons, and explicit references to the AI's rationale occur only in explained conditions. The authors interpret the trust/agreement divergence as richer explanations inviting scrutiny. Limitations disclosed include a single-repository stimulus set, always-correct AI recommendations, and a non-uniform shuffling bug that imbalanced task-to-condition assignment.

Significance. If the results hold, this is a useful early empirical contribution to an understudied intersection: XAI and trust in LLM-based code review. The trust–agreement divergence (full explanations raising self-reported trust while intermediate feedback maximizes behavioral alignment) is an interesting, actionable observation for tool designers, and the qualitative finding that explicit engagement with the AI's rationale occurred only in explained conditions adds texture. Methodological strengths worth naming: an a priori power analysis, a within-subjects design with Bonferroni-corrected post-hocs, item-level reporting of the TXAI scale (Table 4), a published and psychometrically justified trust instrument, multi-round qualitative coding with reported Krippendorff's α, commendably candid disclosure of a randomization flaw (§4.7), and a complete replication package including stimuli, prompts, raw data, and analysis code. The study is honestly reported; the open questions concern confounding and claim calibration rather than transparency.

major comments (3)
  1. [§4.7 (Threats to Validity) / §3.2.2 (Task Assignment)] §4.7 discloses that task-to-condition assignment used sort(Math.random() - 0.5), which does not generate uniform permutations, so 'some sample treatment combinations occurr[ed] less often than expected.' The paper stops at disclosure, but this is potentially load-bearing for both headline results, because the AI recommendation always matches the historical merge outcome (§3.2.1, §4.2): agreement with the AI equals matching the true PR outcome, and the nine PRs differ in difficulty and decision (6 reject / 3 accept, Table 1). If Condition B happened to receive disproportionately many easy or transparently-correct PRs, the B–C gap (the only Bonferroni-surviving agreement contrast, Table 5) could partly reflect task composition rather than explanation level; likewise, TXAI items Q1/Q3 ('works well,' 'I can count on it to be correct') — the items driving the RQ1 effect (Table 4) — are sensit
  2. [Abstract / §1 Results / §4.2 (Table 5)] The central novelty of the paper — the divergence between perceived trust (highest in A) and behavioral agreement (highest in B), and the 'scrutiny' interpretation in §4.2/§4.6 — rests on the A–B ordering. But the A–B agreement contrast does not survive the Bonferroni correction the authors themselves apply (raw p = 0.0209, corrected p = 0.0628, Table 5), and the A–B trust contrast is also non-significant (corrected p = 0.2808, Table 3). The abstract and RQ2 Summary state 'moderate explanations (B) achieve the highest agreement (89.22%)' and the introduction's Results paragraph asserts 'participants followed the AI recommendations most often... for Condition B' without flagging that the A–B difference is not significant after correction. The descriptively observed ordering is fine to report, but the claim that explanation depth *prompts more scrutiny* (and the trust–agreement trade-off f
  3. [§3.3 (RQ2 analysis) / §4.2] Agreement per participant per condition is the proportion of 3 binary decisions matching the AI (§3.3, RQ2), i.e., a 4-level discrete variable (0, 1/3, 2/3, 1). A repeated-measures ANOVA on this variable is a coarse instrument, and with only 3 tasks per condition a single flipped decision moves a participant's score by 33 percentage points — nearly the entire A–C gap. Combined with Major Comment 1, I recommend re-analyzing RQ2 at the level of the 306 individual decisions with a logistic mixed-effects model (participant and PR as random effects). This also subsumes the task-composition concern and would materially strengthen (or appropriately temper) the RQ2 conclusion.
minor comments (7)
  1. [Abstract / §3.2.2] Terminology drift between the abstract/§3.2.2 and the rest of the paper: Condition B is described as 'review feedback only' in the abstract but as 'some explanations' in §1 and §3.2.2, and as 'moderate explanations' in the abstract's Results sentence. A single consistent label (e.g., 'Phase-1 feedback only') would avoid confusion, especially since the A vs. B distinction (Phase 2 inline explanations) is the subtle one.
  2. [§4.1 (Table 4)] Table 4 reports seven item-level repeated-measures ANOVAs with no multiplicity control. Either apply a correction across the seven tests or present the item-level analysis explicitly as exploratory.
  3. [§3.2.1 (Table 1) / §4.2] The PR set is imbalanced in decision (6 reject, 3 accept; Table 1). Since agreement = matching the AI/ground-truth outcome, a participant with a default tendency to reject will show high 'agreement' regardless of condition. Discuss whether decision-imbalance interacts with the agreement measure, or show agreement broken down by accept-PRs vs. reject-PRs.
  4. [§3.3 / §4.1–4.3] No sphericity diagnostics (Mauchly's test or Greenhouse–Geisser correction) are reported for the three ANOVAs, despite assuming ε = 0.75 in the power analysis (§3.2.5). With three levels the impact is limited, but please report the test or applied correction.
  5. [§3.2.5] Quality control on Prolific: the fastest complete study took 4 min 10 s against a 37.9 min mean (§3.2.5), and compensation was $7/hour, below typical platform norms. No attention checks or speeder screening are described. Please state what (if any) data-quality screening was applied, or list its absence as a threat.
  6. [§4.4 (Figure 4) / Figure 1] Figure 4's y-axis label ('0 10 20% of all coded reasons') is ambiguous about whether percentages are within-condition, within-decision, or global; §4.4 mixes denominators (e.g., '55.37% of all coded reasons,' '13.36% of all reasons' vs. '23.94% overall' in the figure). Unify the denominator and state it in the caption. Also a typo in Figure 1: 'Gener ating model outputs.'
  7. [§4.2 (Discussion, second paragraph)] The sentence 'because we did not include recommendations where the AI recommendation didn't match the real-world outcome, we are unable to draw any conclusions about the match between AI recommendations and real-world outcomes' is confusingly worded — the point is that agreement and accuracy are confounded by construction, so the design cannot separate trust in the AI from accuracy on these tasks. Rephrase along those lines.

Circularity Check

0 steps flagged

No circularity: empirical between-condition user study; outcomes are measured, not derived from fitted or self-defined inputs.

full rationale

The paper’s load-bearing claims are statistical contrasts (repeated-measures ANOVA and post-hoc tests) on externally measured outcomes—TXAI questionnaire scores, accept/reject agreement with a fixed AI recommendation, task time, and coded free-text reasons—across three explanation conditions. The AI accept/reject labels were held constant and verified against historical PR merge outcomes rather than fitted to produce the trust or agreement results. The TXAI instrument is an external published scale (Hoffman et al., with Perrig et al. psychometric revision), not defined in terms of the study’s dependent variables. There is no self-definitional loop, no parameter fit re-labeled as prediction, no uniqueness theorem imported from the authors, and no ansatz smuggled via self-citation. The admitted non-uniform task shuffle (§4.7) is an internal-validity confound, not a by-construction reduction of the claimed effects to their inputs. The derivation chain is therefore self-contained empirical comparison; circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

As a controlled user study, the claim rests on standard statistical and measurement assumptions plus several domain and design choices (always-correct AI, single-repo PRs, TXAI as trust, Prolific programmers as developers). No physical free parameters; the main ‘knobs’ are design choices that define the three explanation levels and the task set.

free parameters (4)
  • Explanation depth levels (A/B/C pipeline cutoffs) = A=full Phase1+Phase2; B=Phase1 only; C=binary only
    Hand-designed cumulative exposure of Phase-1 recommendation/reasons vs Phase-2 inline explanations; not derived from a theory of optimal explanation length.
  • TXAI item subset (7 items, reverse item dropped) = 7-item revised Hoffman TXAI
    Scale composition follows Perrig et al. recommendation; choice affects the numeric trust means that ground RQ1.
  • PR difficulty labeling and 9-PR set = 9 PRs, 3 per difficulty
    Manual selection and easy/medium/hard labels from TheAlgorithms/Python with <55 LOC changes; defines the stimulus distribution.
  • GPT-4o temperature and prompts = temperature=0; prompts in replication package
    Temperature 0 and iteratively hand-tuned prompts determine the fixed review text shown in all conditions.
axioms (6)
  • standard math Repeated-measures ANOVA and paired t-tests with Bonferroni correction are appropriate for per-participant trust, agreement, and time scores across three conditions.
    Invoked in §3.3 analysis procedures for RQ1–RQ3.
  • domain assumption Averaged TXAI questionnaire responses after brief interaction measure developers’ trust in the AI code-review system.
    Construct operationalization in §3.2.3; Hoffman et al. scale with one item removed per Perrig et al.
  • domain assumption Holding AI recommendation correctness fixed (always matching historical merge outcome) isolates the effect of explanation format on trust and agreement.
    Stated in methodology and revisited in §4.2 and §5; excludes study of explanation aid in detecting errors.
  • domain assumption Prolific participants screened for English, programming, and Python familiarity adequately stand in for software engineers doing code review.
    Recruitment §3.2.5; external validity threat noted in §4.7.
  • ad hoc to paper LLM-generated post-hoc rationales (GPT-4o) count as XAI support levels for studying explanation effects, even without guaranteed faithfulness.
    §3.1 acknowledges post-hoc rationalization risk; still treats A/B/C as XAI conditions.
  • ad hoc to paper Broken random comparator still left the within-subjects structure intact enough that condition effects can be attributed to explanation level.
    Authors argue this in §4.7 after discovering Math.random()-0.5 shuffle bias; remains a load-bearing internal-validity hope.

pith-pipeline@v1.2.0-grok45-kimik3 · 26646 in / 3639 out tokens · 95086 ms · 2026-07-31T10:50:39.940507+00:00 · methodology

0 comments
read the original abstract

Background: Large language models (LLMs) are increasingly used to automate code review, but the reasoning behind their decisions remains hard to understand. Developers struggle to assess the validity of LLM-generated reviews, making it difficult to gauge how much trust to place in them. The role of Explainable AI (XAI) in code review and its impact on trust remain underexplored. Objective: We study the influence of XAI on developer trust in AI-assisted code reviews. Method: We conducted a within-subjects user study with 34 participants, comparing three LLM-based code review systems with varying levels of XAI support: Condition A (detailed explanation and review feedback), Condition B (review feedback only), and Condition C (no explanations). Participants reviewed real-world code change requests alongside the AI-generated reviews. We measured trust perceptions, agreement with the AI recommendation, the reasoning given for each decision, and the time taken. Results: The level of explanation significantly influences both trust and agreement with AI recommendations, but in different ways. Full explanations (A) yield the highest perceived trust (M = 3.99/5) but not the highest agreement, whereas moderate explanations (B) achieve the highest agreement (89.22%). This could suggest that more explanation prompts developers to question AI recommendations more frequently. No explanations (C) results in the lowest trust and agreement. Explanation level did not significantly affect review time. The most commonly cited reasons for decisions were code readability and correctness. Conclusion: Incorporating XAI into code review significantly changes trust perceptions and agreement with AI recommendations. These results inform the design and evaluation of trustworthy AI-based code review systems, as well as studies on the human factors of AI-assisted software development.

Figures

Figures reproduced from arXiv: 2607.24601 by Daniel Graziotin, Marvin Mu\~noz Bar\'on, Stefan Wagner, Umm-e Habiba, Zhenhan Gao.

Figure 1
Figure 1. Figure 1: Overview of the research workflow reasons, but does not include any Phase 2 output. Condition A includes the same Phase 1 feedback as Condition B and additionally exposes the Phase 2 output through inline explanations and highlighted code sections. Thus, Condition A is not simply a more detailed version of Condition B. Rather, it supplements the Phase 1 recommendation and reasons with a distinct Phase 2 ex… view at source ↗
Figure 2
Figure 2. Figure 2: Outputs generated by the XAI system for a single pull request. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Code Review Task Interface Upon being presented with a code change, participants are instructed to review it, decide whether to accept or reject it by clicking a button, and provide a reason for their decision in a text box. We collect both their responses and the time taken to provide them in the web application for each code change. After the participant has seen all three code changes for an experimenta… view at source ↗
Figure 4
Figure 4. Figure 4: Frequency of codes assigned to responses, across the three conditions. [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 7 canonical work pages · 1 internal anchor

  1. [1]

    Shahin Atakishiyev, Housam KB Babiker, Jiayi Dai, Nawshad Farruque, Teruaki Hayashi, Nafisa Sadaf Hriti, Md Abed Rahman, Iain Smith, Mi-Young Kim, Osmar R Zaïane, et al. 2025. Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations.arXiv preprint arXiv:2510.17256(2025). doi:10.48550/ arXiv.2510.17256

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2510.17256 2025

  2. [2]

    Shahin Atakishiyev, Mohammad Salameh, Hengshuai Yao, and Randy Goebel. 2024. Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions.IEEE Access12 (01 2024), 101603–101625. doi:10.1109/ACCESS.2024.3431437

    arXiv 2024

  3. [3]

    Fannar Steinn Aðalsteinsson, Björn Borgar Magnússon, Mislav Milicevic, Adam Nirving Davidsson, and Chih-Hong Cheng. 2025. Rethinking Code Review Workflows with LLM Assistance: An Empirical Study. In2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 488–497. doi:10.1109/ESEM64174. 2025.00013

    arXiv 2025

  4. [4]

    Sebastian Baltes, Timo Speith, Brenda Chiteri, Seyedmoein Mohsenimofidi, Shalini Chakraborty, and Daniel Buschek

  5. [5]

    Amiangshu Bosu and Jeffrey C Carver. 2013. Impact of peer code review on peer impression formation: A survey. In2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, 133–142. doi:10.1109/ESEM.2013.23

    work page doi:10.1109/esem.2013.23 2013

  6. [6]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models.ACM transactions on intelligent systems and technology15, 3 (2024), 1–45. doi:10.1145/3641289

    doi:10.1145/3641289 2024

  7. [7]

    Umut Cihan, Vahid Haratian, Arda İçöz, Mert Kaan Gül, Ömercan Devran, Emircan Furkan Bayendur, Baykal Mehmet Uçar, and Eray Tüzün. 2025. Automated code review in practice. In2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 425–436. doi:10.1109/ICSE-SEIP66354.2025. 00043

    arXiv 2025

  8. [8]

    Umut Cihan, Arda İçöz, Vahid Haratian, and Eray Tüzün. 2025. Evaluating Large Language Models for Code Review. arXiv preprint arXiv:2505.20206(2025). doi:10.48550/arXiv.2505.20206

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.20206 2025

  9. [9]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. In2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 31–53. doi:10.1109/ICSE-FoSE59343.2023.00008

    arXiv 2023

  10. [10]

    Franz Faul, Edgar Erdfelder, Axel Buchner, and Albert-Georg Lang. 2009. Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses.Behavior Research Methods41 (11 2009), 1149–1160. doi:10.3758/BRM.41. 4.1149

    work page doi:10.3758/brm.41 2009

  11. [11]

    Franz Faul, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner. 2007. G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences.Behavior research methods39, 2 (2007), 175–191. doi:10.3758/BF03193146

    doi:10.3758/bf03193146 2007

  12. [12]

    2026.Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review - Data, Code and Diagrams

    Zhenhan Gao, Marvin Muñoz Barón, Umm-e Habiba, Daniel Graziotin, and Stefan Wagner. 2026.Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review - Data, Code and Diagrams. doi:10.5281/zenodo.21457282 Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA093. Publication date: October 2026. ISSTA093:22 Zhenhan Gao, Marvin Muñoz Barón, ...

    work page doi:10.5281/zenodo.21457282 2026

  13. [13]

    GitHub. 2023. GitHub REST API documentation. https://docs.github.com/en/rest?apiVersion=2022-11-28

    2023

  14. [14]

    David Gunning and David Aha. 2019. DARPA’s Explainable Artificial Intelligence (XAI) Program.AI Magazine40 (06 2019), 44–58. doi:10.1609/aimag.v40i2.2850

    doi:10.1609/aimag.v40i2.2850 2019

  15. [15]

    Vincent J Hellendoorn, Jason Tsay, Manisha Mukherjee, and Martin Hirzel. 2021. Towards automating code review at scale. InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1479–1482. doi:10.1145/3468264.3473134

    arXiv 2021

  16. [16]

    Hoffman, Shane T

    Robert R. Hoffman, Shane T. Mueller, Gary Klein, and Jordan Litman. 2023. Measures for explainable AI: Explanation goodness, user satisfaction, mental models, curiosity, trust, and human-AI performance.Frontiers in Computer Science Volume 5 - 2023 (2023). doi:10.3389/fcomp.2023.1096257

    arXiv 2023

  17. [17]

    Md Imran Hossain, Ghada Zamzmi, Peter R Mouton, Md Sirajus Salekin, Yu Sun, and Dmitry Goldgof. 2025. Explainable AI for medical data: Current methods, limitations, and future directions.Comput. Surveys57, 6 (2025), 1–46. doi:10. 1145/3637487

    2025

  18. [18]

    Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. 2021. Formalizing trust in artificial intelligence: Prerequi- sites, causes and goals of human trust in AI. InProceedings of the 2021 ACM conference on fairness, accountability, and transparency. 624–635. doi:10.1145/3442188.3445923

    arXiv 2021

  19. [19]

    Jiun-Yin Jian, Ann Bisantz, and Colin Drury. 2000. Foundations for an Empirically Determined Scale of Trust in Automated Systems.International Journal of Cognitive Ergonomics4 (03 2000), 53–71. doi:10.1207/S15327566IJCE0401_04

    doi:10.1207/s15327566ijce0401_04 2000

  20. [20]

    2018.Content analysis: An introduction to its methodology

    Klaus Krippendorff. 2018.Content analysis: An introduction to its methodology. Sage publications

    2018

  21. [21]

    Stefano Lambiase, Gemma Catolino, Fabio Palomba, Filomena Ferrucci, and Daniel Russo. 2025. Investigating the role of cultural values in adopting large language models for software engineering.ACM Transactions on Software Engineering and Methodology35, 1 (2025), 1–43. doi:10.1145/3725529

    doi:10.1145/3725529 2025

  22. [22]

    Benedikt Langenberg, Markus Janczyk, Valentin Koob, Reinhold Kliegl, and Axel Mayer. 2023. A tutorial on using the paired t test for power calculations in repeated measures ANOVA with interactions.Behavior Research Methods55, 5 (2023), 2467–2484. doi:10.3758/s13428-022-01902-8

    work page doi:10.3758/s13428-022-01902-8 2023

  23. [23]

    Evan Li. 2025. GitHub Ranking: Top 100 Python Repositories. https://github.com/EvanLi/Github-Ranking/blob/master/ Top100/Python.md

    2025

  24. [24]

    Heng-Yi Li, Shu-Ting Shi, Ferdian Thung, Xuan Huo, Bowen Xu, Ming Li, and David Lo. 2019. Deepreview: automatic code review using deep multi-instance learning. InPacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, 318–330. doi:10.1007/978-3-030-16145-3_25

    work page doi:10.1007/978-3-030-16145-3_25 2019

  25. [25]

    Microsoft. 2022. CodeReviewer. https://huggingface.co/microsoft/codereviewer

    2022

  26. [26]

    Avash Palikhe, Zhenyu Yu, Zichong Wang, and Wenbin Zhang. 2025. Towards Transparent AI: A Survey on Explainable Large Language Models.ArXivabs/2506.21812 (2025). doi:10.48550/arXiv.2506.21812

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.21812 2025

  27. [27]

    Sebastian AC Perrig, Nicolas Scharowski, and Florian Brühlmann. 2023. Trust issues with trust scales: examining the psychometric quality of trust measures in the context of AI. InExtended abstracts of the 2023 CHI Conference on human factors in computing systems. 1–7. doi:10.1145/3544549.3585808

    arXiv 2023

  28. [28]

    Chanathip Pornprasit and Chakkrit Tantithamthavorn. 2024. Fine-tuning and prompt engineering for large language models-based code review automation.Inf. Softw. Technol.175, C (Nov. 2024), 12 pages. doi:10.1016/j.infsof.2024.107523

    arXiv 2024

  29. [29]

    Prolific. 2025. Prolific Participant Recruitment Platform. https://www.prolific.com/. Accessed: 2025-06-15

    2025

  30. [30]

    Waddah Saeed and Christian Omlin. 2023. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities.Know.-Based Syst.263, C (March 2023), 24 pages. doi:10.1016/j.knosys.2023.110273

    arXiv 2023

  31. [31]

    Alan Said. 2025. On explaining recommendations with Large Language Models: a review.Frontiers in Big Data7 (2025), 1505284. doi:10.3389/fdata.2024.1505284

    arXiv 2025

  32. [32]

    Jaydeb Sarker, Sayma Sultana, Steven R Wilson, and Amiangshu Bosu. 2023. ToxiSpanSE: An explainable toxicity detection in code review comments. In2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1–12. doi:10.1109/ESEM56168.2023.10304855

    arXiv 2023

  33. [33]

    Shu-Ting Shi, Ming Li, David Lo, Ferdian Thung, and Xuan Huo. 2019. Automatic code review by learning the revision of source code. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 4910–4917. doi:10.1609/aaai. v33i01.33014910

    doi:10.1609/aaai 2019

  34. [34]

    Stack Overflow. 2025. 2025 Stack Overflow Developer Survey. https://survey.stackoverflow.co/2025. Accessed: 2026-01-26

    2025

  35. [35]

    Jiao Sun, Q Vera Liao, Michael Muller, Mayank Agarwal, Stephanie Houde, Kartik Talamadupula, and Justin D Weisz

  36. [36]

    Tao Sun, Jian Xu, Yuanpeng Li, Zhao Yan, Ge Zhang, Lintao Xie, Lu Geng, Zheng Wang, Yueyan Chen, Qin Lin, et al

  37. [37]

    The Algorithms. 2025. TheAlgorithms/Python. https://github.com/TheAlgorithms/Python. Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA093. Publication date: October 2026. Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review ISSTA093:23

    2025

  38. [38]

    Rosalia Tufano, Ozren Dabić, Antonio Mastropaolo, Matteo Ciniselli, and Gabriele Bavota. 2024. Code review automa- tion: strengths and weaknesses of the state of the art.IEEE Transactions on Software Engineering50, 2 (2024), 338–353. doi:10.1109/TSE.2023.3348172

    arXiv 2024

  39. [39]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. InProceedings of the 31st International Conference on Neural Information Processing Systems(Long Beach, California, USA)(NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 6000–6010

    2017

  40. [40]

    Miku Watanabe, Yutaro Kashiwa, Bin Lin, Toshiki Hirao, Ken’Ichi Yamaguchi, and Hajimu Iida. 2024. On the use of chatgpt for code review: Do developers like reviews by chatgpt?. InProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 375–380. doi:10.1145/3661167.3661183

    arXiv 2024

  41. [41]

    David Gray Widder, Laura Dabbish, James D Herbsleb, Alexandra Holloway, and Scott Davidoff. 2021. Trust in collaborative automation in high stakes software engineering work: A case study at NASA. InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–13. doi:10.1145/3411764.3445650

    arXiv 2021

  42. [42]

    Xuansheng Wu, Haiyan Zhao, Yaochen Zhu, Yucheng Shi, Fan Yang, Lijie Hu, Tianming Liu, Xiaoming Zhai, Wenlin Yao, Jundong Li, Mengnan Du, and Ninghao Liu. 2026. Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era.ACM Trans. Knowl. Discov. Data20, 6, Article 100 (July 2026), 57 pages. doi:10.1145/3816150

    work page doi:10.1145/3816150 2026

  43. [43]

    Lanxin Yang, Jinwei Xu, Yifan Zhang, He Zhang, and Alberto Bacchelli. 2023. Evacrc: Evaluating code review comments. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 275–287. doi:10.1145/3611643.3616245 Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA093. Publicat...

    arXiv 2023

  44. [2022]

    InProceedings of the 27th International Conference on Intelligent User Interfaces

    Investigating explainability of generative AI for code through scenario-based design. InProceedings of the 27th International Conference on Intelligent User Interfaces. 212–228. doi:10.1145/3490099.3511119

    arXiv

  45. [2025]

    InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering

    Bitsai-cr: Automated code review via llm in practice. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 274–285. doi:10.1145/3696630.3728552

    arXiv

  46. [2026]

    doi:10.1109/TSE.2026.3659804

    On the Need to Rethink Trust in AI Assistants for Software Development: A Critical Review .IEEE Transactions on Software Engineering52, 04 (April 2026), 1265–1281. doi:10.1109/TSE.2026.3659804

    arXiv 2026