REVIEW 5 major objections 5 minor 15 references
Rethinking Code Review Workflows with LLM Assistance: An Empirical Study
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Developers prefer AI-led code reviews for large or unfamiliar pull requests, but the preference is context-dependent.
desk verdict A small, honest qualitative study with a genuinely new mode comparison, but the headline 'large PR' preference is speculative and the paper needs a preference tally and PR complexity metrics before the claims can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The experimental vehicle is a web-based chat interface wrapping a large language model with retrieval-augmented generation (RAG). The assistant is built on three semantic search tools: search_pr (pull-request diffs and metadata), search_code (full source files), and search_requirements (the requirement ticket motivating the change). In Mode A (AI-led co-reviewer), a sub-agent called start_review is added; it receives the entire PR context and produces a structured summary before the reviewer begins, and the reviewer can then ask follow-up questions. In Mode B (on-demand assistant), no upfront summary is generated and the reviewer must prompt the assistant explicitly. This difference in proactivity is the independent variable the study varies.
What would settle it
A controlled experiment that assigns the same developers to review multiple pull requests with measured size, complexity, and clarity (for example, comparable lines changed, files touched, and cyclomatic complexity), and that records both stated preference and actual review performance such as defects found and time spent, could determine whether the observed mode preference is driven by the interaction mode or by differences in the pull requests themselves.
Extended reading notes
Core claim
The paper's central claim is that the preferred interaction with an LLM code-review assistant is not a single mode but depends on review context. In the field experiment, participants reviewed two pull requests of similar size and complexity, using the AI-led mode for one and the on-demand mode for the other, with mode assignment rotated. Overall, the AI-led mode was preferred, and it was seen as especially valuable for large or unfamiliar pull requests, where it provided quick orientation and surfaced issues a human might miss. Yet several participants said that for a codebase they already know well, or for changes that carry risk, they would rather keep control with an on-demand assistant. The paper interprets this as evidence that LLM assistance should be adaptive and embedded in existing developer workflows, augmenting rather than replacing human review.
Load-bearing premise
The comparison between the two interaction modes assumes that the two selected pull requests are similar in size and complexity, yet the paper provides no quantitative measures of lines changed, files touched, or requirement complexity, and the pilot used only two internal developers.
Editorial extensions
If this is right
- Tool builders should offer an AI-led summary as the default for large or unfamiliar pull requests, while keeping an on-demand mode available for reviewers who want full control.
- AI review assistance should be embedded directly into the developer's normal platforms (for example, code-hosting platforms, IDEs, or chat tools) rather than a separate interface, since participants cited integration friction as a barrier.
- Response latency and output conciseness are decisive for adoption; at least one participant stated they would not use a tool that takes minutes to reply.
- The usefulness of the assistant depends on access to diff, source, and requirement-ticket context, so production tools should invest in automated retrieval pipelines for these artifacts.
- The same assistant could serve as a pre-review aid, letting authors catch simple issues before submitting a pull request.
Reading between the lines
- If mode preference is genuinely context-dependent, an adaptive tool that automatically selects between proactive and reactive modes based on PR size, codebase familiarity, and risk metadata should outperform any fixed-mode tool; this is a testable extension the paper does not evaluate.
- The participants' worry about over-reliance in the AI-led mode suggests a measurable anchoring effect: a reviewer might fixate on AI-flagged issues and miss unflagged ones. An experiment comparing defect detection rates with and without an upfront AI summary could quantify this risk.
- The paper's RAG pipeline is manually indexed per pull request, so the qualitative benefits it reports may depend on careful indexing rather than on the LLM itself. Automating the indexing step and re-running the experiment would separate these two contributions.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a two-phase, qualitative empirical study conducted at WirelessCar Sweden AB. In Phase 1, semi-structured interviews with seven developers (thematic analysis) identify challenges in existing code review practice: context switching, lack of context, delayed and superficial reviews of large PRs, and mixed views on AI assistance. In Phase 2, ten developers used two variants of an LLM-assisted code review tool in a field experiment: an AI-led co-reviewer mode (Mode A) that proactively generates summaries and findings, and an interactive assistant mode (Mode B) that responds only to on-demand queries. Both modes use a RAG pipeline over PR diffs, source code, and Jira tickets. The paper claims that developers generally preferred the AI-led mode, especially for large or unfamiliar pull requests, but that preferences are context-dependent, with familiarity and perceived risk shaping choices. The authors also report design implications: embed AI in existing tools, keep output concise, support both proactive and reactive modes, and consider pre-review use.
Significance. If the findings are accepted as stated, the paper offers practical guidance for building LLM-assisted code review tools and contributes qualitative evidence from a real industrial setting. Notable strengths include the field context with practicing developers, the use of both observation notes and post-session interviews, the rotation of mode assignment to mitigate ordering effects, and the release of the prototype artifacts under GPLv3. The thematic analysis is careful in places and the paper is candid about its qualitative nature. However, the central preference claim is stronger than the evidence: there is no systematic preference tally, the two PRs are only asserted to be comparable, and the 'large or unfamiliar' scope is based on hypothetical self-reports rather than experimental variation. These issues affect the reliability of the headline results and need to be addressed, either by additional evidence or by appropriately qualified wording.
major comments (5)
- [Section V-B4 and Abstract] The abstract and Section V-B4 state that 'AI-led reviews are overall more preferred,' but the paper does not report a systematic tally of mode preferences across the ten participants. The results present a thematic selection of quotes (e.g., P7, P11, P12 for Mode A; P8 for Mode B in familiar code) and do not tell the reader how many participants preferred Mode A, Mode B, or a combination. Without such a distribution, 'overall more preferred' is indistinguishable from 'a few vivid quotes.' Please provide a participant-level preference summary (e.g., a table or explicit counts) or weaken the claim to 'several participants expressed a preference for the AI-led mode.'
- [Section IV-B2] The comparability of the two pull requests is load-bearing for the comparison of Mode A versus Mode B, yet it is only asserted: the paper says the PRs are 'of similar size and complexity' and were validated by a pilot with two internal developers, but gives no metrics such as lines changed, files touched, requirement complexity, or review effort. If the two PRs differ in difficulty, clarity, or domain, the observed mode preferences could be explained by the PRs themselves. Please report the actual PR characteristics and, ideally, a quantitative similarity justification, or explicitly discuss this threat as a limitation of the mode comparison.
- [Section V-B4 and Section VII] The claim that AI-led reviews are especially preferred 'for large or unfamiliar pull requests' is not supported by the experimental design: both PRs are described as moderate in size, and 'unfamiliar' was not manipulated—the paper compares participants from the owning team with those from other teams, but no systematic familiarity-group comparison is reported. The supporting statements in Section V-B4 are hypothetical self-reports ('if I were in a new team'; 'especially for large PRs, it's nice to get the breakdown'), which express expectations rather than observed behavior in large-PR or objectively unfamiliar contexts. Please reframe these as participant expectations, or add a condition that varies PR size and measures familiarity quantitatively.
- [Section V-A, footnote 6] The paper's own footnote 6 states that 'quotes collected from the interview as well as detailed findings are not listed' for Phase 1. This is a significant transparency limitation because the thematic analysis and the link from RQ1 findings to the design of the two modes cannot be independently assessed. Please provide an appendix or supplementary material containing the full coding scheme, theme definitions, and representative quotes. In addition, the paper should discuss whether the five Phase 2 participants who also took part in Phase 1 may have been influenced by the researchers' earlier design discussions; this potential bias is not acknowledged.
- [Section IV-B2 and Section V-B2] The study has no no-AI baseline condition: participants compared the two AI-assisted modes with each other and with their memory of traditional reviews, not with a contemporaneous manual review of the same PRs. This is acceptable for exploring mode preference, but the paper's language in several places goes further, e.g., 'the AI assistant could speed up the review process' and 'LLMs can meaningfully augment' (Sections V-B2 and I). Because there is no measured comparison against manual review, such statements should be explicitly framed as participant perceptions or hypotheses, not as demonstrated improvements in efficiency or review quality.
minor comments (5)
- [Section V-B2] In the paragraph beginning 'Additionally, interviewers felt that the assistant would be particularly helpful...', the word 'interviewers' should be 'interviewees' or 'participants'.
- [Section IV-B2 and Figure 2] The experiment used two PRs from WirelessCar's codebase, but Figure 2 shows a review of an open-source project (ogen-go/ogen PR 1440). Please clarify in the caption or text whether this screenshot is illustrative and taken from a demo session rather than from one of the experimental PRs.
- [Section V-A] The first sentence of Section V-A has typographical issues: 'When reviewing the informal review process at WirelessCar and their practices ,' contains stray spaces and an awkward phrase; please rephrase and check punctuation throughout the manuscript.
- [Section IV-B2] The sentence 'By having all participants conduct the same two code reviews with alternating modes, this approach allowed for a more controlled comparison...' has a dangling modifier; consider rewriting as 'Having all participants conduct... allowed for...'.
- [References] Reference [12] contains a typo: 'V . Braun and V . C. and' should be 'V. Braun and V. Clarke'.
Circularity Check
No circularity: the paper is an empirical qualitative study with no fitted-input-as-prediction, no self-citation chain, and no derivation that reduces to its own inputs.
full rationale
I reviewed the paper for the seven circularity patterns. This is an empirical, qualitative study with no equations, fitted parameters, or formal derivation chain. The two LLM-assisted review modes were designed from Phase 1 interview findings and then evaluated in Phase 2; that design origin does not make the observed preferences equivalent to the design inputs, because participants' preferences, trust concerns, and mode comparisons were collected independently and are reported alongside disconfirming quotes. There are no self-citations in the reference list, so no self-citation load-bearing or uniqueness-imported-from-authors pattern applies. The main risks are validity threats, not circularity: the two pull requests are asserted to be of similar size and complexity without quantitative support (Sec. IV-B2), and some Phase 2 participants had also participated in Phase 1, which could bias their evaluation. Also, the 'especially for large or unfamiliar pull requests' finding rests partly on hypothetical self-reports rather than directly manipulated large-PR conditions. These are legitimate correctness and generalizability concerns, but they are not cases where a prediction is equivalent to its inputs by construction, and the paper does not present a derivation that reduces to itself. Therefore the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (5)
- domain assumption Thematic analysis is a valid method for deriving themes from the interview data.
- domain assumption Self-reported preferences and think-aloud comments reflect actual review behavior.
- domain assumption Data saturation was reached after seven Phase 1 interviews.
- domain assumption The two selected pull requests are comparable in size and complexity.
- domain assumption The convenience sample of volunteers is representative of the wider reviewer population.
Cite this review
Pith. "Pith review of Rethinking Code Review Workflows with LLM Assistance: An Empirical Study." pith.science (2026). https://pith.science/paper/OXXLX75J
@misc{pith2026250516339,
author = {Pith},
title = {Pith review of: Rethinking Code Review Workflows with LLM Assistance: An Empirical Study},
year = {2026},
howpublished = {\url{https://pith.science/paper/OXXLX75J}},
note = {Machine review of arXiv:2505.16339}
}
read the original abstract
Code reviews are a critical yet time-consuming aspect of modern software development, increasingly challenged by growing system complexity and the demand for faster delivery. This paper presents a study conducted at WirelessCar Sweden AB, combining an exploratory field study of current code review practices with a field experiment involving two variations of an LLM-assisted code review tool. The field study identifies key challenges in traditional code reviews, including frequent context switching, insufficient contextual information, and highlights both opportunities (e.g., automatic summarization of complex pull requests) and concerns (e.g., false positives and trust issues) in using LLMs. In the field experiment, we developed two prototype variations: one offering LLM-generated reviews upfront and the other enabling on-demand interaction. Both utilize a semantic search pipeline based on retrieval-augmented generation to assemble relevant contextual information for the review, thereby tackling the uncovered challenges. Developers evaluated both variations in real-world settings: AI-led reviews are overall more preferred, while still being conditional on the reviewers' familiarity with the code base, as well as on the severity of the pull request.
Figures
Reference graph
Works this paper leans on
-
[1]
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023
arXiv 2023
-
[2]
A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei et al. , “Qwen2.5 technical report,” arXiv preprint arXiv:2412.15115, 2024
arXiv 2024
-
[3]
Gemma: Open models based on gemini research and technology,
T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivi `ere, M. S. Kale, J. Love et al. , “Gemma: Open models based on gemini research and technology,” arXiv preprint arXiv:2403.08295 , 2024
arXiv 2024
-
[4]
Large language models for software engi- neering: A systematic literature review,
X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 8, pp. 1–79, 2024
2024
-
[5]
Deep learning-based code reviews: A paradigm shift or a double-edged sword?
R. Tufano, A. Martin-Lopez, A. Tayeb, S. Haiduc, G. Bavota et al. , “Deep learning-based code reviews: A paradigm shift or a double-edged sword?” arXiv preprint arXiv:2411.11401 , 2024
arXiv 2024
-
[6]
Improving automated code reviews: Learning from experience,
H. Y . Lin, P. Thongtanunam, C. Treude, and W. Charoenwet, “Improving automated code reviews: Learning from experience,” in International Conference on Mining Software Repositories (MSR) . ACM, 4 2024, pp. 278–283
work page 2024
-
[7]
AI-assisted assessment of coding practices in modern code review,
M. Vijayvergiya, M. Salawa, I. Budiseli ´c, D. Zheng, P. Lamblin, M. Ivankovi´c, J. Carin, M. Lewko, J. Andonov, G. Petrovi ´c, D. Tarlow, P. Maniatis, and R. Just, “AI-assisted assessment of coding practices in modern code review,” in Proceedings of the 1st ACM International Conference on AI-Powered Software . Association for Computing Machinery, 2024, pp. 85–93
work page 2024
-
[8]
AI-powered code review with LLMs: Early results,
Z. Rasheed, M. A. Sami, M. Waseem, K.-K. Kemell, X. Wang, A. Nguyen, K. Syst ¨a, and P. Abrahamsson, “AI-powered code review with LLMs: Early results,” arXiv preprint arXiv:2404.18496 , 2024
arXiv 2024
Show all 15 references
-
[9]
Human and machine: How software engineers perceive and engage with AI-assisted code reviews compared to their peers,
A. Alami and N. A. Ernst, “Human and machine: How software engineers perceive and engage with AI-assisted code reviews compared to their peers,” arXiv preprint arXiv:2501.02092 , 2025
2025 arXiv
-
[10]
Towards automating code review activities,
R. Tufano, L. Pascarella, M. Tufano, D. Poshyvanyk, and G. Bavota, “Towards automating code review activities,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 163–174
2021
-
[11]
A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,
L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin et al. , “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” ACM Transactions on Information Systems , vol. 43, no. 2, pp. 1–55, 2025
2025
-
[12]
Using thematic analysis in psychology,
V . Braun and V . C. and, “Using thematic analysis in psychology,” Qualitative Research in Psychology , vol. 3, no. 2, pp. 77–101, 2006
2006
-
[13]
Research and scholarly methods: Semi-structured interviews,
O. A. Adeoye-Olatunde and N. L. Olenik, “Research and scholarly methods: Semi-structured interviews,” JACCP, vol. 4, no. 10, pp. 1358– 1367, 2021
2021
-
[14]
Comparison of convenience sampling and purposive sampling,
I. Etikan, S. A. Musa, R. S. Alkassim et al., “Comparison of convenience sampling and purposive sampling,” American journal of theoretical and applied statistics, vol. 5, no. 1, pp. 1–4, 2016
2016
-
[15]
How many interviews are enough?: An experiment with data saturation and variability,
G. Guest, A. Bunce, and L. Johnson, “How many interviews are enough?: An experiment with data saturation and variability,” Field Methods, vol. 18, no. 1, pp. 59–82, 2006
2006
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.