Automated Code Review Using Large Language Models at Ericsson: An Experience Report

A K Raghavan; Giriprasad Sridhara; Hamender Singh; Joy Bose; Nishrith Saini; Ricardo Britto; Shweta Ramesh; Sujoy Roychowdhury

REVIEW 4 major objections 5 minor 10 references

Automated Code Review Using Large Language Models at Ericsson: An Experience Report

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a lightweight pipeline—program analysis to extract the enclosing method of changed lines plus engineered prompts to an open-source LLM—can generate useful code reviews, with no fine-tuning, at about five to six…

desk verdict A useful but thinly evaluated industry experience report on LLM code review; the reported numbers do not back the 'significantly enhance' claim. read the letter →

arxiv 2507.19115 v2 pith:Y7ZI2YVC submitted 2025-07-25 cs.SE cs.AI

Shweta Ramesh , Joy Bose , Hamender Singh , A K Raghavan , Sujoy Roychowdhury , Giriprasad Sridhara , Nishrith Saini , Ricardo Britto This is my paper · ORCID

classification cs.SEcs.AI

keywords automatedcodereviewlargelanguagemodelsstaticprogramanalysisenclosingmethodextractionpromptengineeringtoolJavaexpertsurvey

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is an experience report arguing that automated code review can be made practical without expensive model training or commercial APIs. The authors built a lightweight tool that takes Java code changes from a version-control system, uses static program analysis to find the enclosing method of each changed line, and sends that method to an open-source LLM with prompts designed to force short, critical, human-like comments. They report that this context is the key: naive whole-diff prompts produce verbose or hallucinated reviews, while the enclosing-method prompt keeps the LLM focused on the actual changes. If the report is right, teams can add a review assistant that returns feedback in seconds, at near-zero marginal cost, and that can be tuned by prompt feedback from senior engineers rather than by fine-tuning.

What carries the argument

The central mechanism is the enclosing-method extraction step. The tool parses the Java file, builds an abstract syntax tree, visits method declarations, and identifies the method that contains each changed line; the LLM then receives that method together with the diff and a prompt that restricts the output to short critical comments and forbids new code generation. The enclosing method, not the whole file, is what keeps the review relevant and cheap. Prompt variants—simple, detailed, security-focused, few-shot, and issue-topic—and a post-processing step that stores, summarizes, and ranks the generated reviews complete the pipeline.

What would settle it

Run a blind controlled study where the tool and experienced reviewers examine the same set of real changes, and count actionable true-positive comments and missed defects; if the tool's precision is no higher than random comment generation, or if independent senior reviewers rate its comments as no better than a simple baseline, the central claim would be refuted.

Watch

Extended reading notes

Core claim

In the paper's own terms, the central discovery is that a small amount of contextual information—the enclosing Java method of the modified lines, obtained by parsing the code into an abstract syntax tree—makes an open-source LLM's review useful. With that context and disciplined prompts, the model generates concise review comments that address the changed lines rather than describing the code or inventing names. In a survey of eight senior engineers who rated reviews of ten real code snippets, comments were mostly positive or neutral, with negative comments tied to wrong variable types, missing abstractions, and verbosity. A pairwise comparison among four open-source models favored Code Llama 13B. The paper also reports that adversarial injections of easy logical bugs were caught in all tested snippets, that comments stayed on changed lines, and that generation took roughly five to six seconds even for large snippets.

Load-bearing premise

The whole case for usefulness rests on whether the responses of a handful of senior engineers at one organization—who had already helped recalibrate the prompts—are a fair measure of code-review quality.

Editorial extensions

If this is right

A development team can deploy a pre-commit review bot that returns comments in seconds without fine-tuning a model or sending code to an external API.
The approach's cost and latency stay roughly constant for snippets of different lengths, because the model sees the enclosing method rather than the whole file.
Prompt calibration using feedback from senior engineers can substitute for fine-tuning in settings where annotated review datasets are unavailable.
The same enclosing-method strategy should transfer to other languages whose abstract syntax trees expose function boundaries.
Automated reviews can catch easy logical bugs and avoid commenting on unchanged lines, which removes a common source of noise in AI-assisted review.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper measures perceived usefulness, not defect-detection precision; a natural next step is a controlled study that counts how many AI comments are actionable true positives against a known set of introduced bugs.
If the enclosing-method trick is the main driver of quality, then adding richer context—call graphs, history, or related functions—may hit diminishing returns, which would keep the approach cheap to operate.
A testable extension is to measure whether review comments produced this way change developer behavior: for example, whether code submitted after such reviews has measurably fewer follow-up fixes than code reviewed without the tool.
Because the evaluation is small and the raters helped tune the prompts, the strongest inference available is that the approach is deployable, not that one model is intrinsically better than another.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is an honest experience report about a lightweight LLM code-review pipeline at Ericsson, and the pipeline itself—diff plus Tree-sitter enclosing method plus prompted open-weight LLM—is a sensible engineering composition. The new material is the deployment story and the survey numbers. But the headline claim, that the approach can 'significantly enhance the quality of code reviews,' is not supported by the paper's own evidence. Table I shows more negative than positive expert comments (10 vs 6, with 8 neutral), and RQ3 found only 4 of 9 users thought it saved time. Those are not encouraging numbers on their face, and there is no baseline or control, so the comparative 'enhance' is undefined.

What the paper does well: it is transparent about being preliminary, it describes the prompts and pipeline in enough detail to reproduce, it reports honest user feedback including negatives, and it gives timing (~5–6 seconds) and a small adversarial check. The decision to avoid fine-tuning and external APIs is a real practical constraint in industry, and the enclosing-method context via TreeSitter is a simple idea that readers building similar tools will try. The citation pattern is fine; related work on fine-tuning versus prompting and prior industrial deployments is relevant, and the contrast with the black-box tool in [2] is apt.

Soft spots, in order of severity. First, the central claim overreaches; soften it or bring in an independent benchmark. Second, Table II's row totals sum to 22, not the 36 pairwise evaluations stated in the text; that needs correcting. Third, the evaluation loop is partly circular: prompts were re-calibrated using expert feedback (Section III.B) before the same kind of expert judgment was used as the oracle. The stress-test note is right that expert contamination is secondary; the bigger issue is that even taken at face value, the numbers do not support the claim. Fourth, no released artifacts, which limits outside verification but is common for industrial reports.

This paper is for practitioners building similar in-house review tools and for researchers who want a realistic picture of LLM review adoption constraints. It deserves a serious referee, but the referee should send it back for revision: fix the arithmetic, restate the claims in line with the evidence, and ideally add a fresh independent evaluation before publication.

Referee Report

4 major / 5 minor

Summary. The paper reports on Ericsson's experience building a lightweight automated code review pipeline that combines static program analysis with open-source large language models (Llama 2 and Code Llama variants). The pipeline extracts the enclosing method of changed Java lines using Tree-sitter, feeds it to an LLM with engineered prompts, and presents the generated review in Gerrit and VS Code. The evaluation consists of three surveys: expert comments on LLM-generated reviews for ten code snippets, pairwise comparisons of four LLMs, and a fifteen-day usage study with nine developers. The authors claim the approach can significantly enhance code review quality, reduce cognitive load, and streamline development, while emphasizing the lightweight, low-cost, and integration-friendly nature of the solution.

Significance. If the central claim were supported, the paper would offer a useful practical data point for industrial adoption of LLM-based code review without fine-tuning or external cloud APIs. The paper's strengths include a clearly described pipeline with detailed prompts, explicit reporting of latency (about five to six seconds per review), and checks that the model comments only on changed lines. It is also transparent that the work is preliminary. However, the evidence base is thin and partly self-referential: the number of expert respondents is small, no baseline or manual-review comparison is reported, the reported vote totals are internally inconsistent, and the authors' own data show mixed and sometimes negative results. The significance of the paper is therefore contingent on the authors reframing the claims to match the evidence or substantially strengthening the evaluation.

major comments (4)

[§I and §VI.A/§VI.C] The introduction claims that the approach 'can significantly enhance the quality of code reviews,' but the reported evidence does not support that wording. In Table I, negative expert comments (10) outnumber positive comments (6), and in the RQ3 survey only 4 of 9 developers agreed that the tool saved time, only 4 of 9 rated it effective or highly effective, and only 2 of 9 used it regularly. These are not the numbers one would expect to support a claim of significant enhancement. Please soften the central claim to a preliminary feasibility result, or provide stronger evidence such as independent expert ratings and a comparison against a human-review baseline.
[§VI.B, Table II] The text states that 9 experts each made 4 comparisons, totaling 36 pairwise evaluations, but the vote counts in Table II sum to 22. This arithmetic inconsistency is material to RQ2 and must be corrected. Moreover, each row of Table II contains at most five votes, so the conclusion that 'the Code llama 13B model seemed to be better' is anecdotal; no statistical test is reported. Please either report the complete, consistent vote counts and apply an appropriate significance test, or explicitly label the finding as preliminary and non-quantitative.
[§III.B and §III.C] There is a validation loop that weakens the causal claim: the prompts were re-calibrated using expert feedback (Section III.B) before the same kind of expert judgment was used as the evaluation oracle (Section III.C), and no external benchmark or manual-review baseline is reported. Because the tool's output is compared only against expert impressions of that same output, the study cannot establish the comparative claim that the tool 'enhances' review quality relative to existing human review. Please clarify the independence of the evaluation experts from the prompt-tuning process, add a baseline condition, or restrict the conclusions to feasibility and usability rather than quality enhancement.
[§VI 'other experiments'] The additional experiments on adversarial prompting, unchanged-line commenting, and latency are reported too thinly to be assessed. In particular, the adversarial-prompting experiment gives no protocol, no sample details beyond snippet lengths, and no comparison condition, while the unchanged-line check is described as covering only 5 change IDs. Please provide the details needed to interpret these results, or remove them from the evaluation section.

minor comments (5)

[§VI.C] The paper mentions that users can give feedback via buttons in the tool, but no results from that feedback mechanism are reported; either present the data or remove the mention.
[§VII] The discussion states that 'Real-world developer surveys validate that our method reduces redundant feedback,' but no survey question in Section VI.C asks about redundant feedback, and the reported results do not directly support this claim.
[§I] There is a typo in the paper organization paragraph: 'Section VII synthesizes the analyze of our preliminary results' should presumably read 'analysis.'
[Table I] The counts in Table I should be tied to the number of respondents and snippets more explicitly; as printed, the row and column totals are not clearly explained, and the reader cannot tell whether the counts are comments, snippets, or experts.
[References] The reference list uses inconsistent formatting, with some entries containing mixed straight and curly quotes and incomplete venue information; please standardize according to the journal style.

Circularity Check

1 steps flagged · score 3.0 of 10

Mild validation loop: prompts re-calibrated on expert feedback, then the same kind of expert feedback is used as the evaluation oracle; otherwise no significant circularity.

fitted input called prediction [Section III.B (Solution Development) and Section VI.A (RQ1)]
"We further developed a validation mechanism to fine-tune the reviews generated, including summarizing and Ranking reviews, storing reviews generated by LLMs and validating them through human experts, and re-calibrating prompts using expert feedback. ... The experts were invited to provide their comments for each of these 10 reviews: such as which parts of the review are relevant and which parts are not."

The prompts and validation mechanism were tuned using expert feedback before evaluation, and the evaluation then uses the same kind of expert judgment as the oracle for review quality. Thus the concluding claim that the approach 'can significantly enhance the quality of code reviews' is assessed against the very signal used to select the prompts. This is a partial validation loop rather than a formal derivation: the experts could have rejected the outputs (and the reported data are mixed), so it is not definitional, but it means the evaluation does not provide independent evidence of quality.

full rationale

This is an experience report rather than a formal derivation, so most circularity patterns do not apply. There is no self-citation chain: all references are external. There is no imported uniqueness theorem, no ansatz smuggled through citations, and no renaming of a known result as a new one. The only noticeable circular element is the evaluation loop: Section III.B states that prompts were 're-calibrating prompts using expert feedback' as part of the system's validation mechanism, and Section VI.A evaluates the LLM-generated reviews by asking expert developers to comment on relevance and accuracy. The 'encouraging' conclusion in Section VI is therefore partly a reflection of the tuning objective rather than an independent confirmation. The effect is mild because the expert ratings were not forced to be positive (Table I records more negative than positive comments, and RQ3 is mixed), so the central claim is not equivalent to an input by construction. Score 3 reflects one partial validation loop; the rest of the paper's contribution (the enclosing-method pipeline and its deployment) is self-contained.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim depends on engineering assumptions rather than mathematical axioms: the enclosing-method context suffices for LLM review quality, expert survey responses are a valid oracle, and prompt engineering removes the need for fine-tuning. There are no fitted numerical parameters and no invented entities.

assumptions (3)

domain assumption The enclosing method of changed lines, extracted via TreeSitter, provides sufficient context for the LLM to generate relevant reviews.
Section V step 3 makes this the core input; no controlled variation of context is reported.
domain assumption Expert developer survey responses are a valid oracle for code review quality and usefulness.
Section VI uses judgments from eight or nine experienced developers as the evidence that the tool works, with no independent metric such as defect detection rate.
domain assumption Open-source LLMs with engineered prompts can produce accurate, concise reviews without fine-tuning.
The entire approach avoids fine-tuning, as stated in Sections III.B and V; the survey provides qualitative support only.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automated Code Review Using Large Language Models at Ericsson: An Experience Report." pith.science (2026). https://pith.science/paper/Y7ZI2YVC

@misc{pith2026250719115,
  author       = {Pith},
  title        = {Pith review of: Automated Code Review Using Large Language Models at Ericsson: An Experience Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y7ZI2YVC}},
  note         = {Machine review of arXiv:2507.19115}
}

read the original abstract

Code review is one of the primary means of assuring the quality of released software along with testing and static analysis. However, code review requires experienced developers who may not always have the time to perform an in-depth review of code. Thus, automating code review can help alleviate the cognitive burden on experienced software developers allowing them to focus on their primary activities of writing code to add new features and fix bugs. In this paper, we describe our experience in using Large Language Models towards automating the code review process in Ericsson. We describe the development of a lightweight tool using LLMs and static program analysis. We then describe our preliminary experiments with experienced developers in evaluating our code review tool and the encouraging results.

Figures

Figures reproduced from arXiv: 2507.19115 by the authors.

**Figure 1.** Research design structure. ”automated code review using LLMs”, ”review code using large language models”, ”LLM in software engineering” and ”extracting the bounding functions of diff lines in a change request”. Our objective was to identify gaps in existing solutions for automated code review, both in academic and industrial contexts. Based on the results, we formulated our research questions, focusing on how LLMs … view at source ↗

**Figure 2.** Steps of the code review pipeline. V. SOLUTION OVERVIEW The code review pipeline is illustrated in figure 2. The steps of the end to end code review pipeline are as follows [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Web based user interface for the code review tool. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Screenshot of the popup menu in the VS Code plugin of the code [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 6 canonical work pages

[2]

A survey on evaluating large language models in code generation tasks,

L. Chen, Q. Guo, H. Jia, Z. Zeng, X. Wang, Y. Xu, J. Wu, Y. Wang, Q. Gao, J. Wang, and W. Ye, "A survey on evaluating large language models in code generation tasks," arXiv preprint arXiv:2408.16498, Aug. 2024

arXiv 2024
[1]

Usage: java MainExample <input-file> <output-directory>

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page
[3]

Automated Code Review In Practice,

U. Cihan, V. Haratian, A. Ic¸oz, M. K. G ¨ ul, D. ¨ O. Devran, E. F. Bayendur, B. M. Uc¸ar, and E. Tuz¨ un, "Automated Code Review In Practice," arXiv preprint arXiv:2412.18531, Dec. 2024

arXiv 2024
[4]

Large language models for code analysis: Do LLMs really do their job?

C. Fang, N. Miao, S. Srivastav, J. Liu, R. Zhang, R. Fang, R. Tsang, N. Nazari, H. Wang, and H. Homayoun, "Large language models for code analysis: Do LLMs really do their job?" in 33rd USENIX Security Symposium (USENIX Security 24), pp. 829-846, 2024

work page 2024
[5]

Fine-tuning and prompt engineering for large language models-based code review automation,

C. Pornprasit and C. Tantithamthavorn, "Fine-tuning and prompt engineering for large language models-based code review automation," Information and Software Technology, vol. 175, p. 107523, Nov. 2024

work page 2024
[6]

Ai-powered code review with llms: Early results,

Z. Rasheed, M. A. Sami, M. Waseem, K. K. Kemell, X. Wang, A. Nguyen, K. Systa¨, and P. Abrahamsson, "Ai-powered code review with llms: Early results," arXiv preprint arXiv:2404.18496, Apr. 2024

arXiv 2024
[7]

Significant productivity gains through programming with large language models,

T. Weber, M. Brandmaier, A. Schmidt, and S. Mayer, "Significant productivity gains through programming with large language models," Proceedings of the ACM on Human-Computer Interaction, vol. 8, no. EICS, pp. 1-29, Jun. 2024

work page 2024
[8]

Where Are Large Language Models for Code Generation on GitHub?,

X. Yu, L. Liu, X. Hu, J. W. Keung, J. Liu, and X. Xia, "Where Are Large Language Models for Code Generation on GitHub?," arXiv preprint arXiv:2406.19544, Jun. 2024

arXiv 2024

Show all 10 references

[9]

Fine-tuning large language models to improve accuracy and comprehensibility of automated code review,

Y. Yu, G. Rong, H. Shen, H. Zhang, D. Shao, M. Wang, Z. Wei, Y. Xu, and J. Wang, "Fine-tuning large language models to improve accuracy and comprehensibility of automated code review," ACM Transactions on Software Engineering and Methodology, vol. 34, no. 1, pp. 1-26, Dec. 2024

2024
[10]

The use of large language models for program repair,

F. Zubair, M. Al-Hitmi, and C. Catal, "The use of large language models for program repair," Computer Standards and Interfaces, vol. 103951, Nov. 2024

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [2]

A survey on evaluating large language models in code generation tasks,

L. Chen, Q. Guo, H. Jia, Z. Zeng, X. Wang, Y. Xu, J. Wu, Y. Wang, Q. Gao, J. Wang, and W. Ye, "A survey on evaluating large language models in code generation tasks," arXiv preprint arXiv:2408.16498, Aug. 2024

arXiv 2024

[2] [1]

Usage: java MainExample <input-file> <output-directory>

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page

[3] [3]

Automated Code Review In Practice,

U. Cihan, V. Haratian, A. Ic¸oz, M. K. G ¨ ul, D. ¨ O. Devran, E. F. Bayendur, B. M. Uc¸ar, and E. Tuz¨ un, "Automated Code Review In Practice," arXiv preprint arXiv:2412.18531, Dec. 2024

arXiv 2024

[4] [4]

Large language models for code analysis: Do LLMs really do their job?

C. Fang, N. Miao, S. Srivastav, J. Liu, R. Zhang, R. Fang, R. Tsang, N. Nazari, H. Wang, and H. Homayoun, "Large language models for code analysis: Do LLMs really do their job?" in 33rd USENIX Security Symposium (USENIX Security 24), pp. 829-846, 2024

work page 2024

[5] [5]

Fine-tuning and prompt engineering for large language models-based code review automation,

C. Pornprasit and C. Tantithamthavorn, "Fine-tuning and prompt engineering for large language models-based code review automation," Information and Software Technology, vol. 175, p. 107523, Nov. 2024

work page 2024

[6] [6]

Ai-powered code review with llms: Early results,

Z. Rasheed, M. A. Sami, M. Waseem, K. K. Kemell, X. Wang, A. Nguyen, K. Systa¨, and P. Abrahamsson, "Ai-powered code review with llms: Early results," arXiv preprint arXiv:2404.18496, Apr. 2024

arXiv 2024

[7] [7]

Significant productivity gains through programming with large language models,

T. Weber, M. Brandmaier, A. Schmidt, and S. Mayer, "Significant productivity gains through programming with large language models," Proceedings of the ACM on Human-Computer Interaction, vol. 8, no. EICS, pp. 1-29, Jun. 2024

work page 2024

[8] [8]

Where Are Large Language Models for Code Generation on GitHub?,

X. Yu, L. Liu, X. Hu, J. W. Keung, J. Liu, and X. Xia, "Where Are Large Language Models for Code Generation on GitHub?," arXiv preprint arXiv:2406.19544, Jun. 2024

arXiv 2024