{"id":"7141658a-f49a-452e-9a93-e4023a7d6409","arxiv_id":"2411.11401","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Starting from a ChatGPT-generated code review biases reviewers toward flagged code locations, increases low-severity findings, and does not save time or raise confidence compared with manual review.","lead":"This paper ran a controlled experiment with 29 professional developers to test whether starting from an LLM-generated code review changes what reviewers find, how long they take, and how confident they feel. It found that automated reviews steer attention to the locations the LLM flagged, add low-severity findings, do not improve high-severity issue detection, and do not save time.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Strong negative claims (no saved time, no confidence gain, no high-severity benefit) rest on non-significant results in a study the authors themselves call underpowered; absence of evidence is treated as evidence of absence.","rationale":"The reader's weakest_assumption identifies exactly the most load-bearing concern: non-significant results in an underpowered sample are being reported as no effect. This concern applies to the central negative claims of the paper, not just to a peripheral detail. The abstract's factual phrasing 'did not result in saved time' and 'did not increase the reviewers' confidence' goes beyond what a failure to reject the null can establish, especially when the observed time differences point in the opposite direction (ACR and CCR took longer than MCR). The same statistical fragility affects the RQ1 conclusion that automated review does not help identify high-severity issues: the logistic regression coefficient for ACR is essentially zero, but with 24 reviews per treatment and multiple controls, the confidence interval is wide. A well-powered equivalence analysis using the released per-review data would settle whether 'no effect' is a defensible claim or merely an absence-of-evidence statement. The paper is otherwise careful: the manipulation is clearly described, the replication package is provided, and the authors explicitly flag the power limitation in Section IV. However, the abstract and conclusion do not carry that caveat, which is why the concern is load-bearing rather than merely cosmetic. I agree with the reader's conditional verdict; the central empirical claims are plausible but the strongest negative claims require either a reanalysis with equivalence bounds or more data, so the verdict should remain CONDITIONAL.","tokens_in":20301,"tokens_out":7789,"duration_ms":83032,"concrete_test":"Using the replication package's per-review data, re-analyze RQ2 and RQ3 with two one-sided tests (TOST) for equivalence: set a priori equivalence bounds of ±5 minutes for total time and ±0.5 points for confidence (1-5 scale). If the 90% CI for the treatment difference falls entirely within these bounds, the 'no effect' phrasing is supported; if it does not, the result is indeterminate and the abstract must be reworded to 'we found no statistically significant effect' rather than 'did not result in saved time.' For RQ1, compute the 95% CI for the ACR-vs-MCR odds ratio on identifying injected issues; if the CI includes OR=1.5, the claim of no high-severity benefit is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline negative claims—that automated review 'did not result in saved time,' 'did not increase the reviewers' confidence,' and did not help identify more high-severity issues—are all statements of no effect derived from frequentist non-significance. The authors themselves concede in Section IV (External validity) that 'our study could be statistically underpowered, thus leading to biased conclusions.' With 24 reviews per treatment and high variance (total time means: MCR=42 min, ACR=56, CCR=57; confidence means 3.5/3.7/3.8), the design has limited power to detect moderate but practically meaningful effects. For RQ1, the ACR coefficient is -0.004 (SE=0.351), so the 95% CI for the odds ratio likely spans values corresponding to both meaningful harm and meaningful benefit. Interpreting these non-significant results as evidence of 'no difference' conflates 'statistically indistinguishable' with 'no effect.' This is the load-bearing step: if a larger or better-powered study found e.g., a one-third reduction in high-severity misses or a five-minute time saving, the abstract's conclusion that automation is a double-edged sword without efficiency benefits would need substantial revision.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a controlled experiment with 29 professional (or formerly professional) developers who performed 72 code reviews across six Java/Python projects under three treatments: fully manual review (MCR), review with a ChatGPT-generated starting review (ACR), and review with a 'comprehensive' starting review containing all injected issues (CCR). The authors measure review quality, review time, and reviewer confidence, and report three classes of findings: (i) reviewers kept, on average, 89% of the issues in the automated review; (ii) providing an automated review anchors reviewers to the code locations highlighted by the tool, leading to more low-severity issues but not more injected high-severity issues; and (iii) automated support did not save time and did not increase confidence. The paper concludes with recommendations for reviewers, tool designers, and researchers.","tokens_in":20514,"tokens_out":6804,"duration_ms":69505,"significance":"If the findings hold, the paper makes a timely and important empirical contribution: it is among the first controlled studies of LLM-generated reviews as starting points in human code review, and its observational evidence about anchoring and issue acceptance is valuable for both practitioners and tool designers. The study has notable strengths: 29 industrial participants, over 50 hours of IDE-monitored review activity, double-coding of the 72 submitted reviews, three treatments including a best-case 'comprehensive' reference condition, and a public replication package. The descriptive results, especially the 89% acceptance rate and the line-coverage Venn analysis, support the behavioral claim that reviewers tend to stay close to the LLM-provided comments. However, the headline negative claims about time savings and confidence are inferred from non-significant regression results in a small sample, which the authors themselves acknowledge may be underpowered (Section IV). The paper's contribution would be strengthened substantially by calibrating those claims to what the design can actually support.","major_comments":[{"comment":"The headline negative claims rest on non-significant results in a small sample. For RQ2, the time models are only reported in the replication package; the text states that none of the independent variables is statistically significant and then concludes that automation 'did not result in saved time' and 'debunks one of the motivations for automated code review.' For RQ3, Table V gives ACR=0.099 (SE=0.211) and CCR=0.180 (SE=0.212); the statement 'does not have a significant effect' is statistically accurate, but the abstract's 'did not increase the reviewers' confidence' goes beyond it. The same issue appears in RQ1: the ACR coefficient in Table IV is -0.004 (SE=0.351), so the confidence interval is compatible with both meaningful benefit and meaningful harm for high-severity issue identification. The authors themselves write in Section IV that 'our study could be statistically underpowered, thus leading to biased conclusions.' With 24 reviews per treatment, non-significance is not evidence of absence. I ask the authors to re-express these results with effect sizes and confidence intervals (or equivalence tests) and to soften the abstract and conclusions to forms such as 'we did not detect a significant time saving' rather than 'the automated support did not result in saved time.'","section":"III-C, III-D, IV (and abstract/conclusion)"},{"comment":"The claim that ACR identifies more low-severity issues but not more high-severity issues depends on the severity ratings of two external developers. The reported weighted kappa is 0.315, which is conventionally only 'fair' agreement, making the conclusion fragile. In addition, the severity comparison is made across different pools of issues (injected issues vs. additional ChatGPT issues vs. additional manual issues), which is not the same as a direct comparison of severity as a function of treatment. The logistic model with severity as a cofactor is said to leave results unchanged, but the model output is not shown in the paper. I recommend reporting the full model, detailed rater-agreement information, and a sensitivity analysis that treats the two raters' severity scores separately.","section":"III-B (severity analysis)"},{"comment":"The behavioral anchoring claim is central to the paper, but it is supported only by a descriptive Venn diagram and a single example. The statement that reviews written without automated support 'had a higher variability' is not backed by any statistical comparison of line-coverage overlap or dispersion across treatments. I suggest adding an explicit metric (for example, per-review overlap or Jaccard similarity, variance of covered locations, or a mixed-effects model on coverage) with confidence intervals, or explicitly labeling this result as a descriptive observation rather than a strong behavioral conclusion.","section":"III-A (RQ0, Figs. 3 and 4)"}],"minor_comments":[{"comment":"The text reports removed data points as '4 for NCR, 5 for ACR, and 4 for CCR'; 'NCR' appears to be a typo for 'MCR' and should be corrected.","section":"IV (Threats to validity)"},{"comment":"The abstract says '29 experts,' while Section II-A describes 29 participants, including one who is not currently a professional developer; please use consistent terminology.","section":"Abstract and II-A"},{"comment":"The text uses 'weighted k agreement' and cites Cohen [41]; please use the standard symbol kappa and, if conventional benchmarks are used, describe 0.315 as 'fair' agreement.","section":"III-B"},{"comment":"Reference [26] is given as 'M. V. et al.' with an incomplete author list; please provide the full author names.","section":"References"},{"comment":"The numbers around the Venn diagram are difficult to parse in the printed figure; a legend or a small table of overlap values would improve readability.","section":"III-A, Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is valid and lands directly on the abstract and conclusions: the paper's negative claims about time, confidence, and high-severity issues are inferred from non-significant results in a study the authors themselves describe as possibly underpowered. The descriptive contributions (issue acceptance, anchoring, line coverage) are solid and could justify acceptance after the inferential claims are recalibrated and the severity analysis is made more transparent. I do not see a circularity issue; the empirical design is appropriate for the behavioral questions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — the headline is that this is a carefully run controlled experiment with a genuinely new independent variable, and the part that will survive contact with referees is the anchoring result. Prior work manipulated file order, gamification, or the presence of existing human comments; nobody had manipulated the presence of an LLM-generated review as the starting point. The authors recruited professional developers, monitored IDE activity, manually inspected all 72 reviews, and shipped the material. The finding that reviewers kept 89% of the LLM's comments and concentrated on the code locations it flagged, rather than exploring the rest of the code, is descriptive, visible in the line-coverage Venn diagram, and corroborated by the inflated number of low-severity issues. That is a real contribution.\n\nThe soft spot is the negative claims. The abstract says the automated support 'did not result in saved time' and 'did not increase the reviewers' confidence,' but those are non-significant regression coefficients in a study the authors themselves describe as possibly underpowered. With 24 reviews per treatment and large variance (42 vs. 56 vs. 57 minutes), the confidence intervals are wide enough that moderate effects would not be detected. The high-severity comparison in RQ1 has the same issue: ACR's coefficient is -0.004 with SE 0.351, so the interval covers both meaningful harm and meaningful benefit. The framing in the conclusion goes beyond what a null result can support. The anchoring story does not depend on these nulls, so a careful revision that explicitly labels them as inconclusive and softens the abstract would fix most of the problem.\n\nMinor flags: the CCR treatment deceptively tells participants that a rephrased human review was machine-generated, and the paper does not mention debriefing. That deserves an ethical note in the methods. The severity rating has a weighted kappa of 0.3, which is weak agreement, though the authors report it transparently and the severity comparison is not the main load-bearing result.\n\nWho should read it: anyone studying human-AI interaction in software engineering, and anyone designing co-review tools. The value is in the behavioral evidence about anchoring, not in the efficiency nulls. I'd send it to a serious referee, and I'd cite the anchoring finding. For the next reading group, I'd put it on the list — it's a good example of a well-scoped experiment with clearly reported limitations, and a useful case study in how non-significance gets overinterpreted.","headline":"A genuinely new controlled experiment on LLM-assisted code review, where the anchoring result is solid and the 'no time saved / no confidence gain' claims outrun the statistical power.","tokens_in":21044,"tokens_out":2458,"would_cite":true,"duration_ms":60834,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-generated code reviews anchor reviewers to flagged lines without saving time or boosting confidence.","keywords":["code review","large language models","controlled experiment","reviewer bias","anchoring","review quality","review time","reviewer confidence"],"falsifier":"A pre-registered replication with a larger sample (e.g., at least 100 reviews per treatment) that observes ACR reviewers finding significantly more injected high-severity issues than MCR reviewers, or completing reviews in significantly less time, would undercut the paper's null conclusions. A cheaper check: if reviewers in the ACR condition, when their gaze is tracked, spend a substantial share of time outside the LLM-flagged lines and find new issues there, the anchoring mechanism would not explain the results.","tokens_in":20123,"feed_emoji":"🔍","tokens_out":5786,"duration_ms":55981,"temperature":0.7,"pith_summary":"An automated code review generated by an LLM, when handed to a professional developer as a starting point, does not work as an accelerator or accuracy booster: it changes where the reviewer looks. In a controlled experiment with 29 professional developers and 72 reviews, reviewers kept 89% of the LLM-flagged issues on average, but concentrated on those flagged code locations instead of searching the rest of the code. Reviews produced with the LLM starting point contained more low-severity issues than fully manual reviews, yet did not uncover more high-severity (injected) issues. The automated support also did not reduce review time, even when the provided review was perfect, and did not increase reviewers' confidence. The paper concludes that current LLM co-reviewers are best used as a complement after the human review, not as the starting point.","feed_headline":"LLM code reviews anchor developers but save no time","feed_subtitle":"A 29-developer experiment finds AI review comments boost low-severity findings, not high-severity ones.","key_machinery":"Three treatments form the experiment's core mechanism: MCR (no support), ACR (a ChatGPT Plus review as the starting point), and CCR (a knowingly perfect review, rephrased by ChatGPT, so issues could not be dismissed as LLM errors). Into 12 small Java and Python programs the authors injected 48 quality issues; this lets them measure hit rates on known high-severity defects and separately count additional issues found. IDE telemetry records time on reading, writing, and code inspection, and every submitted review is manually inspected by two coders to count issues, locations, and kept comments. The CCR condition is what carries the time-and-confidence argument: because even a perfect automated review did not save time or raise confidence, the cost is attributed to the cognitive work of interpreting and checking generated comments, not to the LLM's errors.","core_discovery":"The paper's central claim is that an automatically generated review used as a starting point changes the reviewer's behavior without delivering the benefits claimed for automation. Reviewers accepted most LLM-generated comments (89% on average) and their final reviews reported more issues overall than manual reviews, but the extra issues were predominantly low severity; the odds of spotting the injected high-severity issues were statistically the same for the ChatGPT-assisted and fully manual treatments. Even when the provided review was perfect (the CCR treatment), reviewers did not finish faster than manual reviewers, because they spent time reading and verifying the provided comments. Confidence scores were likewise indistinguishable across treatments. The paper frames this as evidence that LLM co-reviewers, as currently used, should be positioned as a final complementary check rather than as the starting point of a review.","pith_inferences":["The anchoring effect is likely not specific to code review; any AI tool that outputs a list of candidate locations before a human inspects a large space could narrow attention in the same way, and that generalization is worth testing in other domains.","A direct testable extension would compare 'before' versus 'after' presentation of the same LLM review, as well as a condition where the LLM review is hidden until after the reviewer submits, to quantify how much of the lost coverage comes from order alone.","The 89% acceptance rate and the severity gap between injected and LLM-found issues suggest that benchmark metrics based only on issue counts may overstate the practical value of LLM reviews; reporting severity distributions of findings is a cheap improvement.","The null time result, if it holds in larger samples, would shift the business case for LLM co-reviewers from productivity to thoroughness or training, which are not measured here."],"forward_implications":["Adopting an LLM review as the first step in code review will likely raise the number of low-severity issues reported without increasing detection of high-severity defects.","Review time savings should not be assumed; teams may need to budget time for reading and checking generated comments, which the experiment found offset any speed gain.","Reviewer confidence is not a useful internal signal for whether automated support is helping, since confidence did not move across treatments.","Because reviewers anchor on flagged locations, showing the automated review after the human draft (or using it only as a final checklist) is a concrete way to preserve independent inspection.","Tool designers should target high-severity issue types and keep generated reviews concise, since verbosity adds contributor cost without expanding code coverage."],"supporting_citations":[{"why":"ChatGPT Plus is the specific LLM used to generate the ACR starting-point reviews and to rephrase the manually written CCR reviews.","marker":"[24]"},{"why":"Documents open-source projects adopting LLMs as co-reviewers, which motivates the ACR treatment as the current industrial practice.","marker":"[18]"},{"why":"Supplies the taxonomy of code review issues that inspired the types and proportions of issues injected into the object programs.","marker":"[23]"},{"why":"Provides the issue-type classification used to label injected issues as evolvability or functional defects.","marker":"[31]"},{"why":"The Tako IDE monitoring plugin supplies the behavioral data used to measure time spent reviewing, writing, and reading comments.","marker":"[36]"},{"why":"The closest prior controlled experiment on how existing review comments influence bug identification, providing a comparison point for the anchoring result.","marker":"[66]"},{"why":"Represents the motivation that code review automation can boost reviewer productivity, which the paper's time results directly challenge.","marker":"[16]"},{"why":"Exemplifies the current state of the art in automated review comment generation that the ACR treatment is meant to represent.","marker":"[15]"}],"fun_headline_variants":["AI code review steers focus but fails to save time","LLM co-reviewers shift attention, not outcomes","Automated reviews boost trivial findings only","AI review anchors devs, not severity or speed","LLM review comments don't speed or sharpen"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the study's sample (24 reviews per treatment) was large enough that the non-significant differences in high-severity issue detection, time, and confidence really mean 'no effect'; the authors themselves note that the study could be statistically underpowered and thus lead to biased conclusions.","fun_headline_variants_meta":{"raw":{"variants":["AI code review steers focus but fails to save time","LLM co-reviewers shift attention, not outcomes","Automated reviews boost trivial findings only","AI review anchors devs, not severity or speed","LLM review comments don't speed or sharpen"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1305,"prompt_tokens":1035,"completion_tokens":270,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":197}},"tokens_in":651,"tokens_out":270,"duration_ms":3226,"temperature":1.0,"reasoning_tokens":197,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:32:38.719049+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A pre-registered replication with a larger sample (e.g., at least 100 reviews per treatment) that observes ACR reviewers finding significantly more injected high-severity issues than MCR reviewers, or completing reviews in significantly less time, would undercut the paper's null conclusions. A cheaper check: if reviewers in the ACR condition, when their gaze is tracked, spend a substantial share of time outside the LLM-flagged lines and find new issues there, the anchoring mechanism would not explain the results.","supporting_citations":[{"cited_title":"ChatGPT,","cited_arxiv_id":null,"evidence_quote":"ChatGPT Plus is the specific LLM used to generate the ACR starting-point reviews and to rephrase the manually written CCR reviews."},{"cited_title":"Unveiling chatgpt’s usage in open source projects: A mining-based study,","cited_arxiv_id":null,"evidence_quote":"Documents open-source projects adopting LLMs as co-reviewers, which motivates the ACR treatment as the current industrial practice."},{"cited_title":"What types of defects are really dis- covered in code reviews?","cited_arxiv_id":null,"evidence_quote":"Supplies the taxonomy of code review issues that inspired the types and proportions of issues injected into the object programs."},{"cited_title":"Tako - Visual Studio Marketplace,","cited_arxiv_id":null,"evidence_quote":"The Tako IDE monitoring plugin supplies the behavioral data used to measure time spent reviewing, writing, and reading comments."},{"cited_title":"Primers or reminders? the effects of existing review comments on code review,","cited_arxiv_id":null,"evidence_quote":"The closest prior controlled experiment on how existing review comments influence bug identification, providing a comparison point for the anchoring result."},{"cited_title":"Using pre-trained models to boost code review au- tomation,","cited_arxiv_id":null,"evidence_quote":"Represents the motivation that code review automation can boost reviewer productivity, which the paper's time results directly challenge."},{"cited_title":"Automating code review activities by large-scale pre-training,","cited_arxiv_id":null,"evidence_quote":"Exemplifies the current state of the art in automated review comment generation that the ACR treatment is meant to represent."}],"review_version":1}