{"id":"11a1f30c-4cdf-444a-b827-de13bf722c17","arxiv_id":"2508.02732","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A production system using two fine-tuned Llama3 models plus hand-crafted filters automatically detects code quality issues in large codebases, with reported high precision and a 60% user helpfulness rate.","lead":"This paper describes an industrial system that automatically reviews code changes using two fine-tuned Llama3 language models, one that finds quality problems and one that writes critiques, plus hand-built filters that block likely hallucinations. The authors report strong precision offline and a 60% user helpfulness rate in production, which matters because code review is a bottleneck in large engineering organizations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed effectiveness hinges on evaluation labels that are independent of the system's own rules and on an unbiased helpfulness metric; neither is specified in the abstract, so the headline numbers are unverifiable.","rationale":"The reader's UNVERDICTED verdict is appropriate for an abstract-only review. My stress-test identifies the same load-bearing concern: without independent ground-truth labels and a well-defined helpfulness metric, the headline precision and helpfulness numbers cannot be checked. The concern does not call for changing the verdict, because UNVERDICTED already encodes the absence of evidence rather than a demonstrated flaw. The system architecture described in the abstract is plausible, and the claims are not internally inconsistent, but publication-level confidence would require the evaluation protocol to be disclosed or independently replicable. I agree with the reader's weakest_assumption; no additional load-bearing issue appears in the abstract beyond that.","tokens_in":781,"tokens_out":2896,"duration_ms":38042,"concrete_test":"Obtain the full evaluation details and perform one replication: have independent engineers annotate a held-out set of code changes for quality issues, compute precision on unfiltered model outputs, then recompute precision after applying the hand-crafted hallucination filters. If unfiltered precision is near chance while filtered precision is high, the learned models are not the source of the claimed accuracy. Additionally, retrieve the exact definition and denominator of the 60% helpfulness rate; if the full paper discloses these numbers and confirms label independence, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CQS achieves high precision for valid issues and a 60% weekly helpfulness rate. The load-bearing assumption is that both numbers measure the deployed system fairly and independently. For offline precision, the architecture explicitly layers hand-crafted rules to filter hallucinations; if the ground-truth labels for the evaluation were produced using those same rules, or if precision is computed only on the already-filtered outputs, then the reported rate mostly reflects the filters rather than the fine-tuned models, and it cannot be cited as evidence that the LLMs detect issues. Nothing in the abstract states the labeling protocol, the evaluation set size, the actual precision value, or whether the metric is computed before or after filtering. For the production number, '60% week over week user helpfulness rate' is undefined: it could be the fraction of critiques receiving a positive vote, a click-through rate, or an engineer-reported usefulness score, each with different bias characteristics, and no denominator or confidence interval is given. These omissions do not make the system internally incoherent, but they leave the claimed real-world effectiveness unverified rather than established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This abstract-only paper introduces Code Quality Score (CQS), a system for automatically detecting code quality issues in large, concurrent codebases. CQS is built from two fine-tuned Llama3 models: one trained with SFT and offline RL to detect issues, and one trained to generate review critiques. Hand-crafted rules are layered on top to filter hallucinations. The abstract reports that offline evaluations achieve an 'impressive precision rate' for identifying valid issues and that production rollout has 'consistently achieved 60% week over week user helpfulness rate.' It also promises to present system details and learnings from curating developer feedback for LLM fine-tuning.","tokens_in":990,"tokens_out":1962,"duration_ms":25612,"significance":"If the reported effectiveness numbers are credible, CQS represents a practically valuable deployment of fine-tuned LLMs for code review at industrial scale, with a useful external signal (user helpfulness) tying offline quality to real-world acceptance. The description of using developer feedback to create training data is a transferable contribution for practitioners. However, the evidence as presented in the abstract is insufficient to verify these claims: the key numerical result is unnamed, the evaluation protocol is unspecified, and the production metric is undefined. These omissions currently prevent the paper from being assessed on its central claims, though they are potentially fixable with added detail.","major_comments":[{"comment":"The central empirical claim—'impressive precision rate for identifying valid issues'—is not quantified. No precision value, evaluation set size, baseline, confidence interval, or error bar is given. This makes the offline evaluation unverifiable and prevents comparison with prior work. The full paper must report the actual precision, the dataset scale, the baseline used, and the exact definition of a 'valid issue' and its labeling protocol.","section":"Abstract"},{"comment":"The production claim, 'consistently achieved 60% week over week user helpfulness rate,' is undefined. It is not stated how helpfulness is measured (e.g., explicit user feedback, click-through, engineer survey), what the denominator is, whether the rate is a median, mean, or endpoint of a range, or over how many weeks and engineers it was measured. Without this definition, the number cannot be interpreted as evidence of effectiveness.","section":"Abstract"},{"comment":"There is a potential circularity in the offline evaluation. The system layers hand-crafted rules to filter hallucinations, and if the ground-truth labels for precision were constructed using those same rules—or if precision is computed only on outputs that already passed the filters—then the reported precision largely measures the filters rather than the fine-tuned models. The abstract does not state whether precision is computed before or after filtering, nor how the human-labeled ground truth was collected independently of the system's own rules. This must be clarified for the claim to be load-bearing.","section":"Abstract"}],"minor_comments":[{"comment":"The phrase 'week over week user helpfulness rate' is ambiguous: it could mean the rate is measured weekly and is stable, or that the rate is a change from week to week. Please rephrase to indicate which is intended.","section":"Abstract"},{"comment":"The list items in the description of the two models are inconsistent in parallel structure: 'a) detect common code quality issues... and b) to provide good critiques...'. Consider rewriting as 'a) detects ... and b) provides ...'.","section":"Abstract"},{"comment":"The abstract states that the full paper presents 'details of the CQS system along with some learnings,' but does not mention any availability of code, data, or a more detailed evaluation appendix. If such materials exist, citing them would help reviewers and readers.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"This review is based solely on the abstract because the full text was not available. The abstract's headline numbers are not self-contained: the precision is not given, and the helpfulness metric is undefined. These are not mere presentational issues; they are load-bearing for the paper's contribution. In a full submission, the authors would need to provide a complete evaluation section. I would also flag that the paper's phrasing 'impressive precision rate' is rhetorically unusual in a formal venue; the actual number should be present in the abstract itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my read. The paper is a system note, and on the abstract the only genuinely new thing is the specific combination: two fine-tuned Llama3 models (a detector plus a critique writer) with hand-crafted hallucination filters, and the report on turning developer feedback into training data. That curation recipe is often the missing piece in this literature, so the full paper could be useful. The 60% week-over-week helpfulness figure is also a real industrial deployment signal, and if it is well-defined it is worth taking seriously.\n\nThe soft spots come from what the abstract does not say. “Impressive precision rate” is not a number, no evaluation size, no baseline, no definition of helpfulness, no labeling protocol. The stress-test's circularity worry is legitimate: since the filters are explicitly layered before the model output reaches the user, offline precision labels need to be independent of those filters. The abstract does not state that, so the headline precision could be mostly a measure of the filters rather than of the fine-tuned models. The production helpfulness number likewise needs a denominator and a description of what counts as helpful. Without that, it is a marketing number, not evidence.\n\nThese problems are about missing detail, not about internal incoherence. The architecture is plausible and the claims are not absurd. But the abstract alone cannot support the conclusions. If the full text specifies the evaluation — how precision was computed, with counts and a labeling protocol that does not reuse the filter rules, plus a concrete definition of the production metric — this is the kind of industrial evidence we should have in the literature. If the full text just repeats the abstract's vague language, it is not referee-ready.\n\nWho should read it: people building LLM-based code review for large codebases, and anyone studying human feedback curation for RL fine-tuning. Not our theory crowd. My recommendation: send it to peer review, but with the explicit request that the reviewers audit the evaluation protocol. The stakes are modest and the default should be a hard look at the numbers before any acceptance.","headline":"System note with a plausible architecture and real deployment, but the abstract hides all the numbers that matter; the full text needs a hard look at the evaluation protocol.","tokens_in":1544,"tokens_out":2291,"would_cite":false,"duration_ms":28883,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automated code review reaches 60 percent weekly helpfulness by splitting detection, critique, and filtering across two fine-tuned LLMs and hand-crafted guardrails.","keywords":["code quality","LLM code review","fine-tuning","Llama3","offline reinforcement learning","hallucination filtering","industrial deployment","developer feedback"],"falsifier":"A concrete check is to obtain the labeling protocol for the offline precision evaluation: if the 'valid issue' labels come from the same hand-crafted rules that filter the model outputs, the precision number is circular. A second check is to inspect the production definition of 'helpful'; if any click, open, or non-dismissal counts as helpful, the 60 percent figure would not measure actual usefulness. Independent ground truth from engineers who have never seen the system's outputs, applied to a held-out set of change requests, would settle whether the reported precision holds.","tokens_in":571,"feed_emoji":"🧑💻","tokens_out":4803,"duration_ms":54952,"temperature":0.7,"pith_summary":"This paper describes a deployed system, Code Quality Score (CQS), that automatically flags code-quality problems in large codebases and writes review critiques for engineers. The system is built from two fine-tuned Llama3 models, one aimed at detecting concrete quality issues and one at generating readable critique text, with hand-crafted rules layered on top to discard hallucinated or off-topic outputs. Offline evaluation reports high precision at identifying valid issues, and the production rollout reports a consistent 60 percent week-over-week helpfulness rate from developers. The paper's main message is that a detector-critic-filter pipeline can make LLM-based code review useful at industrial scale, and that developer feedback can be curated into training data for further fine-tuning.","feed_headline":"Two LLMs flag code issues at 60 percent weekly helpfulness","feed_subtitle":"Detector and critic models, plus guardrails, catch quality issues in code changes at industrial scale.","key_machinery":"The load-bearing mechanism is the detector-critic-filter split. A detector model, fine-tuned with supervised fine-tuning and offline reinforcement learning, identifies concrete code-quality problems; a separate critic model produces the developer-facing critique; and a hand-crafted filtering layer removes responses that miss the change set or state unsupported claims. This division of labor lets each model specialize, and the deterministic filter supplies the trust boundary that pure neural generation on its own does not. The work treats that three-part separation as what makes the reported precision and helpfulness numbers achievable.","core_discovery":"The paper claims that a code-quality review service can be made reliable enough for industrial use by separating the job into two model roles: one fine-tuned Llama3 model detects common code-quality issues in a set of changes, and a second fine-tuned Llama3 model writes critiques that explain those issues to developers. Around both models sits a layer of hand-crafted rules that filters out incorrect or hallucinated responses before users see them. Built this way, the system reports high offline precision on valid issues and a steady 60 percent weekly helpfulness rate among developers in a large-scale rollout. The result, if it holds, is that LLM-generated code review assistance is practical on large concurrent codebases when detection, explanation, and output filtering are treated as separate components.","pith_inferences":["My inference: the same detector-critic-filter design may transfer to other high-stakes LLM writing tasks, such as medical chart review or legal document checking, where hallucinated findings are the main barrier to trust.","My inference: the week-over-week helpfulness metric is more likely a measure of user engagement than of code-quality outcomes; tracking whether flagged issues reduce rework or defect density would test whether 60 percent helpfulness actually improves maintainability.","A testable extension: run the detector on a labeled benchmark of changes where independent human reviewers, not the system's own filters, define ground truth, then compare precision by issue category to find where the hand-crafted rules are doing the work."],"forward_implications":["If the 60 percent weekly helpfulness rate is real, organizations can delegate routine code-quality review of large change sets to automated agents while human reviewers concentrate on design and architecture.","The detector-critic-filter structure indicates that LLM review tools do not need one model to do everything; separating detection from explanation and adding deterministic guardrails is a workable production pattern.","The reported training-data lesson implies that developer feedback on generated critiques can be turned into fine-tuning signals, so the system can improve from its own deployment.","A 60 percent helpfulness plateau suggests that the remaining two of every five weekly outputs still miss the mark, and product effort should target those failure categories rather than the overall average."],"supporting_citations":[],"fun_headline_variants":["LLM duo catches code issues with 60% helpfulness","Two fine-tuned Llama3s review code at scale","Code review via two Llama3 models, 60% weekly helpful","Detector plus critic LLMs filter hallucinations for code review"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation numbers are meaningful only if the ground truth used to measure precision is independent of the hand-crafted filtering rules, and only if 60 percent weekly helpfulness is measured from an unbiased, clearly defined developer signal; the abstract does not describe either protocol.","fun_headline_variants_meta":{"raw":{"variants":["LLM duo catches code issues with 60% helpfulness","Two fine-tuned Llama3s review code at scale","Code review via two Llama3 models, 60% weekly helpful","Detector plus critic LLMs filter hallucinations for code review"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1230,"prompt_tokens":901,"completion_tokens":329,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":257}},"tokens_in":517,"tokens_out":329,"duration_ms":3621,"temperature":1.0,"reasoning_tokens":257,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:51:12.156553+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to obtain the labeling protocol for the offline precision evaluation: if the 'valid issue' labels come from the same hand-crafted rules that filter the model outputs, the precision number is circular. A second check is to inspect the production definition of 'helpful'; if any click, open, or non-dismissal counts as helpful, the 60 percent figure would not measure actual usefulness. Independent ground truth from engineers who have never seen the system's outputs, applied to a held-out set of change requests, would settle whether the reported precision holds.","supporting_citations":[],"review_version":1}