{"id":"ea49eae5-33a9-481a-9725-b8a8a5a22cda","arxiv_id":"2412.20340","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Desiview identifies desired review comments from code review datasets using the perplexity difference of the actual fix with and without each comment, and the distilled data improves LLaMA-based code review models.","lead":"Desiview is a method that automatically finds useful code review comments by measuring how much a large language model's surprise at the eventual fix drops when the comment is included. It uses the filtered comments to fine-tune LLaMA models, which then produce more accurate review comments than models trained on the unfiltered dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The uncalibrated perplexity proxy in Eq. (2) is load-bearing; the 600-entry validation may not detect systematic bias from fix size or comment length, and the downstream evaluation inherits Desiview's own labels.","rationale":"The reader's weakest_assumption correctly identifies Eq. (2) as the load-bearing premise. I agree that the perplexity-reduction proxy is uncalibrated at scale, but I would sharpen the concern in two ways. First, the 'without comment' baseline is not a true no-comment condition: Fig. 4 shows an empty comment string in the same template, so the comparison may reflect template artifacts rather than the semantic contribution of R. Second, the subsequent fix Cr is the whole commit diff, not the portion attributable to R; the proxy therefore conflates comment usefulness with the predictability of unrelated concurrent changes, and it rewards comments that lexically overlap the fix. The 600-entry human evaluation for RQ1 is real evidence and should be credited, but it is a single aggregate accuracy figure; it does not show that the proxy behaves uniformly across fix sizes, comment lengths, or comment styles. The downstream RQ2 evaluation is also structured so that the test set and the human-evaluation sample are both subsets that Desiview itself labels as DRCs, so improvement over base models on those subsets does not establish that the models generate fewer non-DRCs or are better on unselected review comments. These are fixable concerns: release the distilled dataset and scoring code, report stratified accuracy of the proxy, and evaluate the fine-tuned models on a human-annotated sample that includes both DRCs and non-DRCs according to an independent labeling protocol. Because the paper's central contribution is a data distillation method whose value depends on the validity of its labeling signal, and because the current evidence, while suggestive, does not yet rule out a systematic proxy bias, conditional acceptance with these required checks is the appropriate outcome. This does not change the reader's verdict, so I recommend UNCHANGED.","tokens_in":18472,"tokens_out":7904,"duration_ms":82880,"concrete_test":"Hold out the 600 human-annotated entries and add 300 fresh annotations stratified by fix size (e.g., <10, 10-50, >50 changed lines) and comment length. Recompute Desiview's DS>0 accuracy within each stratum, and also compute DS>0 accuracy on control pairs where R is replaced by an unrelated comment of matched length. If accuracy drops below, say, 75% in the large-fix stratum or if the control comments yield DS>0 at a rate comparable to real comments, the Eq. (2) proxy is confounded and the distilled dataset needs re-labeling with a calibrated threshold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Desiview's core labeling rule, Eq. (2): DS = -[PPL(Cr|Co,R) - PPL(Cr|Co)], with DS>0 meaning desired, is never calibrated against human judgments beyond a single 600-entry sample. The baseline PPL(Cr|Co) is computed with a prompt containing an empty comment (Fig. 4), not a true no-comment condition, so DS can be inflated or deflated by prompt artifacts. More importantly, Cr is the entire subsequent fix commit, which often contains changes unrelated to R; PPL reduction from R is therefore diluted for large fixes and inflated when R lexically overlaps Cr. This can systematically mislabel the 150,406 training entries, and the error propagates: RQ2's test set is precisely the 5,727 entries Desiview labels as DRCs, and its human evaluation samples only those entries. Thus the reported downstream improvement is consistent with the model having learned Desiview's biased notion of desiredness rather than a genuinely better review ability. The 600-entry RQ1 accuracy (86.67%) is encouraging but does not resolve whether the proxy is valid across fix-size/comment-length strata, since no stratified results or error analysis are reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Desiview, a method that automatically identifies \"desired review comments\" (DRCs) from a code review dataset by computing a desiredness score based on perplexity differences: DS = -[PPL(Cr|Co,R) - PPL(Cr|Co)], where Co is the original code, R the review comment, and Cr the subsequent fix. The method labels entries with DS>0 as desired, then distills a fine-tuning dataset from CodeReviewer. The authors fine-tune LLaMA-3 and LLaMA-3.1 on this distilled set (Desiview4FT) and further apply KTO alignment using the non-DRC entries (Desiview4FA). They report that Desiview achieves 86.67% accuracy at DRC identification, outperforming GPT-4o (76.50%), and that the resulting models generate DRCs with higher BLEU-4 scores and better human-evaluated position and description accuracy than the LLaMA-Reviewer baseline.","tokens_in":18773,"tokens_out":2799,"duration_ms":27572,"significance":"If the claims hold, the contribution is practically and scientifically valuable: it offers a scalable, low-cost way to build high-quality code review training data without manual annotation, and it shows a concrete use of KTO alignment for code review. The RQ1 evaluation includes an independent manual annotation of 600 entries, which is a genuine strength, and the human evaluation on 300 entries provides some evidence that the distilled-data models do not merely memorize the distillation labels. The main concerns are the uncalibrated perplexity proxy that generates the training labels, the partial circularity of the RQ2 automated evaluation, and the lack of stratified error analysis. These are load-bearing issues rather than cosmetic ones, because the entire pipeline depends on the validity of the DS proxy and on the evaluation not rewarding models for learning Desiview's own label biases.","major_comments":[{"comment":"The desiredness score DS = -[PPL(Cr|Co,R) - PPL(Cr|Co)] is used to label all 150,406 training entries, but the proxy is never calibrated against human judgments before use. The paper only reports a 600-entry RQ1 accuracy of 86.67%, which is a single aggregate number; it does not report whether the proxy's accuracy varies by fix size, comment length, or lexical overlap between the comment and the fix. The baseline in Fig. 4 uses an empty comment string ('The comment is: ''), which is not a true no-comment condition and may itself introduce prompt-format artifacts into PPL(Cr|Co). Moreover, Cr is the entire subsequent fix commit, which often contains changes unrelated to R, so DS can be diluted for large fixes and inflated when R lexically overlaps Cr. Because every downstream experimental result inherits these labels, the proxy's validity should be demonstrated with a stratified calibration study or an explicit error analysis before the method's headline claims are accepted.","section":"§III.A1, Eq. (2); Fig. 4"},{"comment":"The automated evaluation for RQ2 uses as ground truth the 5,727 test entries that Desiview itself labeled as DRCs (Table I), and the BLEU-4 scores compare generated comments against the existing comments in that labeled set. This is partially circular: models trained on Desiview's distilled data are evaluated against Desiview's own labels, so part of the reported BLEU gains (+42.5% to +97.8% in Table IV) may reflect train/test label alignment rather than a genuinely better code review ability. The paper should evaluate on a test set that is independently annotated, or at least report results on a human-verified subset of both DRC and non-DRC test entries, and show that the improvement is not an artifact of learning the distillation classifier's boundary.","section":"§IV.B.a, Table IV"},{"comment":"The human evaluation samples 300 entries from the Desiview-labeled DRC test set only, and the two evaluators are the same graduate-student population that annotated the RQ1 data. This design cannot measure how the models behave on non-DRC reviews, nor on Desiview's false-positive and false-negative entries, where a distilled-data model might overfit to the proxy's biases. In addition, the agreement checks are reported as chi-squared p-values (0.965 and 0.887) without a kappa statistic or per-stratum agreement; the phrase 'rejecting the hypothesis of inconsistency' is also an unusual interpretation, since a high p-value for a chi-squared test does not establish strong agreement. The paper should report inter-annotator agreement (e.g., Cohen's kappa) and provide a breakdown of human evaluation results by Desiview's true-positive, false-positive, true-negative, and false-negative categories, to show that the downstream gains are not solely driven by the proxy's confident correct cases.","section":"§IV.A.2, §IV.B, Table IV; §VI Threats"}],"minor_comments":[{"comment":"The statement that 'the proportions of DRCs in the training and test sets are close to each other, somewhat indicating the reliability of Desiview' is not a validity argument; the proportions being close could simply reflect random splitting of the data that was already labeled by the same method.","section":"§III.A2, Table I"},{"comment":"The description of KTO says λD=1.7 and λU=1.0, but the formula in §III.C lists a constraint λD nD / (λU nU) ∈ [1, 4/3]; the paper should show that the chosen values satisfy the constraint for the actual class counts in the distilled dataset, or explain why the constraint is not binding.","section":"§II.B"},{"comment":"The annotation procedure reports a margin of error of less than 4% at a 95% confidence level for the 600-entry sample and less than 6% for the 300-entry sample, but the exact sample-size formula and the underlying population size (150,406 vs. 5,727) are not given; the paper should cite the formula or present the confidence intervals explicitly.","section":"§IV.A.1"},{"comment":"The threat-to-validity paragraph says 'the amount of such data is minimal and unlikely to have a significant impact,' but the RQ1 accuracy of 86.67% implies a 13.33% error rate on the labeled subset; the paper should reconcile this with the claim of minimal noise, and ideally quantify the noise in the test set used for RQ2.","section":"§VI, 'Noise in the distilled dataset'"},{"comment":"There are several typographical and formatting issues, including inconsistent subscript rendering in the abstract ('Desiview4F A'), missing whitespace in equations and templates, and the phrase 'GPT4o' in §VI which should be 'GPT-4o' for consistency.","section":"General (paper text)"}],"recommendation":"major_revision","confidential_remarks":"The paper has a sound central idea and a credible set of experiments, but the two main concerns—the uncalibrated perplexity proxy and the partial circularity of the RQ2 automated test set—are load-bearing. I recommend major revision rather than rejection because both concerns are addressable within the manuscript's scope: the authors can add a calibration analysis of Eq. (2) against human labels across strata, build an independently annotated test set for RQ2, and report stratified human results. I would also suggest the editor ask for a data-availability statement; no code or data release is mentioned, which matters for reproducibility of the distillation pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper to know about. The core idea is simple and useful: score each review comment by how much it reduces an LLM's perplexity on the subsequent fix commit, then keep the ones that help. The four-LLM median voting is a sensible guard against single-model quirks, and using KTO with the rejected comments as undesirable data is a nice extension. The RQ1 manual annotation (600 entries, with inter-annotator agreement) shows the method beats GPT-4o by a wide margin, which is honestly encouraging.\n\nThe soft spots are real but fixable. The automated RQ2 evaluation uses the 5,727 entries that Desiview itself labeled as DRCs as ground truth, so part of the BLEU improvement may just reflect train/test label agreement. The human evaluation on 300 entries supports the direction and is independent, but it only samples from Desiview's positive labels. The bigger conceptual concern is that Eq. (2) is never calibrated: the baseline condition uses an empty comment rather than truly no comment, and the fix commit often contains unrelated changes, so the perplexity difference may be inflated or diluted by artifacts. The paper doesn't report stratified results by comment length or fix size, so we don't know whether those biases are severe. Also, no code or data is released, which limits verification.\n\nNone of this sinks the paper. The central claim—that you can distill a large training set without manual labels—is plausible and would be valuable to the code review community if it holds. The authors acknowledge the quality-definition risk and the noise in the distilled dataset. What's missing is a calibration study: sample a few hundred entries across fix-size/comment-length strata, compare Desiview's labels to human labels, and show the proxy isn't just picking up lexical overlap or fix size. Release the distilled data and the labeling code, and rerun the automated evaluation on human-labeled ground truth.\n\nBottom line: this deserves a serious referee. I'd send it to review, but with a request for artifact release and a calibration analysis. If I were working on code review automation, I'd cite it.","headline":"A cheap perplexity-based distiller for code review comments that works better than I'd expect, but the evaluation is partly circular and the proxy is uncalibrated; worth reviewing, not definitive.","tokens_in":19286,"tokens_out":2321,"would_cite":true,"duration_ms":22379,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Desiview automatically identifies code review comments that trigger fixes by measuring how much a language model's perplexity drops when the comment is included in the fix-prediction prompt.","keywords":["code review","dataset distillation","desired review comments","perplexity","fine-tuning","large language models","KTO alignment","LLM alignment"],"falsifier":"Take a fresh sample of review comments labeled by developers as either causing the fix or not, compute Desiview's perplexity-gap score for each comment, and measure the agreement; if the score disagrees with the human labels beyond the error margin reported in the paper, the proxy fails. A sharper version is to paraphrase a helpful comment while keeping the same fix suggestion and check whether the positive gap survives the rewording; if it flips, the score is partly measuring wording rather than desiredness.","tokens_in":18284,"feed_emoji":"🔍","tokens_out":8286,"duration_ms":70096,"temperature":0.7,"pith_summary":"This paper claims that useful code review comments—comments that actually cause a developer to fix the code—can be identified automatically, at scale, by measuring how much a language model's perplexity drops when the comment is added to the prompt that predicts the subsequent fix. The proposed distillation method, Desiview, labels a review comment as desired when that perplexity gap is positive, using the median vote of four code-savvy language models. On a widely used dataset of more than 150,000 review entries, Desiview reaches about 87% accuracy in identifying desired comments, outperforming the 10-line rule, GPT-3.5, and GPT-4o. The distilled dataset, which is less than half the size of the original, is then used to fine-tune the LLaMA series and to align it with KTO, yielding review models that produce more accurate and better-described issue comments than the base models. If this holds, high-quality code review training data can be produced automatically without manual annotation.","feed_headline":"Perplexity gap finds review comments that trigger fixes","feed_subtitle":"A new distillation method filters useful reviews automatically, beating GPT-4o and sharpening LLaMA-based review models.","key_machinery":"The carrying object is the desiredness score $DS$ from Eq. (2), computed as the negative difference between two perplexities: the perplexity of the actual code fix when the review comment is included in the prompt, and the perplexity of the same fix when the comment is omitted. Perplexity is defined by $PPL(X)=\\exp\\left(-\\frac{1}{N}\\sum_{i=1}^{N}\\log P(x_i \\mid x_{<i})\\right)$, and the paper takes the median desiredness score from four different code-savvy language models as the final label. This score does double duty: it produces the distilled fine-tuning set, and it splits the dataset into desired and undesired halves that serve as positive and negative examples for KTO alignment.","core_discovery":"The central claim is that a review comment is desired exactly when adding it to the prompt that predicts the developer's subsequent fix lowers that fix's perplexity. The desiredness score is $DS = -(\\mathrm{PPL}(C_r \\mid C_o,R) - \\mathrm{PPL}(C_r \\mid C_o))$, and a comment is labeled desired when $DS > 0$ and undesired when $DS \\le 0$. The authors apply this labeling to more than 150,000 entries of a widely used code review dataset, distill the desired subset, and then fine-tune LLaMA-3 and LLaMA-3.1 models on it, followed by KTO alignment with the undesired comments as negative examples. They report that this training regime yields models that generate more accurate and better-described issue comments than models fine-tuned on the original unlabeled dataset, as measured by BLEU and by human evaluation.","pith_inferences":["An immediate testable extension is to use the perplexity-gap score as an online filter while a review model is decoding, dropping candidate comments whose own desiredness score is negative before showing them to a developer.","The same proxy could generalize to other human-written artifacts paired with later edits—e.g., bug reports, API documentation revisions, or design-review comments—because it only needs a before/after pair and an intervening text.","If perplexity gaps were calibrated against developer rankings of comment helpfulness, the threshold at zero could be replaced by a per-project threshold, making the distillation robust to repositories where reviewers write polite or verbose comments that do not directly cause fixes."],"forward_implications":["A high-quality code review training set can be built automatically from any repository that records comment–fix pairs, removing the manual annotation bottleneck.","Because the distilled set is less than half the size of the original yet yields stronger models, dataset quality outweighs quantity for review fine-tuning.","Fine-tuned and aligned models on the distilled data localize and describe code issues more accurately in human evaluation, making automated review feedback more actionable.","Explicitly feeding undesired comments as negatives during KTO alignment further improves the generated comments, so data-distillation products are useful for both phases of LLM training."],"supporting_citations":[{"why":"It supplies the code review dataset of original code, review comments, and subsequent fixes that Desiview distills.","marker":"[5]"},{"why":"It introduces perplexity, the metric at the core of the desiredness score.","marker":"[44]"},{"why":"It provides the 10-line rule baseline that Desiview is compared against.","marker":"[17]"},{"why":"It provides the LLaMA-Reviewer baseline and the prompt template used for the fine-tuning comparison.","marker":"[2]"},{"why":"It supplies the KTO alignment objective used to build Desiview4FA from the desired and undesired splits.","marker":"[39]"},{"why":"It supplies LoRA, the parameter-efficient training method used for both fine-tuning and alignment.","marker":"[46]"},{"why":"It supplies the LLaMA-3 base model family used as the starting point for fine-tuning.","marker":"[29]"}],"fun_headline_variants":["Perplexity drop finds review comments that trigger fixes","Distilling code review comments via perplexity signals","LLM review training: keep comments that lower fix perplexity","Perplexity-guided distillation sharpens code review LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a review comment is useful exactly when adding it lowers a language model's perplexity for the developer's actual fix; if that gap instead reflects comment style or dataset artifacts, the distilled data inherit those errors.","fun_headline_variants_meta":{"raw":{"variants":["Perplexity drop finds review comments that trigger fixes","Distilling code review comments via perplexity signals","LLM review training: keep comments that lower fix perplexity","Perplexity-guided distillation sharpens code review LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000332,"raw_usage":{"total_tokens":1922,"prompt_tokens":1094,"completion_tokens":828,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":710,"completion_tokens_details":{"reasoning_tokens":763}},"tokens_in":710,"tokens_out":828,"duration_ms":6327,"temperature":1.0,"reasoning_tokens":763,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:24:20.679910+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh sample of review comments labeled by developers as either causing the fix or not, compute Desiview's perplexity-gap score for each comment, and measure the agreement; if the score disagrees with the human labels beyond the error margin reported in the paper, the proxy fails. A sharper version is to paraphrase a helpful comment while keeping the same fix suggestion and check whether the positive gap survives the rewording; if it flips, the score is partly measuring wording rather than desiredness.","supporting_citations":[],"review_version":1}