{"id":"1d5690b7-d3e4-49d7-8bda-bace0cf8ac00","arxiv_id":"2506.20558","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A two-stage detector-plus-LLM-fixer trained on a new, LLM-filtered dataset reports state-of-the-art code-comment inconsistency detection (F1 89.54%) and 18.84% relative GLEU gain in repair.","lead":"This paper's authors quantified labeling errors in a widely used code-comment inconsistency dataset and built a cleaner one, CCIBench. They then built a two-stage system that detects stale comments with a lightweight model and repairs them with a fine-tuned LLM, reporting state-of-the-art accuracy with lower inference cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CCIBench's full-test-set labels come from the same LLM voters used to build the dataset, so the headline F1/GLEU numbers may reflect agreement with the labeler rather than real CCI; the 300-case validated subset is unanimous-only and cannot independently validate the full set.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: CCIBench labels are produced by LLM voting, and the validated test set is restricted to unanimous votes. I agree this is the most central issue. The 300 validated cases are human-checked, but they are selected from the unanimous subset, so they cannot validate the full test set; the full-test-set F1 and GLEU in the headline therefore encode agreement with the LLM labelers. Human evaluation of fixes is independent but covers only 150 validated cases, so it does not rescue the full-set metrics. A random human-labeled sample from the full test set would directly settle whether the LLM labels are accurate and whether the reported gains are real. The existing CONDITIONAL verdict is appropriate because the concern is addressable by such a check; no change in verdict is needed. Secondary weaknesses, such as test-set hyperparameter selection in RQ2 and the absence of released code, are real but less decisive. Note also that the 'loss-function sign error' flagged by the reader is not obviously an error: Equation 2 as written reads L = BCE + lambda*(1 - mean cosine), which is nonnegative; the similarity term is class-agnostic and its semantic role is questionable, but that is a different issue from the label circularity that is the main threat to the central claim.","tokens_in":30350,"tokens_out":9678,"duration_ms":116448,"concrete_test":"Sample 300 cases uniformly at random from the CCIBench full test set (preserving class balance, not the unanimous-only subset) and have two annotators label them using the Section 2.2 protocol, reporting Cohen's kappa. Recompute CCIDetector's F1 and CCIFixer's GLEU/human-success on this random subset, and also report human-versus-LLM-majority agreement. If F1 or GLEU drops by more than about 3 absolute points relative to Tables 5 and 7, the LLM-vote labels are driving the headline results and the absolute SOTA claim must be re-qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim rests on CCIBench's test labels. Section 3.0.3 constructs CCIBench by keeping cases where at least two of three LLMs (GPT-4o, Claude3.5-Sonnet, LLaMA3.1-405b) judge the old comment inconsistent with the new code. Section 3.0.4 then defines the validated test set as only the unanimously flagged cases, manually verified. Consequently, the full-test-set metrics in Tables 5 and 7 (F1 89.54, GLEU 72.77) are scored against LLM-majority labels, and the only human-verified subset is a high-confidence unanimity slice rather than a random sample of the full test set. Since the detector and fixer are trained on data filtered by the same kind of LLM judgment, a model that learns the labelers' stylistic preferences can inflate all reported numbers without genuinely resolving CCIs. This is testable, not a reason to reject outright: the relative comparison to baselines on the same benchmark is still informative, but the absolute real-world performance claim is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper first performs a manual study of 600 JITDATA cases and reports that 45.67% of positive labels are false positives, attributing them to information add/delete, typo fixes, case changes, and lexical changes. It then constructs a new dataset, CCIBench, by de-duplicating JITDATA, applying syntactic cleaning rules, and filtering with majority voting among GPT-4o, Claude3.5-Sonnet, and LLaMA3.1-405b, with a 300-case validated test set built from unanimously flagged cases and manually verified. The proposed CCISolver combines a deep-learning detector (CCIDetector), which uses UnixCoder, Bi-GRU, multi-head attention, a similarity-based classifier, and iterative LLM-based data enhancement, with an LLM fixer (CCIFixer) based on Qwen2.5-Coder-14B fine-tuned with LoRA and KTO alignment. The paper reports state-of-the-art detection (F1 89.54% on the full CCIBench test set and 92.20% on the validated set), fixing quality (GLEU 72.77 on the full inconsistent test set, an 18.84% relative gain over C4RLLaMA), a human-evaluated fix success rate of 0.6533, and about 36% faster end-to-end inference than C4RLLaMA.","tokens_in":30620,"tokens_out":13716,"duration_ms":145142,"significance":"The potential contributions are real: the quantitative mislabeling study is a concrete empirical finding; CCIBench is a substantial new resource; the two-stage detector-fixer architecture is a sensible engineering response to the cost of LLM-only pipelines; and the evaluation is extensive, including five-run medians, annotation protocols with reported Cohen's kappa, ablation studies, a case study with failures, and a public data link. As currently presented, however, the headline numbers are not fully supported. The benchmark labels are produced by the same LLM voting process that filters the training data, so absolute F1/GLEU values measure agreement with the labelers rather than with an independent ground truth, and the validated set is not a random sample of the full test set. In addition, the default hyperparameters appear to be selected on the test sets used for the final numbers. If the authors add a random human-labeled evaluation sample and a proper validation protocol, the relative comparisons on CCIBench would remain a meaningful contribution even if the absolute real-world performance numbers change.","major_comments":[{"comment":"The ground-truth labels of CCIBench are produced by majority voting among GPT-4o, Claude3.5-Sonnet, and LLaMA3.1-405b (§3.0.3), and the validated test set is composed of cases unanimously flagged as inconsistent and then manually verified (§3.0.4). Because the detector and fixer are trained on data filtered by the same voting process, the full-test-set F1 and GLEU numbers in Tables 5 and 7 quantify agreement with the labelers' judgment rather than with an independent ground truth. The 300-case validated subset cannot resolve this on its own: it is a unanimity slice, not a random sample of the full test set, so it does not measure performance on non-unanimous or negative cases. To support the absolute claims, the authors should human-annotate a random stratified sample of the full test set (including non-unanimous and negative cases) and report detection and fix metrics on that sample; the relative comparisons to baselines on the same benchmark remain informative regardless.","section":"§3.0.3, §3.0.4, Tables 5 and 7"},{"comment":"The default hyperparameters (epoch = 10, loss weight λ = 1, iterative-enhancement iterations = 10) are reported as producing the highest F1 on both the full and validated test sets in Figure 7 and Table 6. This indicates that the hyperparameters were selected on the test sets whose numbers are later presented as final results, and no separate validation split is used. The reported 89.54/92.20 F1 scores are therefore optimistically biased, and the comparison against baselines, whose hyperparameters were not tuned on these same test sets, is not on equal footing. Please introduce a validation split for hyperparameter selection and early stopping, or otherwise demonstrate that the chosen configurations were not selected by test-set performance.","section":"§5.3, §6.3 (Figure 7, Table 6)"},{"comment":"The end-to-end fix metrics in Table 8 are not computed under the same evaluation protocol as the fixer metrics in Table 7. RQ3 deliberately removes consistent cases from the fix test set (Section 6.4.2) because unchanged comments in consistent cases make BLEU-4 and GLEU equal to 1 and inflate scores; RQ5, however, feeds the entire test set through the pipeline and reports fix metrics for the pipeline as a whole. As a result, the Table 8 GLEU values (CCISolver 76.31 vs. C4RLLaMA 63.24) are not comparable with the Table 7 values (72.77 vs. 61.23) and may be inflated by correct \"no-change\" outputs on consistent cases or by missed inconsistent cases whose old comments are scored as predictions. The paper should state exactly how fix metrics are computed in RQ5 and report them either on the flagged subset or under the RQ3 protocol so that the end-to-end comparison is apples-to-apples.","section":"§6.6 RQ5, Table 8"},{"comment":"The construction of CCIBench's negative instances and of the validated set is under-specified and appears internally inconsistent. The semantic filtering in §3.0.3 is described as keeping only cases in which at least two of three LLMs detect an inconsistency; if applied to all instances, this would leave no negative cases, yet the detection experiments in Table 5 require negative labels, and RQ3 (Section 6.4.2) reports that filtering out consistent cases leaves 150 cases in the validated test set, implying the validated set originally contained 150 consistent cases. The paper should state explicitly how negative labels are generated and verified, and how the 300 validated cases were selected. If the negative cases are inherited unchanged from JITDATA, they may contain false negatives under the very heuristic the paper argues is unreliable, which would bias the detection F1 scores.","section":"§3.0.3, §3.0.4, §6.4.2"}],"minor_comments":[{"comment":"Equation (2) is mathematically equivalent to L = BCE + λ(1 − mean cosine similarity); rewriting it in this form would remove the confusing outer minus sign and the phrase \"constraint ensuring the total loss remains positive.\" In addition, Section 6.3.2 refers to an \"α\" hyperparameter in the loss function, while Equation (2) and Figure 7 use λ; the notation should be unified.","section":"§4.2.3, Eq. (2)"},{"comment":"There are several typos that should be corrected in revision: \"Eavaluation\" in the Section 5.2 heading, \"Preious\" in Section 6.6.1, \"deimensions\" in Section 6.6.2, \"contribusions\" in Section 1, \"maintainance\" in the Index Terms, and \"framkework\" in Section 8.1.","section":"Throughout"},{"comment":"The paper reports medians over five runs but does not report standard deviations or confidence intervals for any metric; given the large performance differences claimed, at least for the smaller validated test sets it would be useful to know the spread across runs.","section":"§5.3, §6.2, §6.4"},{"comment":"The phrase \"significantly surpasses\" is used for the human-evaluation success rate (0.6533 vs. 0.5867 and 0.4867), but no statistical significance test is reported; consider softening the wording or adding a test such as McNemar's test for paired outcomes.","section":"Abstract and §6.4"}],"recommendation":"major_revision","confidential_remarks":"The relative comparisons on the same benchmark are useful and the dataset release is a service to the community, but the absolute claims should be treated as benchmark-relative until an independent human-validated evaluation sample and a proper validation protocol are provided. The manuscript is within scope for a software engineering venue; I would not reject it on the current evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, useful paper on code-comment inconsistency (CCI), but treat the absolute numbers as agreement with the LLM labelers rather than ground truth. The relative comparison against baselines is the trustworthy part.\n\nWhat's genuinely new: the first quantitative label-error analysis of JITDATA (45.67% false positives in a 600-case manual sample, with a taxonomy), the CCIBench dataset with a cleaning pipeline, and the two-stage architecture that decouples a cheap detector from an LLM fixer. The paper also does something rare: it reports failure cases for both detector and fixer, and it openly discusses why text metrics like BLEU are inadequate for fix evaluation. The human evaluations are done with reported Cohen's kappa (0.91, 0.95, 0.86), which is better practice than most.\n\nThe main soft spot is exactly what the reader flagged: the ground truth comes from majority voting among three LLMs, and the validated test set is only the unanimously flagged subset. That means the full-test-set numbers (F1 89.54, GLEU 72.77) are essentially measuring agreement with the labelers' judgment. The relative comparisons are still meaningful because all baselines are scored on the same labels, but the absolute real-world claim is not established. A random sample of the full test set, human-labeled, would fix this; the 300 unanimous cases can't validate the whole set.\n\nSecond issue: default hyperparameters (epoch=10, lambda=1, iteration=10) were chosen because they produce the best test-set numbers. That's a selection-on-the-test-set problem, and it inflates the reported scores a bit. It's a common issue in SE papers, but worth flagging.\n\nThird: the printed loss function in Eq. 2 has a sign/notation problem that makes it hard to see what was actually optimized. That's a minor fix; the ablation for lambda suggests the cosine term is helpful either way.\n\nNo code or data artifacts are released beyond a Google Drive link; reproducibility would be much stronger with the actual pipeline scripts.\n\nWho this is for: researchers working on comment maintenance, dataset construction, or SE tooling that uses a cheap prefilter before an LLM. If I were in that area I'd cite it for the dataset and the label-error analysis, and maybe for the two-stage architecture idea. It deserves a serious referee; the concerns are addressable. I'd send it to review with a request for (1) a random human-labeled test subset, (2) a held-out hyperparameter selection procedure, and (3) cleaner loss notation.","headline":"Solid CCI paper with a real dataset and honest analysis, but the absolute numbers measure agreement with LLM labelers, not ground truth.","tokens_in":31132,"tokens_out":1969,"would_cite":true,"duration_ms":22344,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage detector-plus-fixer pipeline detects and repairs Java code-comment mismatches, reaching F1 89.54% and a 65.33% human-judged fix success rate.","keywords":["code-comment inconsistency","just-in-time detection","comment repair","LLM voting","dataset cleaning","Java method comments","LoRA fine-tuning","deep learning detector"],"falsifier":"Take a random sample of, say, 300 CCIBench test cases plus 100 CCIFixer outputs and have two developers who did not build the dataset independently label whether the comment contradicts the code and whether the fix is correct; if their labels match CCIBench's labels and the paper's human success annotations only slightly better than chance, then the reported F1, GLEU, and success rates are artifacts of the labeling procedure.","tokens_in":30192,"feed_emoji":"💬","tokens_out":10294,"duration_ms":94612,"temperature":0.7,"pith_summary":"The paper sets out to show that code-comment inconsistency (CCI) work has been held back by dirty labels and single-model solutions. A manual audit of 600 positive cases from the widely used JITDATA dataset finds that 45.67% are false positives, mostly from comment additions/deletions, typo fixes, case changes, and lexical rewording. To replace that resource, the authors build CCIBench, a cleaned Java method-level dataset filtered by syntactic rules and by majority voting among three LLMs, and they use it to train and evaluate CCISolver, a two-stage pipeline with a lightweight deep-learning detector (CCIDetector) and a fine-tuned LLM fixer (CCIFixer). They report that CCISolver outperforms all compared baselines on detection (F1 89.54% on the full test set, 92.20% on the validated subset), on repair quality (GLEU 72.77, an 18.84% relative gain over C4RLLaMA), and on human-judged fix success (0.6533), while cutting end-to-end inference time by about 36% relative to the monolithic C4RLLaMA. The paper therefore claims that separating cheap detection from expensive repair is both more accurate and more practical than using one large model for both tasks.","feed_headline":"Two-stage code-comment fixer beats LLM-only baseline by 19%","feed_subtitle":"Splitting detection from repair lifts GLEU to 72.77 and runs 36% faster end-to-end than a monolithic LLM.","key_machinery":"The central mechanism is the two-stage pipeline itself: CCIDetector scans all code-comment pairs cheaply and forwards only predicted inconsistencies to CCIFixer. The detector's input representation matters as much as the architecture: code diffs are rendered as add/del/rep/keep edit actions with replace spans, raw comment and new code are embedded with UniXcoder, and a classifier head scores consistency with a cosine-similarity term in the loss. The fixer is a LoRA fine-tuned code LLM (Qwen2.5-Coder-14B) put through KTO alignment so its rewrites stay close to the dataset's comment style. The dataset pipeline is equally load-bearing: syntactic filters remove trivial edits and a three-LLM voting rule with 4-shot in-context examples decides which semantic changes remain as genuine inconsistencies.","core_discovery":"On its own terms, the paper's central discovery is that the CCI problem decomposes: a small, fast model can decide whether an old comment contradicts new code, and a larger generative model needs to run only on the cases flagged. The authors support this with CCIBench, whose construction is itself part of the contribution: after de-duplication, syntactic rules strip typo/case/stopword/lexical changes, and three LLM voters with dynamic 4-shot prompting decide semantic inconsistencies, keeping cases with at least two-thirds agreement and manually validating a 300-case test subset (Cohen's kappa 0.95). The detector encodes old comment, new code, and a structured code diff through UniXcoder, Bi-GRU, and multi-head attention, with a classifier that combines binary cross-entropy with a comment-code cosine similarity term, and is iteratively boosted on LLM-synthesized versions of its own mistakes. The fixer is Qwen2.5-Coder-14B fine-tuned with LoRA and then aligned with KTO on the cleaned inconsistent cases. With this design the paper reports F1 of 89.54% and 92.20% (full and validated test sets), GLEU of 72.77 and 74.90, human fix success of 0.6533, and per-instance inference time of 0.6164 s versus 0.9618 s for C4RLLaMA.","pith_inferences":["Beyond the paper, the same cheap-filter/expensive-repair split could be applied to inline comments, class comments, or other languages, since nothing in the architecture is Java-specific; the gains would need testing, but the bottleneck it removes is generic.","Beyond the paper, the reliance on LLM voters for ground truth means CCIBench's main risk is shared blind spots; publishing an independently human-labeled sample alongside the dataset would let users calibrate all downstream numbers.","Beyond the paper, because the paper itself notes that BLEU/GLEU scores do not reliably track fix correctness, an LLM-as-judge metric calibrated against the human annotations would be a natural, lower-cost way to scale fix evaluation.","Beyond the paper, conditioning the fixer on the detector's identified inconsistency type (return type, signature, or logic) could improve fixes, since the case study shows the fixer's errors differ across those categories."],"forward_implications":["If the CCIBench labels are trusted, prior evaluations on JITDATA were inflated, since 45.67% of sampled positive labels were false positives; future CCI work should be measured on the cleaned benchmark.","A developer integrating CCISolver into a just-in-time commit workflow could scan a batch of changed Java methods, have the detector flag candidate inconsistencies, and send only those to the LLM fixer, obtaining better fixes than C4RLLaMA while spending about 36% less inference time per instance.","The fixer's human-evaluated success rate of 0.6533 implies that roughly two of every three generated comments are found acceptable by two independent annotators, setting a higher bar than DeepJIT (0.4867) and C4RLLaMA (0.5867).","The iterative enhancement results (F1 86.6 without augmentation versus 89.54 with 10 iterations) imply that synthesizing the detector's own hard cases is a usable substitute for collecting more labeled data.","Because only flagged cases reach the LLM, real-world savings should grow as the true inconsistency rate drops below the test set's rate, a point the authors make in their discussion."],"supporting_citations":[{"why":"Provides the OCD baseline for obsolete comment detection and update against which CCIDetector and CCIFixer are compared.","marker":"[5]"},{"why":"Supplies the JITDATA dataset, the DeepJIT baselines, and the code-diff edit-action representation that CCIDetector builds on.","marker":"[8]"},{"why":"Prior work on label quality in obsolete-comment detection whose adversarial framework is a baseline and whose cleaning ideas motivate CCIBench's syntactic rules.","marker":"[9]"},{"why":"Defines C4RLLaMA, the monolithic fine-tuned LLM baseline used for detection, fixing, and end-to-end efficiency comparisons.","marker":"[10]"},{"why":"GPT-4o is one of the three LLM voters in CCIBench's semantic filtering and the teacher LLM in iterative enhancement.","marker":"[13]"},{"why":"Claude 3.5-Sonnet is one of the three LLM voters whose agreement determines which cases stay in CCIBench.","marker":"[14]"},{"why":"LLaMA 3.1 405B is one of the three LLM voters and also serves as a general LLM baseline in detection.","marker":"[15]"},{"why":"UniXcoder supplies the encoder that embeds raw code and comment for CCIDetector's similarity classifier.","marker":"[30]"},{"why":"LoRA is the parameter-efficient fine-tuning method used to train CCIFixer on the cleaned inconsistent cases.","marker":"[40]"},{"why":"Kahneman-Tversky optimization provides the alignment objective that keeps CCIFixer's generated comments close to the dataset's preferred style.","marker":"[41]"}],"fun_headline_variants":["Fast detector + LLM fixer boosts code-comment repair by 19%","Split detection from repair: 19% better code-comment fixes","CCISolver: end-to-end code-comment fixer, 19% better","Code-comment fixer runs 36% faster, improves GLEU 19%","Detector gates LLM: 19% better fixes, 36% faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every reported performance number assumes the CCIBench labels are correct, but most labels come from majority votes among three large language models, so if those models systematically disagree with developer judgment, the F1, GLEU, and success-rate figures measure agreement with the labelers rather than real code-comment inconsistency.","fun_headline_variants_meta":{"raw":{"variants":["Fast detector + LLM fixer boosts code-comment repair by 19%","Split detection from repair: 19% better code-comment fixes","CCISolver: end-to-end code-comment fixer, 19% better","Code-comment fixer runs 36% faster, improves GLEU 19%","Detector gates LLM: 19% better fixes, 36% faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00117,"raw_usage":{"total_tokens":4920,"prompt_tokens":1110,"completion_tokens":3810,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":726,"completion_tokens_details":{"reasoning_tokens":3704}},"tokens_in":726,"tokens_out":3810,"duration_ms":26878,"temperature":1.0,"reasoning_tokens":3704,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:45:52.681818+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 300 CCIBench test cases plus 100 CCIFixer outputs and have two developers who did not build the dataset independently label whether the comment contradicts the code and whether the fix is correct; if their labels match CCIBench's labels and the paper's human success annotations only slightly better than chance, then the reported F1, GLEU, and success rates are artifacts of the labeling procedure.","supporting_citations":[{"cited_title":"Just-In-Time Obsolete Comment Detection and Update,","cited_arxiv_id":null,"evidence_quote":"Provides the OCD baseline for obsolete comment detection and update against which CCIDetector and CCIFixer are compared."},{"cited_title":"Automatic Detection of Outdated Comments During Code Changes,","cited_arxiv_id":null,"evidence_quote":"Supplies the JITDATA dataset, the DeepJIT baselines, and the code-diff edit-action representation that CCIDetector builds on."},{"cited_title":"Deep Just- In-Time Inconsistency Detection Between Comments and Source Code,","cited_arxiv_id":null,"evidence_quote":"Prior work on label quality in obsolete-comment detection whose adversarial framework is a baseline and whose cleaning ideas motivate CCIBench's syntactic rules."},{"cited_title":"Data Quality Matters: A Case Study of Obsolete Comment Detection,","cited_arxiv_id":null,"evidence_quote":"Defines C4RLLaMA, the monolithic fine-tuned LLM baseline used for detection, fixing, and end-to-end efficiency comparisons."},{"cited_title":"Automatic programming: Large language models and beyond,","cited_arxiv_id":null,"evidence_quote":"GPT-4o is one of the three LLM voters in CCIBench's semantic filtering and the teacher LLM in iterative enhancement."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"Claude 3.5-Sonnet is one of the three LLM voters whose agreement determines which cases stay in CCIBench."},{"cited_title":"Claude 3.5: Sonnet,","cited_arxiv_id":null,"evidence_quote":"LLaMA 3.1 405B is one of the three LLM voters and also serves as a general LLM baseline in detection."},{"cited_title":"Learning to Update Natural Language Comments Based on Code Changes,","cited_arxiv_id":null,"evidence_quote":"UniXcoder supplies the encoder that embeds raw code and comment for CCIDetector's similarity classifier."},{"cited_title":"Self-instruct: Aligning language models with self- generated instructions,","cited_arxiv_id":null,"evidence_quote":"LoRA is the parameter-efficient fine-tuning method used to train CCIFixer on the cleaned inconsistent cases."},{"cited_title":"Lora: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"Kahneman-Tversky optimization provides the alignment objective that keeps CCIFixer's generated comments close to the dataset's preferred style."}],"review_version":1}