{"id":"6396990f-88fa-436a-86ff-e9decbfa2ce1","arxiv_id":"2504.12608","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLM-generated code frequently contains repetitive patterns at character, statement, and block levels, and a rule-based pruner named DeRep reduces this repetition and improves functional correctness on HumanEval and MBPP.","lead":"This paper measures how often LLM-generated code falls into repetitive loops and duplicate blocks, finding it across all 19 tested models. It then presents a rule-based post-processor, DeRep, that cuts repeated code and reportedly boosts the share of correct solutions substantially.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DeRep's keep-first repair may inflate Pass@1 by deleting behavior-affecting code (e.g., failing self-tests) rather than pure redundancy; no regression audit is provided.","rationale":"I agree with the reader's weakest assumption: the Pass@1 improvement hinges on the first-occurrence-is-correct and duplicates-are-redundant premise. The proposed confusion-matrix audit is the decisive check: it directly measures whether repair ever turns a passing program into a failing one. The paper's existing evidence (89.9% truncation) supports that many failures are truncation-induced, but it does not rule out regressions or the self-test confound. This is the single most load-bearing concern because if regressions are non-negligible, the headline 'enhancing code quality' claim is overstated; if regressions are rare, the claim is credible. The reader's CONDITIONAL verdict is appropriate and should remain unless the audit is run. I am not raising a new objection beyond the reader's, hence UNCHANGED and agreement.","tokens_in":20430,"tokens_out":14417,"duration_ms":136813,"concrete_test":"Run a regression audit on the RQ3 setup (HumanEval-Python + MBPP, six DeepSeekCoder variants): for every problem, execute both the raw greedy output and DeRep's repaired output against the official hidden test suite, and tabulate a 2x2 confusion matrix (both fail / raw-fail repaired-pass / raw-pass repaired-fail / both pass). Report the count of raw-pass repaired-fail cases (regressions). If any regressions occur, quantify their fraction relative to recoveries; if regressions exceed 2% of recoveries or include any case where the only deleted code is a failing assertion, the claim that deleted duplicates are pure redundancy is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract; Section VI-A Table V) is that DeRep, a rule-based post-processor, cuts repetition metrics by 88–97% and lifts Pass@1 from 10.8 to 33.3, a 208.3% relative gain. The mechanism (Section V-B) is: keep the first occurrence of each repeated unit, delete all later duplicates. This is safe only if the first occurrence is a correct, complete implementation and the deleted duplicates are behaviorally redundant. The paper does not establish either. In the standard HumanEval harness, generated top-level statements are executed; repeated self-test assertions are a common degeneration pattern (Table IV). If later assertions are wrong, they crash the raw module, and DeRep deletes them, so the repaired module loads and passes hidden tests even though the function itself was never repaired. Without a control that separates 'truncation repair' from 'behavioral redundancy deletion', the 208.3% Pass@1 gain conflates repeated-code removal with removal of failing code. Section VIII (Threats to Validity) does not discuss this.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates repetitive or degenerate code generated by LLMs. It first conducts an empirical study across 19 code LLMs on HumanEval-Python, HumanEval-Java, and MBPP, quantifying repetition with three metrics (rep-n, rep-line, and sim-line) and deriving a taxonomy of 20 repetition patterns at character, statement, and block levels. It then proposes DeRep, a rule-based post-processor that detects repeated units and repairs them by keeping the first occurrence and deleting later duplicates. The evaluation claims that DeRep reduces repetition by 88.3% on a combined rep metric and improves Pass@1 from 10.8 to 33.3 (a 208.3% relative gain) over greedy search across six DeepSeekCoder variants, and that it also improves the Pass@1 of existing repetition-mitigation baselines by 53.7% to 215.7%. The paper additionally reports a deployment in an industrial code-completion setting.","tokens_in":20639,"tokens_out":4950,"duration_ms":54163,"significance":"If the DeRep results hold, the paper would make a useful and practical contribution: repetition in code LLMs is shown to be pervasive, and a cheap rule-based post-processor appears to fix a large part of it without retraining. The prevalence study is a genuine strength: it covers 19 models, uses three benchmarks, follows a consistent greedy-decoding protocol, and the resulting taxonomy should be useful to future work. The Pass@1 improvement is an important independent signal, because it shows that the method does something beyond lowering self-defined repetition metrics. The industrial deployment, though reported at a high level, adds real-world relevance. The main weaknesses lie in the evaluation of DeRep: the abstract reports numbers that do not appear in the body, the repair strategy can delete behavior-affecting code rather than only redundancy, and the detection thresholds appear to be tuned on the evaluation set without validation. These issues must be resolved before the central claims can be accepted.","major_comments":[{"comment":"The abstract claims average improvements of 91.3%, 93.5%, and 79.9% in rep-3, rep-line, and sim-line, but these numbers do not appear anywhere in the body. Table V reports only a combined rep metric, with an average reduction of 88.3% for DeRep applied to greedy search. Because the abstract frames the central result, the authors must either provide the per-metric breakdown with the exact protocol or correct the abstract so that it matches the evaluation actually reported.","section":"Abstract and Section VI-A"},{"comment":"The Pass@1 improvement may be inflated by deleting behavior-affecting code rather than pure redundancy. The repair strategy keeps the first occurrence of each detected repeated unit and deletes all later duplicates. In the standard HumanEval harness, repeated assertions and self-test statements (which Table IV identifies as common patterns) execute as top-level code, and a later wrong assertion can crash the generated module. If DeRep deletes such failing assertions, the repaired module can load and pass hidden tests even though the function itself was never repaired. The paper provides no regression audit, such as comparing DeRep against a control that truncates at the first repeated block, or verifying that the repaired code preserves the behavior of the model's own generated assertions. Section VIII does not discuss this threat, and it is load-bearing for the 208.3% Pass@1 claim.","section":"Section V-B and Table V"},{"comment":"The detection thresholds appear to be tuned on the same data used for evaluation. The cosine similarity threshold of 0.65 is described as \"based on empirical tuning,\" and the block-length upper bound Lmax is described as \"empirically set\" to 2n/3. No validation split, threshold sensitivity analysis, or procedure for choosing these values is reported. Since the evaluation metrics are computed with the same line- and block-similarity machinery, the numerical reductions in Table V may be optimistically biased. The authors should describe how the thresholds were selected, use a held-out split, and report how Table V changes across a range of threshold values.","section":"Section V-A"},{"comment":"The repetition-reduction results are partly by construction. rep-line is the proportion of exact duplicate lines and sim-line is based on Levenshtein similarity, while DeRep detects exactly the repeated lines and blocks flagged by TF-IDF cosine similarity and deletes them. Large reductions in these metrics are therefore expected regardless of whether the deleted code is semantically redundant. The Pass@1 signal mitigates this concern, but it does not fully resolve it in light of the assertion-deletion issue raised above. An independent assessment, such as human readability ratings, functional-equivalence checks, or a test of whether repaired code still passes the model's own generated assertions, is needed to support the claim that DeRep improves code quality rather than only lowering the self-defined metrics.","section":"Section III-C and Section VI-A"}],"minor_comments":[{"comment":"The preliminary analysis reports 10,399 snippets with repetition and 9,346 cases (89.9%) exceeding the token limit, but no methodology is given for how these counts were obtained; add the model, dataset, and decoding details.","section":"Section I"},{"comment":"The parenthesized percentages (e.g., \"+1144.1%\" and \"+5400.4%\") are never defined; state explicitly what baseline they are computed against.","section":"Table III"},{"comment":"Equation (1) labels the metric rep-n but the formula does not show the value of n; state that n is fixed (e.g., n=3) or make the dependence on n explicit in the equation.","section":"Section III-C"},{"comment":"There is a typo in Algorithm 4 (\"blcoks\"), and Figure 6 refers to a \"red-boxed area\" that is not visible in grayscale; fix the typo and use a distinguishable annotation or describe the area in the caption.","section":"Algorithm 4 and Figure 6"},{"comment":"The findings contain grammar errors that obscure the claims, e.g., \"DeRep significant outperformexisting repetition mitigation techniques\" in Finding 1; proofread the findings and the surrounding text.","section":"Section VI-A"},{"comment":"The industrial evaluation reports only repetition-metric deltas and mentions \"sampling interviews\" with over 50 users, but provides no interview protocol, no quantitative user outcomes, and no Pass@1 or correctness measurement; add these details or temper the industrial-effectiveness claim.","section":"Section VI-C"},{"comment":"The repair description says \"for each group of consecutive repeated units,\" but Algorithm 3 returns only the longest contiguous block; clarify whether DeRep repairs multiple separated repeated groups and, if so, how they are identified.","section":"Section V-B"}],"recommendation":"major_revision","confidential_remarks":"The prevalence study and taxonomy are solid and likely publishable on their own. The DeRep evaluation, however, has an internal abstract/body discrepancy and a potentially serious confound: the Pass@1 gain may come from deleting failing self-test code rather than from eliminating semantically redundant repetition. I recommend major revision rather than rejection because the Pass@1 improvement is an independent signal and could be made convincing with controlled experiments and a regression audit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look: this is the first systematic study of repetition in code LLMs, with a sensible 20-pattern taxonomy, new line-level metrics, and a wide 19-model sweep. The prevalence numbers are credible. But the paper's headline Pass@1 gain for their rule-based pruner DeRep is not yet clean evidence that pruning repetition improves correctness; the keep-first repair strategy can delete repeated self-tests that would otherwise crash the module, and there is no control or regression audit separating that effect from pure redundancy removal.\n\nThe empirical study (Sections III-IV) is the strongest part. Running 19 models on HumanEval-Python, HumanEval-Java, and MBPP with three metrics is a solid contribution, and the ground-truth comparison gives a useful baseline. The taxonomy in Table IV is well organized and practical; the new rep-line and sim-line metrics are simple but fit the purpose. The replication package is a plus.\n\nThe trouble is in the evaluation of DeRep. The abstract reports 91.3/93.5/79.9 percent improvements on rep-3/rep-line/sim-line, but Table V only shows a combined rep metric with an 88.3 percent average reduction; those three numbers never appear in the body. That mismatch alone needs fixing. More substantively, DeRep's repair is keep-first: for each repeated block it deletes all later copies. In code, repeated units are often self-test assertions, and if the first copy is benign but later copies fail, deleting them makes the module importable and hidden tests pass. The paper does not check whether that is happening, nor does it report any case where DeRep turns a passing module into a failing one. Without a control that removes the same number of lines at random or strips top-level tests, the 208.3 percent Pass@1 gain cannot be attributed to repetition repair specifically. The industrial results (Table VII) are also much weaker - single-digit to 15 percent reductions - which the paper does not reconcile with the benchmark numbers.\n\nThe similarity threshold is tuned on the same benchmarks without a validation split, and there are no error bars, so some overfitting risk. These are minor compared to the keep-first control issue.\n\nBottom line: this is a genuinely useful empirical paper, and I'd send it to review, but only with a request for a regression audit and a control analysis on Pass@1. The repetition prevalence and taxonomy stand; the mitigation claim needs sharper evidence.","headline":"Useful first map of repetition in code LLMs, but the Pass@1 gain for their pruner needs a control for deleting behavior-affecting code before the mitigation claim is clean.","tokens_in":21190,"tokens_out":4582,"would_cite":true,"duration_ms":49194,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Repetition in LLM-generated code is pervasive, and DeRep, a rule-based post-processor that keeps the first occurrence and deletes later copies, cuts repetition metrics by roughly 80–97% and raises Pass@1 by 208.3% over greedy search.","keywords":["code generation","repetition","large language models","repetition patterns","rule-based post-processing","Pass@1","code quality","empirical study"],"falsifier":"Apply DeRep only to generated outputs that did not reach the token limit: if the Pass@1 gain disappears, the reported improvement comes from recovering truncated outputs rather than from removing repetition per se. Alternatively, inspect the removed duplicate units: if a later duplicate is sometimes the only occurrence that passes the unit tests while the kept first copy fails, then the keep-first rule destroys correct solutions.","tokens_in":20212,"feed_emoji":"✂️","tokens_out":10267,"duration_ms":91166,"temperature":0.7,"pith_summary":"Repetition in LLM-generated code is common enough to bloat outputs and hurt readability, and this paper argues it is also cheap to fix. The authors study 19 open code LLMs on three benchmarks, find repetition at character, statement, and block levels, and distill 20 recurring repetition patterns. They then build DeRep, a rule-based post-processor that detects repeated units and prunes every copy after the first. If the reported numbers hold, a post-processing pass can cut repetition metrics by roughly 80–97 percent and roughly triple the fraction of tasks solved on the first sample, with no retraining and no change to how the model samples. The practical stake is that production code-completion tools can remove a large share of repetitive output after generation.","feed_headline":"Dedup pass cuts AI code repetition by ~90%","feed_subtitle":"A dedup pass that keeps only the first copy raises first-try solve rates by 208 percent.","key_machinery":"The carrying mechanism is DeRep, a lightweight rule-based pipeline that detects repetition at three granularities and then repairs it. Detection is cascading: character-level rules inspect overlength or end-marker lines, statement-level detection scans for the longest contiguous run of lines whose TF-IDF cosine similarity exceeds 0.65, and block-level detection slides over candidate block lengths to find repeated multi-line units, with Tree-sitter used for syntax support. Repair is a single pruning rule: for each group of consecutive repeated units, retain the first valid occurrence and delete all later duplicates, including incomplete trailing copies cut off by the token limit. This keep-the-first-copy rule is what converts the detection taxonomy into a working post-processor.","core_discovery":"The paper reports two findings. First, repetition is pervasive: across 19 open code LLMs on HumanEval-Python, HumanEval-Java, and MBPP, generated code shows far higher repetition than human-written ground truth, at character, statement, and block granularities, and the authors catalog 20 repetition patterns. Second, much of that repetition can be removed after the fact by DeRep, a rule-based post-processor that detects repetitive units and keeps only the first occurrence of each. On six DeepSeekCoder variants, DeRep lowers the composite repetition metric by 88.3 percent over greedy search and raises average Pass@1 from 10.8 to 33.3, a 208.3 percent increase; it also improves five general repetition-mitigation baselines when applied to their outputs, with Pass@1 gains of 53.7 to 215.7 percent. The same method was deployed in an industrial code-completion tool, where it reduced repetition metrics by about 3 to 15 percent across five languages.","pith_inferences":["Inference: If repetition is largely a training-data artifact, as the paper's discussion of code clones and commented-out code suggests, then deduplicating pretraining corpora could attack the root cause; the paper only proposes this as future work.","Inference: The industrial deployment shows much smaller metric reductions (about 3 to 15 percent) than the benchmarks, which suggests real production completions are less repetition-dominated; a natural extension would be to report industrial Pass@1, which the paper does not.","Inference: Because the similarity thresholds (0.65 for detection, 0.8 for sim-line) are tunable, the reported gains are threshold-dependent; a practical next step would be to map how Pass@1 and repetition metrics move as these thresholds vary."],"forward_implications":["Because 89.9 percent of repetitive snippets in the paper's preliminary sample hit the token limit, DeRep's pruning should recover many otherwise truncated outputs as usable code.","DeRep's per-snippet detection takes about 50 milliseconds, so it can run as a real-time post-processing step inside interactive code completion.","Layering DeRep on top of beam search, top-k, top-p, contrastive search, or repetition penalty raises their Pass@1 by 53.7 to 215.7 percent, so existing mitigation strategies become stronger when combined with it.","The 20-pattern taxonomy gives code-quality tooling a concrete checklist for recognizing repetition at character, statement, and block granularity."],"supporting_citations":[{"why":"Supplies the StarCoder model family whose generated code is measured for repetition.","marker":"[1]"},{"why":"Supplies the CodeLlama model family used to show repetition across model sizes and instruction tuning.","marker":"[2]"},{"why":"Provides the DeepSeekCoder backbone on which DeRep and all mitigation baselines are evaluated.","marker":"[3]"},{"why":"DITTO is the existing general repetition-mitigation method that DeRep builds on and improves via post-processing.","marker":"[6]"},{"why":"Supplies the HumanEval benchmark and the observation that LLMs repeat common structures from overfitting.","marker":"[11]"},{"why":"Supplies the MBPP benchmark used in both the prevalence study and the DeRep evaluation.","marker":"[13]"},{"why":"Provides the contrastive search baseline and the rep-n metric adapted for measuring code repetition.","marker":"[29]"},{"why":"Supplies the normalized Levenshtein distance used to define the sim-line similarity metric.","marker":"[30]"},{"why":"Provides the top-k and top-p sampling baselines that DeRep is compared against and combined with.","marker":"[33]"},{"why":"Provides the repetition-penalty baseline, the strongest repetition reducer whose Pass@1 trade-off DeRep is weighed against.","marker":"[34]"}],"fun_headline_variants":["DeRep: rule-based fix cuts LLM code repetition ~90%","Post-hoc dedup raises first-try solve rates 208% in code LLMs","Code LLMs repeat themselves; DeRep cuts repetition by 90%","LLM code repetition pervasive across 19 models, but DeRep fixes it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The repair step keeps only the first occurrence of each repeated unit and discards the rest, so the central premise is that the first copy is a correct and complete implementation while the deleted copies are pure redundancy.","fun_headline_variants_meta":{"raw":{"variants":["DeRep: rule-based fix cuts LLM code repetition ~90%","Post-hoc dedup raises first-try solve rates 208% in code LLMs","Code LLMs repeat themselves; DeRep cuts repetition by 90%","LLM code repetition pervasive across 19 models, but DeRep fixes it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000556,"raw_usage":{"total_tokens":2677,"prompt_tokens":1005,"completion_tokens":1672,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":1589}},"tokens_in":621,"tokens_out":1672,"duration_ms":12999,"temperature":1.0,"reasoning_tokens":1589,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:27:01.928617+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply DeRep only to generated outputs that did not reach the token limit: if the Pass@1 gain disappears, the reported improvement comes from recovering truncated outputs rather than from removing repetition per se. Alternatively, inspect the removed duplicate units: if a later duplicate is sometimes the only occurrence that passes the unit tests while the kept first copy fails, then the keep-first rule destroys correct solutions.","supporting_citations":[{"cited_title":"Learning to break the loop: Analyzing and mitigating repetitions for neural text generation,","cited_arxiv_id":null,"evidence_quote":"DITTO is the existing general repetition-mitigation method that DeRep builds on and improves via post-processing."},{"cited_title":"A contrastive framework for neural text generation,","cited_arxiv_id":null,"evidence_quote":"Provides the contrastive search baseline and the rep-n metric adapted for measuring code repetition."}],"review_version":1}