Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generation

T0 review · 4 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Repetition in LLM-generated code is pervasive, and DeRep, a rule-based post-processor that keeps the first occurrence and deletes later copies, cuts repetition metrics by roughly 80–97% and raises Pass@1 by 208.3% over greedy search.

desk verdict Useful first map of repetition in code LLMs, but the Pass@1 gain for their pruner needs a control for deleting behavior-affecting code before the mitigation claim is clean. read the letter →

arxiv 2504.12608 v1 pith:ME6XSYDW submitted 2025-04-17 cs.SE cs.AI

Mingwei Liu , Juntao Li , Ying Wang , Xueying Du , Zuoyu Ou , Qiuyuan Chen , Bingxu An , Zhao Wei
show 4 more authors
Yong Xu Fangming Zou Xin Peng Yiling Lou
This is my paper · ORCID
classification cs.SEcs.AI
keywords codegenerationrepetitionlargelanguagemodelspatternsrule-basedpost-processingPass@1qualityempiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Repetition in LLM-generated code is common enough to bloat outputs and hurt readability, and this paper argues it is also cheap to fix. The authors study 19 open code LLMs on three benchmarks, find repetition at character, statement, and block levels, and distill 20 recurring repetition patterns. They then build DeRep, a rule-based post-processor that detects repeated units and prunes every copy after the first. If the reported numbers hold, a post-processing pass can cut repetition metrics by roughly 80–97 percent and roughly triple the fraction of tasks solved on the first sample, with no retraining and no change to how the model samples. The practical stake is that production code-completion tools can remove a large share of repetitive output after generation.

What carries the argument

The carrying mechanism is DeRep, a lightweight rule-based pipeline that detects repetition at three granularities and then repairs it. Detection is cascading: character-level rules inspect overlength or end-marker lines, statement-level detection scans for the longest contiguous run of lines whose TF-IDF cosine similarity exceeds 0.65, and block-level detection slides over candidate block lengths to find repeated multi-line units, with Tree-sitter used for syntax support. Repair is a single pruning rule: for each group of consecutive repeated units, retain the first valid occurrence and delete all later duplicates, including incomplete trailing copies cut off by the token limit. This keep-the-first-copy rule is what converts the detection taxonomy into a working post-processor.

What would settle it

Apply DeRep only to generated outputs that did not reach the token limit: if the Pass@1 gain disappears, the reported improvement comes from recovering truncated outputs rather than from removing repetition per se. Alternatively, inspect the removed duplicate units: if a later duplicate is sometimes the only occurrence that passes the unit tests while the kept first copy fails, then the keep-first rule destroys correct solutions.

Watch

Extended reading notes

Core claim

The paper reports two findings. First, repetition is pervasive: across 19 open code LLMs on HumanEval-Python, HumanEval-Java, and MBPP, generated code shows far higher repetition than human-written ground truth, at character, statement, and block granularities, and the authors catalog 20 repetition patterns. Second, much of that repetition can be removed after the fact by DeRep, a rule-based post-processor that detects repetitive units and keeps only the first occurrence of each. On six DeepSeekCoder variants, DeRep lowers the composite repetition metric by 88.3 percent over greedy search and raises average Pass@1 from 10.8 to 33.3, a 208.3 percent increase; it also improves five general repetition-mitigation baselines when applied to their outputs, with Pass@1 gains of 53.7 to 215.7 percent. The same method was deployed in an industrial code-completion tool, where it reduced repetition metrics by about 3 to 15 percent across five languages.

Load-bearing premise

The repair step keeps only the first occurrence of each repeated unit and discards the rest, so the central premise is that the first copy is a correct and complete implementation while the deleted copies are pure redundancy.

Editorial extensions

If this is right

  • Because 89.9 percent of repetitive snippets in the paper's preliminary sample hit the token limit, DeRep's pruning should recover many otherwise truncated outputs as usable code.
  • DeRep's per-snippet detection takes about 50 milliseconds, so it can run as a real-time post-processing step inside interactive code completion.
  • Layering DeRep on top of beam search, top-k, top-p, contrastive search, or repetition penalty raises their Pass@1 by 53.7 to 215.7 percent, so existing mitigation strategies become stronger when combined with it.
  • The 20-pattern taxonomy gives code-quality tooling a concrete checklist for recognizing repetition at character, statement, and block granularity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: If repetition is largely a training-data artifact, as the paper's discussion of code clones and commented-out code suggests, then deduplicating pretraining corpora could attack the root cause; the paper only proposes this as future work.
  • Inference: The industrial deployment shows much smaller metric reductions (about 3 to 15 percent) than the benchmarks, which suggests real production completions are less repetition-dominated; a natural extension would be to report industrial Pass@1, which the paper does not.
  • Inference: Because the similarity thresholds (0.65 for detection, 0.8 for sim-line) are tunable, the reported gains are threshold-dependent; a practical next step would be to map how Pass@1 and repetition metrics move as these thresholds vary.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper investigates repetitive or degenerate code generated by LLMs. It first conducts an empirical study across 19 code LLMs on HumanEval-Python, HumanEval-Java, and MBPP, quantifying repetition with three metrics (rep-n, rep-line, and sim-line) and deriving a taxonomy of 20 repetition patterns at character, statement, and block levels. It then proposes DeRep, a rule-based post-processor that detects repeated units and repairs them by keeping the first occurrence and deleting later duplicates. The evaluation claims that DeRep reduces repetition by 88.3% on a combined rep metric and improves Pass@1 from 10.8 to 33.3 (a 208.3% relative gain) over greedy search across six DeepSeekCoder variants, and that it also improves the Pass@1 of existing repetition-mitigation baselines by 53.7% to 215.7%. The paper additionally reports a deployment in an industrial code-completion setting.

Significance. If the DeRep results hold, the paper would make a useful and practical contribution: repetition in code LLMs is shown to be pervasive, and a cheap rule-based post-processor appears to fix a large part of it without retraining. The prevalence study is a genuine strength: it covers 19 models, uses three benchmarks, follows a consistent greedy-decoding protocol, and the resulting taxonomy should be useful to future work. The Pass@1 improvement is an important independent signal, because it shows that the method does something beyond lowering self-defined repetition metrics. The industrial deployment, though reported at a high level, adds real-world relevance. The main weaknesses lie in the evaluation of DeRep: the abstract reports numbers that do not appear in the body, the repair strategy can delete behavior-affecting code rather than only redundancy, and the detection thresholds appear to be tuned on the evaluation set without validation. These issues must be resolved before the central claims can be accepted.

major comments (4)
  1. [Abstract and Section VI-A] The abstract claims average improvements of 91.3%, 93.5%, and 79.9% in rep-3, rep-line, and sim-line, but these numbers do not appear anywhere in the body. Table V reports only a combined rep metric, with an average reduction of 88.3% for DeRep applied to greedy search. Because the abstract frames the central result, the authors must either provide the per-metric breakdown with the exact protocol or correct the abstract so that it matches the evaluation actually reported.
  2. [Section V-B and Table V] The Pass@1 improvement may be inflated by deleting behavior-affecting code rather than pure redundancy. The repair strategy keeps the first occurrence of each detected repeated unit and deletes all later duplicates. In the standard HumanEval harness, repeated assertions and self-test statements (which Table IV identifies as common patterns) execute as top-level code, and a later wrong assertion can crash the generated module. If DeRep deletes such failing assertions, the repaired module can load and pass hidden tests even though the function itself was never repaired. The paper provides no regression audit, such as comparing DeRep against a control that truncates at the first repeated block, or verifying that the repaired code preserves the behavior of the model's own generated assertions. Section VIII does not discuss this threat, and it is load-bearing for the 208.3% Pass@1 claim.
  3. [Section V-A] The detection thresholds appear to be tuned on the same data used for evaluation. The cosine similarity threshold of 0.65 is described as "based on empirical tuning," and the block-length upper bound Lmax is described as "empirically set" to 2n/3. No validation split, threshold sensitivity analysis, or procedure for choosing these values is reported. Since the evaluation metrics are computed with the same line- and block-similarity machinery, the numerical reductions in Table V may be optimistically biased. The authors should describe how the thresholds were selected, use a held-out split, and report how Table V changes across a range of threshold values.
  4. [Section III-C and Section VI-A] The repetition-reduction results are partly by construction. rep-line is the proportion of exact duplicate lines and sim-line is based on Levenshtein similarity, while DeRep detects exactly the repeated lines and blocks flagged by TF-IDF cosine similarity and deletes them. Large reductions in these metrics are therefore expected regardless of whether the deleted code is semantically redundant. The Pass@1 signal mitigates this concern, but it does not fully resolve it in light of the assertion-deletion issue raised above. An independent assessment, such as human readability ratings, functional-equivalence checks, or a test of whether repaired code still passes the model's own generated assertions, is needed to support the claim that DeRep improves code quality rather than only lowering the self-defined metrics.
minor comments (7)
  1. [Section I] The preliminary analysis reports 10,399 snippets with repetition and 9,346 cases (89.9%) exceeding the token limit, but no methodology is given for how these counts were obtained; add the model, dataset, and decoding details.
  2. [Table III] The parenthesized percentages (e.g., "+1144.1%" and "+5400.4%") are never defined; state explicitly what baseline they are computed against.
  3. [Section III-C] Equation (1) labels the metric rep-n but the formula does not show the value of n; state that n is fixed (e.g., n=3) or make the dependence on n explicit in the equation.
  4. [Algorithm 4 and Figure 6] There is a typo in Algorithm 4 ("blcoks"), and Figure 6 refers to a "red-boxed area" that is not visible in grayscale; fix the typo and use a distinguishable annotation or describe the area in the caption.
  5. [Section VI-A] The findings contain grammar errors that obscure the claims, e.g., "DeRep significant outperformexisting repetition mitigation techniques" in Finding 1; proofread the findings and the surrounding text.
  6. [Section VI-C] The industrial evaluation reports only repetition-metric deltas and mentions "sampling interviews" with over 50 users, but provides no interview protocol, no quantitative user outcomes, and no Pass@1 or correctness measurement; add these details or temper the industrial-effectiveness claim.
  7. [Section V-B] The repair description says "for each group of consecutive repeated units," but Algorithm 3 returns only the longest contiguous block; clarify whether DeRep repairs multiple separated repeated groups and, if so, how they are identified.

Circularity Check

1 steps flagged · score 6.0 of 10

Repetition-reduction results are entailed by DeRep's delete-duplicates repair; Pass@1 is an independent signal that keeps the paper from full circularity.

  1. self definitional [Section V-B (Repetition Repair) and Section VI-A (RQ3.a, Table V)]
    ""The core strategy is straightforward: for each group of consecutive repeated units, we retain only the first valid occurrence and remove all subsequent duplicates." ... "Applying our DeRep method to greedy search yields average improvement of 88.3% in rep metric.""

    The rep metric used in evaluation (Section VI-A) is the average of rep-n, rep-line, and sim-line, which measure the proportion of repeated n-grams, duplicate lines, and edit-distance-similar lines. DeRep's repair phase deletes every detected repeated unit, keeping only the first occurrence. Removing duplicates mechanically lowers all three components: the number of repeated n-grams shrinks, duplicate lines are deleted, and similar lines are pruned. Therefore the reported 88.3%–97.0% repetition reductions are not empirical discoveries but the direct output of the algorithm's own definition.

full rationale

DeRep's headline repetition-reduction numbers (88.3%–97.0%) are largely self-definitional. The evaluation metric 'rep' is defined as the average of rep-n, rep-line, and sim-line, which quantify repeated n-grams, duplicate lines, and edit-distance-similar lines. The repair mechanism is literally 'retain only the first valid occurrence and remove all subsequent duplicates' for detected repeated units. Removing duplicates cannot increase uniqueness; it mechanically reduces rep-line, sim-line, and n-gram repetition. Thus the Table V comparison on rep is not a test of a prediction but a restatement of the algorithm's own operation. The Pass@1 improvement (10.8 to 33.3 average) is an independent outcome not entailed by the repetition metrics alone, which prevents the whole paper from being tautological. However, the paper does not isolate whether Pass@1 gains come from removing genuinely redundant code or from deleting failing self-test assertions that would otherwise crash the module; that is an unaddressed confound, though not itself a circularity. No load-bearing self-citation or imported uniqueness theorem appears; the taxonomy and prevalence analysis are grounded in externally generated benchmark outputs. Overall, this is partial circularity: the central repetition-reduction claim reduces by construction, while the correctness claim retains independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rely on hand-tuned detection thresholds (line length 150, cosine 0.65, Lmax 2n/3), the assumption that greedy decoding reveals typical repetition, and the assumption that the first repeated unit is correct. The repetition metrics are newly proposed in this paper and are used both to define the problem and to measure the fix, creating a partial circularity burden.

free parameters (4)
  • Line over-length threshold = 150 characters
    Algorithm 2 uses 150 as the threshold to flag a line as possibly containing character-level repetition.
  • TF-IDF cosine similarity threshold = 0.65
    Used in statement-level and block-level detection; described as empirically tuned in Section V-A.
  • Maximum block length Lmax = 2n/3
    Empirically set in Algorithm 4 to balance detection coverage and computational efficiency.
  • Minimum block length Lmin = 2
    Default minimum contiguous block size for block-level repetition detection in Algorithm 4.
assumptions (4)
  • domain assumption Greedy decoding exposes typical repetition behavior
    The study sets maximum window length to 512 and uses greedy search to characterize repetition; the paper does not test sampling-based decoding in the empirical study.
  • domain assumption Repetition metrics rep-line and sim-line validly capture code quality
    These metrics are newly proposed in this paper and are used as the ground truth for prevalence and as the evaluation criteria for DeRep, without prior validation.
  • ad hoc to paper First occurrence of a repeated unit is correct and complete
    Section V-B states the repair keeps only the first valid occurrence and removes all subsequent duplicates; the whole Pass@1 improvement depends on this being true for the evaluated cases.
  • ad hoc to paper Thresholds chosen by empirical tuning without a held-out set are representative
    The line length threshold (150) and cosine similarity threshold (0.65) are described as empirically tuned, but no validation split is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generation." pith.science (2026). https://pith.science/paper/ME6XSYDW

@misc{pith2026250412608,
  author       = {Pith},
  title        = {Pith review of: Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ME6XSYDW}},
  note         = {Machine review of arXiv:2504.12608}
}
read the original abstract

Despite recent advances in Large Language Models (LLMs) for code generation, the quality of LLM-generated code still faces significant challenges. One significant issue is code repetition, which refers to the model's tendency to generate structurally redundant code, resulting in inefficiencies and reduced readability. To address this, we conduct the first empirical study to investigate the prevalence and nature of repetition across 19 state-of-the-art code LLMs using three widely-used benchmarks. Our study includes both quantitative and qualitative analyses, revealing that repetition is pervasive and manifests at various granularities and extents, including character, statement, and block levels. We further summarize a taxonomy of 20 repetition patterns. Building on our findings, we propose DeRep, a rule-based technique designed to detect and mitigate repetition in generated code. We evaluate DeRep using both open-source benchmarks and in an industrial setting. Our results demonstrate that DeRep significantly outperforms baselines in reducing repetition (with an average improvements of 91.3%, 93.5%, and 79.9% in rep-3, rep-line, and sim-line metrics) and enhancing code quality (with a Pass@1 increase of 208.3% over greedy search). Furthermore, integrating DeRep improves the performance of existing repetition mitigation methods, with Pass@1 improvements ranging from 53.7% to 215.7%.

Figures

Figures reproduced from arXiv: 2504.12608 by the authors.

Figure 1
Figure 1. Examples of Endless Repetition Cases of Code LLMs [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. rep_n across Different n-grams on H-P+MBPP [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distribution of rep-3, rep-line, and sim-line Metrics [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Trends of Metrics by Input Tokens Number Intervals [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Trends of Repetition Metrics by Ground Truth Tokens [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Examples of Code Repair Using Our Approach [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Repetition Pattern Proportions Across Different Code LLMs [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Repetition Pattern per LLM industrial applications, enhancing the overall code completion experience. VII. DISCUSSIONS AND FUTURE DIRECTIONS In this section, we explore the distinct characteristics and prevalence of repetition in code generation compared to general tex…
Figure 9
Figure 9. Figure 9: Examples of Repetitive Patterns in Training Corpus of Code LLMs [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficient Black-Box Fault Localization for System-Level Test Code Using Large Language Models

    cs.SE 2025-06 unverdicted novelty 7.0 of 10

    A fully static, LLM-driven pipeline estimates the execution trace from a single failure log, prunes the test code, and ranks faulty locations at function, block, and line levels, tested on 785 industrial Python test cases.

Reference graph

Works this paper leans on

35 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, and Y . Z. et al., “Starcoder: may the source be with you!” CoRR, vol. abs/2305.06161, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2305.06161

    arXiv doi:10.48550/arxiv.2305.06161 2023

  2. [2]

    Code llama: Open foundation models for code,

    B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. Canton-Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve, “Code llama: Open foundation models for code,” CoRR, vol. abs/2...

    arXiv doi:10.48550/arxiv.2308.12950 2023

  3. [3]

    Deepseek-coder: When the large language model meets programming - the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . K. Li, F. Luo, Y . Xiong, and W. Liang, “Deepseek-coder: When the large language model meets programming - the rise of code intelligence,” CoRR, vol. abs/2401.14196, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2401.14196

    arXiv doi:10.48550/arxiv.2401.14196 2024

  4. [4]

    Automated repair of programs from large language models,

    Z. Fan, X. Gao, M. Mirchev, A. Roychoudhury, and S. H. Tan, “Automated repair of programs from large language models,” in 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 2023, pp. 1469–

    work page 2023

  5. [5]

    Repetition in repetition out: Towards understanding neural text degeneration from the data perspective,

    H. Li, T. Lan, Z. Fu, D. Cai, L. Liu, N. Collier, T. Watanabe, and Y . Su, “Repetition in repetition out: Towards understanding neural text degeneration from the data perspective,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, ...

    work page 2023

  6. [6]

    Learning to break the loop: Analyzing and mitigating repetitions for neural text generation,

    J. Xu, X. Liu, J. Yan, D. Cai, H. Li, and J. Li, “Learning to break the loop: Analyzing and mitigating repetitions for neural text generation,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , S. Koyejo, S. Mohamed,...

    work page 2022

  7. [7]

    [Online]

    Code repetition on github. [Online]. Available: https://anonymous.4open. science/r/CodeRepetition-30F4/

    work page

  8. [8]

    Can large language models write good property-based tests?

    V . Vikram, C. Lemieux, and R. Padhye, “Can large language models write good property-based tests?” CoRR, vol. abs/2307.04346, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2307.04346

    arXiv doi:10.48550/arxiv.2307.04346 2023

Show all 35 references
  1. [9]

    Large language models are few-shot testers: Exploring llm-based general bug reproduction,

    S. Kang, J. Yoon, and S. Yoo, “Large language models are few-shot testers: Exploring llm-based general bug reproduction,” in 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 2023, pp. 2312–

    2023

  2. [10]

    Explainable automated debugging via large language model-driven scientific debugging,

    S. Kang, B. Chen, S. Yoo, and J. Lou, “Explainable automated debugging via large language model-driven scientific debugging,” CoRR, vol. abs/2304.02195, 2023. [Online]. Available: https://doi.org/10.48550/ arXiv.2304.02195

    2023 doi arXiv

  3. [11]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, and H. J. et al., “Evaluating large language models trained on code,” CoRR, vol. abs/2107.03374, 2021. [Online]. Available: https://arxiv.org/abs/2107.03374

    2021 arXiv

  4. [12]

    Repocoder: Repository-level code completion through iterative retrieval and generation,

    F. Zhang, B. Chen, Y . Zhang, J. Keung, J. Liu, D. Zan, Y . Mao, J. Lou, and W. Chen, “Repocoder: Repository-level code completion through iterative retrieval and generation,” pp. 2471–2484, 2023. [Online]. Available: https://aclanthology.org/2023.emnlp-main.151

    2023

  5. [14]

    Testeval: Benchmarking large language models for test case generation,

    W. Wang, C. Yang, Z. Wang, Y . Huang, Z. Chu, D. Song, L. Zhang, A. R. Chen, and L. Ma, “Testeval: Benchmarking large language models for test case generation,” arXiv preprint arXiv:2406.04531 , 2024

    2024 arXiv

  6. [15]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,

    T. Y . Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul et al., “Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,” arXiv preprint arXiv:2406.15877, 2024

    2024 arXiv

  7. [16]

    [Online]

    Classeval on github. [Online]. Available: https://github.com/FudanSELab/ ClassEval JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13

    2021

  8. [17]

    Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion,

    Y . Ding, Z. Wang, W. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth et al. , “Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  9. [18]

    Evocodebench: An evolving code generation benchmark aligned with real-world code repositories,

    J. Li, G. Li, X. Zhang, Y . Dong, and Z. Jin, “Evocodebench: An evolving code generation benchmark aligned with real-world code repositories,” arXiv preprint arXiv:2404.00599 , 2024

    2024 arXiv

  10. [19]

    Bugs in large language models generated code,

    F. Tambon, A. M. Dakhel, A. Nikanjam, F. Khomh, M. C. Desmarais, and G. Antoniol, “Bugs in large language models generated code,” arXiv preprint arXiv:2403.08937, 2024

    2024 arXiv

  11. [20]

    Exploring and evaluating hallucinations in llm-powered code generation,

    F. Liu, Y . Liu, L. Shi, H. Huang, R. Wang, Z. Yang, and L. Zhang, “Exploring and evaluating hallucinations in llm-powered code generation,” arXiv preprint arXiv:2404.00971 , 2024

    2024

  12. [21]

    Beyond functional correctness: Investigating coding style inconsistencies in large language models,

    Y . Wang, T. Jiang, M. Liu, J. Chen, and Z. Zheng, “Beyond functional correctness: Investigating coding style inconsistencies in large language models,” arXiv preprint arXiv:2407.00456 , 2024

    2024 arXiv

  13. [22]

    Santacoder: don’t reach for the stars!

    L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey, L. K. Umapathi, C. J. Anderson, Y . Zi, J. Lamy-Poirier, H. Schoelkopf, S. Troshin, D. Abulkhanov, M. Romero, M. Lappert, F. D. Toni, B. G. del Río, Q. Liu, S. Bose, U...

    2023 doi arXiv

  14. [23]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Wei, T. Liu, M. Tian, D. Kocetkov, A. Zucker, Y . Belkada, Z. Wang, Q. Liu, D. Abulkhanov, I. Paul, Z. Li, W. Li, M. Risdal, J. Li, J. Zhu, T. Y . Zhuo, E. Zheltonozhskii, N....

    2024 doi arXiv

  15. [24]

    Wizardcoder: Empowering code large language models with evol-instruct,

    Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang, “Wizardcoder: Empowering code large language models with evol-instruct,” CoRR, vol. abs/2306.08568, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2306.08568

    2023 doi arXiv

  16. [25]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, and G. I. et al., “Llama: Open and efficient foundation language models,” CoRR, vol. abs/2302.13971, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2302.13971

    2023 doi arXiv

  17. [26]

    Magicoder: Source code is all you need,

    Y . Wei, Z. Wang, J. Liu, Y . Ding, and L. Zhang, “Magicoder: Source code is all you need,” CoRR, vol. abs/2312.02120, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2312.02120

    2023 doi arXiv

  18. [27]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” CoRR, vol. abs/2305.01210, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2305.01210

    2023 doi arXiv

  19. [28]

    Multi- lingual evaluation of code generation models,

    B. Athiwaratkun, S. K. Gouda, and Z. W. et al., “Multi- lingual evaluation of code generation models,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. [Online]. Available: https://openreview...

    2023

  20. [29]

    A contrastive framework for neural text generation,

    Y . Su, T. Lan, Y . Wang, D. Yogatama, L. Kong, and N. Collier, “A contrastive framework for neural text generation,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, N...

    2022

  21. [30]

    A normalized levenshtein distance metric,

    L. Yujian and L. Bo, “A normalized levenshtein distance metric,” IEEE transactions on pattern analysis and machine intelligence , vol. 29, no. 6, pp. 1091–1095, 2007

    2007

  22. [31]

    An information-theoretic perspective of tf–idf measures,

    A. Aizawa, “An information-theoretic perspective of tf–idf measures,” Information Processing & Management , vol. 39, no. 1, pp. 45–65, 2003

    2003

  23. [32]

    Beam search strategies for neural machine translation,

    M. Freitag and Y . Al-Onaizan, “Beam search strategies for neural machine translation,” arXiv preprint arXiv:1702.01806 , 2017

    2017 arXiv

  24. [33]

    The curious case of neural text degeneration,

    A. Holtzman, J. Buys, L. Du, M. Forbes, and Y . Choi, “The curious case of neural text degeneration,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. [Online]. Available: https://openrevie...

    2020

  25. [34]

    CTRL: A conditional transformer language model for controllable generation,

    N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher, “CTRL: A conditional transformer language model for controllable generation,” CoRR, vol. abs/1909.05858, 2019. [Online]. Available: http://arxiv.org/abs/1909.05858

    1909 arXiv

  26. [2021]

    Available: https://arxiv.org/abs/2108.07732

    [Online]. Available: https://arxiv.org/abs/2108.07732

    arXiv

  27. [2323]

    Available: https://doi.org/10.1109/ICSE48619.2023.00194

    [Online]. Available: https://doi.org/10.1109/ICSE48619.2023.00194

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.