Pith. sign in

REVIEW 3 major objections 6 minor 26 references

On the Possibility of Breaking Copyleft Licenses When Reusing Code Generated by ChatGPT

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ChatGPT reproduces copyleft code more than five times as often when given the full class as context.

desk verdict Solid measurement study: the context effect on reproducing copyleft code is real and new, but a couple of secondary claims are overgeneralized. read the letter →

arxiv 2502.05023 v1 pith:ZDFNJQHD submitted 2025-02-07 cs.SE

Gaia Colombo , Leonardo Mariani , Daniela Micucci , Oliviero Riganelli This is my paper · ORCID
classification cs.SE
keywords AI-assistedcodingcodegenerationcopyleftlicensesintellectualpropertyChatGPTGPT-4-turbomemorizationplagiarismdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether code generated by ChatGPT can accidentally reproduce copyleft-licensed code, and what makes that more or less likely. Using more than 70,000 generated Java method implementations, the authors find that while individual requests rarely match copyleft code, providing the rest of the class as context raises the worst-case similarity rate from 3.35% to 19.50%—more than a fivefold increase. Even just including access methods roughly doubles the risk. Raising the temperature to 2 reduces similarity, but explicitly instructing the model not to copy known implementations has no meaningful effect. The results matter because developers who accept one suspicious recommendation may unwittingly increase the chance that subsequent suggestions also carry copyleft restrictions.

What carries the argument

The study centers on a dataset of 7,347 Java methods extracted from 146 copyleft-licensed GitHub repositories that existed before December 2020, guaranteeing the code predates GPT-4-turbo's training cutoff. Similarity between generated and original methods is measured with JPlag's token-level similarity and a character-level fuzzy ratio based on Levenshtein distance, with suspicious cases defined as similarity above 0.7. The experimental machinery systematically varies prompts: baseline uses signature plus JavaDoc, the full-class condition embeds the rest of the class, the access-method condition adds only getters and setters, temperature is set to 0, 1, or 2, and one prompt explicitly asks the model not to copy. Wilcoxon signed-rank tests with effect sizes distinguish real factor effects from noise.

What would settle it

Replicate the experiment with a set of copyleft-licensed Java methods published after GPT-4-turbo's training cutoff (for example, created in 2024). If the full-class context still raises the fraction of generated methods with similarity above 0.7 to roughly 19%, then the amplification effect does not require the methods to have been in training data, contradicting the memorization interpretation; if the rate stays near the baseline 3.35%, then the context effect is indeed tied to training set exposure.

Watch

Extended reading notes

Core claim

The central claim is that the likelihood ChatGPT returns code closely resembling copyleft-licensed implementations depends strongly on the context supplied in the prompt. When the prompt contains only the method signature and JavaDoc, 3.35% of methods exceed a maximum similarity of 0.7 to the original copyleft code. When the prompt includes the rest of the class, this rises to 19.50%; with only access methods as context, it rises to 18.70% on the restricted sample. Differences are statistically significant with large effect sizes for the class context. A temperature of 2 lowers similarity compared to temperature 1, while temperature 0 has a mild effect. An explicit request to avoid copying does not shift similarity in a statistically significant way.

Load-bearing premise

The study assumes that the GitHub methods selected from repositories existing before December 2020 were actually part of GPT-4-turbo's training data, an assumption it cannot verify directly; if many were not, the observed similarity increases could be due to in-context pattern completion rather than memorization of copyleft code.

Editorial extensions

If this is right

  • Developers who accept one code recommendation that matches copyleft code are substantially more likely to receive further such recommendations when they later ask for other methods in the same class.
  • Using a high temperature setting (for example 2) reduces the chance of reproducing copyleft code, though the paper notes this must be balanced against evidence that high temperatures degrade code correctness.
  • Telling ChatGPT not to copy known implementations is not an effective safeguard, so technical and process-level defenses are needed rather than prompt-based ones.
  • The measured risk is likely an under-approximation because the dataset only covers GitHub repositories selected before December 2020, and other code sources are not considered.
  • The observed context amplification is expected to transfer beyond GPT-4-turbo and beyond Java, since the authors attribute it to the memorization properties of large language models rather than to a model-specific quirk.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of whether high similarity truly reflects training-data memorization would be to repeat the protocol on methods published after GPT-4-turbo's training cutoff; if the context effect persists for post-cutoff methods, the mechanism would be in-context pattern completion rather than memorization.
  • The cascade effect implies a kind of licensing ratchet: the more copyleft-like code a developer incorporates, the harder it becomes to keep later suggestions clean, which raises the practical importance of checking every generated method before acceptance.
  • The same measurement protocol could be applied to open-weight models with inspectable training data to separate memorization from other causes, and to estimate how much variability across models is due to training-set overlap rather than architecture.
  • One could design a lightweight run-time detector that flags generated methods whose surrounding class context matches a copyleft repository, using the paper's observed 5x amplification as motivation for checking context similarity before inserting a generated method.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper empirically investigates whether ChatGPT (gpt-4-turbo) generates Java methods that are similar to copyleft-licensed implementations, and how context, temperature, and explicit prompts affect this behavior. The authors build a dataset of 7,347 methods from 146 copyleft-licensed GitHub repositories, generate more than 70,000 method implementations across five research questions, and measure similarity using JPlag and a character-level fuzzy ratio. They report that, with only signature and JavaDoc context, 3.35% of methods have a max similarity above 0.7 (RQ1); adding the full class context raises this to 19.50% (RQ2, ~5.8X); adding only access methods raises it about 2X on a subpopulation (RQ3); higher temperature (2) reduces similarity (RQ4); and an explicit request not to copy has no significant effect (RQ5).

Significance. The paper has notable strengths: a large stratified sample, five responses per request, multiple similarity metrics, exploration of the full threshold range, Wilcoxon signed-rank tests with effect sizes, and a public replication package. The RQ2 finding that full-class context multiplies the rate of high-similarity outputs by a factor of five is practically relevant for developers and for designers of AI-assisted coding tools, and the RQ4 result on temperature is a concrete mitigation hypothesis. If the results are interpreted carefully, the study advances the empirical understanding of license-related risks in LLM code generation. The main caveats are that training-data membership of the selected methods is unverified, and the RQ3 result is computed on a non-representative subset, limiting the generality of the headline 2X factor.

major comments (3)
  1. [III-B (Method Selection for RQ3) and Table IV] The RQ3 analysis is performed only on the 631 methods that were already flagged as likely plagiarized under the full-class context of RQ2. This selection makes the reported 2X increase (from 9.19% to 18.70% for max similarity) conditional on prior flagging, and it does not establish that access methods increase plagiarism risk for the general population of methods. The Answer to RQ3 and the Conclusion state that 'a matching set of access methods ... increases the chance ... by a 2X factor' without this qualification. Please either re-run RQ3 on a random sample of methods not pre-filtered by RQ2, or explicitly frame the result as applying only to methods already at high risk under full-class context.
  2. [IV-F (Threats to Validity) and IV-A (RQ1)] The paper states that methods from repositories existing before December 2020 are 'almost certainly' processed by GPT-4-turbo, but no verification is provided. This assumption is load-bearing for the claim that high similarity indicates 'reproducing' copyleft code from training data. In particular, for RQ2 the prompt includes the rest of the class with the target method removed, so the model could generate a similar method by in-context completion from the provided code alone, without any memorization of the training data. Please add a training-membership check or, alternatively, consistently soften the language from 'reproduces' to 'produces code similar to' and explicitly discuss in-context completion as a possible mechanism for the RQ2 effect.
  3. [IV-E (RQ5)] The Conclusion and the Answer to RQ5 claim that ChatGPT 'cannot be asked, through the prompt, to avoid reusing existing code.' This is too broad: the experiment tested a single prompt variant ('not copy any known implementation') on a small selected subset (239 methods with max similarity above 0.9 in RQ1). The result shows that this particular instruction has no significant effect on that subset, but it does not warrant the general conclusion that no prompt can reduce reuse. Please limit the claim to the specific instruction tested, or add additional prompt variants to support the stronger claim.
minor comments (6)
  1. [III-B (Plagiarism detection)] The text says that results are discussed for thresholds above 0.7 and that every threshold value with a step of 0.05 was considered, but the paper only presents tables and figures for the 0.7 threshold. It would be helpful to show how the conclusions change across the threshold sweep, even in an appendix.
  2. [III-A (Repository Selection)] The description of the repository selection criteria says 'by the end of 2020' and later 'before December 2020.' Please make the cutoff phrasing consistent.
  3. [IV-B (RQ2 Results)] The text describes the difference distributions as 'skewed on the left,' but does not explicitly state that negative differences correspond to higher similarity with the broader context. Adding axis labels and a sentence explaining the sign convention would improve readability.
  4. [IV-D (RQ4)] For the temperature 0 versus 1 comparison, the paper reports p-values but not effect sizes, while effect sizes are reported for the temperature 2 comparison. Please report effect sizes for all statistically significant comparisons, including the fuzzy ratio results.
  5. [IV-F (Threats to Validity)] The sentence 'we do expect the observed degree of reuse of existing code to be an intrinsic characteristic of the architecture of the model' is speculative and not supported by the single-model design. It would be more accurate to say that the findings are limited to GPT-4-turbo and should be replicated on other models.
  6. [References] There are a few typos: '2,3X increase' in Section IV-C should be '2.3X'; reference [2] has 'EEE/ACM' instead of 'IEEE/ACM'; and in reference [8] the author name appears as 'Y . V orobeychik' with a stray space.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the study's claims are direct measurements of ChatGPT outputs against a fixed copyleft dataset, with no fitted parameters and no load-bearing self-citations.

full rationale

The paper reports an empirical measurement study: it samples 7,347 copyleft-licensed Java methods, prompts GPT-4-turbo under controlled conditions, and compares returned code to the dataset using JPlag similarity and fuzzy ratio. The headline results (e.g., 3.35% to 19.50% max similarity above 0.7 when full class context is provided) are counts of observed similarity scores, not outputs of any model fitted to the data. The 0.7 threshold is explicitly treated as a design choice and the authors state they consider every threshold value with a step of 0.05, so the threshold does not define the result by construction. The paper's acknowledged threat that training-data membership cannot be verified is an empirical assumption, not a logical circularity: even if some methods were absent from training data, the measured context effect is still a real observation about the model's behavior in the tested conditions. The self-citations (references [4], [5], and [17]) concern prior work on code generation configuration and are used only as related work, not as load-bearing justification for the present claims. No equation is defined in terms of its own output, no fitted input is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. The derivation chain is therefore self-contained with respect to the paper's actual claims.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The measurements depend on the assumption that the target repositories appear in the model's training data and that JPlag similarity is a valid proxy for copyright-relevant copying. The first is acknowledged in Threats to Validity; the second is partially supported by manual inspection. The only hand-chosen number in the pipeline, the 0.7 similarity cutoff, is varied across all thresholds in the analysis, so it does not constitute a fitted parameter.

free parameters (1)
  • Similarity threshold for 'potentially plagiarized' = 0.70
    Hand-chosen cutoff used to count suspicious cases (Tables II-IV); the authors report they inspected cases above and below 0.7 and also present results for every 0.05 step, so the threshold does not drive a fitted conclusion.
assumptions (3)
  • domain assumption Selected GitHub repositories were part of GPT-4-turbo's training data
    The authors assume that methods from repositories created before December 2020 were seen during training because the model's cutoff is 2023. This cannot be verified externally and is load-bearing for the interpretation that high similarity constitutes memorization of training data.
  • domain assumption JPlag token similarity above 0.7 indicates copyright-relevant copying
    The study treats similarity as a proxy for plagiarism; the authors manually inspected a sample to justify the threshold, but no legal or ground-truth definition of copying is used.
  • domain assumption Five responses per request capture the model's output distribution
    The paper aggregates five completions per method and reports mean and max; this sample size is arbitrary but reasonable for a nondeterministic API.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Possibility of Breaking Copyleft Licenses When Reusing Code Generated by ChatGPT." pith.science (2026). https://pith.science/paper/ZDFNJQHD

@misc{pith2026250205023,
  author       = {Pith},
  title        = {Pith review of: On the Possibility of Breaking Copyleft Licenses When Reusing Code Generated by ChatGPT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZDFNJQHD}},
  note         = {Machine review of arXiv:2502.05023}
}
read the original abstract

AI assistants can help developers by recommending code to be included in their implementations (e.g., suggesting the implementation of a method from its signature). Although useful, these recommendations may mirror copyleft code available in public repositories, exposing developers to the risk of reusing code that they are allowed to reuse only under certain constraints (e.g., a specific license for the derivative software). This paper presents a large-scale study about the frequency and magnitude of this phenomenon in ChatGPT. In particular, we generate more than 70,000 method implementations using a range of configurations and prompts, revealing that a larger context increases the likelihood of reproducing copyleft code, but higher temperature settings can mitigate this issue.

Figures

Figures reproduced from arXiv: 2502.05023 by the authors.

Figure 1
Figure 1. Distribution of similarity scores for mean similarity [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Distribution of similarity scores for max similarity. syntactic elements could be found even between rather diverse implementations. Yet the vast majority of the code has a low fuzzy ratio. We zoom into the cases with suspicious similarity levels reporting in Table II the percentage of methods with a similar￾ity above 0.7. Specifically, 102 methods (1.48%) have a mean similarity above 0.70, while 232 methods (3.35%)… view at source ↗
Figure 4
Figure 4. Difference between JavaDoc comment and whole class in the case of [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Similarity Scores for JavaDoc comment and whole class contexts in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 3
Figure 3. Figure 3: Difference between JavaDoc comment and whole class in the case of [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]
Figure 6
Figure 6. Figure 6: Similarity Scores for JavaDoc comment and whole class contexts in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Differences in similarity for cases of max similarity. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Similarity Scores for different contexts in the case of mean similarity. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 11
Figure 11. Figure 11: Difference between temperature 0 and 1 in the case of mean [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Difference between temperature 0 and 1 in the case of max similarity. [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]
Figure 13
Figure 13. Figure 13: Difference between temperature 1 and 2 in the case of mean [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 15
Figure 15. Figure 15: Similarity scores generated without and with explicit instructions to [PITH_FULL_IMAGE:figures/full_fig_p011_15.png]
Figure 16
Figure 16. Figure 16: Max similarity scores for outputs generated w/o explicit instructions. [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 21 canonical work pages

  1. [1]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Trans. Softw. Eng. Methodol., Sep. 2024. [Online]. Available: https://doi.org/10.1145/ 3695988

    work page 2024

  2. [2]

    Large language models for software engineering: Sur- vey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Sur- vey and open problems,” in Proceedings of the EEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), 2023

    work page 2023

  3. [3]

    On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot,

    A. Mastropaolo, L. Pascarella, E. Guglielmi, M. Ciniselli, S. Scalabrino, R. Oliveto, and G. Bavota, “On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot,” in Proceedings of the International Conference on Software Engineering (ICSE) , 2023

    work page 2023

  4. [4]

    Gen- erating java methods: An empirical assessment of four ai-based code assistants,

    V . Corso, D. Mariani, Leonardo Micucci, , and O. Riganelli, “Gen- erating java methods: An empirical assessment of four ai-based code assistants,” in Proceedings of the International Conference on Program Comprehension (ICPC), 2024

    work page 2024

  5. [5]

    Analyzing prompt influence on automated method generation: An empirical study with copilot,

    I. D. Fagadau, L. Mariani, D. Micucci, and O. Riganelli, “Analyzing prompt influence on automated method generation: An empirical study with copilot,” in Proceedings of the International Conference on Pro- gram Comprehension (ICPC) , 2024

    work page 2024

  6. [6]

    Traces of memorisation in large language models for code,

    A. Al-Kaswan, M. Izadi, and A. van Deursen, “Traces of memorisation in large language models for code,” in Proceedings of the International Conference on Software Engineering (ICSE) , 2024

    work page 2024

  7. [7]

    Unveil- ing memorization in code models,

    Z. Yang, Z. Zhao, C. Wang, J. Shi, D. Kim, D. Han, and D. Lo, “Unveil- ing memorization in code models,” in Proceedings of the International Conference on Software Engineering (ICSE) , 2024

    work page 2024

  8. [8]

    Codeipprompt: Intellectual property infringement assessment of code language models,

    Z. Yu, Y . Wu, N. Zhang, C. Wang, Y . V orobeychik, and C. Xiao, “Codeipprompt: Intellectual property infringement assessment of code language models,” in Proceedings of the International Conference on Machine Learning (ICML) , 2023

    work page 2023

Show all 26 references
  1. [9]

    Large Language Models for Software Engineering: Survey and Open Problems ,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “ Large Language Models for Software Engineering: Survey and Open Problems ,” in Proceedings of the International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), 2023

    2023

  2. [10]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 8, 2024

    2024

  3. [11]

    Exploring the effectiveness of llms in automated logging statement generation: An empirical study,

    Y . Li, Y . Huo, Z. Jiang, R. Zhong, P. He, Y . Su, L. C. Briand, and M. R. Lyu, “Exploring the effectiveness of llms in automated logging statement generation: An empirical study,” IEEE Transactions on Software Engineering , vol. 50, no. 12, pp. 3188–3207, 2024

    2024

  4. [12]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,” IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85–105, 2023

    2023

  5. [13]

    Automated program repair in the era of large pre-trained language models,

    C. S. Xia, Y . Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in Proceedings of the International Conference on Software Engineering (ICSE) , 2023, pp. 1482–1494

    2023

  6. [14]

    Assessing the quality of github copilot’s code generation,

    B. Yetistiren, I. Ozsoy, and E. Tuzun, “Assessing the quality of github copilot’s code generation,” in Proceedings of the Conference on Predic- tive Models and Data Analytics in Software Engineering (PROMISE) , 2022

    2022

  7. [15]

    Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models,

    P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models,” in Proceedings of the Conference on Human Factors in Computing Systems (CHI) , 2022

    2022

  8. [16]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  9. [17]

    Studying How Configurations Impact Code Generation in LLMs: the Case of Chat- GPT,

    B. Donato, L. Mariani, D. Micucci, and O. Riganelli, “Studying How Configurations Impact Code Generation in LLMs: the Case of Chat- GPT,” in Proceedings of the International Conference on Program Comprehension (ICPC), 2025

    2025

  10. [18]

    To what extent do deep learning-based code recommenders generate predictions by cloning code from the training set?

    M. Ciniselli, L. Pascarella, and G. Bavota, “To what extent do deep learning-based code recommenders generate predictions by cloning code from the training set?” in Proceedings of the International Conference on Program Comprehension (ICPC) , 2022

    2022

  11. [19]

    GitHub, “Github,” 2024, https://github.com/

    2024

  12. [20]

    A complexity measure,

    T. J. McCabe, “A complexity measure,” IEEE Transactions on Software Engineering, vol. SE-2, no. 4, pp. 308–320, 1976

    1976

  13. [21]

    Jplag: State-of-the-art software plagiarism & collusion detec- tion,

    JPlag, “Jplag: State-of-the-art software plagiarism & collusion detec- tion,” 2024, https://github.com/jplag/JPlag

    2024

  14. [22]

    (2024) Thefuzz

    TheFuzz. (2024) Thefuzz. https://github.com/seatgeek/thefuzz

    2024

  15. [23]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945

    1945

  16. [24]

    Cohen, Statistical power analysis for the behavioral sciences

    J. Cohen, Statistical power analysis for the behavioral sciences . Rout- ledge, 2013

    2013

  17. [25]

    Optimizing large language model hyperparameters for code generation,

    C. Arora, A. I. Sayeed, S. Licorish, F. Wang, and C. Treude, “Optimizing large language model hyperparameters for code generation,” 2024. [Online]. Available: https://arxiv.org/abs/2408.10577

    2024 arXiv

  18. [26]

    An empirical study of the non-determinism of chatgpt in code generation,

    S. Ouyang, J. M. Zhang, M. Harman, and M. Wang, “An empirical study of the non-determinism of chatgpt in code generation,” ACM Transactions on Software Engineering and Methodology , vol. 34, no. 2, 2025

    2025

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.