Pith. sign in

REVIEW 3 major objections 6 minor 75 references

I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read C code carries a strong enough stylistic fingerprint to identify which LLM generated it.

desk verdict Valuable new benchmark for LLM code attribution, but template overlap between train and test likely inflates the headline accuracies. read the letter →

arxiv 2506.17323 v1 pith:IHDXDADT submitted 2025-06-18 cs.LG cs.AIcs.SE

Tamas Bisztray , Bilel Cherif , Richard A. Dubniczky , Nils Gruschka , Bertalan Borsos , Mohamed Amine Ferrag , Attila Kovacs , Vasileios Mavroeidis
show 1 more author
Norbert Tihanyi
This is my paper
classification cs.LGcs.AIcs.SE
keywords LLM-generatedcodeauthorshipattributionstylometryTransformerencoderT5LLMfingerprintingCsourcedigitalforensics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to prove that the specific large language model which generated a C program can be identified from the code alone, even when the candidate models are close relatives. It contributes a dataset of 32,000 compilable C programs written by eight LLMs and a custom encoder-only Transformer, CodeT5-Authorship, built from the CodeT5+ encoder. On the hardest pairwise test, GPT-4.1 versus GPT-4o, the model reaches 97.56% accuracy, and among five major models it reaches 95.40%. The authors conclude that reliable authorship attribution of LLM-generated C code is feasible for moderately sized Transformer encoders. The practical stake is that generated code can be traced to its source model without watermarks, supporting forensics and accountability.

What carries the argument

The central object is CodeT5-Authorship: the pretrained CodeT5+ encoder with the decoder removed, whose first-token output embedding is passed through a linear, GELU, dropout, and linear classification head to produce a probability distribution over possible authors. It carries the argument by showing that an encoder-only model working on 512-token snippets captures style cues better than larger decoder-only models or long-context encoders. The other load-bearing component is LLM-AuthorBench, a corpus of 32,000 gcc-compilable C programs generated from 300 parameterized templates instantiated into about 4,000 unique tasks, with only exact duplicates removed and Type 1 and Type 2 near-duplicates retained to preserve stylistic variation.

What would settle it

Train the same classifier on LLM-AuthorBench under a template-disjoint split, holding out entire templates rather than individual programs, and compare accuracy; a substantial drop below the reported 97.56% and 95.40% figures would show the current numbers are inflated by near-duplicate tasks. A second test is to generate several hundred C programs from completely new prompts outside the 300 templates and measure attribution accuracy on that unseen material.

Watch

Extended reading notes

Core claim

CodeT5-Authorship, a 24-layer encoder-only Transformer built by removing the decoder from CodeT5+ (770M) and adding a two-layer GELU-activated classification head over the first-token embedding, can attribute C programs to their generating LLM with 97.56% accuracy in binary classification (GPT-4.1 versus GPT-4o) and 95.40% accuracy in five-way classification among Gemini 2.5 Flash, Claude 3.5 Haiku, GPT-4.1, Llama 3.3, and DeepSeek-V3. It outperforms seven traditional ML classifiers and eight fine-tuned Transformers, including BERT, RoBERTa, CodeBERT, ModernBERT, DistilBERT, DeBERTa-V3, Longformer, and LoRA-fine-tuned Qwen2-1.5B. The experiments find that comment style is the single richest cue, that removing comments hurts accuracy more in the multi-class setting, and that extending context beyond 512 tokens brings only marginal gains. The paper's conclusion is that reliable authorship attribution of LLM-generated C code is feasible with moderately sized Transformer encoders.

Load-bearing premise

The load-bearing premise is that the train/test separation, made over individual programs, measures model-specific style rather than template-specific familiarity, because each of the 300 task templates is reused about 13 times per model and only exact duplicates are removed.

Editorial extensions

If this is right

  • Generated C code is traceable to one of several known candidate LLMs at over 95% accuracy, so provenance checking can work as a post-hoc forensic step rather than requiring watermarks.
  • Comment handling matters: removing comments drops multi-class accuracy by 7.2 percentage points, so attribution systems should treat comment phrasing as a first-class signal.
  • A moderate 512-token encoder outperforms larger decoder-only and long-context models, suggesting the decisive style cues are local and that architecture matters more than context length.
  • A tuned XGBoost on hand-crafted features reaches 90.8% accuracy in under a minute, giving a low-cost attribution option for smaller candidate sets.
  • DeepSeek-V3 is the most confusable author in the five-way test, so model families with converging output styles may need additional discriminative signals.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: a template-disjoint evaluation, holding out entire task templates rather than individual programs, would test whether the reported numbers reflect author style or reuse of near-duplicate tasks across the train and test splits.
  • Beyond the paper: because comments are the dominant cue, paraphrasing or stripping comments is a plausible low-effort evasion, and a direct attack test would quantify how much real-world security the method provides.
  • Beyond the paper: if the same encoder approach transfers to Python, Java, or Rust, the benchmark could be extended to cross-language attribution, but comment and formatting conventions differ enough that transfer is not guaranteed.
  • Beyond the paper: an encoder-based attribution model could serve as an independent check on generative-watermark claims, since it relies on natural style drift rather than an embedded marker.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces LLM-AuthorBench, a benchmark of 32,000 C programs generated by eight LLMs from 300 parameterized templates, and CodeT5-Authorship, an encoder-only variant of CodeT5+ with a two-layer classification head. The authors evaluate CodeT5-Authorship against seven classical ML classifiers and eight transformer baselines. They report 97.56% accuracy for binary attribution between GPT-4.1 and GPT-4o, and 95.40% accuracy for five-way attribution among Gemini 2.5 Flash, Claude 3.5 Haiku, GPT-4.1, Llama 3.3, and DeepSeek-V3. The paper addresses two research questions: whether attribution generalizes to arbitrary programming tasks (RQ1) and which model families work best (RQ2). The central claim is that reliable authorship attribution of LLM-generated C code is feasible with moderately sized Transformer encoders.

Significance. If the results withstand closer evaluation, the paper would be a valuable contribution: it provides a public benchmark for LLM code attribution, a strong encoder-only architecture, and a systematic comparison of many baselines. The open release of the dataset and training code is a strength. However, the current evaluation design has a serious template-leakage problem: the train/test split is over programs, not templates, so near-duplicates of the same task appear in both training and validation. This threatens the generalizability claim in RQ1. The paper's small out-of-distribution probe is insufficient to rule out this alternative explanation. The contribution is therefore promising but not yet fully supported.

major comments (3)
  1. [Section 3.1.1 and Section 3.1] The train/test split is performed over individual programs, not templates. Since the 4,000 tasks are generated from only 300 templates (each template appears ≈13.3 times per model) and only Type 0 (byte-identical) duplicates are removed, Type 1/2 near-duplicates of the same template can appear in both training and validation. A transformer encoder can then memorize template–author associations (e.g., 'bubble-sort tasks from GPT-4o tend to have comments phrased like this') instead of learning model-specific style. This directly threatens RQ1, which asks whether attribution works for arbitrary programming tasks. Please re-evaluate with a template-disjoint split (e.g., assign entire templates to train/test) or provide strong evidence that accuracy degrades only mildly under such a split. The current cross-check in Section 4.3 is too small to rule out this leakage.
  2. [Section 4.3] The out-of-distribution cross-check uses only a few new prompts and only two models, and it reports confidence values (99% and 100%) rather than accuracy over a meaningful sample. Confidence is not a substitute for classification accuracy. To support the claim of generalization to arbitrary tasks, evaluate on a larger set of held-out prompts (e.g., 50–100 new templates per model) and report accuracy, precision, and recall, and specify the number of prompts used.
  3. [Section 4.1 and Section 4.2, Tables 5 and 6] All reported accuracies come from a single run per configuration, with no error bars or multiple seeds. Since the top models are separated by small margins (e.g., 97.56% vs. 97.00% vs. 96.88% in Table 5), it is unclear whether these differences are statistically significant. Please report results over multiple random seeds (at least 3) with mean and standard deviation, or otherwise justify why single runs suffice.
minor comments (6)
  1. [Section 6, Conclusion] The sentence 'In the harder multi-class scenario it again edges out (96%), the best long-context baseline (Longformer, 95%)' should read 95.4% per Table 6, not 96%.
  2. [Section 4.2, first paragraph] The phrase 'To gasp how significant' should be 'To gauge how significant'.
  3. [Section 4.1, item (v)] The word 'additinal' should be 'additional'.
  4. [Section 3.1] The phrase 'only unique C programs are included' is contradicted by Section 3.1.1, which explicitly retains Type 1/2 clones; please rephrase to avoid confusion.
  5. [Tables 5 and 6] The 'Time' column does not specify units; add 'minutes' to the header or legend.
  6. [Section 4.1] The claim that structural metrics from Table 4 shift decision boundaries by at most 0.4 pp is stated without supporting details; consider providing a supplementary table or a more concrete description of the ablation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported accuracies are empirical measurements on held-out programs, and the template-overlap issue is a benchmark-validity concern rather than a self-referential derivation.

full rationale

The paper's derivation chain is a standard supervised-learning pipeline: construct LLM-AuthorBench from 300 parameterized templates, prompt eight LLMs to produce 32,000 labeled C programs, split 80/20 by program, fine-tune CodeT5-Authorship and baselines on the training split, and report accuracies on the validation split. The reported 97.56% binary and 95.40% multi-class accuracies are empirical measurements on held-out programs whose labels are the known generating LLM; they are not quantities obtained by plugging fitted parameters back into the loss or by identifying the output with the input by construction. No step in the derivation equates a prediction with a fitted value or with the training labels. The main weakness is Section 3.1.1's template reuse: each of the 300 templates is used on average 13.3 times per model, only fully identical (Type 0) programs are removed, and the split is over programs rather than templates, so Type 1/2 near-duplicates can appear in both train and validation. This means validation accuracy may partly reflect template-specific artifacts rather than model style, which is a threat to the RQ1 generalization to 'arbitrary programming tasks.' However, that is an external-validity/data-leakage limitation, not a circular step: validation labels are independent of the training fit, and the paper does not define the attribution target in terms of the model's own outputs. The only self-citation ([58], a prior empirical study by overlapping authors on vulnerability rates in AI-generated C code) is used as background motivation and is not load-bearing for the attribution result. Thus, under the circularity definition used here, there is no significant circularity; the template-overlap concern belongs in a correctness/robustness review rather than a circularity score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central contribution is an empirical benchmark and model evaluation, not a derivation. The listed free parameters are human-chosen hyperparameters or dataset-construction choices that shape the reported accuracy. The axioms are standard domain assumptions about compilation, API provenance, and data splitting; the last one is only partially satisfied, which is the paper's main weakness.

free parameters (5)
  • Dropout rate in classification head = 0.20
    Chosen by hand (Section 3.2); affects regularization and final accuracy.
  • Activation function = GELU
    Selected experimentally over alternatives (Section 3.2), not derived from theory.
  • Pooling strategy = first token embedding
    Assumed without comparison to other pooling methods (Section 3.2).
  • Context window = 512 tokens
    Chosen for CodeT5-Authorship; the paper shows longer windows give only small gains (Section 4.1).
  • Template count N = 300
    Manual choice in benchmark construction; determines how often templates repeat and affects train/test overlap (Section 3.1).
assumptions (4)
  • domain assumption gcc -c compilation is a valid filter for well-formed C programs
    Used to exclude invalid or non-C outputs before training (Section 3.1.2).
  • domain assumption Programs from the same OpenRouter API endpoint represent the named LLM
    Labels come from API provenance; proxy artifacts are only partially checked in Section 4.3.
  • domain assumption NiCad Type 1 and Type 2 clones are distinct enough for stylometry
    The deduplication step keeps these near-duplicates, changing the effective task distribution (Section 3.1.1).
  • domain assumption Random program-level split yields independent training and test sets
    With templates reused across splits this is questionable; see related red flag.

how reviews work

0 comments
Cite this review

Pith. "Pith review of I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution." pith.science (2026). https://pith.science/paper/IHDXDADT

@misc{pith2026250617323,
  author       = {Pith},
  title        = {Pith review of: I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IHDXDADT}},
  note         = {Machine review of arXiv:2506.17323}
}
read the original abstract

Detecting AI-generated code, deepfakes, and other synthetic content is an emerging research challenge. As code generated by Large Language Models (LLMs) becomes more common, identifying the specific model behind each sample is increasingly important. This paper presents the first systematic study of LLM authorship attribution for C programs. We released CodeT5-Authorship, a novel model that uses only the encoder layers from the original CodeT5 encoder-decoder architecture, discarding the decoder to focus on classification. Our model's encoder output (first token) is passed through a two-layer classification head with GELU activation and dropout, producing a probability distribution over possible authors. To evaluate our approach, we introduce LLM-AuthorBench, a benchmark of 32,000 compilable C programs generated by eight state-of-the-art LLMs across diverse tasks. We compare our model to seven traditional ML classifiers and eight fine-tuned transformer models, including BERT, RoBERTa, CodeBERT, ModernBERT, DistilBERT, DeBERTa-V3, Longformer, and LoRA-fine-tuned Qwen2-1.5B. In binary classification, our model achieves 97.56% accuracy in distinguishing C programs generated by closely related models such as GPT-4.1 and GPT-4o, and 95.40% accuracy for multi-class attribution among five leading LLMs (Gemini 2.5 Flash, Claude 3.5 Haiku, GPT-4.1, Llama 3.3, and DeepSeek-V3). To support open science, we release the CodeT5-Authorship architecture, the LLM-AuthorBench benchmark, and all relevant Google Colab scripts on GitHub: https://github.com/LLMauthorbench/.

Figures

Figures reproduced from arXiv: 2506.17323 by the authors.

Figure 1
Figure 1. Attribution goals for source code and text. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The five-step methodology for LLM authorship attribution in C code. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. CodeT5-Authorship architecture 3.3 Traditional ML and Transformer Models To identify the most effective approach for LLM authorship attri￾bution without architectural modifications, first we evaluate both traditional machine learning baselines and modern Transformer￾based architectures. Authorship attribution methods generally fall into two broad categories: (a) classical stylometric approaches that rely on engineer… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Binary classification: 50% + 𝜖? [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Confusion Matrix for CodeT5-Authorship [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: shows that for the five way classification, instead of using 4000 C programs per mode, approximately 2000 would have been sufficient to archive similar levels of accuracy. In summary, across different model families, author attribution is highly feasible. These finding…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

75 extracted references · 41 canonical work pages

  1. [1]

    Mohammed Abuhamad, Tamer AbuHmed, Aziz Mohaisen, and DaeHun Nyang

    work page

  2. [2]

    Mohammed Abuhamad, Tamer Abuhmed, David Mohaisen, and Daehun Nyang

    work page

  3. [3]

    Mohammed Abuhamad, Ji-su Rhim, Tamer AbuHmed, Sana Ullah, Sanggil Kang, and DaeHun Nyang. 2019. Code authorship identification using convolutional neural networks. Future Generation Computer Systems 95 (June 2019), 104–115. doi:10.1016/j.future.2018.12.038

    work page doi:10.1016/j.future.2018.12.038 2019

  4. [4]

    Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2018. code2vec: Learning Distributed Representations of Code. doi:10.48550/arXiv.1803.09473 arXiv:1803.09473 [cs]

    arXiv doi:10.48550/arxiv.1803.09473 2018

  5. [5]

    Bander Alsulami, Edwin Dauber, Richard Harang, Spiros Mancoridis, and Rachel Greenstadt. 2017. Source Code Authorship Attribution Using Long Short-Term Memory Based Networks. In Computer Security – ESORICS 2017 , Simon N. Foley, Dieter Gollmann, and Einar Snekkenes (Eds.). Springer International Publishing, Cham, 65–82. doi:10.1007/978-3-319-66402-6_6

    work page doi:10.1007/978-3-319-66402-6_6 2017

  6. [6]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv doi:10.48550/arxiv.2309.16609 2023

  7. [7]

    Dimitris Bamidis, Ilias Kalouptsoglou, Apostolos Ampatzoglou, and Alexandros Chatzigeorgiou. 2024. Software Skills Identification: A Multi-Class Classification on Source Code Using Machine Learning. Global Clinical Engineering Journal 6, SI6 (Dec. 2024), 74–77. doi:10.31354/globalce.v6iSI6.278

    work page doi:10.31354/globalce.v6isi6.278 2024

  8. [8]

    Parinaz Bayrami and Jacqueline E. Rice. 2021. Code Authorship Attribution using content-based and non-content-based features. In 2021 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE) . 1–6. doi:10.1109/ CCECE53047.2021.9569061 ISSN: 2576-7046

    work page arXiv 2021

Show all 75 references
  1. [9]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long- Document Transformer. doi:10.48550/arXiv.2004.05150 arXiv:2004.05150 [cs]

    2020 doi arXiv

  2. [10]

    Benedikt Boenninghoff, Steffen Hessler, Dorothea Kolossa, and Robert M. Nickel

  3. [11]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Ka- plan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey...

    2020

  4. [12]

    Aylin Caliskan, Fabian Yamaguchi, Edwin Dauber, Richard Harang, Konrad Rieck, Rachel Greenstadt, and Arvind Narayanan. 2018. When Coding Style Survives Compilation: De-anonymizing Programmers from Executable Binaries. In Proceedings 2018 Network and Distributed System Security...

    2018

  5. [13]

    Aylin Caliskan-Islam, Richard Harang, Andrew Liu, Arvind Narayanan, Clare Voss, Fabian Yamaguchi, and Rachel Greenstadt. 2015. De-anonymizing Pro- grammers via Code Stylometry. 255–270. https://www.usenix.org/conference/ usenixsecurity15/technical-sessions/presentation/caliskan-islam

    2015

  6. [14]

    Soohyeon Choi, Yong Kiam Tan, Mark Huasong Meng, Mohamed Ragab, Soumik Mondal, David Mohaisen, and Khin Mi Mi Aung. 2025. I Can Find You in Seconds! Leveraging Large Language Models for Code Authorship Attribution. doi:10.48550/arXiv.2501.08165 arXiv:2501.08165 [cs] version: 1

    2025 doi arXiv

  7. [15]

    Cordy and Chanchal Kumar Roy

    James R. Cordy and Chanchal Kumar Roy. 2011. The NiCad Clone Detector.2011 IEEE 19th International Conference on Program Comprehension (2011), 219–220. https://api.semanticscholar.org/CorpusID:2991109

    2011

  8. [16]

    Tugce Coskun, Rusen Halepmollasi, Khadija Hanifi, Ramin Fadaei Fouladi, Pinar Comak De Cnudde, and Ayse Tosun. 2022. Profiling developers to pre- dict vulnerable code changes. In Proceedings of the 18th International Confer- ence on Predictive Models and Data Analytics in Soft...

    2022

  9. [17]

    Desmarais, and Foutse Khomh

    Arghavan Moradi Dakhel, Michel C. Desmarais, and Foutse Khomh. 2023. Dev2vec: Representing Domain Expertise of Developers in an Embedding Space. Information and Software Technology 159 (July 2023), 107218. doi:10.1016/j.infsof. 2023.107218 arXiv:2207.05132 [cs]

    2023

  10. [18]

    Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, Jamie Hayes, Nidhi Vyas, Majd Al Merey, Jonah Brown-Cohen, Rudy Bunel, Borja Balle, Taylan Cemgil, Zahra Ahmed, Ki...

  11. [19]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human...

    2019

  12. [20]

    Ahmet Emir Dirik. 2013. Source Attribution Based on Physical Defects in Light Path. In Digital Image Forensics: There is More to a Picture than Meets the Eye , Husrev Taha Sencar and Nasir Memon (Eds.). Springer, New York, NY, 219–236. doi:10.1007/978-1-4614-0757-7_7

    2013 doi

  13. [21]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. doi:10.48550/ arXiv.2002.08155 arXiv:2002.08155 [cs]

    2020 doi arXiv

  14. [22]

    Alberto Ferrante, Eric Medvet, Francesco Mercaldo, Jelena Milosevic, and Cor- rado Aaron Visaggio. 2016. Spotting the Malicious Moment: Characterizing Mal- ware Behavior Using Dynamic Features. In 2016 11th International Conference on A vailability, Reliability and Security (A...

    2016 doi

  15. [23]

    Georgia Frantzeskou, Efstathios Stamatatos, Stefanos Gritzalis, and Sokratis Katsikas. 2006. Effective identification of source code authors using byte-level information. In Proceedings of the 28th international conference on Software engi- neering (ICSE ’06) . Association for...

    2006

  16. [24]

    Georgia Frantzeskou, Efstathios Stamatatos, Stefanos Gritzalis, and Sokratis Katsikas. 2006. Source Code Author Identification Based on N-gram Author Profiles. In Artificial Intelligence Applications and Innovations, Ilias Maglogiannis, Kostas Karpouzis, and Max Bramer (Eds.)....

    2006 doi

  17. [25]

    Jason Gray, Daniele Sgandurra, Lorenzo Cavallaro, and Jorge Blasco Alis. 2024. Identifying Authorship in Malicious Binaries: Features, Challenges & Datasets. ACM Comput. Surv. 56, 8, Article 212 (April 2024), 36 pages. doi:10.1145/3653973

    2024 doi

  18. [26]

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training Code Rep...

    2021 doi arXiv

  19. [27]

    Dixiao Guo, Anmin Zhou, Liang Liu, Shan Liao, and Lei Zhang. 2022. A Method of Source Code Authorship Attribution Based on Graph Neural Network. In Proceedings of 2021 Chinese Intelligent Automation Conference , Zhidong Deng (Ed.). Springer, Singapore, 645–657. doi:10.1007/978...

    2022 doi

  20. [28]

    Hanxi Guo, Siyuan Cheng, Kaiyuan Zhang, Guangyu Shen, and Xiangyu Zhang

  21. [29]

    Surbhi Gupta, Neeraj Mohan, and Munish Kumar. 2021. A Study on Source Device Attribution Using Still Images. Archives of Computational Methods in Engineering 28, 4 (June 2021), 2209–2223. doi:10.1007/s11831-020-09452-y

    2021 doi

  22. [30]

    Andrea Gurioli, Maurizio Gabbrielli, and Stefano Zacchiroli. 2025. Is This You, LLM? Recognizing AI-written Programs with Multilingual Code Stylome- try . In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE Computer Society, Lo...

    2025

  23. [31]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTaV3: Improv- ing DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing. doi:10.48550/arXiv.2111.09543 arXiv:2111.09543 [cs]

    2023 doi arXiv

  24. [32]

    Xie He, Arash Habibi Lashkari, Nikhill Vombatkere, and Dilli Prasad Sharma. 2024. Authorship Attribution Methods, Challenges, and Future Research Directions: A Comprehensive Survey. Information 15, 3 (March 2024), 131. doi:10.3390/ info15030131 Number: 3 Publisher: Multidiscip...

    2024

  25. [33]

    Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine- tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Iryna Gurevych and Yusuke Miyao (Eds.). Association for C...

    2018 doi

  26. [34]

    Vaibhavi Kalgutkar, Ratinder Kaur, Hugo Gonzalez, Natalia Stakhanova, and Alina Matyukhina. 2019. Code Authorship Attribution: Methods and Challenges. ACM Comput. Surv. 52, 1 (Feb. 2019), 3:1–3:36. doi:10.1145/3292577

    2019 doi

  27. [35]

    Avila, Jacob Brunelle, and Baba Mamadou Camara

    Raphaël Khoury, Anderson R. Avila, Jacob Brunelle, and Baba Mamadou Camara

  28. [36]

    Jungin Kim, Shinwoo Park, and Yo-Sub Han. 2025. Marking Code Without Breaking It: Code Watermarking for Detecting LLM-Generated Code. doi:10. 48550/arXiv.2502.18851 arXiv:2502.18851 [cs]

    2025 doi

  29. [37]

    Tharindu Kumarage, Garima Agrawal, Paras Sheth, Raha Moraffah, Aman Chadha, Joshua Garland, and Huan Liu. 2024. A Survey of AI-generated Text Forensic Systems: Detection, Attribution, and Characterization. doi:10.48550/ arXiv.2403.01152 arXiv:2403.01152 [cs] version: 1

    2024 doi arXiv

  30. [38]

    Boquan Li, Mengdi Zhang, Peixin Zhang, Jun Sun, and Xingmei Wang. 2024. Resilient Watermarking for LLM-Generated Codes. doi:10.48550/arXiv.2402. 07518 arXiv:2402.07518 [cs] version: 1

    2024 doi arXiv

  31. [39]

    Paula Lissón and Nicolas Ballier. 2018. Investigating Lexical Progression through Lexical Diversity Metrics in a Corpus of French L3.Discours. Revue de linguistique, psycholinguistique et informatique. A journal of linguistics, psycholinguistics and computational linguistics 2...

    2018 doi

  32. [40]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang

  33. [41]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. doi:10.48550/arXiv.1907.11692 arXiv:1907.11692 [cs]

    2019 doi arXiv

  34. [42]

    T.J. McCabe. 1976. A Complexity Measure. IEEE Transactions on Software Engineering SE-2, 4 (1976), 308–320. doi:10.1109/TSE.1976.233837

    1976

  35. [43]

    Manning, and Chelsea Finn

    Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature. doi:10.48550/arXiv.2301.11305 arXiv:2301.11305 [cs]

    2023 doi arXiv

  36. [44]

    Nguyen, Juri Di Rocco, Claudio Di Sipio, Riccardo Rubei, Davide Di Ruscio, and Massimiliano Di Penta

    Phuong T. Nguyen, Juri Di Rocco, Claudio Di Sipio, Riccardo Rubei, Davide Di Ruscio, and Massimiliano Di Penta. 2024. GPTSniffer: A CodeBERT-based classifier to detect source code written by ChatGPT. Journal of Systems and Software 214 (Aug. 2024), 112059. doi:10.1016/j.jss.20...

    2024

  37. [45]

    Timothy Paek and Chilukuri Mohan. 2025. Detection of LLM-Generated Java Code Using Discretized Nested Bigrams. doi:10.48550/arXiv.2502.15740 arXiv:2502.15740 [cs]

    2025 doi arXiv

  38. [46]

    Wei Hung Pan, Ming Jie Chok, Jonathan Leong Shan Wong, Yung Xin Shin, Yeong Shian Poon, Zhou Yang, Chun Yong Chong, David Lo, and Mei Kuan Lim

  39. [47]

    Ad- vances in Neural Information Processing Systems 36 (Dec

    Is Your Code Generated by ChatGPT Really Correct? Rigor- ous Evaluation of Large Language Models for Code Generation. Ad- vances in Neural Information Processing Systems 36 (Dec. 2023), 21558– 21572. https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 43e9d647ccd3e4b7b...

    2023

  40. [48]

    Erwin Quiring, Alwin Maier, and Konrad Rieck. 2019. Misleading Authorship Attribution of Source Code using Adversarial Learning. 479–496. https://www. usenix.org/conference/usenixsecurity19/presentation/quiring

    2019

  41. [49]

    Nathan Rosenblum, Xiaojin Zhu, and Barton P. Miller. 2011. Who Wrote This Code? Identifying the Authors of Program Binaries. In Computer Security – ESORICS 2011, Vijay Atluri and Claudia Diaz (Eds.). Springer, Berlin, Heidelberg, 172–189. doi:10.1007/978-3-642-23822-2_10

    2011 doi

  42. [50]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. doi:10.48550/arXiv.1910.01108 arXiv:1910.01108 [cs]

    2020 doi arXiv

  43. [51]

    Yanir Seroussi, Ingrid Zukerman, and Fabian Bohnert. 2014. Authorship Attribu- tion with Topic Models. Computational Linguistics 40, 2 (June 2014), 269–310. doi:10.1162/COLI_a_00173 Place: Cambridge, MA Publisher: MIT Press

    2014 doi

  44. [52]

    Kanishka Silva, Ingo Frommholz, Burcu Can, Fred Blain, Raheem Sarwar, and Laura Ugolini. 2024. Forged-GAN-BERT: Authorship Attribution for LLM- Generated Forged Novels. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguisti...

    2024

  45. [53]

    Qige Song, Yongzheng Zhang, Linshu Ouyang, and Yige Chen. 2022. BinMLM: Binary Authorship Verification with Flow-aware Mixture-of-Shared Language Model . In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE Computer Society, Lo...

    2022

  46. [54]

    doi:10.48550/arXiv.2401.03676 arXiv:2401.03676 [cs]

    Assessing AI Detectors in Identifying AI-Generated Code: Implications for Education. doi:10.48550/arXiv.2401.03676 arXiv:2401.03676 [cs]

    doi arXiv

  47. [55]

    Shinwoo Park, Hyundong Jin, Jeong-won Cha, and Yo-Sub Han. 2025. Detection of LLM-Paraphrased Code and Identification of the Responsible LLM Using Coding Style Features. doi:10.48550/arXiv.2502.17749 arXiv:2502.17749 [cs] version: 2

    2025 doi

  48. [56]

    Tarun Suresh, Shubham Ugare, Gagandeep Singh, and Sasa Misailovic. 2025. Is The Watermarking Of LLM-Generated Code Robust? doi:10.48550/arXiv.2403. 17983 arXiv:2403.17983 [cs]

    2025 doi arXiv

  49. [57]

    N. N. Thathsarani. 2024. A Comprehensive Software Complexity Metric Based on Cyclomatic Complexity. In 2024 4th International Conference of Science and Information Technology in Smart Administration (ICSINTESA). 90–95. doi:10.1109/ ICSINTESA62455.2024.10748227

    2024

  50. [58]

    Cordeiro

    Norbert Tihanyi, Tamas Bisztray, Mohamed Amine Ferrag, Ridhi Jain, and Lu- cas C. Cordeiro. 2024. How secure is AI-generated code: a large-scale comparison of large language models. Empirical Software Engineering 30, 2 (Dec. 2024), 47. doi:10.1007/s10664-024-10590-1

    2024 doi

  51. [59]

    Farhan Ullah, Muhammad Rashid Naeem, Hamad Naeem, Xiaochun Cheng, and Mamoun Alazab. 2022. CroLSSim: Cross-language software similarity detector using hybrid approach of LSA-based AST-MDrep features and CNN-LSTM model. International Journal of Intelligent Systems 37, 9 (2022),...

    2022 doi

  52. [60]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems , Vol. 30. Curran Associates, Inc. https://papers.nips.cc/paper_f...

    2017

  53. [61]

    Ningfei Wang, Shouling Ji, and Ting Wang. 2018. Integration of Static and Dynamic Code Stylometry Analysis for Programmer De-anonymization. In Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security (AISec ’18). Association for Computing Machinery, New Yor...

    2018

  54. [62]

    Qige Song, Yongzheng Zhang, Linshu Ouyang, and Yige Chen. 2022. BinMLM: Binary Authorship Verification with Flow-aware Mixture-of-Shared Language Model. doi:10.48550/arXiv.2203.04472 arXiv:2203.04472 [cs]

    2022 doi arXiv

  55. [63]

    Hyunjae Suh, Mahan Tafreshipour, Jiawei Li, Adithya Bhattiprolu, and Iftekhar Ahmed. 2024. An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far Are We? doi:10.48550/arXiv.2411.04299 arXiv:2411.04299 [cs] version: 1

    2024 doi arXiv

  56. [64]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https://arxiv.org/abs/2201.11903v6

    2022 arXiv

  57. [65]

    Zhenyu Xu and Victor S. Sheng. 2025. CodeVision: Detecting LLM-Generated Code Using 2D Token Probability Maps and Vision Models. doi:10.48550/arXiv. 2501.03288 arXiv:2501.03288 [cs] version: 1

    2025 doi arXiv

  58. [66]

    Sarim Zafar, Muhammad Usman Sarwar, Saeed Salem, and Muhammad Zubair Malik. 2020. Language and Obfuscation Oblivious Source Code Authorship Attribution. IEEE Access 8 (2020), 197581–197596. doi:10.1109/ACCESS.2020. 3034932

    2020 doi

  59. [67]

    David Álvarez Fidalgo and Francisco Ortin. 2025. CLAVE: A deep learning model for source code authorship verification with contrastive learning and transformer encoders. Inf. Process. Manage. 62, 3 (April 2025). doi:10.1016/j.ipm.2024.104005 Received *; revised *; accepted *

    2025

  60. [70]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Under- standing and Generation. doi:10.48550/arXiv.2109.00859 arXiv:2109.00859 [cs]

    2021 doi arXiv

  61. [71]

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hall- ström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, Better, Faster, Longer: A Modern Bidirec...

    2024 arXiv

  62. [2018]

    In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS ’18)

    Large-Scale and Language-Oblivious Code Authorship Identification. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS ’18) . Association for Computing Machinery, New York, NY, USA, 101–114. doi:10.1145/3243734.3243738

    2018

  63. [2019]

    IEEE Computer Society, 36–45

    Explainable Authorship Verification in Social Media via Attention-based Similarity Learning. IEEE Computer Society, 36–45. doi:10.1109/BigData47090. 2019.9005650

    2019

  64. [2021]

    ACM Trans

    Large-scale and Robust Code Authorship Identification with Deep Feature Learning. ACM Trans. Priv. Secur. 24, 4 (July 2021), 23:1–23:35. doi:10.1145/ 3461666

    2021

  65. [2023]

    In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC)

    How Secure is Code Generated by ChatGPT?. In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC) . 2445–2451. doi:10.1109/ SMC53992.2023.10394237 ISSN: 2577-1655

    2023

  66. [2024]

    Nature 634, 8035 (Oct

    Scalable watermarking for identifying large language model outputs. Nature 634, 8035 (Oct. 2024), 818–823. doi:10.1038/s41586-024-08025-4 Publisher: Nature Publishing Group

    2024 doi

  67. [2025]

    doi:10.48550/arXiv.2506

    CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs. doi:10.48550/arXiv.2506. 11059 arXiv:2506.11059 [cs]

    doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.