REVIEW 3 major objections 6 minor 45 references
MultiAIGCD: A Comprehensive dataset for AI Generated Code Detection Covering Multiple Languages, Models,Prompts, and Scenarios
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper introduces MultiAIGCD, a 153,419-snippet benchmark showing that detectors for AI-written code perform far better on code generated from scratch than on code where an AI fixed a human's bug.
desk verdict A genuinely useful dataset with fixable but load-bearing problems: the abstract contradicts its own Table 3 about ADA embeddings, and the dataset itself is not yet available. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the scenario-by-prompt-by-language grid built on 800 coding problems with matched human and AI versions of the same task, which MultiAIGCD organizes into a single dataset. For each problem the authors take human submissions that are accepted, that fail at runtime, or that give wrong answers, and ask six LLMs to solve the problem from scratch, to fix the runtime-error submission, and to fix the wrong-output submission, each under three prompting strategies. Problem-level train/validation/test splitting keeps the same problem out of more than one split, so performance differences can be attributed to scenario, language, model, and prompt rather than to problem difficulty. The detector stack consists of an SVM on text-embedding features, an SVM on CodeT5+ embeddings, and a fine-tuned CodeBERTa classifier, which serve as measurement instruments rather than as the contribution itself.
What would settle it
Collect a few hundred snippets in which a programmer pasted an LLM suggestion and then edited it by hand, split each snippet at the edited lines, and run a detector trained on MultiAIGCD with whole-snippet labels. If accuracy on those edited snippets falls below the roughly 0.60–0.86 F1 band the paper reports for fix scenarios, the benchmark overstates real-world detection accuracy.
Extended reading notes
Core claim
The central claim is that detection quality depends heavily on the usage scenario, not just on the detector or the LLM. On MultiAIGCD, the best detector reaches 0.9815 F1 for Java scratch generation, while all three detectors fall to roughly 0.60–0.86 F1 for runtime-error and wrong-output fix scenarios; cross-language transfer is sharply worse, with Go test F1 as low as about 0.07–0.31 when Go is absent from training. The paper also reports that LLMs differ in detectable style, and that certain models are harder to spot, so results from a single model or a single language do not generalize. These findings are presented as evidence that MultiAIGCD is a needed resource: a controlled setup where problem identity is fixed and the same problem never appears in both train and test sets.
Load-bearing premise
The load-bearing premise is that using LLM outputs verbatim and limiting usage to scratch generation, runtime-error fixing, and wrong-output fixing captures how people actually use code assistants; if real users edit or mix AI code with their own, detector accuracy on this dataset will overstate real-world accuracy.
Editorial extensions
If this is right
- A detector evaluated on MultiAIGCD should be reported per scenario, because fix scenarios are materially harder than scratch generation for all three models tested.
- Cross-language claims need per-language training data: when the target language is absent at training time, F1 falls sharply, especially for Go.
- Single-model detector results do not generalize across LLMs; code from some models is harder to detect, so a robust detector must be tested against several generators.
- Prompting strategies affect the code that detectors must recognize, so benchmark prompts should be documented and varied rather than fixed.
Reading between the lines
- Beyond the paper, because MultiAIGCD deliberately removes formatting tells such as triple backticks and language names, its accuracy numbers reflect content-based detection; in the wild, detectors could do better on unedited outputs that keep those tells, and worse on outputs that are lightly edited.
- The paper's own future-work note on blended code implies that the benchmark's practical ceiling is untested: real use often mixes AI and human lines, and whole-snippet labels cannot tell a detector which lines are AI's. A natural extension is a line-level provenance benchmark built from the same 800 problems.
- Beyond the paper, the execution-quality analysis hints at a cheap complementary signal: response behavior itself, such as o3-mini's frequent failures to answer, could be used as a detection feature alongside code content.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MultiAIGCD, a dataset for AI-generated code detection containing 121,271 AI-generated and 32,148 human-written code snippets in Python, Java, and Go. Code samples are generated from 800 CodeNet problems using six LLMs (Llama-3.3-70B, Qwen2.5-Coder-32B, GPT-4o, DeepSeek-V3, o3-mini, Claude 3.5 Sonnet v2), three prompting strategies (Lazy, Role, Rephrase & Respond), and three usage scenarios (scratch generation, runtime-error fixing, and incorrect-output correction). The authors also benchmark three detectors (SVM with Ada embeddings, SVM with CodeT5+ embeddings, and fine-tuned CodeBERTa) under within-scenario, cross-LLM, and cross-language evaluation protocols, and provide a qualitative and quantitative analysis of LLM-generated code style and correctness.
Significance. If the dataset is released and the presented splits are followed, MultiAIGCD could serve as a valuable standardized benchmark, particularly for the relatively underexplored AI-assisted debugging/fix scenarios. The paper's strengths include problem-level train/test splits, syntactic filtering for quality control, inclusion of a reasoning model (o3-mini), and a careful analysis of LLM code accuracy across languages and prompts. The benchmark results provide a useful baseline, and the observed drop in detection accuracy for fix scenarios is an important finding for the community. However, the contribution is contingent on the dataset being publicly available and on the correction of the statistical claims in the abstract.
major comments (3)
- [Footnote 1] The paper states that 'The URL for the data and code will be shared when the paper is published' (footnote 1), yet the paper's central contribution is the dataset itself. Without access to the data, the reported numbers in Tables 2–8 cannot be reproduced or audited, including the problem-level splits described in Section 5.1. For a dataset paper, the artifact must be available at least in a controlled form; as presented, the contribution is currently unverifiable.
- [Abstract and Section 5.2.1] The abstract and introduction claim that 'OpenAI's ADA embeddings consistently yield the highest prediction accuracy across most scenarios,' but Table 3 shows that CodeBERTa achieves the highest F1 in 6 of 9 language-scenario cells, while SVMAda only leads for Go. The paper's own Section 5.2.1 states the opposite: 'CodeBERTa achieves the highest F1 scores across all scenarios for Java and in most cases for Python.' This contradiction is misleading and must be resolved.
- [Section 2 (end)] The claim that MultiAIGCD is 'the most comprehensive dataset for the AI-generated code detection problem' is not supported by the paper's own comparison in Table 1. For example, Orel et al. (2025a) list 246,581 LLM-generated samples across more languages, and Guo et al. (2025) cover 10 languages with 200,000 LLM-generated samples. The claim should be qualified (e.g., 'most comprehensive in terms of usage scenarios') or removed.
minor comments (6)
- [Section 4.3] The phrase 'reiably evaluate' should be corrected to 'reliably evaluate.'
- [Section 5.2.1] The word 'hyperparemeters' should be corrected to 'hyperparameters.'
- [Table 2] The abbreviation 'DS' for DeepSeek is not defined in the table caption or surrounding text; please define it or use the full model name.
- [Section 3.2] The three prompting strategies are listed as 'Role, Lazy, and Rephrase & Respond' here, but later they are described in the order 'Lazy, Role, Rephrase & Respond.' Please make the ordering consistent.
- [Figure 1] Figure 1 uses a green marker to indicate the average, which may be unclear in grayscale reproduction; consider using a distinct marker shape or adding a label to the legend.
- [Section 6, Limitations] The last bullet says the dataset is 'one of the most comprehensive,' which contradicts the unqualified 'most comprehensive' claim in Section 2; please harmonize the wording.
Circularity Check
No circularity: MultiAIGCD is an empirical dataset and benchmark whose numbers are held-out measurements, not refitted inputs; the only self-citation is non-load-bearing.
full rationale
MultiAIGCD is not a derivation paper: its load-bearing claims are the dataset construction (sampling CodeNet submissions, prompting six LLMs, and post-processing to 121,271 AI-generated and 32,148 human snippets) and benchmark measurements on problem-disjoint training, validation, and test splits. No equation or claimed result is defined in terms of another claimed result. The only self-citation, AIGCodeSet (Demirok and Kutlu 2024), is used as a scope comparison and does not carry any load-bearing argument or uniqueness claim. Detector accuracies in Tables 3-8 are empirical measurements, not quantities fitted from the test set; hyperparameters are tuned on the validation set and problem-level splits prevent leakage. Footnote 1's deferred data and code URL and the abstract's overstatement that Ada embeddings 'consistently yield the highest' accuracy (contradicted by CodeBERTa's higher F1 in most Table 3 cells) are reproducibility and correctness risks, but neither reduces any claim to its own inputs. No circular step is present.
Assumptions & free parameters
assumptions (4)
- domain assumption CodeNet submissions from 2021 are human-written and predate AI code assistants.
- domain assumption LLM API outputs are used verbatim as AI-generated code; no human editing or blending is modeled.
- domain assumption Six selected LLMs and three prompting strategies are representative of the AI code that detectors will face in practice.
- domain assumption Go code validity is adequately checked by heuristic rules rather than a full parser.
Cite this review
Pith. "Pith review of MultiAIGCD: A Comprehensive dataset for AI Generated Code Detection Covering Multiple Languages, Models,Prompts, and Scenarios." pith.science (2026). https://pith.science/paper/XEBTXYGO
@misc{pith2026250721693,
author = {Pith},
title = {Pith review of: MultiAIGCD: A Comprehensive dataset for AI Generated Code Detection Covering Multiple Languages, Models,Prompts, and Scenarios},
year = {2026},
howpublished = {\url{https://pith.science/paper/XEBTXYGO}},
note = {Machine review of arXiv:2507.21693}
}
read the original abstract
As large language models (LLMs) rapidly advance, their role in code generation has expanded significantly. While this offers streamlined development, it also creates concerns in areas like education and job interviews. Consequently, developing robust systems to detect AI-generated code is imperative to maintain academic integrity and ensure fairness in hiring processes. In this study, we introduce MultiAIGCD, a dataset for AI-generated code detection for Python, Java, and Go. From the CodeNet dataset's problem definitions and human-authored codes, we generate several code samples in Java, Python, and Go with six different LLMs and three different prompts. This generation process covered three key usage scenarios: (i) generating code from problem descriptions, (ii) fixing runtime errors in human-written code, and (iii) correcting incorrect outputs. Overall, MultiAIGCD consists of 121,271 AI-generated and 32,148 human-written code snippets. We also benchmark three state-of-the-art AI-generated code detection models and assess their performance in various test scenarios such as cross-model and cross-language. We share our dataset and codes to support research in this field.
Reference graph
Works this paper leans on
-
[1]
In: 2024 IEEE International Conference on Cyber Security and Resilience (CSR), IEEE, pp 1--8
Ambati SH, Ridley N, Branca E, et al (2024) Navigating (in) security of ai-generated code. In: 2024 IEEE International Conference on Cyber Security and Resilience (CSR), IEEE, pp 1--8
work page 2024
-
[2]
In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V
Becker BA, Denny P, Finnie-Ansley J, et al (2023) Programming is hard-or at least it used to be: Educational opportunities and challenges of ai code generation. In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, pp 500--506
work page 2023
-
[3]
Bukhari S, Tan B, De Carli L (2023) Distinguishing ai-and human-generated code: a case study. In: Proceedings of the 2023 Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses, pp 17--25
work page 2023
-
[4]
Bulla L, Midolo A, Mongiov \` M, et al (2024) Ex-code: A robust and explainable model to detect ai-generated code. Information 15(12):819
work page 2024
-
[5]
Information and Software Technology 177:107572
Cotroneo D, De Luca R, Liguori P (2025) Devaic: A tool for security assessment of ai-generated code. Information and Software Technology 177:107572
work page 2025
-
[6]
arXiv preprint arXiv:241216594
Demirok B, Kutlu M (2024) Aigcodeset: A new annotated dataset for ai generated code detection. arXiv preprint arXiv:241216594
work page 2024
-
[7]
arXiv preprint arXiv:231104205
Deng Y, Zhang W, Chen Z, et al (2023) Rephrase and respond: Let large language models ask better questions for themselves. arXiv preprint arXiv:231104205
work page 2023
-
[8]
arXiv preprint arXiv:200208155
Feng Z, Guo D, Tang D, et al (2020) Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:200208155
work page 2020
Show all 45 references
-
[9]
In: 2025 5th International Conference on Advanced Research in Computing (ICARC), IEEE, pp 1--6
Gunawardhana R, Wijayasiriwardhane TK (2025) An approach to detect large language model generated firmware for arduino platform. In: 2025 5th International Conference on Advanced Research in Computing (ICARC), IEEE, pp 1--6
2025
-
[10]
arXiv preprint arXiv:250611059
Guo H, Cheng S, Zhang K, et al (2025) Codemirage: A multi-lingual benchmark for detecting ai-generated and paraphrased source code from production-level llms. arXiv preprint arXiv:250611059
2025
-
[11]
arXiv preprint arXiv:241214611
Gurioli A, Gabbrielli M, Zacchiroli S (2024) Is this you, llm? recognizing ai-written programs with multilingual code stylometry. arXiv preprint arXiv:241214611
2024
-
[12]
In: Proceedings of the 55th ACM Technical Symposium on Computer Science Education V
Hoq M, Shi Y, Leinonen J, et al (2024) Detecting chatgpt-generated code submissions in a cs1 course using machine learning models. In: Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1. Association for Computing Machinery, New York, NY, USA, SI...
2024
-
[13]
ACM Transactions on Software Engineering and Methodology 33(8):1--79
Hou X, Zhao Y, Liu Y, et al (2024) Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33(8):1--79
2024
-
[14]
In: Proceedings of the 21st International Conference on Mining Software Repositories, pp 394--406
Idialu OJ, Mathews NS, Maipradit R, et al (2024) Whodunit: Classifying code as human authored or gpt-4 generated-a case study on codechef problems. In: Proceedings of the 21st International Conference on Mining Software Repositories, pp 394--406
2024
-
[15]
In: 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), IEEE, pp 563--575
Jesse K, Ahmed T, Devanbu PT, et al (2023) Large language models and simple, stupid bugs. In: 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), IEEE, pp 563--575
2023
-
[16]
arXiv preprint arXiv:241219437
Liu A, Feng B, Xue B, et al (2024 a ) Deepseek-v3 technical report. arXiv preprint arXiv:241219437
2024
-
[17]
ACM Transactions on Software Engineering and Methodology 33(5):1--26
Liu Y, Le-Cong T, Widyasari R, et al (2024 b ) Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology 33(5):1--26
2024
-
[18]
Journal of Systems and Software 214:112059
Nguyen PT, Di Rocco J, Di Sipio C, et al (2024) Gptsniffer: A codebert-based classifier to detect source code written by chatgpt. Journal of Systems and Software 214:112059
2024
-
[19]
arXiv e-prints pp arXiv--2405
Oedingen M, Engelhardt RC, Denz R, et al (2024) Chatgpt code detection: Techniques for uncovering the source of code. arXiv e-prints pp arXiv--2405
2024
-
[20]
arXiv preprint arXiv:250313733
Orel D, Azizov D, Nakov P (2025 a ) Codet-m4: Detecting machine-generated code in multi-lingual, multi-generator and multi-domain settings. arXiv preprint arXiv:250313733
2025
-
[21]
://arxiv.org/abs/2507.10583, https://arxiv.org/abs/2507.10583 arXiv:2507.10583
Orel D, Paul I, Gurevych I, et al (2025 b ) Droid : A resource suite for ai-generated code detection. ://arxiv.org/abs/2507.10583, https://arxiv.org/abs/2507.10583 arXiv:2507.10583
2025 arXiv
-
[22]
In: Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training, pp 1--11
Pan WH, Chok MJ, Wong JLS, et al (2024) Assessing ai detectors in identifying ai-generated code: Implications for education. In: Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training, pp 1--11
2024
-
[23]
Perry N, Srivastava M, Kumar D, et al (2023) Do users write more insecure code with ai assistants? In: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp 2785--2799
2023
-
[24]
IEEE Access
Pham H, Ha H, Tong V, et al (2024) Magecode: Machine-generated code detection method using large language models. IEEE Access
2024
-
[25]
In: International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, Springer, pp 44--64
Pordanesh S, Bukhari S, Tan B, et al (2025) Hiding in plain sight: On the robustness of ai-generated code detection. In: International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, Springer, pp 44--64
2025
-
[26]
it’s weird that it knows what i want
Prather J, Reeves BN, Denny P, et al (2023) “it’s weird that it knows what i want”: Usability and interactions with copilot for novice programmers. ACM Transactions on Computer-Human Interaction 31(1):1--31
2023
-
[27]
Puri R, Kung D, Janssen G, et al (2021) Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks
2021
-
[28]
arXiv preprint arXiv:240901382
Rahman M, Khatoonabadi S, Abdellatif A, et al (2024) Automatic detection of llm-generated code: A case study of claude 3 haiku. arXiv preprint arXiv:240901382
2024
-
[29]
arXiv preprint arXiv:240606608
Schulhoff S, Ilie M, Balepur N, et al (2024) The prompt report: A systematic survey of prompting techniques. arXiv preprint arXiv:240606608
2024
-
[30]
Suh H, Tafreshipour M, Li J, et al (2024) An empirical study on automatically detecting ai-generated source code: How far are we? arXiv e-prints pp arXiv--2411
2024
-
[31]
In: Proceedings of the 19th International Conference on Predictive Models and Data Analytics in Software Engineering, pp 33--43
Tihanyi N, Bisztray T, Jain R, et al (2023) The formai dataset: Generative ai in software security through the lens of formal verification. In: Proceedings of the 19th International Conference on Predictive Models and Data Analytics in Software Engineering, pp 33--43
2023
-
[32]
arXiv preprint arXiv:230405193
Wang J, Liu S, Xie X, et al (2023 a ) Evaluating aigc detectors on code content. arXiv preprint arXiv:230405193
2023
-
[33]
arXiv preprint arXiv:240702395
Wang J, Luo X, Cao L, et al (2024) Is your ai-generated code really safe? evaluating large language models on secure code generation with codeseceval. arXiv preprint arXiv:240702395
2024
-
[34]
arXiv preprint arXiv:231000746
Wang ZM, Peng Z, Que H, et al (2023 b ) Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv:231000746
2023
-
[35]
ACM Transactions on Software Engineering and Methodology
Xu X, Ni C, Guo X, et al (2024) Distinguishing llm-generated from human-written code by contrastive learning. ACM Transactions on Software Engineering and Methodology
2024
-
[36]
In: Proceedings of the AAAI Conference on Artificial Intelligence, pp 23155--23162
Xu Z, Sheng VS (2024) Detecting ai-generated code assignments using perplexity of large language models. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp 23155--23162
2024
-
[37]
arXiv preprint arXiv:250103288
Xu Z, Sheng VS (2025) Codevision: Detecting llm-generated code using 2d token probability maps and vision models. arXiv preprint arXiv:250103288
2025
-
[38]
arXiv preprint arXiv:231005103
Yang X, Zhang K, Chen H, et al (2023) Zero-shot detection of machine-generated codes. arXiv preprint arXiv:231005103
2023
-
[39]
arXiv preprint arXiv:240516133
Ye T, Du Y, Ma T, et al (2024) Uncovering llm-generated code: A zero-shot synthetic code detector via code rewriting. arXiv preprint arXiv:240516133
2024
-
[40]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[41]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[42]
WZ5Zyz ; <8 F^zŋM<yI_d H`=
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
1999
-
[43]
sn-basic.bst
FUNCTION identify.basic.version "sn-basic.bst" " [2024/07/19 v1.1 bibliography style]" * top ENTRY address archive author booktitle chapter doi edition editor eid eprint howpublished institution journal key keywords month note number organization pages publisher school series ...
2024
-
[44]
write newline
" write newline "" before.all 'output.state := FUNCTION add.period duplicate empty 'skip "." * add.blank if FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap dupl...
-
[45]
write newline
" write newline "" before.all 'output.state := FUNCTION output.doi doi empty skip "doi:" doi * "" * output if FUNCTION format.archive archivePrefix empty "" archivePrefix ":" * if FUNCTION format.primaryClass primaryClass empty "" " [" primaryClass * "] " * if FUNCTION format....
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.