Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Detection of LLM-Generated Java Code Using Discretized Nested Bigrams

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Discretized nested-bigram features detect GPT-rewritten Java code with over 96% accuracy.

desk verdict Practical feature-engineering advance with a likely leakage problem in the evaluation: the accuracy numbers need a file-disjoint re-run before they can be trusted. read the letter →

arxiv 2502.15740 v1 pith:DXQKDWRY submitted 2025-02-07 cs.SE cs.AIcs.CLcs.LG

Timothy Paek , Chilukuri Mohan This is my paper · ORCID
classification cs.SEcs.AIcs.CLcs.LG
keywords LLM-GeneratedCodeAuthorshipAttributionGPTDetectionLargeLanguageModelsStylometryFeaturesAbstractSyntaxTreeNestedBigramsDiscretization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that LLM-generated Java code can be reliably told apart from human-written code by looking at the shape of its abstract syntax tree (AST). The proposed features, called Equal Width Discretized Nested Bigram Frequencies (EWD-NB-F), count small two-node AST patterns and then pack those sparse counts into a few dense bins; an optional variant mixes in CodeBERT's CLS-token means. Across three datasets—976 files rewritten by GPT 3.5/GPT 4, a 40-author corpus, and 76,089 Google Code Jam files rewritten by GPT 4o—the method reports 96-99% accuracy and an AUC up to 0.999, beating a commercial GPT detector and three previous authorship-attribution approaches. If true, it means a cheap, low-dimensional feature set could screen individual code fragments for LLM authorship, which matters for academic integrity and for detecting AI-written snippets inside otherwise human files.

What carries the argument

The central objects are Nested Bigrams—two-node subtrees of an AST that capture a parent-child syntactic pattern with attribute information—and Equal Width Discretization, which maps the very large, sparse vocabulary of nested bigrams into a small number of dense bins by summing normalized frequencies (or CLS-token means) within each bin. The binning collapses tens of thousands of rare features into 12-227 dense columns, which is what lets off-the-shelf tree ensembles such as Random Forest, XGBoost, LGBM, and CatBoost chew through the data. The paper's load-bearing move is formulas (1)-(2): the linear index mapping and the per-bin sum, normalized by code-group character count, that turn sparse indicators into soft membership values.

What would settle it

Run the same EWD-NB-F pipeline but split the data by source file (or by original author) rather than randomly across code groups, and measure accuracy on files never seen in training; if accuracy falls well below 96%, the reported numbers depend on within-file leakage. A second check: train on GPT-3.5/GPT-4 rewrites and test on GPT-4o rewrites (or vice versa) to see whether the features capture a general signal of LLM authorship or just a model-specific style.

Watch

Extended reading notes

Core claim

The central claim is that dense binning of sparse AST bigram frequencies is a strongly discriminative signal for code authorship. Raw nested-bigram features (NB-F) are too sparse to be effective, and compressing them by stripping attribute information (CNB-F) helps but loses information; the paper's EWD-NB-F instead keeps the full nested-bigram vocabulary and merges its counts into equal-width bins, producing just 12 features on the GPT dataset. Adding CodeBERT-embedded CLS means of nested bigrams (EWD-CBNB-CM) raises accuracy about another percentage point. The paper reports this outperforms the ZeroGPT API (72-73% accuracy) and prior Java authorship methods, and maintains 98-99% accuracy on a 40-author task and 99% accuracy (AUC 0.999) on a 76,089-file scale-up. It positions the gain as coming from representing sparse information in dense membership bins rather than from any particular classifier.

Load-bearing premise

The reported accuracies assume that splitting code groups randomly into training and test sets yields independent samples; if groups from the same file, or human and GPT-rewritten versions of the same file, appear in both partitions, the classifier can memorize file-specific patterns and the 96-99% numbers would drop on truly held-out files.

Editorial extensions

If this is right

  • Code fragments, not just whole files, can be screened for LLM authorship, matching real scenarios where a student pastes one LLM-written method into a mostly human file.
  • The 12-feature EWD-NB-F representation makes LLM-code detection feasible on ordinary hardware, since feature extraction takes minutes and inference runs on small tabular data rather than per-fragment transformer sweeps.
  • The method transfers to a large number of authors: 99% accuracy across 1,000+ Google Code Jam participants suggests the signal is not just memorizing a few authors' styles.
  • Because adding CodeBERT-embedded features buys only about one percentage point over EWD-NB-F alone, the cheaper feature set is nearly sufficient for practical deployment.
  • Commercial detectors built for prose text, such as ZeroGPT, are shown to be far weaker on code, indicating that code-specific syntactic features are necessary for this task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication not tested in the paper is whether the same binning trick transfers to other sparse syntactic code features, such as AST paths or code2vec-style embeddings, where it might similarly convert sparse occurrences into dense discriminative columns.
  • The evaluation's random train/test split on code groups—rather than a file- or author-disjoint split—leaves open how much of the 96-99% accuracy survives on fully unseen files; a strict-split benchmark would settle whether the detector generalizes to unseen files or only to unseen chunks of the same files.
  • A natural next stress test is cross-model generalization: train on GPT-3.5/GPT-4 rewrites and probe on rewrites from a different LLM (e.g., Claude or Gemini). The paper explicitly notes robustness to unseen coding styles as future work.
  • Because the discretized features are cheap to compute, a practical detector could screen student submissions as a pre-filter, flagging fragments for closer human review; this deployment path is implied by the paper's fragment-level design but not worked out.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes a stylometric approach for detecting LLM-generated Java code. It introduces discretized nested bigram frequency features (EWD-NB-F) and a combined CodeBERT-embedded version (EWD-CBNB-CM), and evaluates them on two newly created datasets (GPT Dataset with 976 files, GPT GCJ Dataset with 76,089 files) plus the 40-author Yang et al. dataset. The authors report accuracy exceeding 96% on the GPT Dataset, 98-99% on the 40-author dataset, and 99% (AUC 0.999) on the GPT GCJ Dataset, outperforming the ZeroGPT API and three prior authorship-attribution methods. The paper also releases the datasets publicly and uses deterministic ensemble configurations.

Significance. If the reported results hold under a properly disjoint evaluation, this is a valuable contribution: a low-dimensional (12-239 feature) representation of sparse AST bigram frequencies appears to be highly discriminative for LLM code detection, and the public datasets will be useful to the community. The paper is also commendable for reporting multiple metrics, comparing with an external API, and fixing random seeds for reproducibility. However, the significance of the numerical claims depends on resolving the evaluation-protocol concerns below.

major comments (3)
  1. [Section 4.1 / Table 2] The evaluation protocol is described only as 'random training vs. testing data splitting' (Section 4.1 and the Table 2 note). Because Section 3.1 creates multiple code groups per file and generates GPT-rewritten counterparts of the same files, a random group-level split can place chunks from the same source file, and chunks from a human file and its GPT rewrite, in both training and test partitions. EWD-NB-F features are frequencies of AST nested bigrams and therefore carry file- and author-specific structure; a tree ensemble can memorize these patterns rather than learn a general 'LLM-generated' signal. The reported low variance across random re-splitting (Table 2; Section 4.3's mean 0.9855, std 0.0003) is consistent with such leakage. The paper's own conclusion (Section 5) states that robustness to unseen coding styles 'needs to be assessed,' which concedes that the current protocol does not establish generalization to new files or authors. The central claims in Tables 1, 3, and 5 are therefore not interpretable until the authors clarify (or fix) the split: it must be disjoint by file, by original/rewrite pair, and by author.
  2. [Section 4.2 / Table 4] The comparison with prior work in Table 4 is not apples-to-apples. Prior methods (code2seq [Omi et al., 2021], Bigram [Al-Ahmad et al., 2023], PSOBP [Yang et al., 2017]) perform multi-class full-file authorship attribution, while the proposed approach performs binary classification on code groups. The text acknowledges this difference but still concludes that the approach 'performed substantially better.' To support the claim of outperforming prior work, the authors should either adapt the prior methods to the same binary code-group protocol or report the proposed features in a multi-class full-file setting. As presented, the accuracy/F1 values in Table 4 are not comparable across tasks.
  3. [Section 3.1 / Section 5] In the GPT Dataset, GPT-rewritten files come from only 5 of the 11 authors, while the human class includes all 11 authors. Even under a clean author-disjoint split, a classifier could learn to separate those 5 authors' coding styles from the other 6, rather than to detect LLM-generated code in general. The GPT GCJ dataset (with 1,000+ authors and 17,565 rewritten files) is better suited to support the generalization claim. The paper should discuss this limitation explicitly and, if feasible, report results on a split where the authors whose code was rewritten are held out from training.
minor comments (4)
  1. [Section 4.1 / Section 5] The service is called 'ZeroGPT' in Section 4.1 but 'GPTZero API' in Section 5; these are different products. Please use a consistent name and verify which service was actually used.
  2. [Section 3.1 / Section 5] The GPT Dataset is described as having 11 authors in Section 3.1 but '15 human authors' in the Conclusion. Please correct this inconsistency.
  3. [Table 1] The text says group sizes 20, 30, 50, and 60 are omitted, but the CBN row includes a group size 20 entry. Please align the table with the text.
  4. [Section 4.2] The 40-author dataset is described as 'with 15 authors considered the positive class' but the binary classification setup is not fully defined. Please clarify which authors are positive and how the split is made.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proposed features are empirically evaluated on held-out splits against external baselines, not derived from the target labels.

full rationale

The paper's central claims are empirical performance measurements, not derivations from assumptions that already contain the result. EWD-NB-F and EWD-CBNB-CM are new feature constructions applied to AST nested bigrams; their discriminative power is tested by training standard ensembles on training groups and reporting accuracy, F1, AUC, and precision on test groups, with comparisons to the ZeroGPT API and three prior authorship-attribution methods. No predicted quantity is defined in terms of a fitted parameter in a way that forces the reported accuracy; the bin width is a hyperparameter, and the paper states that results were stable over a range of bin widths. The nested-bigram base feature is cited from prior work by the same group, but the discretized variants and their evaluation are independent of that citation, so this is not load-bearing circularity. The paper's own conclusion notes that robustness to unseen coding styles still needs to be assessed; that is a stated limitation about generalization, not a circular step in the derivation. Concerns about random group-level splitting potentially leaking file-level or author-level information are validity threats about data independence, not instances of a claim being equivalent to its inputs by construction, and therefore do not raise the circularity score under the specified criteria.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The evaluation rests on synthetic rewrite datasets, random train/test splitting, and the prior nested-bigram representation. The hand-chosen numeric controls are bin width and code-group size; no invented physical or conceptual entities are introduced.

free parameters (2)
  • bin_width = 3000 for GPT GCJ; optimized for accuracy on GPT dataset
    Controls how many nested-bigram features are summed into one dense bin (Eq. 1-2). The authors state accuracy is insensitive once width exceeds about 2000.
  • code_group_size = 10-70 lines (varied)
    Determines the granularity of code chunks; results are averaged over group sizes, and performance was only marginally affected.
assumptions (4)
  • domain assumption LLM rewriting of existing human Java code with a 'maintain functionality' prompt produces samples representative of real LLM-generated code.
    Section 3.1. All three datasets are constructed this way; Section 5 admits real-world prompts and unseen styles remain unassessed.
  • domain assumption Randomly splitting code groups into train/test sets yields independent evaluation samples.
    Section 4.1 uses 'random training vs. testing data splitting'; no file/author-disjoint partition is described, and multiple groups per file can leak.
  • domain assumption Nested bigrams extracted from ASTs are a valid, discriminative stylometric representation.
    Section 2.2 adopts this from Hozhabrierdi et al. 2020 rather than proving it.
  • domain assumption CodeBERT CLS token means are useful for Java code-group representation.
    Section 3.2. The choice of mean over max/min is heuristic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Detection of LLM-Generated Java Code Using Discretized Nested Bigrams." pith.science (2026). https://pith.science/paper/DXQKDWRY

@misc{pith2026250215740,
  author       = {Pith},
  title        = {Pith review of: Detection of LLM-Generated Java Code Using Discretized Nested Bigrams},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DXQKDWRY}},
  note         = {Machine review of arXiv:2502.15740}
}
read the original abstract

Large Language Models (LLMs) are currently used extensively to generate code by professionals and students, motivating the development of tools to detect LLM-generated code for applications such as academic integrity and cybersecurity. We address this authorship attribution problem as a binary classification task along with feature identification and extraction. We propose new Discretized Nested Bigram Frequency features on source code groups of various sizes. Compared to prior work, improvements are obtained by representing sparse information in dense membership bins. Experimental evaluation demonstrated that our approach significantly outperformed a commonly used GPT code-detection API and baseline features, with accuracy exceeding 96% compared to 72% and 79% respectively in detecting GPT-rewritten Java code fragments for 976 files with GPT 3.5 and GPT4 using 12 features. We also outperformed three prior works on code author identification in a 40-author dataset. Our approach scales well to larger data sets, and we achieved 99% accuracy and 0.999 AUC for 76,089 files and over 1,000 authors with GPT 4o using 227 features.

Figures

Figures reproduced from arXiv: 2502.15740 by the authors.

Figure 1
Figure 1. Feature Extraction Flowchart captured by nested bigrams. However, their sparsity (small number of occurences in the dataset) appeared to diminish performance. Using Principal Component Analysis and Autoencoding did not help, motivating the exploration of new features. Increased feature specificity usually implies that the number of features increases. For example, a dataset with 1,000 files split into code groups of… view at source ↗
Figure 2
Figure 2. Two major components of the Feature Extraction process [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A fine-tuned encoder-only CodeT5 model attributes LLM-generated C code to its source model with up to 97.56% binary and 95.40% five-class accuracy on a new 32,000-program benchmark.

Reference graph

Works this paper leans on

13 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Pegah Hozhabrierdi, Dunai Fuentes Hitos, and Chilukuri K. Mohan. Python source code de-anonymization using nested bigrams. In 2018 IEEE International Conference on Data Mining Workshops (ICDMW) , pages 23–28,

    work page 2018

  2. [6]

    Bilal Al-Ahmad, Nailah Al-Madi, Abdullah Alzaqebah, Rami S

    doi:10.1109/QRS57517.2022.00059. Bilal Al-Ahmad, Nailah Al-Madi, Abdullah Alzaqebah, Rami S. Alkhawaldeh, Khaled Aldebei, Md Faisal Kabir, Ismail Altaharwa, Mua’ad Abu-Faraj, and Ibrahim Aljarah. Meta-heuristic guided feature optimization for en- hanced authorship attribution in java source code. IEEE Access , 11:141657–141673,

    arXiv 2022

  3. [10]

    Dan Kondratyuk, Mingxing Tan, Matthew Brown, and Boqing Gong

    URL https://arxiv.org/abs/2103.03806. Dan Kondratyuk, Mingxing Tan, Matthew Brown, and Boqing Gong. When ensembling smaller models is more efficient than single large models,

    arXiv

  4. [11]

    Leo Breiman

    URL https://arxiv.org/abs/2005.00570. Leo Breiman. Random forests. Mach. Learn., 45(1):5–32, oct

    arXiv 2005

  5. [2001]

    doi:10.1023/A:1010933404324

    ISSN 0885-6125. doi:10.1023/A:1010933404324. URL https://doi.org/10.1023/A:1010933404324. Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD ’16, page 785–794, New York, NY , USA,

    doi:10.1023/a:1010933404324

  6. [2015]

    ISBN 9781931971232

    USENIX Association. ISBN 9781931971232. Pegah Hozhabrierdi, Dunai Fuentes Hitos, and Chilukuri K. Mohan. Zero-shot source code author identification: A lexicon and layout independent approach. In 2020 International Joint Conference on Neural Networks (IJCNN) , pages 1–8,

    work page 2020

  7. [2016]

    ISBN 9781450342322

    Association for Computing Machinery. ISBN 9781450342322. doi:10.1145/2939672.2939785. URL https://doi.org/10.1145/2939672.2939785. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: a highly efficient gradient boosting decision tree. In Proceedings of the 31st International Conference on Neural Info...

    work page arXiv

  8. [2017]

    Abdul Mannan Omi, Monir Hossain, Md Nahidul Islam, and Tanni Mittra

    doi:10.1371/journal.pone.0187204. Abdul Mannan Omi, Monir Hossain, Md Nahidul Islam, and Tanni Mittra. Multiple authors identification from source code using deep learning model. In 2021 International Conference on Electronics, Communications and Information Technology (ICECIT), pages 1–4,

    work page doi:10.1371/journal.pone.0187204 2021

Show all 13 references
  1. [2018]

    Xinyu Yang, Guoai Xu, Qi Li, Yanhui Guo, and Miao Zhang

    doi:10.1109/ICDMW.2018.00011. Xinyu Yang, Guoai Xu, Qi Li, Yanhui Guo, and Miao Zhang. Authorship attribution of source code by using back propagation neural network based on particle swarm optimization. PLOS ONE , 12:e0187204, 11

    2018

  2. [2020]

    Timothy Paek

    doi:10.1109/IJCNN48605.2020.9207647. Timothy Paek. Gpt java dataset: A dataset for llm-generated code detection. GitHub Repository, 2024a. URL https://github.com/tipaek/GPT-Java-Dataset . Timothy Paek. Gpt java dataset: The largest llm-generated code dataset from google code j...

    2020

  3. [2021]

    Gabriela Czibula, Mihaiela Lupea, and Anamaria Briciu

    doi:10.1109/ICECIT54077.2021.9641497. Gabriela Czibula, Mihaiela Lupea, and Anamaria Briciu. Enhancing the performance of software authorship attribution using an ensemble of deep autoencoders. Mathematics, 10(15),

    2021

  4. [2022]

    doi:10.3390/math10152572

    ISSN 2227-7390. doi:10.3390/math10152572. URL https://www.mdpi.com/2227-7390/10/15/2572. Pengnan Hao, Zhen Li, Cui Liu, Yu Wen, and Fanming Liu. Towards improving multiple authorship attribution of source code. In 2022 IEEE 22nd International Conference on Software Quality, Re...

    2022 doi

  5. [2023]

    doi:10.1109/ACCESS.2023.3341395

    ISSN 2169-3536. doi:10.1109/ACCESS.2023.3341395. Publisher Copyright: © 2013 IEEE. Aylin Caliskan-Islam, Richard Harang, Andrew Liu, Arvind Narayanan, Clare V oss, Fabian Yamaguchi, and Rachel Greenstadt. De-anonymizing programmers via code stylometry. In Proceedings of the 24...

    2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.