AIDetx: a compression-based method for identification of machine-learning generated text

Armando J. Pinho; Diogo Magalh\~aes; Diogo Pratas; Leonardo Almeida; Pedro Rodrigues

REVIEW 4 major objections 4 minor 14 references

AIDetx: a compression-based method for identification of machine-learning generated text

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A compression trick tells human-written from AI-generated text at over 97% F1.

desk verdict AIDetx is a credible compression-based detector with strong in-distribution F1, but the general claim needs cross-domain and baseline evidence before I would trust it. read the letter →

arxiv 2411.19869 v1 pith:2FX7J2GZ submitted 2024-11-29 cs.CL cs.LG

Leonardo Almeida , Pedro Rodrigues , Diogo Magalhães , Armando J. Pinho , Diogo Pratas This is my paper · ORCID

classification cs.CLcs.LG

keywords AI-generatedtextdetectionmachine-generatedfinite-contextmodelsclassificationdatacompressionMarkovcross-entropyHC3dataset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a simple compression-based classifier can reliably tell human-written from AI-generated text. It builds two finite-context models, one trained on human reference texts and one on AI reference texts, and labels a new document by whichever model compresses it into fewer bits. On the HC3 benchmark the method reaches an F1 score above 0.97, and on the AI-human-text benchmark above 0.99, while running on a single CPU core with no GPU. The authors argue this makes automated text detection cheaper, faster, and more interpretable than deep-learning classifiers, which need large language models and substantial training resources.

What carries the argument

The load-bearing object is a finite-context model (FCM), a Markov model that estimates the probability of each next symbol given the previous $k$ symbols. Equation (2) gives a smoothed count estimate $P(x_i \mid x_{i-1},\dots,x_{i-k}) = (N(\text{context},x_i)+\alpha)/(\sum_j N(\text{context},j)+\alpha|\Sigma|)$, and Equation (1) sums $-\log_2 P$ over the text to get the bits required. With $k=8$, $\alpha=0.5$, and an alphabet reduced to digits, letters, and spaces, the two class models are built from reference texts of up to 7 million characters; classifying a target text is then just comparing the two bit totals and choosing the smaller.

What would settle it

Retrain AIDetx on HC3 only, then run it on a held-out collection of, say, news articles or technical abstracts written by humans and by several recent large language models; if accuracy on that collection drops to near chance, the central transfer claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the cross-entropy between a candidate text and two class-specific finite-context models acts as a high-accuracy discriminator between human and machine text. Specifically, for an order-8 Markov model with smoothing factor $\alpha=0.5$ over a trimmed 37-character alphabet, the number of bits needed to encode the text under each class model is computed by Equation (1), with symbol probabilities given by Equation (2). The text is assigned to the class whose model requires fewer bits. On held-out test data this decision rule reaches an accuracy of 0.9757 and an F1 score of 0.9752 on HC3, and an accuracy of 0.9963 and an F1 score of 0.9963 on AI-human-text. The authors present this as evidence that recognisable statistical differences between human and AI prose can be captured by a small Markov model without any neural network.

Load-bearing premise

The load-bearing premise is that the statistical patterns encoded in the two reference texts transfer to texts outside the training domains; if a new corpus of human and AI writing has different distributional features, the bit-count comparison will not separate the classes.

Editorial extensions

If this is right

AIDetx can classify text on commodity hardware: model creation for HC3 takes 22 seconds and inference processes roughly 3 million characters per second on one CPU core.
Detection no longer requires training or loading a large language model; a two-model comparison gives a transparent, reproducible decision.
Performance grows with the amount of reference text and with target length, so users can trade reference size against accuracy in low-resource settings.
The method can be applied to any two text collections, not just human-versus-AI, suggesting the same compressor-based decision rule could serve other binary text-classification tasks.
Being single-threaded, the method can be sped up further by parallelisation, cutting inference time below the reported rates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the same rule could be trained per genre or per author, turning compression distance into a lightweight stylometric tool.
A likely stress test is adversarial AI text explicitly designed to match human perplexity; the fixed-order Markov model may be more vulnerable than neural detectors to such manipulation.
Since the classifier produces bit counts, it can be combined with a threshold to flag uncertain texts for human review, something the paper does not discuss.
The dependence on reference length suggests diminishing returns past a few million characters, so practitioners could estimate a data budget for their own domain.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a clean, honest application of finite-context-model compression classification to AI-text detection. The method is not new—the authors themselves cite the same framework for authorship attribution, language trees, and digit classification—but applying it to machine-generated text is a legitimate new application, and the paper does it carefully. The F1 scores are high (0.9752 on HC3, 0.9963 on AI-human-text), the ablations on reference length and target length are sensible, and the CPU-only efficiency is real and useful. Code is public, and the parameter search is described clearly enough to follow.

The main soft spot is exactly what the stress-test note flags: the reported accuracy comes only from test splits of the same two datasets used to train the reference models, and both datasets are ChatGPT-era Q&A-style text. The classifier is therefore likely exploiting n-gram regularities specific to these generators and this genre. The paper does concede in the conclusions that performance may vary on other datasets, but the abstract and introduction frame AIDetx as a general, competitive method. That mismatch matters. No cross-dataset, cross-generator, or cross-domain evaluation is provided, so the scope of the central claim is not established.

A second soft spot is the absence of any baseline comparisons on the same data. Saying AIDetx is more efficient than LLM-based detectors is fine, but comparing against BERT's training cost is not a fair runtime comparison. I would want to see AIDetx next to a fine-tuned or zero-shot detector on the HC3 test set. A third, smaller issue is preprocessing: "removing duplicates and short samples, and balancing character counts" is vague, and the public repo has no commit hash, which makes exact reproduction harder than it should be.

On the positive side, the paper does not oversell its internal results: the F1 numbers are computed on held-out test sets after validation-based hyperparameter selection, and the target-length experiments give a real sense of when the method works. The citation pattern is honest, and the reliance on the authors' own FCM work is legitimate given that the method builds directly on it.

Who is this for? People who need a lightweight, interpretable, no-GPU detector on short-to-medium texts and are willing to accept that it may not transfer across domains without retraining. It deserves a serious referee, but the referee should ask for cross-domain/cross-generator evaluation, proper baselines, and exact preprocessing details. I would not desk-reject it.

Referee Report

4 major / 4 minor

Summary. The paper introduces AIDetx, a compression-based classifier that distinguishes human-written from machine-generated text using finite-context models (FCMs). The method trains one FCM on human reference text and another on AI reference text, then assigns a target text to the class whose model yields the lower cross-entropy, formalized by Equations (1) and (2). The authors perform a grid search over the Markov order k and smoothing factor alpha, then select the alphabet Sigma_2 after trimming experiments. On held-out test splits of HC3 and AI-human-text, the method achieves F1 scores of 0.9752 and 0.9963, respectively, with CPU-only inference at several million characters per second. The paper also includes ablations on reference-text length and target-text length, and the implementation is publicly available.

Significance. If the reported results hold beyond the specific evaluation setting, AIDetx is an interesting contribution because it is lightweight, interpretable, requires no GPU, and is backed by public code. The information-theoretic framing is clean, and the ablations on reference and target length are useful practical analyses. However, the paper currently demonstrates only in-distribution performance on two ChatGPT-era Q&A-style datasets, with no comparison against existing detectors and no cross-domain or cross-generator evaluation. The general claim of a competitive method for detecting machine-generated text is therefore not yet established. The central derivation is internally sound, but the empirical scope is the main weakness.

major comments (4)

[Results, Classifier testing (Table 2) and Conclusions] The reported F1 scores are measured only on test splits of the same datasets from which the reference models were trained, so the evaluation does not establish the method as a general detector. For HC3, the AI-class model is trained on ChatGPT-style Q&A text and tested on the same distribution; the paper explicitly concedes in the Conclusions that 'performance may vary on other datasets,' yet the abstract and introduction frame AIDetx as a general, competitive method. Please add cross-dataset, cross-generator, or cross-genre experiments, or explicitly narrow the central claim to the evaluated setting.
[Benchmark, Datasets] The train/validation/test split is not described at the granularity needed to rule out topic leakage. HC3 contains human and ChatGPT answers to the same questions; if the split is performed at the level of individual answers rather than source questions or prompts, the same question can appear in both training and test partitions, allowing the model to exploit topic-specific n-gram matches and inflate the reported F1 scores. The authors should specify the splitting unit and, if the current split is sample-level, re-evaluate with a prompt-aware split.
[Results, Classifier testing (Table 2)] No quantitative comparison with existing AI-text detectors is provided. The abstract claims that AIDetx offers advantages 'compared to current methods,' but the only concrete comparison is an anecdotal remark about BERT training cost. At minimum, the authors should report the F1 of one or more strong baselines (e.g., GPTZero, OpenAI detector, a fine-tuned BERT classifier, or a gzip-based compression classifier) on the same test splits, so that the absolute F1 values in Table 2 can be interpreted.
[Benchmark, Alphabet Trimming (Table 1)] The statement that 'Sigma_2 was the one that averaged the best performance on both datasets' is contradicted by Table 1. The average F1 across the two datasets is approximately 0.9836 for Sigma_2, 0.9839 for Sigma_1, and 0.9874 for Sigma_3. This should be corrected, or the actual selection criterion should be stated, since the current wording misrepresents the reported numbers.

minor comments (4)

[Methodology, Equation (2)] Equation (2) is typeset ambiguously: the summation should be placed in the denominator as the total count over all symbols j in the alphabet, e.g., P(x_i | context) = (N(context, x_i) + alpha) / (sum_{j in Sigma} N(context, j) + alpha * |Sigma|). Please clarify the notation.
[Methodology, hyperparameters] The sentence 'the last one is used to classify the target texts' is imprecise, because the alphabet Sigma also affects model construction through Equation (2). Both the reference models and the target classification use the same alphabet.
[Results, Classifier testing] The paper does not state the exact number of characters used for the final reference models in Table 2, even though the influence of reference length is studied earlier. Reporting the final character counts per class would improve reproducibility.
[Conclusions] There are minor language issues, such as 'substantial less computational power,' which should read 'substantially less computational power.' The sentence comparing BERT training cost also conflates pretraining a general language model with training a detector; this comparison should be reworded.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the FCM classifier is defined by standard cross-entropy equations and evaluated on held-out splits; the two self-citations are background, and the reported F1 is not forced by construction.

full rationale

The derivation chain is self-contained. AIDetx defines a two-class model comparison: Equations (1) and (2) specify the ideal code length under a smoothed finite-context model trained by counting n-grams in class reference texts, and classification assigns a target to the class with lower cross-entropy. The reported F1 scores (0.9752 on HC3 and 0.9963 on AI-human-text) are measured on test splits held out from the training and validation splits used for hyperparameter optimization (k=8, alpha=0.5, Sigma2), so no fitted parameter is renamed as a prediction. The self-citations (refs [7] and [12]) are background examples of compression-based classification and FCMs for DNA; the paper's own equations define the model, so those citations are not load-bearing. The conclusion's caveat that 'performance may vary on other datasets' is an honest external-validity limitation, not a circular step. One non-circular correctness issue is noted: the claim that Sigma2 'averaged the best performance on both datasets' is contradicted by Table 1, where Sigma3 has a higher average F1; this is a factual inconsistency in the alphabet-selection justification, not a reduction of the reported results to the method's inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard information-theoretic and Markovian assumptions. The hyperparameters are empirically fitted to validation data, not derived from first principles. There are no invented physical entities.

free parameters (4)

Markov order k = 8
Selected via grid search on validation F1 and time performance; the paper states this value consistently delivered high F1 without degradation.
Smoothing factor alpha = 0.5
Selected via grid search for optimal balance across datasets and values of k.
Alphabet Sigma_2 = '1234567890 abcdefghijklmnopqrstuvwxyz'
Chosen because it averaged the best performance on both datasets among the four tested alphabets.
Preprocessing thresholds
The minimum length for removing short samples and the exact balancing procedure are not specified, making these implicit free choices.

assumptions (4)

standard math The Markov property: probability of the next symbol depends only on the previous k symbols.
Invoked in Equation (2) to define finite-context model probabilities.
domain assumption Cross-entropy between a model and a target text reflects class membership.
The entire classification rule relies on the information-theoretic argument that lower cross-entropy implies better fit to the class model.
domain assumption The reference texts are representative of their respective classes.
The method assumes that training data captures the statistical regularities of human and AI text for the target distribution; this is load-bearing for generalization.
standard math Smoothing with alpha avoids zero probabilities and stabilizes estimates.
Equation (2) uses additive smoothing, a standard technique.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AIDetx: a compression-based method for identification of machine-learning generated text." pith.science (2026). https://pith.science/paper/2FX7J2GZ

@misc{pith2026241119869,
  author       = {Pith},
  title        = {Pith review of: AIDetx: a compression-based method for identification of machine-learning generated text},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2FX7J2GZ}},
  note         = {Machine review of arXiv:2411.19869}
}

read the original abstract

This paper introduces AIDetx, a novel method for detecting machine-generated text using data compression techniques. Traditional approaches, such as deep learning classifiers, often suffer from high computational costs and limited interpretability. To address these limitations, we propose a compression-based classification framework that leverages finite-context models (FCMs). AIDetx constructs distinct compression models for human-written and AI-generated text, classifying new inputs based on which model achieves a higher compression ratio. We evaluated AIDetx on two benchmark datasets, achieving F1 scores exceeding 97% and 99%, respectively, highlighting its high accuracy. Compared to current methods, such as large language models (LLMs), AIDetx offers a more interpretable and computationally efficient solution, significantly reducing both training time and hardware requirements (e.g., no GPUs needed). The full implementation is publicly available at https://github.com/AIDetx/AIDetx.

Figures

Figures reproduced from arXiv: 2411.19869 by the authors.

**Figure 2.** F1 score for the grid search of the hyperparameters [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Time performance for the grid search of the hyperparameters [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Classifier performance evolution as reference text length increases. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Accuracy in function of the length of the target texts for the datasets HC3 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Confusion matrix for HC3 and AI-human-text datasets. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages

[1]

The genuine problem of fake news,

M Mitchell Waldrop, “The genuine problem of fake news,” Proceedings of the National Academy of Sciences, vol. 114, no. 48, pp. 12631–12634, 2017

work page 2017
[2]

Challenges in deep learning,

Plamen P. Angelov and Alessandro Sperduti, “Challenges in deep learning,” in The European Symposium on Artificial Neural Networks , 2016

work page 2016
[3]

Deep learning: systematic review, models, challenges, and research directions,

Tala Talaei Khoei, Hadjar Ould Slimane, and Naima Kaabouch, “Deep learning: systematic review, models, challenges, and research directions,” Neural Computing and Applications, vol. 35, 09 2023

work page 2023
[4]

Perfect detec- tion of computer-generated text faces fundamental challenges,

Martin Majovsky, Martin ˇCern´ y, David Netuka, and Tomas Mikolov, “Perfect detec- tion of computer-generated text faces fundamental challenges,” Cell Reports Physical Science, 01 2024

work page 2024
[5]

Using literal and grammatical statistics for authorship attribution,

Olga V. Kukushkina, Anatoly A. Polikarpov, and Dmitry V. Khmelev, “Using literal and grammatical statistics for authorship attribution,” Probl. Inf. Transm. , vol. 37, no. 2, pp. 172–184, 2001

work page 2001
[6]

Language trees and zip- ping,

Dario Benedetto, Emanuele Caglioti, and Vittorio Loreto, “Language trees and zip- ping,” Physical Review Letters, vol. 88, no. 4, Jan. 2002

work page 2002
[7]

An Application of Data Compression Models to Handwritten Digit Classification,

Armando J Pinho and Diogo Pratas, “An Application of Data Compression Models to Handwritten Digit Classification,” in Advanced Concepts for Intelligent Vision Systems: 19th International Conference, ACIVS 2018, Poitiers, France, September 24–27, 2018, Proceedings 19. Springer, 2018, pp. 487–495

work page 2018
[8]

Statistical compression-based mod- els for text classification,

Vidya Saikrishna, David L. Dowe, and Sid Ray, “Statistical compression-based mod- els for text classification,” in 2016 Fifth International Conference on Eco-friendly Computing and Communication Systems (ICECCS) , 2016, pp. 1–6

work page 2016

Show all 14 references

[9]

Tweet classi- fication by data compression,

Kyosuke Nishida, Ryohei Banno, Ko Fujimura, and Takahide Hoshide, “Tweet classi- fication by data compression,” in Proceedings of the 2011 International Workshop on DETecting and Exploiting Cultural DiversiTy on the Social Web , New York, NY, USA, 2011, DETECT ’11, p. 29–34, A...

2011
[10]

Timothy C Bell, John G Cleary, and Ian H Witten, Text compression, Prentice-Hall, Inc., 1990

1990
[11]

Khalid Sayood, Introduction to data compression , Morgan Kaufmann, 2017

2017
[12]

Finite-context models for DNA coding,

Armando J Pinho, Ant´ onio JR Neves, Daniel A Martins, Carlos AC Bastos, and PJSG Ferreira, “Finite-context models for DNA coding,” Signal Processing, pp. 117–130, 2010

2010
[13]

How close is chatgpt to human experts? comparison corpus, evaluation, and detection,

Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu, “How close is chatgpt to human experts? comparison corpus, evaluation, and detection,” 2023

2023
[14]

Bert: Pre- training of deep bidirectional transformers for language understanding,

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “Bert: Pre- training of deep bidirectional transformers for language understanding,” 2019

2019

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

The genuine problem of fake news,

M Mitchell Waldrop, “The genuine problem of fake news,” Proceedings of the National Academy of Sciences, vol. 114, no. 48, pp. 12631–12634, 2017

work page 2017

[2] [2]

Challenges in deep learning,

Plamen P. Angelov and Alessandro Sperduti, “Challenges in deep learning,” in The European Symposium on Artificial Neural Networks , 2016

work page 2016

[3] [3]

Deep learning: systematic review, models, challenges, and research directions,

Tala Talaei Khoei, Hadjar Ould Slimane, and Naima Kaabouch, “Deep learning: systematic review, models, challenges, and research directions,” Neural Computing and Applications, vol. 35, 09 2023

work page 2023

[4] [4]

Perfect detec- tion of computer-generated text faces fundamental challenges,

Martin Majovsky, Martin ˇCern´ y, David Netuka, and Tomas Mikolov, “Perfect detec- tion of computer-generated text faces fundamental challenges,” Cell Reports Physical Science, 01 2024

work page 2024

[5] [5]

Using literal and grammatical statistics for authorship attribution,

Olga V. Kukushkina, Anatoly A. Polikarpov, and Dmitry V. Khmelev, “Using literal and grammatical statistics for authorship attribution,” Probl. Inf. Transm. , vol. 37, no. 2, pp. 172–184, 2001

work page 2001

[6] [6]

Language trees and zip- ping,

Dario Benedetto, Emanuele Caglioti, and Vittorio Loreto, “Language trees and zip- ping,” Physical Review Letters, vol. 88, no. 4, Jan. 2002

work page 2002

[7] [7]

An Application of Data Compression Models to Handwritten Digit Classification,

Armando J Pinho and Diogo Pratas, “An Application of Data Compression Models to Handwritten Digit Classification,” in Advanced Concepts for Intelligent Vision Systems: 19th International Conference, ACIVS 2018, Poitiers, France, September 24–27, 2018, Proceedings 19. Springer, 2018, pp. 487–495

work page 2018

[8] [8]

Statistical compression-based mod- els for text classification,

Vidya Saikrishna, David L. Dowe, and Sid Ray, “Statistical compression-based mod- els for text classification,” in 2016 Fifth International Conference on Eco-friendly Computing and Communication Systems (ICECCS) , 2016, pp. 1–6

work page 2016