{"id":"ccd11f7c-6f25-4afb-91b4-24b99b58df26","arxiv_id":"2411.19869","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AIDetx uses finite-context Markov models to classify text as human or AI-written based on which class-specific compression model yields the fewest bits.","lead":"This paper presents AIDetx, a tool that detects AI-generated text by building separate compression models for human and machine text and assigning each new text to the model that compresses it more. It reports F1 scores above 97% on two public datasets while requiring no GPU and only a fraction of the training cost of large language models, making cheap and interpretable AI-text screening potentially practical.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The high F1 results are measured only within the same datasets and generators used for training; the untested assumption that the method transfers to unseen domains and generators is load-bearing for the general claim.","rationale":"The reader's weakest_assumption correctly identifies cross-domain generalization as the fragile point, and the paper's own conclusion acknowledges this limitation. My stress-test sharpens the concern: the issue is not merely 'other datasets' but that the same-distribution evaluation cannot distinguish between detecting intrinsic AI-text properties and detecting dataset- or generator-specific character n-gram artifacts. Since the alphabet is trimmed to lowercase letters, digits, and space, the model cannot use lexical or punctuation cues, so its signal is purely distributional and likely source-specific. The proposed cross-generator and cross-dataset test would settle whether the classifier transfers. This does not undermine the reported within-dataset results, which appear internally consistent, but it does support the reader's CONDITIONAL verdict rather than full acceptance. No new concern requiring a verdict change was found beyond what the reader already flagged, so UNCHANGED is appropriate.","tokens_in":5853,"tokens_out":3369,"duration_ms":36211,"concrete_test":"Using the released AIDetx code with the reported hyperparameters (k=8, alpha=0.5, Sigma2), train models on the HC3 training split. Then evaluate on three conditions: (1) the held-out HC3 test split to reproduce the reported F1; (2) the held-out AI-human-text test split as a cross-dataset transfer check; (3) a new out-of-domain corpus containing human-written text and outputs from a different and unseen generator (e.g., GPT-4 or Llama-3 in news/essay style), with a separate training reference from that corpus if needed. If F1 on condition (3), or even condition (2), drops substantially below the reported 97% and 99%, the generalization claim fails; if it stays high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"AIDetx's decision rule is a two-class model comparison: it trains one order-8 FCM on human reference text and another on AI reference text, then assigns a target to the class whose model gives lower cross-entropy (Eqs. 1 and 2). The reported F1 scores of 0.9752 and 0.9963 come from test splits of the same datasets (HC3 and AI-human-text) from which the training references were drawn. This means the AI-class model is trained on text from the same generator (ChatGPT-family) and the same Q&A-style distribution as the test targets. The classifier may therefore be exploiting stable n-gram regularities specific to that generator and genre, rather than a general property of machine-generated text. The alphabet trimming to a 37-character set (lowercase letters, digits, space) discards capitalization and most punctuation, so the discriminative signal is entirely character-level n-gram frequencies, which are strongly tied to the source distribution. The paper explicitly concedes in the Conclusions that 'performance may vary on other datasets,' but the abstract and introduction frame AIDetx as a general, competitive method for detecting machine-generated text. Thus the load-bearing assumption is that a fixed-order character FCM trained on two ChatGPT-era Q&A corpora transfers to unseen domains, writing styles, and generation models. No cross-dataset, cross-generator, or cross-domain evaluation is provided, so the central claim's scope is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AIDetx, a compression-based classifier that distinguishes human-written from machine-generated text using finite-context models (FCMs). The method trains one FCM on human reference text and another on AI reference text, then assigns a target text to the class whose model yields the lower cross-entropy, formalized by Equations (1) and (2). The authors perform a grid search over the Markov order k and smoothing factor alpha, then select the alphabet Sigma_2 after trimming experiments. On held-out test splits of HC3 and AI-human-text, the method achieves F1 scores of 0.9752 and 0.9963, respectively, with CPU-only inference at several million characters per second. The paper also includes ablations on reference-text length and target-text length, and the implementation is publicly available.","tokens_in":6098,"tokens_out":5631,"duration_ms":52049,"significance":"If the reported results hold beyond the specific evaluation setting, AIDetx is an interesting contribution because it is lightweight, interpretable, requires no GPU, and is backed by public code. The information-theoretic framing is clean, and the ablations on reference and target length are useful practical analyses. However, the paper currently demonstrates only in-distribution performance on two ChatGPT-era Q&A-style datasets, with no comparison against existing detectors and no cross-domain or cross-generator evaluation. The general claim of a competitive method for detecting machine-generated text is therefore not yet established. The central derivation is internally sound, but the empirical scope is the main weakness.","major_comments":[{"comment":"The reported F1 scores are measured only on test splits of the same datasets from which the reference models were trained, so the evaluation does not establish the method as a general detector. For HC3, the AI-class model is trained on ChatGPT-style Q&A text and tested on the same distribution; the paper explicitly concedes in the Conclusions that 'performance may vary on other datasets,' yet the abstract and introduction frame AIDetx as a general, competitive method. Please add cross-dataset, cross-generator, or cross-genre experiments, or explicitly narrow the central claim to the evaluated setting.","section":"Results, Classifier testing (Table 2) and Conclusions"},{"comment":"The train/validation/test split is not described at the granularity needed to rule out topic leakage. HC3 contains human and ChatGPT answers to the same questions; if the split is performed at the level of individual answers rather than source questions or prompts, the same question can appear in both training and test partitions, allowing the model to exploit topic-specific n-gram matches and inflate the reported F1 scores. The authors should specify the splitting unit and, if the current split is sample-level, re-evaluate with a prompt-aware split.","section":"Benchmark, Datasets"},{"comment":"No quantitative comparison with existing AI-text detectors is provided. The abstract claims that AIDetx offers advantages 'compared to current methods,' but the only concrete comparison is an anecdotal remark about BERT training cost. At minimum, the authors should report the F1 of one or more strong baselines (e.g., GPTZero, OpenAI detector, a fine-tuned BERT classifier, or a gzip-based compression classifier) on the same test splits, so that the absolute F1 values in Table 2 can be interpreted.","section":"Results, Classifier testing (Table 2)"},{"comment":"The statement that 'Sigma_2 was the one that averaged the best performance on both datasets' is contradicted by Table 1. The average F1 across the two datasets is approximately 0.9836 for Sigma_2, 0.9839 for Sigma_1, and 0.9874 for Sigma_3. This should be corrected, or the actual selection criterion should be stated, since the current wording misrepresents the reported numbers.","section":"Benchmark, Alphabet Trimming (Table 1)"}],"minor_comments":[{"comment":"Equation (2) is typeset ambiguously: the summation should be placed in the denominator as the total count over all symbols j in the alphabet, e.g., P(x_i | context) = (N(context, x_i) + alpha) / (sum_{j in Sigma} N(context, j) + alpha * |Sigma|). Please clarify the notation.","section":"Methodology, Equation (2)"},{"comment":"The sentence 'the last one is used to classify the target texts' is imprecise, because the alphabet Sigma also affects model construction through Equation (2). Both the reference models and the target classification use the same alphabet.","section":"Methodology, hyperparameters"},{"comment":"The paper does not state the exact number of characters used for the final reference models in Table 2, even though the influence of reference length is studied earlier. Reporting the final character counts per class would improve reproducibility.","section":"Results, Classifier testing"},{"comment":"There are minor language issues, such as 'substantial less computational power,' which should read 'substantially less computational power.' The sentence comparing BERT training cost also conflates pretraining a general language model with training a detector; this comparison should be reworded.","section":"Conclusions"}],"recommendation":"major_revision","confidential_remarks":"This is a straightforward application of finite-context models, and the novelty is incremental relative to the authors' prior work on FCM-based classification. The main concern is the gap between the general framing of AIDetx as a competitive detector and the narrow in-distribution evaluation. A revision with cross-domain and cross-generator experiments, proper prompt-aware splits, and at least one baseline comparison could make the paper acceptable; without those, I would not support publication of the general claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clean, honest application of finite-context-model compression classification to AI-text detection. The method is not new—the authors themselves cite the same framework for authorship attribution, language trees, and digit classification—but applying it to machine-generated text is a legitimate new application, and the paper does it carefully. The F1 scores are high (0.9752 on HC3, 0.9963 on AI-human-text), the ablations on reference length and target length are sensible, and the CPU-only efficiency is real and useful. Code is public, and the parameter search is described clearly enough to follow.\n\nThe main soft spot is exactly what the stress-test note flags: the reported accuracy comes only from test splits of the same two datasets used to train the reference models, and both datasets are ChatGPT-era Q&A-style text. The classifier is therefore likely exploiting n-gram regularities specific to these generators and this genre. The paper does concede in the conclusions that performance may vary on other datasets, but the abstract and introduction frame AIDetx as a general, competitive method. That mismatch matters. No cross-dataset, cross-generator, or cross-domain evaluation is provided, so the scope of the central claim is not established.\n\nA second soft spot is the absence of any baseline comparisons on the same data. Saying AIDetx is more efficient than LLM-based detectors is fine, but comparing against BERT's training cost is not a fair runtime comparison. I would want to see AIDetx next to a fine-tuned or zero-shot detector on the HC3 test set. A third, smaller issue is preprocessing: \"removing duplicates and short samples, and balancing character counts\" is vague, and the public repo has no commit hash, which makes exact reproduction harder than it should be.\n\nOn the positive side, the paper does not oversell its internal results: the F1 numbers are computed on held-out test sets after validation-based hyperparameter selection, and the target-length experiments give a real sense of when the method works. The citation pattern is honest, and the reliance on the authors' own FCM work is legitimate given that the method builds directly on it.\n\nWho is this for? People who need a lightweight, interpretable, no-GPU detector on short-to-medium texts and are willing to accept that it may not transfer across domains without retraining. It deserves a serious referee, but the referee should ask for cross-domain/cross-generator evaluation, proper baselines, and exact preprocessing details. I would not desk-reject it.","headline":"AIDetx is a credible compression-based detector with strong in-distribution F1, but the general claim needs cross-domain and baseline evidence before I would trust it.","tokens_in":6666,"tokens_out":1922,"would_cite":false,"duration_ms":20564,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A compression trick tells human-written from AI-generated text at over 97% F1.","keywords":["AI-generated text detection","machine-generated text","finite-context models","text classification","data compression","Markov models","cross-entropy","HC3 dataset"],"falsifier":"Retrain AIDetx on HC3 only, then run it on a held-out collection of, say, news articles or technical abstracts written by humans and by several recent large language models; if accuracy on that collection drops to near chance, the central transfer claim is refuted.","tokens_in":5618,"feed_emoji":"🤖","tokens_out":4699,"duration_ms":39313,"temperature":0.7,"pith_summary":"This paper claims that a simple compression-based classifier can reliably tell human-written from AI-generated text. It builds two finite-context models, one trained on human reference texts and one on AI reference texts, and labels a new document by whichever model compresses it into fewer bits. On the HC3 benchmark the method reaches an F1 score above 0.97, and on the AI-human-text benchmark above 0.99, while running on a single CPU core with no GPU. The authors argue this makes automated text detection cheaper, faster, and more interpretable than deep-learning classifiers, which need large language models and substantial training resources.","feed_headline":"Compression tells human text from AI text at 97–99% F1","feed_subtitle":"By asking which reference file shrinks the text more, AIDetx detects AI writing on a CPU alone.","key_machinery":"The load-bearing object is a finite-context model (FCM), a Markov model that estimates the probability of each next symbol given the previous $k$ symbols. Equation (2) gives a smoothed count estimate $P(x_i \\mid x_{i-1},\\dots,x_{i-k}) = (N(\\text{context},x_i)+\\alpha)/(\\sum_j N(\\text{context},j)+\\alpha|\\Sigma|)$, and Equation (1) sums $-\\log_2 P$ over the text to get the bits required. With $k=8$, $\\alpha=0.5$, and an alphabet reduced to digits, letters, and spaces, the two class models are built from reference texts of up to 7 million characters; classifying a target text is then just comparing the two bit totals and choosing the smaller.","core_discovery":"The paper's central discovery is that the cross-entropy between a candidate text and two class-specific finite-context models acts as a high-accuracy discriminator between human and machine text. Specifically, for an order-8 Markov model with smoothing factor $\\alpha=0.5$ over a trimmed 37-character alphabet, the number of bits needed to encode the text under each class model is computed by Equation (1), with symbol probabilities given by Equation (2). The text is assigned to the class whose model requires fewer bits. On held-out test data this decision rule reaches an accuracy of 0.9757 and an F1 score of 0.9752 on HC3, and an accuracy of 0.9963 and an F1 score of 0.9963 on AI-human-text. The authors present this as evidence that recognisable statistical differences between human and AI prose can be captured by a small Markov model without any neural network.","pith_inferences":["Beyond the paper, the same rule could be trained per genre or per author, turning compression distance into a lightweight stylometric tool.","A likely stress test is adversarial AI text explicitly designed to match human perplexity; the fixed-order Markov model may be more vulnerable than neural detectors to such manipulation.","Since the classifier produces bit counts, it can be combined with a threshold to flag uncertain texts for human review, something the paper does not discuss.","The dependence on reference length suggests diminishing returns past a few million characters, so practitioners could estimate a data budget for their own domain."],"forward_implications":["AIDetx can classify text on commodity hardware: model creation for HC3 takes 22 seconds and inference processes roughly 3 million characters per second on one CPU core.","Detection no longer requires training or loading a large language model; a two-model comparison gives a transparent, reproducible decision.","Performance grows with the amount of reference text and with target length, so users can trade reference size against accuracy in low-resource settings.","The method can be applied to any two text collections, not just human-versus-AI, suggesting the same compressor-based decision rule could serve other binary text-classification tasks.","Being single-threaded, the method can be sped up further by parallelisation, cutting inference time below the reported rates."],"supporting_citations":[{"why":"Provides the HC3 dataset used for training and evaluation.","marker":"[13]"},{"why":"Establishes compression methods as viable for authorship classification, the conceptual basis for AIDetx.","marker":"[5]"},{"why":"Demonstrates that relative compression can classify texts, the direct precursor of the two-model bit-comparison rule.","marker":"[6]"},{"why":"Applies finite-context models to a classification task, the specific mechanism AIDetx adopts.","marker":"[7]"},{"why":"The BERT baseline that motivates the paper's efficiency and interpretability comparison.","marker":"[14]"},{"why":"Supplies the finite-context model formalism used in Equations (1) and (2).","marker":"[12]"}],"fun_headline_variants":["Compression tells human from AI text at up to 99% F1","AI text detector packs a CPU-only compression trick","Finite-context models spot AI writing with 99% F1","No GPUs needed: compression classifies AI text at 97-99% F1","AIDetx: compression-based AI text detection hits 99% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the statistical patterns encoded in the two reference texts transfer to texts outside the training domains; if a new corpus of human and AI writing has different distributional features, the bit-count comparison will not separate the classes.","fun_headline_variants_meta":{"raw":{"variants":["Compression tells human from AI text at up to 99% F1","AI text detector packs a CPU-only compression trick","Finite-context models spot AI writing with 99% F1","No GPUs needed: compression classifies AI text at 97-99% F1","AIDetx: compression-based AI text detection hits 99% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000762,"raw_usage":{"total_tokens":3359,"prompt_tokens":899,"completion_tokens":2460,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":2362}},"tokens_in":515,"tokens_out":2460,"duration_ms":15451,"temperature":1.0,"reasoning_tokens":2362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:42:30.437927+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain AIDetx on HC3 only, then run it on a held-out collection of, say, news articles or technical abstracts written by humans and by several recent large language models; if accuracy on that collection drops to near chance, the central transfer claim is refuted.","supporting_citations":[{"cited_title":"How close is chatgpt to human experts? comparison corpus, evaluation, and detection,","cited_arxiv_id":null,"evidence_quote":"Provides the HC3 dataset used for training and evaluation."},{"cited_title":"Using literal and grammatical statistics for authorship attribution,","cited_arxiv_id":null,"evidence_quote":"Establishes compression methods as viable for authorship classification, the conceptual basis for AIDetx."},{"cited_title":"Language trees and zip- ping,","cited_arxiv_id":null,"evidence_quote":"Demonstrates that relative compression can classify texts, the direct precursor of the two-model bit-comparison rule."},{"cited_title":"An Application of Data Compression Models to Handwritten Digit Classification,","cited_arxiv_id":null,"evidence_quote":"Applies finite-context models to a classification task, the specific mechanism AIDetx adopts."},{"cited_title":"Bert: Pre- training of deep bidirectional transformers for language understanding,","cited_arxiv_id":null,"evidence_quote":"The BERT baseline that motivates the paper's efficiency and interpretability comparison."},{"cited_title":"Finite-context models for DNA coding,","cited_arxiv_id":null,"evidence_quote":"Supplies the finite-context model formalism used in Equations (1) and (2)."}],"review_version":1}