Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Can You Detect the Difference?

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Diffusion-based language models such as LLaDA produce text whose perplexity and burstiness match human writing so closely that detectors tuned for autoregressive outputs, such as DetectGPT and GPTZero, miss them at high false-negative…

desk verdict One solid rephrase-task observation, an unmeasured detector-evasion claim, and a correctable framing. read the letter →

arxiv 2507.10475 v1 pith:UHLMKKGA submitted 2025-07-14 cs.CL cs.AI

İsmail Tar{i}m , Aytuğ Onan This is my paper · ORCID
classification cs.CLcs.AI
keywords diffusionlanguagemodelsLLaDAMAAItextdetectionstylometryperplexityburstinessfalse-negativerates
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether AI-text detectors, which were built and calibrated for autoregressive models that compose text token by token, still work on diffusion language models that reconstruct text by filling in masked tokens in parallel. The experiments generate 2,000 samples from LLaDA (diffusion) and LLaMA (autoregressive) on rephrasing and title-conditional abstract generation, then measure perplexity, burstiness, lexical diversity, readability, and BLEU/ROUGE as proxies for what detection tools actually see. The paper's central claim is that in rephrasing, LLaDA's perplexity (44.62) is statistically indistinguishable from human abstracts (43.03) and its burstiness stays inside the human range, so detectors keyed on low perplexity and uniform sentence-length variation flag LLaMA but let LLaDA through. A sympathetic reader would take away that generation architecture, not output quality, now determines detectability, and that single-metric heuristics are no longer sufficient for catching machine text.

What carries the argument

The argument runs on the contrast between two generation mechanisms. Autoregressive models factor text as a product of conditional probabilities $P(x_t \mid x_{1:t-1})$ under a causal mask and cannot revise emitted tokens; diffusion models such as LLaDA start from a fully masked sequence and iteratively predict all masked positions in parallel, re-masking low-confidence tokens over 128 steps until the text is recovered. The measured quantities are GPT-2 perplexity, the average inverse likelihood a reference autoregressive model assigns to a text, and burstiness, defined operationally as the coefficient of variation of sentence lengths. Because diffusion never follows the locally smoothest left-to-right probability path, its outputs avoid the low-perplexity, low-burstiness profile that perplexity/burstiness classifiers treat as machine evidence.

What would settle it

Run DetectGPT and GPTZero, or threshold classifiers built from their published perplexity and burstiness criteria, directly on the paper's 2,000 LLaDA and LLaMA samples and compare false-negative rates across the two model families. If LLaDA outputs are flagged at rates close to those for the human abstracts, the claimed evasiveness does not occur; if LLaDA is flagged far less often than LLaMA while the proxy metrics overlap, the claim is confirmed.

Watch

Extended reading notes

Core claim

The paper's discovery is that diffusion generation erases the statistical signature that detectors have been trained to look for. Autoregressive output is conspicuous because the generating model finds its own text too predictable — in the rephrase task, LLaMA reaches a GPT-2 perplexity of $18.37$ against $43.03$ for the human abstracts — while LLaDA's deterministic outputs land at $44.62$, statistically indistinguishable from the humans, with burstiness near the low end of the human range. In the abstract-generation task both models fall below human perplexity, but LLaDA's burstiness ($0.184$) again sits closer to the human profile ($0.334$) than LLaMA's ($0.307$). The authors conclude that perplexity and burstiness alone cannot separate diffusion output from human writing, that AR-focused detectors such as DetectGPT and GPTZero therefore produce high false-negative rates on LLaDA, and that detection must move to diffusion-specific signatures such as reverse-process artifacts.

Load-bearing premise

The load-bearing premise is that GPT-2 perplexity and sentence-length burstiness faithfully stand in for what DetectGPT and GPTZero actually decide — the detectors are never run on the outputs — and that deterministic temperature-0 generation is a fair test case; if either gives way, the high-false-negative conclusion does not follow.

Editorial extensions

If this is right

  • Perplexity-threshold detectors will keep missing LLaDA outputs even under deterministic decoding, because the diffusion samples sit inside the human perplexity range rather than below it.
  • Deploying GPTZero-style detectors against diffusion generators means accepting high false-negative rates; the paper's numbers imply these tools cannot serve as a safety net for LLaDA text.
  • For applications that need surface-faithful, human-like paraphrase, diffusion models look preferable, while autoregressive models remain attractive when lexical novelty is the goal.
  • Detectors that want to catch diffusion output should be built on signatures of the denoising process rather than token-level probability statistics.
  • Any single-metric detection rule, such as a fixed perplexity cutoff, is insufficient once both families of generators are in play.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would expect the evasion to be task-dependent rather than an intrinsic property of diffusion: in the paper's own generation task, LLaDA's perplexity (17.26) is far below the human range, so a detector would catch it there; the human-like perplexity in rephrasing may owe partly to the source text constraining the output.
  • A direct way to test the paper's proxy assumption would be to run DetectGPT and GPTZero verbatim on the same 2,000 outputs; if their false-negative rates fail to track the metric overlap, the conclusion would need re-grounding.
  • If diffusion text is truly this close to human statistics, post-hoc detection is a losing race, and provenance tools such as watermarking become the more viable defense — a direction the paper itself flags.
  • The temperature-0 setting probably cuts both ways: higher temperatures would add randomness to LLaMA and make it less detectable, but could also push LLaDA out of the exactly-human perplexity region reported here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents a stylometric comparison of human-written ArXiv abstracts with outputs from an autoregressive model (LLaMA 7B) and a diffusion model (LLaDA 8B) on two tasks: rephrasing and title-conditional abstract generation. The authors report that LLaDA outputs are 'nearly indistinguishable' from human text in perplexity and burstiness and, on this basis, conclude that AR-focused detectors (DetectGPT, GPTZero) will exhibit high false-negative rates on diffusion outputs. No detector is actually run; the metrics are introduced as proxies in Section 1. The paper also introduces a new dataset of abstracts and makes code publicly available.

Significance. If the central claim held, the paper would provide an important caution about detector blind spots for diffusion LLMs, and the public dataset would be a useful resource. The paper usefully documents a stylometric comparison at zero temperature and provides reproducible code and data. However, the claim of detector vulnerability is unsupported by direct measurements and is contradicted by the paper's own generation-task results; the significance is therefore limited to a preliminary metric comparison that does not substantiate the headline conclusion.

major comments (4)
  1. [§1, §6.1, Eqs. (1)/(7), Table 1] The abstract and §7 claim that LLaDA outputs 'result in high false-negative rates when processed by AR-focused detectors such as DetectGPT and GPTZero,' but no detector is run. Section 1 states that the metrics are 'proxies for detector performance,' yet the paper never validates that mapping. DetectGPT's score is log-probability curvature under perturbations (Eq. 1/7), not raw perplexity, so human-like GPT-2 perplexity does not imply a small DetectGPT score. Likewise, GPTZero's burstiness feature uses variance of sentence-level perplexities (Eq. 5), but Table 1 and the results use coefficient of variation of sentence lengths. The proxy evidence therefore cannot license the detector false-negative conclusion.
  2. [Table 2, §5.2, Abstract] The claim that LLaDA is 'nearly indistinguishable' from human text in perplexity and burstiness is only supported in the rephrase perplexity condition (44.62 vs 43.03, n.s.). In the generation task, Table 2 shows LLaDA perplexity 17.26 vs human 43.03 (p<0.001) and burstiness 0.184 vs human 0.334 (p<0.001), so the two distributions are significantly separated. Furthermore, §5.2 states LLaDA's burstiness 'aligns even more closely' with human than LLaMA's, but LLaMA's 0.307 is numerically closer to the human 0.334 than LLaDA's 0.184; this arithmetic error reverses the stated conclusion.
  3. [Eq. (5), Table 1] The burstiness metric is defined in Eq. (5) as the variance of sentence-level perplexities, but Table 1 defines burstiness as the coefficient of variation of sentence lengths (std/mean). These are different quantities; the latter does not correspond to GPTZero's burstiness feature. The paper therefore does not measure the same burstiness construct that the detectors use, and overlap on the measured quantity does not inform GPTZero's behavior.
  4. [§6.2, Abstract, §7] The paper acknowledges in §6.2 that zero-temperature decoding 'eliminates stochastic variation' and makes outputs 'fully deterministic—and therefore unusually predictable—compared with texts sampled at higher temperatures.' Despite this, the abstract and §7 state the 'nearly indistinguishable' and high-false-negative conclusions without qualification. The conclusions outrun the experimental setting, and the acknowledged limitation directly undercuts the generalization to realistic generation configurations.
minor comments (4)
  1. [Abstract, §1, §4.1] The abstract and §1 report a dataset of 2,000 samples, but §4.1 computes 500×3×2 = 3,000 examples; reconcile the count.
  2. [Table 2] Table 2 uses the abbreviation 'LLMA' for LLaMA; use consistent naming throughout the paper.
  3. [§5.2, §2.3] In §5.2, 'all (p <0.001)' contains a stray parenthesis, and the sentence beginning 'AIhumanizers ... [7]. evaluated' in §2.3 is grammatically broken and should be rewritten.
  4. [References] The reference list contains formatting inconsistencies (e.g., the DetectGPT entry and the LLaDA-related entries) that should be checked against the original sources.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the metric comparisons are independent measurements; the detector-failure inference is an unvalidated proxy mapping, not an equation-level reduction.

full rationale

The paper's quantitative chain is: (i) generate LLaDA and LLaMA samples for two tasks with standard public models; (ii) compute perplexity under GPT-2, coefficient of variation of sentence lengths, TTR, LanguageTool error rates, SBERT coherence, BLEU/ROUGE; (iii) compare against human abstracts via Mann-Whitney U tests; (iv) infer detector evasion. Steps (i)-(iii) are self-contained empirical measurements using external, fixed tools; no parameter is fitted from the data and no prediction is defined in terms of the conclusion. The inference in (iv) is an external assumption that these proxies predict DetectGPT/GPTZero behavior, explicitly acknowledged in Section 1: 'Rather than running existing detectors directly, we extracted key stylometric and linguistic metrics ... as proxies for detector performance and used them to reflect on how distinguishable the two families of outputs are in practice.' That mapping is not established and is arguably undermined by the paper's own equations: Eq. (5) defines burstiness as variance of sentence-level perplexities, while Table 1 operationalizes it as coefficient of variation of sentence lengths, and DetectGPT's Eq. (7) uses log-probability curvature, not raw GPT-2 perplexity. These are correctness and validity defects, not circular reductions. The limitation passage in Section 6.2 concedes that all outputs were generated at temperature 0, making them deterministic and 'unusually predictable'; this undercuts generalization but is not circularity. No load-bearing self-citation or imported uniqueness theorem appears, and the LLaDA architecture citations are background only. Therefore the derivation chain is not equivalent to its inputs by construction, and the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted constants and no invented entities. Its conclusions rest on four domain assumptions, the most fragile being that proxy stylometric metrics faithfully represent the behavior of deployed detectors.

assumptions (4)
  • domain assumption GPT-2 perplexity and sentence-length CV can stand in for the outputs of real detectors such as DetectGPT and GPTZero.
    Section 4.2 defines these as proxies, and Section 6.1 converts metric overlaps into detector failure without running either detector.
  • domain assumption Zero-temperature outputs (T=0) are a fair basis for detectability conclusions.
    Both models use temperature 0 in Section 4.1; Section 6.2 admits this removes stochasticity and makes outputs unusually predictable, which may distort both the human-like and the detectable readings.
  • domain assumption LLaMA 7B and LLaDA-8B represent the autoregressive and diffusion model families generally.
    Only two models are tested, but Section 7 treats the results as applying to family-level differences between AR and diffusion generation.
  • domain assumption The 500 sampled arXiv abstracts are representative human text for both rephrasing and abstract generation.
    Section 4.1 samples 500 papers from one Kaggle dataset, and Section 6.2 limits the scope to English abstracts, which constrains generalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can You Detect the Difference?." pith.science (2026). https://pith.science/paper/UHLMKKGA

@misc{pith2026250710475,
  author       = {Pith},
  title        = {Pith review of: Can You Detect the Difference?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UHLMKKGA}},
  note         = {Machine review of arXiv:2507.10475}
}
read the original abstract

The rapid advancement of large language models (LLMs) has raised concerns about reliably detecting AI-generated text. Stylometric metrics work well on autoregressive (AR) outputs, but their effectiveness on diffusion-based models is unknown. We present the first systematic comparison of diffusion-generated text (LLaDA) and AR-generated text (LLaMA) using 2 000 samples. Perplexity, burstiness, lexical diversity, readability, and BLEU/ROUGE scores show that LLaDA closely mimics human text in perplexity and burstiness, yielding high false-negative rates for AR-oriented detectors. LLaMA shows much lower perplexity but reduced lexical fidelity. Relying on any single metric fails to separate diffusion outputs from human writing. We highlight the need for diffusion-aware detectors and outline directions such as hybrid models, diffusion-specific stylometric signatures, and robust watermarking.

Figures

Figures reproduced from arXiv: 2507.10475 by the authors.

Figure 1
Figure 1. Conceptual diagram of LLaDA’s pre-training, supervised fine-tuning (SFT), and inference (sampling) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Stylometric Metrics by Model for Rephrase Task [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Stylometric Metrics by Model for Generation Task [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Discrete Diffusion Models: A Unified Framework from Tokenization to Generation

    cs.LG 2026-07 unverdicted novelty 4.0 of 10

    Discrete diffusion models are re-framed as instances of a tokenization-centric, four-component design space (corruption, denoiser, objective, sampler) in a broad survey with no new experimental or theoretical results.

Reference graph

Works this paper leans on

20 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Detectgpt: Zero-shot machine-generated text detection.arXiv preprint arXiv:2301.11305, 2023

    Michael Mitchell et al. Detectgpt: Zero-shot machine-generated text detection.arXiv preprint arXiv:2301.11305, 2023

    arXiv 2023

  2. [2]

    Gptzero: Ai content detector.https://gptzero.me/, 2023

    work page 2023

  3. [3]

    Gptzero: 15130070230551-how-do-i-interpret-burstiness-or-perplexity. 2023

    work page 2023

  4. [4]

    Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    Shang Nie et al. Large language diffusion models.arXiv preprint arXiv:2502.09992, 2025

    arXiv 2025

  5. [5]

    Ml-gsai/llada.https://github.com/ML-GSAI/LLaDA, 2025

    Shang Nie et al. Ml-gsai/llada.https://github.com/ML-GSAI/LLaDA, 2025

    work page 2025

  6. [6]

    Gsai-ml / llada.https://huggingface.co/GSAI-ML, 2025

    Shang Nie et al. Gsai-ml / llada.https://huggingface.co/GSAI-ML, 2025

    work page 2025

  7. [7]

    DAMAGE: Detecting adversarially modified ai generated text

    Elyas Masrour, Bradley Emi, and Max Spero. DAMAGE: Detecting adversarially modified ai generated text. arXiv preprint arXiv:2501.03437, 2025

    arXiv 2025

  8. [8]

    Best AI content detectors of 2025 (free & paid)

    Eram Shaikh. Best AI content detectors of 2025 (free & paid). https://www.demandsage.com/ ai-content-detectors/, 2024. Accessed: 2025-06-24

    work page 2025

Show all 20 references
  1. [9]

    A watermark for large language models

    John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. InProceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 17061–17084, 2023

    2023

  2. [10]

    A robust semantics- based watermark for large language model against paraphrasing

    Jie Ren, Han Xu, Yiding Liu, Yingqian Cui, Shuaiqiang Wang, Dawei Yin, and Jiliang Tang. A robust semantics- based watermark for large language model against paraphrasing. InFindings of the Association for Computational Linguistics: NAACL 2024, pages 613–625, Jun 2024

    2024

  3. [11]

    In-context watermarks for large language models.arXiv preprint arXiv:2505.16934, 2025

    Yepeng Liu, Xuandong Zhao, Christopher Kruegel, Dawn Song, and Yuheng Bu. In-context watermarks for large language models.arXiv preprint arXiv:2505.16934, 2025

    2025 arXiv

  4. [12]

    Scalable watermarking for identifying large language model outputs.Nature, 634(8035):818–823, 2024

    Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, Jamie Hayes, Nidhi Vyas, Majd AlMerey, Jonah Brown-Cohen, Rudy Bunel, Borja Balle, Taylan Cemgil, Zahra Ahmed, Kit...

    2024

  5. [13]

    Improved unbiased watermark for large language models

    Ruibo Chen, Yihan Wu, Junfeng Guo, and Heng Huang. Improved unbiased watermark for large language models. arXiv preprint arXiv:2502.11268, 2025

    2025 arXiv

  6. [14]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020. Presented at NeurIPS 2020

    2005 arXiv

  7. [15]

    LanguageTool: Ai-based grammar checker and paraphrasing tool.https://languagetool.org/, 2025

    2025

  8. [16]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. InProceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),...

    2019

  9. [17]

    kaggle/arxiv-paper-abstracts

    spsayakpaul. kaggle/arxiv-paper-abstracts. https://www.kaggle.com/datasets/spsayakpaul/ arxiv-paper-abstracts

  10. [18]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics, 2002

    2002

  11. [19]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. InText Summarization Branches Out, pages 74–81, Barcelona, Spain, Jul 2004. Association for Computational Linguistics

    2004

  12. [20]

    MIT Press, Cambridge, MA, 2016

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville.Deep Learning. MIT Press, Cambridge, MA, 2016. Chapter 6 covers softmax and temperature scaling. 11

    2016

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.