{"id":"2a5827b7-a1e4-4a76-b3b6-288ebb1132c3","arxiv_id":"2412.03025","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Human-written texts show higher linguistic variability, richer vocabulary and emotional content, and lower syntactic depth than texts from five LLMs.","lead":"This study compares human-written and AI-generated texts across four domains using 250 linguistic features, finding that human texts are more variable, more emotional, and less syntactically complex. It also shows a logistic classifier using these explainable features can tell the two apart with around 87% accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Length confound undermines central variability claim: Table 3's PCA centroid distances are computed on raw count features despite a 1.7-3.8x human/LLM token gap.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: systematic length differences between human and machine texts are not controlled in the feature analysis. This is the most serious threat because it directly targets the paper's headline contribution, namely that humans are more variable in linguistic features than LLMs. The concern is concrete and checkable: Table 3's variability numbers can be recomputed on length-normalized or length-matched data. I do not see a more fundamental issue: the dataset is public, the features are explainable, and the syntactic-depth and emotion analyses are at least plausibly length-independent. The reported accuracy inconsistency (0.82 in the Introduction vs. 0.87 in Results/Appendix) and the absence of released code are real weaknesses but secondary to the length confound. The verdict should remain CONDITIONAL because the central claim is not yet supported as stated, but the paper could be rescued by a normalization or matching analysis.","tokens_in":11393,"tokens_out":3179,"duration_ms":36716,"concrete_test":"Recompute the PCA and Table 3 using length-normalized features: convert all raw count features (words, characters, spaces, unique words, etc.) to rates per 1,000 tokens, replace raw unique-word counts with MATTR or root type-token ratio, and restrict each domain to a common length band (e.g., 150-400 tokens) with equal numbers per source. If the human/LLM centroid-distance ratios in Table 3 collapse toward 1, the variability claim is a length artifact; if they remain large in the normalized and length-matched analysis, the claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that human-written texts are intrinsically more variable than LLM texts. The evidence for this is Table 3, which measures variability as the distance of each document from its source centroid in a two-dimensional PCA space. But the PCA is run on all 250 LFTK features, many of which are raw counts (words, characters, spaces, unique words, etc.), and Table 1 reports that humans average 706 tokens per document while the five LLMs average 187-412. Because PCA without explicit standardization is dominated by high-variance, high-magnitude count features, and because longer documents naturally have larger raw counts and larger count variance, the centroid distances in Table 3 (e.g., human Wikipedia 296.75 vs. ChatGPT 4.72) may largely encode variation in document length, not intrinsic linguistic variability. The paper acknowledges the length difference in Section 5 but never controls for it in the variability analysis. The claim that humans are 'extremely varied' whenever constraints are flexible is therefore not established until length-normalized features are analyzed. This is an internal consistency issue, not a disagreement with consensus: the paper's own Table 1 creates the confound, and the variability analysis does not remove it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript analyzes human-written and LLM-generated texts from the M4 corpus (SemEval 2024 Task 8 Subtask B), profiling each document with 250 linguistic features from LFTK plus measures of syntactic depth, semantic distance, and emotional content. The authors apply PCA and report that human texts show considerably higher variability than LLM texts, especially in less constrained domains (Wikipedia, Reddit), and that humans produce shallower syntax, more unique words, greater semantic distance, and more negative emotion. A logistic regression on the linguistic features is reported to distinguish human from machine text with high accuracy.","tokens_in":11620,"tokens_out":7175,"duration_ms":66917,"significance":"The paper addresses a timely question—whether interpretable linguistic features can reveal systematic differences between human and machine text. Its strengths are the use of a large public benchmark, the breadth of features (surface, syntactic, semantic, emotional), and domain-stratified analyses. If the variability claim survives length normalization and inferential testing, it would be a useful empirical contribution, with implications for explainable detection of machine-generated text. At present the core evidence for the variability claim is confounded by document length, and the semantic findings contain internal contradictions; these are fixable but require a substantive revision.","major_comments":[{"comment":"The central variability claim rests on Table 3, where variability is measured as distance from each source's centroid in a two-dimensional PCA of all 250 features. Table 1 reports average token counts of 706 for humans versus 187–412 for the five LLMs, and many LFTK features are raw counts (words, characters, spaces, unique words). The paper does not state that features were standardized before PCA, and no length normalization is applied in this analysis. Longer documents therefore contribute larger count values and larger count variance, so the PCA centroid distances (e.g., human Wikipedia 296.75 vs. ChatGPT 4.72) may largely encode length differences rather than intrinsic linguistic variability. I request a reanalysis with length-normalized features (e.g., per 1,000 tokens or residualized on length) and/or feature standardization, with a demonstration that the human-vs-LLM variability gap persists.","section":"§5, Table 3 vs Table 1"},{"comment":"The variability estimates in Table 3 are reported as single numbers without confidence intervals, bootstraps, or significance tests. Because the claim is specifically that human variability is higher, the authors should compare the full distributions of per-document distances from the centroid (e.g., via permutation tests or bootstrap CIs) rather than only the mean distance. Without such tests, differences like the Wikipedia human/ChatGPT gap cannot be distinguished from sampling variation or outlier influence.","section":"§5, Table 3"},{"comment":"The vocabulary and semantic-distance findings are also length-sensitive. Unique-word counts are raw counts, and type-token ratio mechanically declines with text length for natural language, so the statement in the introduction that humans show a lower TTR while having a 'richer vocabulary' is an artifact of comparing longer human documents with shorter LLM documents unless a length-controlled vocabulary richness measure (e.g., MATTR or moving-average TTR) is used. Likewise, semantic distance from pairwise sentence comparisons depends on the number of sentences per document, which varies with length. The analyses should be repeated on length-matched subsamples or with length-normalized features before claiming that humans have richer vocabulary and higher semantic content.","section":"§5, Figures 6–8 and discussion of unique words"},{"comment":"The paper's statements about semantic consistency are mutually contradictory. Section 5 and Figure 2 show that humans have higher average semantic distance (i.e., lower pairwise similarity), and the introduction says humans 'tend to employ less similar semantic content.' Yet Section 6 concludes that 'HWT contain richer semantic content and show greater consistency in meaning than LLMs.' The authors should clarify whether higher semantic distance is interpreted as less consistency or as greater content richness, and align the text across sections.","section":"§5 vs §6"}],"minor_comments":[{"comment":"The accuracy is reported as 0.82 in the abstract and introduction but as 87.15% (0.87) in Section 5 and Table 6; these numbers should be reconciled or the settings for each should be stated.","section":"Abstract/Introduction vs §5"},{"comment":"The phrase 'achieving a prediction accuracy of 93%' after discussing the most prominent features is unclear; Table 6 reports human-class F1 of 0.93, so the text should specify that this is the human-class F1 (or recall/precision), not overall accuracy.","section":"§5"},{"comment":"Please state explicitly whether the 250 features were standardized or scaled before PCA, and report the total variance explained by the two components shown (18.82% + 8.92% = 27.74%), so readers can judge how much variability the centroid-distance measure summarizes.","section":"§4.2"},{"comment":"The terms 'semantic distance' and 'semantic similarity' are used interchangeably; define distance = 1 − cosine similarity in the methodology and use the terms consistently.","section":"§4.1"},{"comment":"The code-release sentence ('All codes will be released in the camera-ready version due to anonymity reasons') is confusing in an arXiv preprint with visible author names; rephrase to a standard data/code availability statement.","section":"§8–9"},{"comment":"The description of the training/test split (random selection of 5,000 elements per model) should clarify whether the test set is balanced across the six classes and whether the classifier is trained on all domains combined or per domain.","section":"§10.1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the length confound in the PCA variability analysis; I recommend requesting the reanalysis as a condition of acceptance. The paper is otherwise a solid descriptive study, and the issues are addressable with normalization and inferential tests rather than requiring rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid descriptive study that overreaches in its central claim. The headline finding—human texts are intrinsically more variable than LLM texts—is probably directionally correct, but the evidence in Table 3 does not support it because the PCA is run on raw count features while humans write 1.7–3.8× longer documents.\n\nWhat's genuinely useful: the paper works on the M4/SemEval-2024 data with a wide feature set (LFTK + syntactic depth, semantic distance, emotion), and it breaks results down by domain, showing that the human–LLM gap shrinks in constrained registers like arXiv. That's a real value-add over the prior results it cites (Guo et al., 2023; Wang et al., 2024b), which already reported higher human variability, richer vocabulary, and stronger negative emotion. The limitations and ethics sections are honest and thoughtful.\n\nWhere it falls short: the variability analysis is the load-bearing part, and it's under-built. Table 1 shows humans average 706 tokens vs 187–412 for LLMs, and many of the 250 features are raw counts. PCA without standardization will be dominated by high-magnitude count features, and the centroid distances in Table 3 are therefore likely to encode length variation rather than intrinsic linguistic variability. The paper notes the length difference but never normalizes or controls for it. Relatedly, the variability numbers come with no confidence intervals or tests. The accuracy figure also floats: 0.82 in the intro, 0.87 in the results/appendix. No code is released, so a referee can't check the exact feature pipeline.\n\nNone of this kills the paper. The descriptive patterns for vocabulary, emotion, and syntactic depth are consistent with prior work, and the Dunn/Kruskal-Wallis tests at least support group differences. But the \"humans are extremely varied when constraints are flexible\" claim should be read as a hypothesis, not a result, until length is handled.\n\nMy recommendation: send it to peer review, but request major revision. The analysis needs length-normalized features, significance tests for variability, and a single consistent accuracy figure. It's a competent paper that would benefit from a firmer statistical core.","headline":"Likely true but not established: length confounding undermines the headline variability claim, though the domain breakdown is a useful extension.","tokens_in":12121,"tokens_out":2083,"would_cite":false,"duration_ms":20034,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that human-written and machine-generated texts differ most sharply in variability: human texts vary far more in their linguistic profiles, and the gap grows in genres with flexible stylistic rules.","keywords":["human-written text","machine-generated text","linguistic features","text variability","syntactic depth","semantic distance","emotional content","authorship attribution"],"falsifier":"Repeat the same feature computation after truncating all human documents to the median LLM length or after regressing each feature on token count and analyzing the residuals; if the human-machine gaps in variability, semantic distance, and negative emotion largely disappear, the paper's central claim is refuted, while if they persist at similar magnitude, the claim survives this direct challenge.","tokens_in":11205,"feed_emoji":"✍️","tokens_out":8523,"duration_ms":72366,"temperature":0.7,"pith_summary":"Using 250 interpretable linguistic features on the aligned M4 corpus of 71,000+ English texts from four domains, this paper tries to show that human writing is not merely harder to imitate but systematically different in kind: humans are far more variable, write shallower sentences, use richer vocabulary, and express more emotion (especially negative emotion) than any of the five tested LLMs. The authors argue that this variability signature is strongest in genres with loose stylistic constraints, such as Wikipedia and Reddit, and weakest in tightly constrained scientific prose (arXiv). As evidence, they report that a logistic regression on the handcrafted features alone separates the six text sources with roughly 87% accuracy in the results table (82% stated in the abstract), and that the features driving classification differ between humans and machines. If correct, the paper's pith is that explainable, feature-based profiling can capture a stable human fingerprint in text.","feed_headline":"Human text shows more variability and emotion than AI text","feed_subtitle":"A 250-feature analysis of 71,000 texts finds the gap widens in casual genres, aiding explainable detection.","key_machinery":"The analytical engine is a 250-dimension linguistic profile: 247 handcrafted features from the LFTK toolkit (covering surface, lexico-semantics, discourse, and syntax), plus three added measures – average syntactic depth from a SpaCy dependency parse, semantic distance from pairwise cosine similarity of sentence embeddings (paraphrase-MiniLM-L6-v2), and emotional content from the NRC Emotion Intensity Lexicon across eight emotion categories. Variability is quantified as each text source's distance from its cluster centroid in a two-dimensional PCA of the full feature set, and a logistic regression on all 250 features is used for authorship classification. The machinery's job is to show that a small set of human-interpretable features, none of them black-box embeddings, can carry both explanation and prediction.","core_discovery":"On its own terms, the paper claims that human-written texts and LLM-generated texts differ not just in style but in statistical behavior. Humans exhibit considerably higher variability across nearly all 250 measured linguistic features, and this variability is domain-dependent: in Wikipedia and Reddit, where style constraints are relaxed, the human cluster in PCA space spreads far wider than the machine clusters (e.g., distance-from-centroid values of 296.8 and 50.7 for humans versus single digits for most LLMs), while in arXiv the gap nearly vanishes (4.7 vs. 1.6–9.7). Beyond variability, humans write shallower syntactic trees (lower average syntactic depth), show greater semantic distance between sentences while using richer vocabulary, and score higher on emotional intensity, particularly for negative emotions such as anger. The authors interpret these results as evidence that LLM outputs are homogenized and emotionally flattened, likely shaped by training and alignment, and that human text is more cognitively economical, meaningful per unit, and emotionally engaged.","pith_inferences":["Because human documents average about 706 tokens versus 187–412 for the five LLMs, and several features are raw unnormalized counts, a length-matched or residualized analysis is needed to test whether the variability gap is intrinsic to human style or partly a length artifact.","If variability is the load-bearing signal, detectors should measure dispersion within authors or genres rather than mean differences, a concrete design principle for future systems.","The finding that all five LLMs cluster tightly while humans scatter wide suggests a general 'machine style' of current training objectives; testing decoder-only models trained without reinforcement learning from human feedback would isolate the role of alignment in this homogenization.","The emotional flattening result implies a testable hypothesis: LLMs instructed to 'express anger' should still show lower anger intensity than humans in equivalent contexts, indicating a model-level bias rather than a prompt artifact."],"forward_implications":["The reported classifier reaches 87% accuracy in the in-domain, same-generator setting, so handcrafted, explainable features can support accurate authorship detection.","Detection will be easier in informal, unconstrained genres and harder in formal, rigid genres, since the human-machine gap shrinks where style constraints are tight.","The consistent machine profile across five different LLMs implies that model families may be identifiable by shared linguistic fingerprints, despite different architectures and training data.","If the suppression of negative emotion in LLM texts is replicated, machine text may be emotionally narrower by design, with implications for affective computing and content moderation.","Variability, rather than any single mean difference, may be the most robust signal for separating humans from machines."],"supporting_citations":[{"why":"Supplies the M4 dataset of aligned human and LLM texts across four domains and the same-generator, same-domain evaluation protocol.","marker":"Wang et al., 2024b"},{"why":"Provides the LFTK toolkit from which the 247 handcrafted linguistic features are drawn.","marker":"Lee and Lee, 2023"},{"why":"Gives the paraphrase-MiniLM-L6-v2 sentence transformer used to compute semantic distance via pairwise cosine similarity.","marker":"Reimers and Gurevych, 2019"},{"why":"Supplies the NRC Emotion Intensity Lexicon used to measure emotional content across eight emotion categories.","marker":"Mohammad and Turney, 2013"},{"why":"Provides prior evidence that LLM texts downplay negative emotions, which the paper's emotionality results align with.","marker":"Guo et al., 2023"},{"why":"Supports the interpretation that lower syntactic depth reflects lower cognitive demand in human writing.","marker":"Hagoort et al., 1999"}],"fun_headline_variants":["Humans vary more than AI text, especially in casual genres","AI text is homogenized, humans show richer emotion","Variability gap: humans vs. LLMs widens in casual writing","Human writers: more variability, emotion, less cognitive load","LLMs flatten style; human text varies more widely"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes human and machine texts are comparable even though human documents average about twice the token count of machine documents, and many features are raw, unnormalized counts; if the extra length of human text drives the variability and emotion gaps, the claim that humans are intrinsically more variable would collapse.","fun_headline_variants_meta":{"raw":{"variants":["Humans vary more than AI text, especially in casual genres","AI text is homogenized, humans show richer emotion","Variability gap: humans vs. LLMs widens in casual writing","Human writers: more variability, emotion, less cognitive load","LLMs flatten style; human text varies more widely"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000324,"raw_usage":{"total_tokens":1834,"prompt_tokens":974,"completion_tokens":860,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":777}},"tokens_in":590,"tokens_out":860,"duration_ms":6370,"temperature":1.0,"reasoning_tokens":777,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:49:59.924759+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the same feature computation after truncating all human documents to the median LLM length or after regressing each feature on token count and analyzing the residuals; if the human-machine gaps in variability, semantic distance, and negative emotion largely disappear, the paper's central claim is refuted, while if they persist at similar magnitude, the claim survives this direct challenge.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the interpretation that lower syntactic depth reflects lower cognitive demand in human writing."}],"review_version":1}