REVIEW 9 cited by
ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
AI generated content (AIGC) presents considerable challenge to educators around the world. Instructors need to be able to detect such text generated by large language models, either with the naked eye or with the help of some tools. There is also growing need to understand the lexical, syntactic and stylistic features of AIGC. To address these challenges in English language teaching, we first present ArguGPT, a balanced corpus of 4,038 argumentative essays generated by 7 GPT models in response to essay prompts from three sources: (1) in-class or homework exercises, (2) TOEFL and (3) GRE writing tasks. Machine-generated texts are paired with roughly equal number of human-written essays with three score levels matched in essay prompts. We then hire English instructors to distinguish machine essays from human ones. Results show that when first exposed to machine-generated essays, the instructors only have an accuracy of 61% in detecting them. But the number rises to 67% after one round of minimal self-training. Next, we perform linguistic analyses of these essays, which show that machines produce sentences with more complex syntactic structures while human essays tend to be lexically more complex. Finally, we test existing AIGC detectors and build our own detectors using SVMs and RoBERTa. Results suggest that a RoBERTa fine-tuned with the training set of ArguGPT achieves above 90% accuracy in both essay- and sentence-level classification. To the best of our knowledge, this is the first comprehensive analysis of argumentative essays produced by generative large language models. Machine-authored essays in ArguGPT and our models will be made publicly available at https://github.com/huhailinguist/ArguGPT
Forward citations
Cited by 9 Pith papers
-
Show Me How You Reason and I'll Tell You Who You Are: Reasoning Graphs for Robust LLM Authorship Attribution
Reasoning graphs extracted by an argument-mining pipeline let a GNN attribute LLM-generated essays to the correct model family with more robustness to paraphrasing/backtranslation and better cross-version generalizati...
-
Relativistic Quantum Thermal Machine: Harnessing Relativistic Effects to Surpass Carnot Efficiency
Relativistic motion of the reservoirs in a three-level maser is claimed to yield a generalized Carnot bound that allows efficiency above the ordinary Carnot limit.
-
RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patterns
A detector that projects a text's hidden neural activations onto a direction learned from human versus AI writing differences reports higher accuracy than baselines, including on unseen generators and attacked texts.
-
A General Method for Detecting Information Generated by Large Language Models
A new LLM-text detector built from twin memory networks and domain-generalization losses outperforms prior detectors on held-out LLMs and domains in the paper's benchmark.
-
Large Language Models as 'Hidden Persuaders': Fake Product Reviews are Indistinguishable to Humans and Machines
Human readers and seven LLMs both fail to reliably distinguish real Amazon product reviews from ChatGPT-o1 generated fakes, performing near chance accuracy.
-
Measuring Human Involvement in AI-Generated Text: A Case Study on Academic Writing
Human involvement in AI-generated academic text can be estimated continuously by training a RoBERTa regressor on BERTScore-derived labels, outperforming binary detectors on a new synthetic dataset.
-
Stylometry recognizes human and LLM-generated texts in short samples
Stylometric features and tree-based classifiers separate human-written Wikipedia summaries from LLM-generated texts with high cross-validated accuracy on a new seven-class benchmark, though performance drops on other ...
-
GenAI Content Detection Task 1: English and Multilingual Machine-Generated Text Detection: AI vs. Human
A COLING 2025 shared task benchmark showing that current machine-generated text detectors reach only moderate accuracy and degrade badly on out-of-domain and humanized AI text.
-
Tracing Thought: Using Chain-of-Thought Reasoning to Identify the LLM Behind AI-Generated Text
Adding chain-of-thought explanations to a fine-tuned BERT improved human-vs-AI text detection from 0.742 to 0.898 F1 in a shared task, but the explanations were generated with the true labels.
Discussion (0). Continue with ORCID to comment.