{"id":"73579e29-a9fa-4dda-8a41-45332941bc44","arxiv_id":"2501.03203","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"XGBoost and Random Forest can distinguish ChatGPT-written cybersecurity paragraphs from human Wikipedia paragraphs with 81 to 83% accuracy, and a narrow XGBoost model beat GPTZero in a three-class test, yet the benchmark rests on a small unreleased dataset.","lead":"This paper builds a 1,000-paragraph dataset of human and ChatGPT-written cybersecurity texts and trains standard machine learning models to tell them apart. The best model, XGBoost, reached 83% accuracy on paragraphs and beat GPTZero in a three-way test, but the dataset and code are not public and the comparisons have caveats.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The GPTZero comparison may be invalid: Section 4.3 does not ensure disjoint source paragraphs between training and test sets, risking data leakage that could inflate the reported 77.5% accuracy.","rationale":"The reader identified dataset representativeness as the weakest assumption, which affects external validity: Wikipedia prose is not necessarily student work, so the academic-integrity interpretation is overclaimed. I agree with that, but the more immediate threat is to the internal validity of the headline comparison. Section 4.3 gives no details on how Mixed observations were built or how the 400/200 split was performed, and with only 600 observations derived from a smaller pool of source paragraphs, the risk of source-paragraph overlap between training and test is high. Such overlap would let the TF-IDF model recognize tokens it has already seen, inflating the 77.5% accuracy while leaving GPTZero's API-based predictions unaffected. This would invalidate the central claim that a narrow fine-tuned model outperforms a general-purpose detector. The binary 83% result may still be valid for the described dataset, but the distinctive contribution of the paper is the GPTZero comparison, and it is the most exposed to this flaw. The appropriate response is to keep the verdict CONDITIONAL: the paper is not yet reproducible or internally verifiable until the dataset and split construction are clarified or the experiment is re-run with a disjoint split.","tokens_in":14036,"tokens_out":7626,"duration_ms":71722,"concrete_test":"Request the CyberHumanAI dataset and the scripts for Section 4.3. Check for any source paragraph (human or ChatGPT) that appears in both the 400 training and 200 test observations (in any Mixed ratio). Alternatively, re-run the 3-class experiment with a split that guarantees source-paragraph disjointness (e.g., group all observations derived from the same human/AI pair into one fold) and report XGBoost accuracy. If accuracy drops materially below 77.5%, the claimed outperformance is an artifact of leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline comparative result (77.5% vs 48.5%, Section 4.3) rests on a 3-class benchmark whose construction is underspecified. The paper says 600 observations were 'created' by combining ChatGPT and human text, then split 400/200 for training/testing, but it never states that the source paragraphs used in a Mixed observation are disjoint between the training and test sets. Because the model uses TF-IDF unigram features, any source paragraph that appears in both a training observation (e.g., a Mixed text) and a test observation (e.g., another ratio of the same human/AI pair) would directly leak token identities and inflate the XGBoost accuracy. GPTZero, being a black-box API, is immune to this leakage, so the comparison would be systematically biased. Without a statement of disjointness or the released data/code, the central claim of outperforming GPTZero is not verifiable. The separate representativeness issue (Wikipedia prose as a proxy for student writing) further limits the academic-integrity interpretation, but the leakage risk is the more immediate threat to the quantitative claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a machine-learning pipeline to distinguish human-written from ChatGPT-generated cybersecurity text, introducing the CyberHumanAI dataset of 500 Wikipedia-derived human paragraphs and 500 ChatGPT paragraphs. The authors evaluate six classifiers (RF, SVM, J48, XGBoost, DNN, CNN) on paragraph-level and article-level classification, report XGBoost (83%) and RF (81%) as the best paragraph classifiers, use LIME to identify discriminative words, and compare their best model with GPTZero on a three-class (Pure AI, Mixed, Pure Human) benchmark, claiming 77.5% accuracy versus GPTZero's 48.5%. The paper's title and abstract frame the work as supporting academic integrity in educational content.","tokens_in":14240,"tokens_out":3149,"duration_ms":31358,"significance":"If the central claims are valid, the paper offers a useful, lightweight baseline: TF-IDF features plus XGBoost can separate ChatGPT from human-written cybersecurity paragraphs at roughly 80% accuracy, and a task-specific detector can outperform a general-purpose commercial detector on a narrow domain. The internal arithmetic is consistent: the confusion matrices in Figures 4 and Tables 6/7 agree with the reported accuracies. The XAI analysis (LIME) is a commendable transparency addition, and the article-versus-paragraph comparison is a legitimate observation. However, the significance is substantially tempered by the lack of dataset/code availability, the absence of any statistical uncertainty quantification, and potential flaws in the GPTZero benchmark construction that currently make the headline comparative claim unverifiable.","major_comments":[{"comment":"The construction of the three-class GPTZero comparison is underspecified to the point that the central comparative claim (77.5% vs 48.5%) is not verifiable. The text says 600 observations were 'created' by combining ChatGPT and human text, then split 400/200 for training/testing, but it never states whether the source paragraphs used in a Mixed observation are disjoint between the training and test sets. Because the XGBoost model uses TF-IDF unigram features, any source paragraph that contributes tokens to both a training observation and a test observation would directly leak lexical identities and inflate the reported accuracy. GPTZero, as a black-box API, would not receive this leakage benefit, so the comparison would be systematically biased. Please either state explicitly and demonstrate that every source paragraph appears in only one observation (training or test), or re-run the comparison with a formally disjoint construction and release the exact splitting procedure.","section":"Section 4.3, Table 4"},{"comment":"The dataset does not match the educational-integrity framing of the paper. The human paragraphs were extracted from Wikipedia using the keyword 'computer security,' and the AI paragraphs were generated by ChatGPT on the same titles. This corpus may resemble encyclopedia prose, but it is not student work, homework, term papers, or typical academic submissions. The title and abstract claim that the tools detect 'AI-generated content in student work' and support 'academic integrity,' yet no evidence is provided that the distribution of Wikipedia text resembles student writing. As a result, the reported accuracies, the XAI feature findings, and the GPTZero comparison all lose direct relevance to the stated application. Please either add an evaluation on student-like text (or clearly scope the claims to Wikipedia-style prose) and discuss the transfer gap.","section":"Section 3.1 / Abstract"},{"comment":"The paragraph-level accuracies (XGBoost 83%, RF 81%) are based on a single 80/20 train/test split, with no cross-validation, repeated splits, or confidence intervals. Given that RF, CNN, and XGBoost differ by only 2–4 percentage points, the reported ordering could easily be produced by noise. The reader cannot assess whether the superiority of XGBoost over RF is stable or whether the 83% figure is an artifact of one favorable split. Please report results over multiple random splits (e.g., 5× or 10× with mean and standard deviation) or use k-fold cross-validation, and state the hyperparameter choices as part of the evaluation protocol.","section":"Section 4.1, Table 3"}],"minor_comments":[{"comment":"There are two tables numbered 'Table (6)' (one for GPTZero and one for the proposed model); renumber the second one as Table (7) and update all references accordingly.","section":"Section 4.3"},{"comment":"The sentence 'This dataset to acts as a fundamental step' contains a grammatical error; it should read 'This dataset acts as a fundamental step...'.","section":"Section 3.1"},{"comment":"The phrase 'uses different the performance metrics' should be corrected to 'uses different performance metrics'.","section":"Section 3.2"},{"comment":"In Section 4.2, the text says the model predicted the class 'with about 99% accuracy,' but this appears to refer to the prediction probability for a single instance, not the model's overall accuracy; please rephrase to avoid confusion.","section":"Section 4.2 and Figure 6"},{"comment":"The comparison would be more convincing if the authors reported GPTZero's behavior on the 32 unrecognized cases and the exact handling of those cases in the accuracy calculation, since the reported 48.5% excludes them while the proposed model has no unrecognized cases.","section":"Section 4.3"},{"comment":"The CyberHumanAI dataset is described as a key contribution, but no repository link, data sharing statement, or code release is provided. For reproducibility and for the academic-integrity community to build on this work, the dataset and preprocessing/feature-engineering code should be made available.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely topic and contains a plausible empirical pipeline, but the most important claim—outperformance of GPTZero—currently rests on an underspecified benchmark that could be compromised by data leakage. The authors can likely fix this by clarifying or re-running the experiment with disjoint source paragraphs, and by adding cross-validated uncertainty estimates. I would also encourage the editor to consider whether the educational-application framing is sufficiently supported by the Wikipedia-based dataset in the absence of a transfer experiment. These concerns are major but not fatal, so major revision seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the CyberHumanAI dataset: 1000 balanced cybersecurity paragraphs, 500 human (via Wikipedia) and 500 ChatGPT, plus a three-class pure/mixed/human comparison against GPTZero. The internal numbers are consistent — the confusion matrices do yield the reported 83%, 81%, and 77.5% accuracies, and the XAI findings (humans use practical words like \"use\" and \"allow,\" ChatGPT prefers \"realm\" and \"employ\") are honestly presented. The \"shorter content is harder\" result is a useful confirmation, not a breakthrough.\n\nBut the central claim — that a narrow XGBoost model beats GPTZero 77.5% to 48.5% — has a real soft spot. Section 4.3 says 600 observations were \"created\" by combining ChatGPT and human text, then split 400/200, but it never says the source paragraphs are disjoint between training and test. Since the model uses TF-IDF unigrams, any overlap between a Mixed training instance and a Mixed test instance would leak token identities and inflate the XGBoost accuracy. GPTZero, being a black-box API, would not benefit from that leakage, so the comparison would be systematically biased. This is not a manufactured flaw; it is a direct consequence of the missing disjointness statement. The paper also does not ship code or data, and there are no error bars or cross-validation. The representativeness issue — Wikipedia prose as a proxy for student work — further limits the academic-integrity interpretation, though it is a minor point compared to the leakage risk.\n\nOn the credit side: the arithmetic is sound, the experimental design is straightforward, and the authors do not overclaim. They explicitly acknowledge that deep learning underperforms and that article-level detection is trivially easy (likely due to formatting artifacts). That is honest. However, the headline comparative result is currently unverifiable from the manuscript alone.\n\nWho is this for? Someone building a small domain-specific detector will find the design lesson useful, and the dataset could be a seed for future work if released. But as it stands, the paper is a conditional pass at best: without data release or an explicit disjointness guarantee, the GPTZero comparison should not be believed. I would send it to peer review because the question is worthwhile and the result is plausible, but I would demand those fixes before acceptance.","headline":"A plausible small benchmark with a useful narrow-beats-general lesson, but the headline GPTZero comparison rests on an underspecified split that risks leakage.","tokens_in":14820,"tokens_out":1133,"would_cite":false,"duration_ms":12152,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A domain-specific XGBoost model trained on TF-IDF word counts classifies ChatGPT-written versus human-written cybersecurity paragraphs with 83% accuracy and outperforms GPTZero (77.5% versus 48.5%) on a three-class mixed benchmark…","keywords":["AI-generated text detection","ChatGPT","XGBoost","LIME","TF-IDF","academic integrity","GPTZero","cybersecurity text"],"falsifier":"Build a corpus of 500 real student cybersecurity essays and 500 ChatGPT replies to the same assignment prompts, then run the reported TF-IDF plus XGBoost pipeline on it; if paragraph accuracy falls to near 50% or to GPTZero's level, the paper's generalization to educational content is not supported.","tokens_in":1795,"feed_emoji":"🤖","tokens_out":1887,"duration_ms":120464,"temperature":0.7,"pith_summary":"The paper tries to establish that a simple, domain-specific machine-learning model can tell ChatGPT-written paragraphs from human-written ones in cybersecurity text, and that such a narrow detector can beat a general-purpose commercial detector on the same task. On a new 1,000-paragraph dataset, XGBoost reaches 83% accuracy and Random Forest 81% for paragraphs, while the same family of models reaches 99-100% for full articles, evidence that shorter units are harder to attribute. Explainable-AI analysis finds a consistent vocabulary split: human writing favors practical words like 'use' and 'allow', while ChatGPT text prefers formal, abstract terms like 'realm' and 'employ'. In a three-way benchmark of pure AI, pure human, and mixed paragraphs, the proposed model reaches about 77.5% accuracy versus GPTZero's 48.5%. If these results hold, educators can build transparent, low-cost detectors tailored to their own subject matter and get better results than generic AI-detection services.","feed_headline":"Small AI detector beats GPTZero: 77.5% to 48.5%","feed_subtitle":"A narrow word-count model catches ChatGPT-written paragraphs far more reliably than the general-purpose detector.","key_machinery":"The pipeline's working parts are the CyberHumanAI dataset, TF-IDF vectorization, XGBoost, and LIME. TF-IDF, term frequency-inverse document frequency, turns each paragraph into a weighted word-count vector, so the model is essentially comparing word-choice profiles rather than reading meaning. XGBoost, a gradient-boosted decision-tree classifier, is the model that reaches the top paragraph accuracy; LIME, a local model-agnostic explanation method, takes a trained classifier and one example and reports which words pushed the prediction toward 'human' or 'ChatGPT.' The article-versus-paragraph comparison isolates input length as a difficulty factor, and the GPTZero comparison pits this fixed vocabulary signal against a commercial detector built on perplexity and burstiness.","core_discovery":"The paper claims that a simple, fixed-feature machine-learning pipeline can reliably separate ChatGPT-written from human-written cybersecurity paragraphs, and that such a narrowly trained model beats a general-purpose detector on that task. The evidence is the CyberHumanAI dataset: 500 human paragraphs pulled from an open encyclopedia via its API and 500 ChatGPT paragraphs on the same cybersecurity titles, compiled in September 2023. On paragraph classification, XGBoost reaches 83% accuracy and Random Forest 81%, while deep networks trail (DNN 69%, CNN 79%); on full articles, accuracy rises to 99-100%. In the three-class comparison of pure AI, pure human, and mixed text, the proposed XGBoost model reaches about 77.5% accuracy with no unrecognized cases, while GPTZero reaches 48.5% and leaves 32 of 200 inputs unclassified, most often labeling pure-AI and pure-human text as mixed. LIME explanations attribute the separation to a vocabulary asymmetry: practical words such as 'use,' 'allow,' and 'virus' mark human text, while formal, abstract words such as 'realm,' 'employ,' and 'establish' mark ChatGPT text.","pith_inferences":["Not tested in the paper: whether the ChatGPT half could be paraphrased or prompted to imitate encyclopedia style; if a simple style-shift prompt erases the 'realm' and 'employ' markers, the 83% result would not generalize to adversarial student use.","The September 2023 data collection makes the lexical signal time-sensitive; newer LLMs or updated GPTZero could narrow or erase the reported 29-point gap, and re-running the same pipeline on current text is a direct check.","The human side is encyclopedia prose, not student essays; until the pipeline is validated on actual assignment submissions, the paper's 'educational content' framing should be treated as a claim about one genre pair, not about student behavior.","A deployment implication the paper leaves implicit: publishing LIME's feature lists helps explain flags to students but also teaches anyone how to avoid detection, so an integrity tool built this way should be continually re-evaluated."],"forward_implications":["A narrow, TF-IDF-based detector can beat a general-purpose commercial detector on its own benchmark: 77.5% versus 48.5% accuracy on pure-AI, pure-human, and mixed paragraphs, with no unrecognized cases.","Shorter text is harder to attribute: the same algorithms that reach 99-100% on full articles drop to 69-83% on paragraphs, so length must be reported alongside any accuracy claim.","Because LIME identifies the exact words driving a prediction, a deployed detector can give educators a student-facing explanation rather than an opaque flag.","Classical machine learning on small corpora is competitive with deep learning for this task: XGBoost at 83% beat the best deep model, CNN, at 79%.","A general-purpose detector's accuracy on short, domain-specific text can be substantially overstated: GPTZero classified most pure-AI and pure-human paragraphs as mixed."],"supporting_citations":[{"why":"Supplies the TF-IDF text vectorizer that converts the paragraphs into the features used by every classifier.","marker":"[21]"},{"why":"Introduces LIME, the model-agnostic explanation tool used to extract the discriminative words for human and ChatGPT classes.","marker":"[42]"},{"why":"The commercial GPTZero detector that the paper benchmarks its proposed model against.","marker":"[43]"},{"why":"Press account documenting GPTZero's launch and the false-positive criticism it drew, framing why the comparison matters for academic integrity.","marker":"[44]"},{"why":"Describes GPTZero's perplexity and burstiness mechanism, the detection approach the paper contrasts with its own.","marker":"[45]"},{"why":"Prior AI-text detection framework in research publications that reports 89.95% accuracy, giving the paper's stated baseline for detection frameworks.","marker":"[10]"},{"why":"Earlier comparison of ML classifiers on ChatGPT versus human text (about 77% accuracy on a 10,000-document set), the result this paper extends to cybersecurity paragraphs.","marker":"[14]"},{"why":"Reports Random Forest reaching about 93% accuracy for human/AI text, the closest prior RF result the paper positions against its 81%.","marker":"[19]"}],"fun_headline_variants":["Focused AI detector beats GPTZero: 77.5% vs 48.5%","Simple model beats GPTZero by 29 points on AI text","Word choice gives away ChatGPT: detector hits 77.5%","Narrow AI detector outperforms GPTZero on student text","Detecting AI essays: focused model beats GPTZero"],"cache_read_input_tokens":16896,"weakest_assumption_plain":"The load-bearing premise is that encyclopedia paragraphs on computer-security topics stand in for the student work that academic-integrity tools would actually screen, and that ChatGPT output on matching titles stands in for how students use AI.","fun_headline_variants_meta":{"raw":{"variants":["Focused AI detector beats GPTZero: 77.5% vs 48.5%","Simple model beats GPTZero by 29 points on AI text","Word choice gives away ChatGPT: detector hits 77.5%","Narrow AI detector outperforms GPTZero on student text","Detecting AI essays: focused model beats GPTZero"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000855,"raw_usage":{"total_tokens":3789,"prompt_tokens":1096,"completion_tokens":2693,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":712,"completion_tokens_details":{"reasoning_tokens":2602}},"tokens_in":712,"tokens_out":2693,"duration_ms":17741,"temperature":1.0,"reasoning_tokens":2602,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:52:48.918697+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a corpus of 500 real student cybersecurity essays and 500 ChatGPT replies to the same assignment prompts, then run the reported TF-IDF plus XGBoost pipeline on it; if paragraph accuracy falls to near 50% or to GPTZero's level, the paper's generalization to educational content is not supported.","supporting_citations":[{"cited_title":"Sentiment classification: Review of text vectorization methods: Bag of words, Tf-Idf, Word2vec and Doc2vec,","cited_arxiv_id":null,"evidence_quote":"Supplies the TF-IDF text vectorizer that converts the paragraphs into the features used by every classifier."},{"cited_title":"‘ Why should i trust you?’ Explaining the predictions of any classifier,","cited_arxiv_id":null,"evidence_quote":"Introduces LIME, the model-agnostic explanation tool used to extract the discriminative words for human and ChatGPT classes."},{"cited_title":"GPTZero | The Trusted AI Detector for ChatGPT, GPT -4, & More","cited_arxiv_id":null,"evidence_quote":"The commercial GPTZero detector that the paper benchmarks its proposed model against."},{"cited_title":"Princeton student creates GPTZero tool to detect ChatGPT-generated text - The Washington Post","cited_arxiv_id":null,"evidence_quote":"Press account documenting GPTZero's launch and the false-positive criticism it drew, framing why the comparison matters for academic integrity."},{"cited_title":"GPTZero - Wikipedia","cited_arxiv_id":null,"evidence_quote":"Describes GPTZero's perplexity and burstiness mechanism, the detection approach the paper contrasts with its own."},{"cited_title":"A Framework for Detecting AI -Generated Text in Research Publications,","cited_arxiv_id":null,"evidence_quote":"Prior AI-text detection framework in research publications that reports 89.95% accuracy, giving the paper's stated baseline for detection frameworks."},{"cited_title":"Distinguishing Human-Written and ChatGPT-Generated Text Using Machine Learning,","cited_arxiv_id":null,"evidence_quote":"Reports Random Forest reaching about 93% accuracy for human/AI text, the closest prior RF result the paper positions against its 81%."}],"review_version":1}