{"id":"d8a8b54e-de48-4bfe-8a2b-59a435326982","arxiv_id":"2412.16302","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"NLP model comparison on Reddit mental-health narratives claims BERT is less dependent on explicit topic words, but small effect sizes and misreported p-values weaken the conclusion.","lead":"This paper compares BERT, MentalBERT, and traditional classifiers on Reddit posts about depression, then tests what happens when topic words are removed or sentences are shuffled. The authors conclude BERT is more robust to missing mental-health keywords, but the accuracy differences are tiny and the statistical reporting is inconsistent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed BERT-vs-MentalBERT contrast in topic-word sensitivity is not supported by Table VI: both models show near-identical accuracy drops on ENS-Depression, and the t-test p-values are internally inconsistent.","rationale":"Good-faith reading: the paper aims to show that BERT generalizes to narratives without mental-health keywords better than MentalBERT. That would require demonstrating a meaningful, robust difference in sensitivity. The data in Table VI do not show this: the accuracy drops on the key ENS-Depression set are virtually identical (-0.6 vs -0.6 for removal, -0.6 vs -0.7 for replacement), so the claimed contrast is driven by p-value thresholding rather than by effect size. The p-value column is also internally inconsistent with the t-statistics, which means the statistical foundation for labeling some models 'sensitive' and BERT 'minimally sensitive' is not credible. The undisclosed WordCloud word list is a second, independent problem: it prevents replication and could fully determine the ranking of models if the selected words happen to be more salient for MentalBERT than for BERT. I give credit for the ENS dataset and for the general research questions about topic-word and sentence-order sensitivity, but the central claim is not supported as reported. The reader's REJECT verdict stands; a corrected resubmission would need fixed statistics, direct model comparisons, and full release of the word list and per-post predictions.","tokens_in":13410,"tokens_out":4697,"duration_ms":42117,"concrete_test":"Release per-post predictions for BERT(128) and MentalBERT(128) on the ENS-Depression test set under raw, word-removed, and word-replaced conditions. Recompute the paired t-tests in Table VI from these vectors, then run a direct comparison of the per-post accuracy-change vectors between BERT and MentalBERT (e.g., a paired bootstrap on the difference of mean accuracy changes, or McNemar's test on discordant pairs). If the printed p-values do not match the t-statistics, or if the between-model difference in mean accuracy change is not significant, the claim of BERT's minimal sensitivity is unsupported. As a secondary check, publish the full WordCloud-derived word list and re-run the manipulation; if a broader or differently selected word list changes the drop ranking, the reported result is an artifact of word selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that BERT(128) shows 'minimal sensitivity' to topic-word removal/replacement in ENS while MentalBERT is significantly dependent on those words (Abstract; Section III.C.1). The decisive evidence is Table VI. On the ENS-Depression testing set, BERT(128) drops from 95.9% to 95.3% (AccDiff = -0.6) for both word removal and replacement; MentalBERT(128) drops from 96.0% to 95.4% (AccDiff = -0.6) for removal and to 95.3% (AccDiff = -0.7) for replacement. Thus the accuracy-based sensitivity is essentially identical. The paper's distinction rests entirely on paired t-test p-values (p = 0.84/0.58 for BERT vs. p < 0.01 for MentalBERT), but a paired t-test tests whether the within-model mean change differs from zero, not whether BERT's change is smaller than MentalBERT's. No direct comparison of the two models' per-post change vectors is provided. Moreover, the p-value column is unreliable: with test-set sizes around 3000, t = 1.43 cannot yield p = 0.67, t = -1.79 cannot yield p < 0.01, and t = -3.00 cannot yield p = 0.05 as two-tailed values. The undisclosed WordCloud word list (Section III.C) is a compounding issue, because any topic-word manipulation inherits the authors' arbitrary word selection, and without the list or code the reader cannot tell whether BERT's apparent robustness is an artifact. But even before that, the reported numbers do not demonstrate a meaningful model-level sensitivity difference.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript studies classification of Reddit posts containing Expressive Narrative Stories (ENS) into depression-related versus control narratives. The authors fine-tune BERT and MentalBERT at several input lengths and compare them with Naive Bayes, Logistic Regression, and SVM on GNS-Depression, ENS-Depression, and Mix-Depression datasets, then probe the models by removing or replacing topic words and by shuffling sentences within and across posts. The paper's central claim is that BERT(128) is minimally sensitive to topic-word absence in ENS, whereas MentalBERT and traditional models depend on explicit topic words, and that BERT(128) is therefore more suitable for real-world mental-health text where keywords are sparse. A secondary claim is that both BERT and MentalBERT maintain accuracy under sentence shuffling to differing degrees across narrative types.","tokens_in":13760,"tokens_out":3842,"duration_ms":33299,"significance":"If the central claim were supported, the paper would provide practically useful guidance for model selection in mental-health NLP, especially for pre-screening where explicit psychiatric terms may be absent. The manuscript also addresses an underexplored data type, ENS, and its manipulation framework is a reasonable way to probe model reliance on surface features. However, the load-bearing BERT-versus-MentalBERT comparison is not supported by the reported accuracy differences, and the reported t-statistics and p-values are internally inconsistent. Because the main conclusion depends on those statistics and on an undisclosed topic-word list, the contribution as stated is not established. The paper's strengths include the separate batch-based data collection to avoid leakage and the inclusion of multiple mental-health generalization sets, but these do not compensate for the central statistical problems.","major_comments":[{"comment":"The claim that BERT(128) exhibited minimal sensitivity to topic-word removal while MentalBERT demonstrated a dependency on topic words is not supported by the accuracy changes. For the ENS-Depression testing set, BERT(128) shows AccDiff = -0.6 for both word removal and replacement, while MentalBERT(128) shows AccDiff = -0.6 for removal and -0.7 for replacement. These accuracy drops are effectively identical. The paired t-test p-values, even if correct, test whether each model's within-model change differs from zero; they do not test whether BERT's change is significantly smaller than MentalBERT's. No direct between-model comparison of the per-post change vectors is provided, so the reported numbers do not demonstrate a model-level sensitivity difference.","section":"Section III.C.1 and Table VI"},{"comment":"The reported t-statistics and p-values are internally inconsistent, which invalidates the statistical evidence. In Table VI, Mix-Depression, BERT(128), Word Replacing has t = 1.43 and p = 0.67; for a paired test on a set of roughly 3,976 posts, this is impossible (the correct two-tailed p would be about 0.15). In Table VII, ENS-Depression BERT(128) Cross-Post Shuffling reports t = -1.79 with p < 0.01, and Within-Post Shuffling reports t = -3.00 with p = 0.05; both are inconsistent with standard paired t-test calculations. Since these p-values are the only evidence distinguishing BERT's sensitivity from MentalBERT's, the key statistical support is unreliable.","section":"Tables VI and VII"},{"comment":"The topic-word list is not reported. The text states that the WordCloud method generated a list of the ten most influential topic words for each training set along with variations, but the actual lists are never given, and no code or data are released. The sensitivity ranking is defined entirely by manipulations of those words, and different word choices could change the measured AccDiff values and significance results. Without the word lists, the experiment cannot be reproduced, and the BERT-versus-MentalBERT ranking may be an artifact of the particular word selection. The authors should provide the lists, specify all variations, and ideally test robustness across multiple independently selected word lists.","section":"Section III.C"}],"minor_comments":[{"comment":"The table header contains 'TDIDF' twice; this should be 'TF-IDF'.","section":"Table IV"},{"comment":"The labels 'Words Removing' and 'Words Replacing' are grammatically awkward; consider 'Word Removal' and 'Word Replacement' for clarity.","section":"Table V"},{"comment":"The sentence 'Despite MentalBERT is design to better handle psychiatric contexts' contains grammatical errors; it should read 'Despite MentalBERT's design to better handle psychiatric contexts'.","section":"Abstract"},{"comment":"Reference [23] gives a World Bank URL, but the text describes a Kaggle dataset for SuicideWatch; the URL appears to be incorrect and should be fixed.","section":"Reference [23]"},{"comment":"The narrative overstates Table VII when it says Cross-Post Shuffling generally showed significant differences with p < 0.01; in several MentalBERT ENS rows the cross-post accuracy actually increases and the p-value is not significant. The prose should be matched to the table entries more carefully.","section":"Section III.D.2"}],"recommendation":"reject","confidential_remarks":"The manuscript addresses a useful question, but the central empirical claim is contradicted by the authors' own accuracy numbers, and the t-test reporting contains inconsistencies that cannot be resolved by minor edits. The undisclosed topic-word list and lack of released code or data compound the problem. I would be willing to reconsider a substantially revised version that reports between-model sensitivity tests, corrects the statistical tables, and releases the word lists; in its current form the paper does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things before reading arXiv:2412.16302. First, the new Reddit ENS dataset and the two perturbation experiments (topic-word removal/replacement, sentence shuffling) are a reasonable, useful extension of the existing BERT-versus-MentalBERT comparison. Second, the paper's headline claim — that BERT(128) is minimally sensitive to missing topic words while MentalBERT depends on them — is not supported by Table VI, and the reported statistics are internally inconsistent.\n\nThe accuracy data is the crux. On the ENS-Depression test set, word removal drops BERT(128) from 95.9 to 95.3 (AccDiff -0.6) and MentalBERT(128) from 96.0 to 95.4 (AccDiff -0.6). Replacement gives -0.6 and -0.7. The sensitivities are essentially identical, so the claimed contrast does not hold up. The paper leans entirely on paired t-test p-values (BERT p=0.84 versus MentalBERT p<0.01), but a paired t-test only asks whether each model's within-model change differs from zero; it does not test whether BERT's change is smaller than MentalBERT's, and no direct comparison of the per-post change vectors is provided. The p-values also do not match the t-statistics: with test sets near 3000 posts, t=1.43 cannot yield p=0.67 (Table VI), t=-1.79 cannot yield p<0.01, and t=-3.00 cannot yield p=0.05 (Table VII). Some signs are inverted as well: MentalBERT word removal on ENS-Depression shows AccDiff -0.6 with t=+3.14.\n\nThe compounding problem is disclosure. The topic-word lists come from a WordCloud procedure that reportedly produced ten key words plus variations per training set, but no list appears anywhere. The entire sensitivity ranking hinges on that selection, and without it the manipulation is not interpretable. No code, no data, no confidence intervals.\n\nCredit where it is due. The data-collection design (batched collection to prevent leakage, consistent preprocessing, 1:1 labels on held-out sets) is careful, and the generalization sets for anxiety, suicide, and bipolar are a plus. The sentence-shuffling finding — order disturbances hurt ENS classification more than GNS — is plausible and is the more defensible half of the paper.\n\nThis paper is for someone studying social-media mental-health screening who wants a new narrative-stories dataset and a reminder of how p-value misuse can undercut an otherwise reasonable study. I would not cite it as evidence about BERT versus MentalBERT until the statistics are corrected and the word lists and code are released.\n\nRecommendation: as submitted, the central claim is unsupported and the tables contain errors. I would still send it to referees — the dataset and the questions are legitimate — but it needs major revision: recompute the statistics, test the between-model sensitivity difference directly, release the word lists and code, and reframe the conclusions to match the accuracy numbers.","headline":"The new ENS dataset and perturbation design are real contributions, but the BERT-superiority claim rests on nearly identical accuracy drops and p-values that do not match the reported t-statistics.","tokens_in":14270,"tokens_out":9181,"would_cite":false,"duration_ms":67090,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Plain BERT keeps its accuracy when mental-health keywords are stripped from Reddit narratives, while MentalBERT and traditional classifiers degrade.","keywords":["mental health text classification","Expressive Narrative Stories","BERT","MentalBERT","topic word sensitivity","sentence shuffling","Reddit depression detection","linguistic nuance"],"falsifier":"Re-run the topic-word manipulation with a published, independently constructed list of depression-related keywords (for example, from a clinical lexicon) and with the exact ten-word WordCloud list disclosed; if BERT(128)'s ENS accuracy drops by more than about one point under that broader removal, the claim of minimal sensitivity would not generalize.","tokens_in":13187,"feed_emoji":"🧠","tokens_out":5387,"duration_ms":45640,"temperature":0.7,"pith_summary":"This paper asks whether text classifiers can recognize mental-health signals in Expressive Narrative Stories (ENS)—emotional, first-person Reddit narratives—when the stories contain no explicit mental-health vocabulary. The authors fine-tune BERT and MentalBERT on depression versus control narratives and compare them with SVM, Naive Bayes, and Logistic Regression, then probe each model by deleting or replacing topic words and by shuffling sentences. Their central finding is that BERT(128) keeps its accuracy when topic words vanish from ENS, while MentalBERT and the traditional models lose accuracy significantly, even though MentalBERT was pretrained on psychiatric text. They also find that shuffling sentences hurts ENS classification accuracy for both BERT and MentalBERT, showing that narrative order carries real signal. If correct, the paper argues for context-aware models rather than keyword matching in real-world mental-health screening.","feed_headline":"BERT outlasts MentalBERT when mental-health words vanish","feed_subtitle":"Removing depression topic words barely moves BERT's accuracy on Reddit stories, while a domain-tuned model loses ground.","key_machinery":"The load-bearing mechanism is the controlled perturbation protocol. Topic words are identified per training set with the WordCloud method, removed or replaced by the word 'nothing' to test surface-word dependence; sentence order is destroyed by within-post and cross-post shuffling to test reliance on narrative coherence. Paired t-tests on accuracy differences between raw and manipulated text decide which models are sensitive. This protocol is what separates BERT(128) from MentalBERT and the shallow classifiers, and it is the evidence behind the paper's recommendation of context-aware models for real-world screening.","core_discovery":"On the paper's own terms, the discovery is that a general-purpose BERT model is more robust to the absence of explicit depression-related topic words than a domain-adapted MentalBERT or any of the traditional bag-of-words classifiers. Using a WordCloud-derived list of ten influential topic words per training set, the authors removed those words from, and then replaced them with the neutral token 'nothing' in, all testing sets. In the ENS-Depression testing set, BERT(128) showed no significant change under either manipulation (P≈0.84 and 0.58), whereas MentalBERT showed a significant drop under word removal (P<0.01) and the shallow models showed scattered but repeated significant drops. The paper reads this as evidence that BERT(128) captures deeper linguistic features than keyword presence. A second discovery is that sentence order matters for distinguishing depression narratives: within-post shuffling significantly reduced accuracy for both BERT(128) and MentalBERT(128) on ENS-related sets, and cross-post shuffling significantly affected BERT(128), while GNS-related sets were essentially unaffected. The authors conclude that ENS classification is driven by narrative coherence and subtle linguistic pattern, not by surface vocabulary.","pith_inferences":["A testable extension is to compare BERT(128) against a large language model prompted with no fine-tuning on the same manipulated sets; the paper's perturbation protocol could serve as a benchmark for how any model relies on surface vocabulary.","The authors leave implicit that the same WordCloud word list being drawn from training sets may leak into the manipulation of testing sets; an independent lexical source would strengthen the conclusion.","If sentence-order sensitivity reflects narrative coherence, then modeling discourse structure explicitly—for example, with a coherence-aware architecture—might improve ENS classification beyond BERT(128)."],"forward_implications":["In a pre-screening tool that reads Reddit-style narratives, BERT(128) would likely keep working when users avoid the words 'depression,' 'anxiety,' or 'suicide,' whereas MentalBERT and traditional classifiers would degrade.","MentalBERT's psychiatric pretraining does not by itself buy robustness to missing keywords; model selection should be tested under input perturbation, not just on held-out accuracy.","Since sentence shuffling significantly lowers ENS accuracy, narrative coherence is itself a signal; tools that preserve sentence order will capture information that bag-of-words or shuffled-pipeline approaches discard.","For general narratives (GNS), classification stays near 99 percent regardless of shuffling, so keyword and surface features are sufficient there; the need for context-aware models is specific to expressive narratives."],"supporting_citations":[{"why":"Supplies the BERT-base model that is fine-tuned and shown to be minimally sensitive to topic-word removal.","marker":"[24]"},{"why":"Supplies MentalBERT, the domain-pretrained model whose topic-word dependency is the paper's main comparative finding.","marker":"[25]"},{"why":"Supplies the SuicideWatch Reddit dataset used to build the suicide-related testing sets for generalization analysis.","marker":"[23]"},{"why":"Grounds the definition of Expressive Narrative Stories in the expressive-writing concept.","marker":"[2]"},{"why":"Provides the psychological evidence that emotion impairs logical reasoning, motivating the sentence-shuffling manipulation.","marker":"[26]"},{"why":"Supports the claim that severe anxiety disrupts reasoning and narrative coherence, framing the interpretation of shuffled-text results.","marker":"[27]"}],"fun_headline_variants":["BERT keeps its edge when depression words are stripped","MentalBERT stumbles without mental-health keywords, BERT doesn't","Removing topic words hurts MentalBERT, not BERT","Narrative order matters: BERT and MentalBERT both lose to shuffling","Keyword removal exposes BERT's resilience over MentalBERT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The key assumption is that the ten topic words chosen by WordCloud for each training set are the right words to remove or replace across all test sets; the paper never lists them, so the measured sensitivities could change with a different word list.","fun_headline_variants_meta":{"raw":{"variants":["BERT keeps its edge when depression words are stripped","MentalBERT stumbles without mental-health keywords, BERT doesn't","Removing topic words hurts MentalBERT, not BERT","Narrative order matters: BERT and MentalBERT both lose to shuffling","Keyword removal exposes BERT's resilience over MentalBERT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000665,"raw_usage":{"total_tokens":3100,"prompt_tokens":1073,"completion_tokens":2027,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":689,"completion_tokens_details":{"reasoning_tokens":1943}},"tokens_in":689,"tokens_out":2027,"duration_ms":12679,"temperature":1.0,"reasoning_tokens":1943,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:42:55.623031+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the topic-word manipulation with a published, independently constructed list of depression-related keywords (for example, from a clinical lexicon) and with the exact ten-word WordCloud list disclosed; if BERT(128)'s ENS accuracy drops by more than about one point under that broader removal, the claim of minimal sensitivity would not generalize.","supporting_citations":[{"cited_title":"New dataset: Reddit r/depression and r/suicidewatch for nlp,","cited_arxiv_id":null,"evidence_quote":"Supplies the SuicideWatch Reddit dataset used to build the suicide-related testing sets for generalization analysis."},{"cited_title":"Confronting a traumatic event: toward an understanding of inhibition and disease","cited_arxiv_id":null,"evidence_quote":"Grounds the definition of Expressive Narrative Stories in the expressive-writing concept."},{"cited_title":"How emo- tions affect logical reasoning: evidence from experiments with mood- manipulated participants, spider phobics, and people with exam anxiety,","cited_arxiv_id":null,"evidence_quote":"Provides the psychological evidence that emotion impairs logical reasoning, motivating the sentence-shuffling manipulation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the claim that severe anxiety disrupts reasoning and narrative coherence, framing the interpretation of shuffled-text results."}],"review_version":1}