{"id":"b22d30f6-131c-4e9f-972c-987cb6a97f59","arxiv_id":"2412.03796","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper converts single-label mental health datasets into multi-label ones by having LLMs annotate additional disorders, producing SPAADE-DR, but the main evaluation validates models against labels the models themselves generated.","lead":"Researchers used five large language models to turn single-label mental health datasets into multi-label ones, then built a new six condition dataset called SPAADE-DR from Reddit posts. The paper is worth reading because it tests a cheap way to annotate overlapping mental health conditions, but its six label evaluation is circular and the dataset is not released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SPAADE-DR's five synthetic labels are generated by the same LLMs later scored against them (Section 4.2.2 vs Table 5); no human validation is reported, so the six-label results and comorbidity findings are not evidence for the central claim.","rationale":"The reader's weakest-assumption identification matches my own: the load-bearing premise is that the five LLM-generated labels in SPAADE-DR are accurate enough to serve as evaluation ground truth. The manuscript provides no human validation, and Table 5 contains direct internal evidence of circularity. The strongest claim is not merely unsupported by an external gold standard; the reported near-perfect Single_Label Llama-3 scores indicate that the evaluation compares the model against its own outputs. This makes the six-label performance and the comorbidity findings artifacts of the annotating model. I do not object to LLM-assisted annotation in principle, nor to the DepSeverity-Dreaddit experiment, which uses human labels and yields a useful comparison of prompt strategies. The objection is specific to the SPAADE-DR construction and its downstream evaluation. A human-validated subset, with comorbidity statistics recomputed on that subset, would be the natural way to test whether the dataset claim survives. Since the reader already recommended REJECT and my analysis supports that conclusion, no verdict adjustment is needed.","tokens_in":17080,"tokens_out":3702,"duration_ms":39316,"concrete_test":"Select a random stratified sample of 200 SPAADE-DR posts, masking all LLM-generated labels, and have two independent clinicians (or trained annotators using a structured DSM-aligned rubric) label each post for all six disorders. Compute per-disorder Cohen's kappa between each clinician and the SPAADE-DR labels, and between the two clinicians. Then recompute the Section 5.1 comorbidity odds ratios on the clinician-labeled subset. If per-disorder kappa is below a pre-registered threshold (e.g., 0.6) for the five LLM-annotated disorders, or if the odds-ratio pattern reverses, SPAADE-DR cannot serve as ground truth and Tables 5 and Figures 6-7 collapse. A direct secondary check: for the Single_Label Llama-3 row in Table 5, measure the fraction of SPAADE-DR labels that are identical to Llama-3's own single-prompt outputs; near-100% identity would confirm the circularity quantitatively.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is SPAADE-DR as a robust multi-label dataset. Section 4.2.2 constructs it by retaining the originating subreddit as the label for one disorder and using the single-label prompt with Llama-3 70b, GPT-4o-mini, and Phi-3.5-MoE to annotate the remaining five disorders. No human or independent clinical validation of these generated labels is reported anywhere in the manuscript. Section 5.2 then uses these same labels as ground truth in Table 5. The internal signature of this problem is the Single_Label Llama-3 row: balanced accuracy and F1 are 0.97-1.00 across all six disorders with Hamming loss 0.01. That near-perfect score is exactly what one expects when an LLM is evaluated against labels produced by the same prompt and model, rather than against independently verified diagnoses. The comorbidity claims in Section 5.1, such as '93% of individuals diagnosed with depression also exhibit symptoms of anxiety,' are therefore properties of the annotating model's labeling tendencies, not evidence about clinical co-occurrence. This is not a disagreement with an external consensus: it is an internal correctness risk, because the evaluation target is defined by the system being evaluated. The DepSeverity-Dreaddit experiment retains value as an independent two-label benchmark, since its labels are human-derived, but it does not license the six-label SPAADE-DR results or the dataset-level claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an LLM-based zero-shot pipeline for converting single-label mental-health social-media datasets into multi-label datasets. On the two-label DepSeverity-Dreaddit corpus, the authors compare five LLMs under single-label, multi-label, and unrestricted prompts, reporting that Llama-3 70b with the single-label prompt performs best (Table 3). They then apply the single-label prompt with Llama-3 70b, GPT-4o-mini, and Phi-3.5-MoE to posts sampled from six RMHD subreddits, retaining the source subreddit as the 'true' label for one disorder and using the models to label the other five, producing SPAADE-DR. The paper reports label distributions (Table 4), comorbidity and odds-ratio analyses (Figures 6 and 7), and a six-label evaluation of the same models (Table 5).","tokens_in":17479,"tokens_out":4344,"duration_ms":45775,"significance":"The two-label DepSeverity-Dreaddit experiment is a genuine, human-labeled benchmark and provides a useful comparison of prompt designs and LLMs for multi-label mental-health classification. If SPAADE-DR's labels were independently verified, the dataset could be a valuable resource for studying comorbidity from social media. However, the manuscript currently offers no such verification: the five non-source labels are generated by the same models later scored on them, so the six-label results and comorbidity findings cannot support the paper's central claims. The paper also does not report code or dataset release details, which limits reproducibility and adoptability.","major_comments":[{"comment":"The evaluation is circular. The five non-source labels of SPAADE-DR are produced by applying the single-label prompt with Llama-3 70b, GPT-4o-mini, and Phi-3.5-MoE, and Table 5 then scores these same models against those self-generated labels. The near-perfect single-label row for Llama-3 70b (GBA = 0.99, HL = 0.01) is the expected signature of this circularity, not evidence of diagnostic ability. No human or clinical validation of the generated labels is reported anywhere in the manuscript, so the six-label results in Table 5, the label distributions in Table 4, and the comorbidity findings in Section 5.1 are unsupported as statements about actual mental-health co-occurrence.","section":"Section 4.2.2 and Table 5"},{"comment":"The dataset size is internally inconsistent. The text states that after cleaning, 500 posts are selected for each of the six disorders, which would give 3,000 posts total, but every column of Table 4 sums to 3,500 for each LLM, implying 3,500 posts per disorder. The authors should clarify the exact sample size, whether control posts are included, and correct the discrepancy, because the reported positive/negative counts are central to the claimed dataset resource.","section":"Section 4.2.1 and Table 4"},{"comment":"The comorbidity percentages and odds ratios are computed on LLM-generated labels, so statements such as '93% of individuals diagnosed with depression also exhibit symptoms of anxiety' describe the annotating models' labeling behavior rather than clinically established co-occurrence. To support these claims, the authors would need a human-annotated validation sample, and preferably a comparison with clinically established comorbidity rates.","section":"Section 5.1"}],"minor_comments":[{"comment":"The column header 'OR' is used for overall recall in Table 3, but 'OR' is also introduced as the symbol for odds ratio in Section 2.3 and used in Figure 7; renaming the Table 3 metric to 'Overall Recall' (or 'ORec') would avoid ambiguity.","section":"Section 2.3 / Table 3"},{"comment":"There is a typo: 'r/legalagvice' should read 'r/legaladvice'.","section":"Section 2.1.3"},{"comment":"The cleaning step says that posts predicted as negative are 'manually reviewed' and true negatives removed, but no review criteria, number of annotators, or inter-annotator agreement are reported; this step should be described in detail.","section":"Section 4.2.1"},{"comment":"The text says the 'optimal combinations' from the DepSeverity-Dreaddit evaluation are used for labeling, but Section 5.2 then evaluates multi-label and unrestricted prompts on SPAADE-DR; please clarify which prompt-LLM combinations were used to generate the labels versus which are being evaluated.","section":"Section 3.4 / Section 4.2.2"},{"comment":"The caption contains a typo: 'ana 6 disorders' should read 'on 6 disorders'.","section":"Table 5 caption"}],"recommendation":"reject","confidential_remarks":"The two-label human-validated experiment is salvageable and could be a useful standalone contribution, but the manuscript's primary contribution, SPAADE-DR and its six-label evaluation, rests on unvalidated, self-generated labels. The count inconsistency in Table 4 and the absence of human validation would require substantial new work to address. I would not recommend acceptance in the current form, but a revised version that adds human validation and reframes the claims could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the two-label DepSeverity-Dreaddit experiment is the honest core of this paper, and it delivers a clean, useful result: single-label prompts beat multi-label prompts for depression-plus-stress detection, and Llama-3 70b leads. That part uses human labels and should survive. The six-label SPAADE-DR contribution does not. The labels for five of the six disorders are generated by the same LLMs that are later scored against those labels (Section 4.2.2 vs Table 5), so the near-perfect Single_Label Llama-3 row is an artifact of self-evaluation, not evidence of accuracy. No human or clinical validation is reported anywhere. The comorbidity findings in Section 5.1, like \"93% of depression-positive posts also anxiety-positive,\" are properties of the annotating model's labeling tendencies, not clinical co-occurrence. There's also an internal inconsistency: the text says 500 posts per disorder after cleaning, but Table 4 sums to 3,500 per label, which would require roughly 583 posts per disorder. And no data or code is released, so nothing can be checked externally.\n\nWhat is genuinely new is the SPAADE-DR dataset itself, but novelty without validity is just a claim. The paper's framing—LLM-driven synthetic labeling to create multi-label mental health resources—is reasonable, and the prompt comparison is executed carefully, with multiple models and sensible metrics. The soft spots are not minor; the central dataset claim is unsupported. That said, the two-label result is worth keeping, and the six-label pipeline could become publishable if the authors add human validation on a sample, fix the sample-size arithmetic, and release the dataset.\n\nThe paper is aimed at NLP and computational mental health researchers. I would send it to peer review rather than desk-reject, because the two-label experiment is legitimate and the six-label flaw is correctable. But as submitted, I would not accept; I would ask for major revision or reject-and-resubmit.","headline":"The two-label prompt comparison is a legitimate, useful result, but the six-label SPAADE-DR dataset is built on self-generated labels and circular evaluation, so the central claim does not hold as written.","tokens_in":17937,"tokens_out":2817,"would_cite":false,"duration_ms":28401,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that zero-shot LLM prompting can turn single-label mental-health datasets into multi-label datasets, producing a six-disorder Reddit corpus.","keywords":["multi-label annotation","large language models","zero-shot learning","mental health","comorbidity","social media","SPAADE-DR","data annotation"],"falsifier":"Take a random sample of about 200 SPAADE-DR posts, have two independent clinicians annotate all six disorders from the post text, and compare their labels with the LLM-generated labels on the five synthetically labeled disorders; if clinician-LLM agreement is near chance (for example, Cohen's kappa below 0.4), the dataset's ground truth, the six-label evaluation, and the comorbidity odds ratios are unsupported.","tokens_in":16831,"feed_emoji":"🧠","tokens_out":5934,"duration_ms":54927,"temperature":0.7,"pith_summary":"The paper proposes a zero-shot prompting method that converts single-label mental-health datasets into multi-label datasets, addressing the shortage of resources that record co-occurring disorders. It merges two human-labeled Reddit datasets, DepSeverity (depression) and Dreaddit (stress), into a two-label testbed, evaluates several LLMs and prompt templates on it, and picks the strongest combination: the single-label binary prompt with Llama-3 70b. That combination is then used to label five additional disorders onto posts from six RMHD subreddits, producing SPAADE-DR, a six-disorder multi-label dataset spanning ADHD, anxiety, depression, eating disorder, PTSD, and suicide. The paper reports that single-label prompts stay robust as the label count grows, that GPT-4o-mini becomes the more consistent model for multi-label and unrestricted prompts, and that the LLM-generated labels reproduce known comorbidity patterns such as depression with suicide and PTSD with anxiety. If the labels are accepted as ground truth, the method gives a cheap, scalable route to multi-label annotation and comorbidity analysis on social-media text.","feed_headline":"Zero-shot LLM prompts build a six-disorder mental-health dataset","feed_subtitle":"SPAADE-DR labels Reddit posts for co-occurring conditions, enabling comorbidity studies without extra human annotation.","key_machinery":"The load-bearing object is the single-label binary prompt template: a short instruction that asks the LLM to answer only Yes or No on whether the post's writer exhibits clear symptoms of one target disorder, with no explanation, and to pick the most probable label when ambiguous. Applied once per disorder, it produces one bit of a six-bit annotation vector per post, converting a subreddit-derived single label into a multi-label profile. The DepSeverity-Dreaddit merged dataset, in which the same posts carry independent human annotations for depression and stress, supplies the two-label testbed used to select prompt and model before scaling to six labels.","core_discovery":"On its own terms, the paper's central claim is that a zero-shot single-label prompting procedure is enough to turn any single-label mental-health dataset into a reliable multi-label dataset. For each post, the original subreddit assignment is kept as the true label for one disorder, and an LLM is asked a separate yes/no question for each of the other five disorders using the single-label prompt template; concatenating the answers produces a six-bit label vector. The authors construct SPAADE-DR this way and then evaluate multi-label and unrestricted prompts on it. They report that GPT-4o-mini outperforms the other models on the six-label multi-label and unrestricted tasks, that Llama-3 70b achieves near-perfect scores on the single-label prompts, and that odds ratios between labels reproduce established clinical associations, including strong depression-suicide and PTSD-anxiety links.","pith_inferences":["My inference: if this zero-shot pipeline generalizes, it also applies to other domains with overlapping categories, since nothing in the method is specific to Reddit or to these six disorders.","My inference: the reverse ranking of models across prompt types (Llama-3 70b strong on single-label, GPT-4o-mini strong on multi-label) suggests that conclusions about LLMs for mental-health annotation should be conditioned on the prompt format used.","My inference: a direct test of the method's value would be to train a classifier on SPAADE-DR and measure its agreement with clinician diagnoses on held-out posts; positive transfer would support the labels, while poor transfer would implicate platform or labeling noise."],"forward_implications":["The method turns any collection of single-label mental-health datasets into a multi-label dataset, removing the need for expensive multi-label human annotation.","Single-label prompts remain the most robust as the number of disorders grows from two to six, at the price of running the LLM once per disorder.","SPAADE-DR provides a six-disorder dataset on which multi-label classifiers and comorbidity studies can be trained and benchmarked.","Comorbidity odds ratios computed from LLM labels align with clinical epidemiology, supporting use of such synthetic labels for association discovery."],"supporting_citations":[{"why":"Supplies the Dreaddit half of the merged two-label testbed, contributing human stress labels on the same Reddit posts.","marker":"[23]"},{"why":"Supplies the DepSeverity half with human depression-severity labels, forming the two-label evaluation set.","marker":"[24]"},{"why":"Source of the six single-disorder subreddit datasets that become SPAADE-DR.","marker":"[29]"},{"why":"The single-label prompt template is inspired by this prior LLM mental-health evaluation, and is the template tested in the experiments.","marker":"[35]"},{"why":"Epidemiological comorbidity results used to check that SPAADE-DR odds ratios match known clinical patterns.","marker":"[39]"},{"why":"Cross-national comorbidity data used as external grounding for the association patterns found in the synthetic labels.","marker":"[40]"}],"fun_headline_variants":["Zero-shot LLMs turn Reddit posts into six-disorder mental health labels","New dataset SPAADE-DR uses LLM prompts to label co-occurring mental illnesses","LLM-driven labeling builds multi-label mental health dataset from Reddit","Automated LLM annotation captures overlapping mental health conditions on Reddit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM's yes/no answers for the five conditions that the original subreddit did not label are accurate enough to count as ground truth; the paper applies the single-label prompt to those five conditions in Section 4.2.2 without human validation and later evaluates all models against those same answers in Section 5.2. If the answers are systematically wrong, the reported six-label performance and the comorbidity findings collapse.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLMs turn Reddit posts into six-disorder mental health labels","New dataset SPAADE-DR uses LLM prompts to label co-occurring mental illnesses","LLM-driven labeling builds multi-label mental health dataset from Reddit","Automated LLM annotation captures overlapping mental health conditions on Reddit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1658,"prompt_tokens":960,"completion_tokens":698,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":616}},"tokens_in":576,"tokens_out":698,"duration_ms":6909,"temperature":1.0,"reasoning_tokens":616,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:05:27.890194+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of about 200 SPAADE-DR posts, have two independent clinicians annotate all six disorders from the post text, and compare their labels with the LLM-generated labels on the five synthetically labeled disorders; if clinician-LLM agreement is near chance (for example, Cohen's kappa below 0.4), the dataset's ground truth, the six-label evaluation, and the comorbidity odds ratios are unsupported.","supporting_citations":[{"cited_title":"Early identification of depression severity levels on reddit using ordinal classification","cited_arxiv_id":null,"evidence_quote":"Supplies the DepSeverity half with human depression-severity labels, forming the two-label evaluation set."},{"cited_title":"Natural language processing reveals vulnerable mental health support groups and heightened health anxiety on reddit during covid-19: An observational study","cited_arxiv_id":null,"evidence_quote":"Source of the six single-disorder subreddit datasets that become SPAADE-DR."},{"cited_title":"Hanafy, and Mohammed E","cited_arxiv_id":null,"evidence_quote":"The single-label prompt template is inspired by this prior LLM mental-health evaluation, and is the template tested in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Epidemiological comorbidity results used to check that SPAADE-DR odds ratios match known clinical patterns."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cross-national comorbidity data used as external grounding for the association patterns found in the synthetic labels."}],"review_version":1}