REVIEW 3 major objections 6 minor 28 references
Su-RoBERTa: A Semi-supervised Approach to Predicting Suicide Risk through Social Media using Base Language Models
T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A 355M-parameter RoBERTa model, augmented with GPT-2-generated posts and pseudo-labeled Reddit data, reaches a 69.84% weighted F1 score on a hidden suicide-risk test set.
desk verdict The externally verified leaderboard score is credible, but the GPT-4 pseudo-label comparison does not support the 'base models vs LLMs' claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The pipeline is carried by three components: (1) GPT-2-base (124M) fine-tuned to generate synthetic posts, upsampling Behavior and Indicator to 190 samples each and Attempt to 82; (2) a 355M-parameter RoBERTa classifier fine-tuned for four-class suicide-risk prediction on the resulting 652-sample labeled set; and (3) an iterative self-training loop that pseudo-labels the 1,500 unlabeled posts only when predicted probability exceeds 0.33, retrains the model from scratch twice, and finally trains on all 2,000 available posts. The competition's weighted F1 metric defines success. The authors report that the full fine-tuning pipeline runs in about 30 minutes on a single 24 GB GPU.
What would settle it
Obtain expert clinician labels for the 100 withheld test posts (or an equivalent expert-annotated Reddit set) and recompute Su-RoBERTa's weighted F1 against those labels. If the score falls far below 69.84%, or if GPT-4's pseudo-labels match the expert labels no better than Su-RoBERTa matches GPT-4 (50%), then the paper's base-model-versus-LLM comparison loses its support.
Extended reading notes
Core claim
Su-RoBERTa is a 355M-parameter RoBERTa classifier trained through a two-stage semi-supervised pipeline. In the first stage, a 124M-parameter GPT-2 model is fine-tuned to generate synthetic posts that raise the minority classes: Behavior and Indicator are upsampled to 190 samples each and Attempt to 82, bringing the labeled training set from 500 to 652 posts. In the second stage, the RoBERTa model is fine-tuned on the augmented labels, used to pseudo-label the 1,500 unlabeled posts with a confidence threshold above 0.33, and retrained from scratch on the expanded corpus for two iterations. The final model reaches 69.84% weighted F1 on the competition's unrevealed test set, placing 10th. Against a GPT-4 few-shot labeling of the same 100 test posts, Su-RoBERTa agrees with GPT-4 on 50% of samples while an SVM baseline agrees on 32%, which the paper uses to argue that a base model captures much of the signal of a state-of-the-art LLM at a fraction of the compute.
Load-bearing premise
The comparison that supports the paper's claim that Su-RoBERTa is comparable to a state-of-the-art LLM assumes that GPT-4's few-shot labels on the 100 withheld test posts are a trustworthy stand-in for the true hidden labels; the paper explicitly concedes GPT-4 'cannot be perfect' and says no validation against the real labels is possible.
Editorial extensions
If this is right
- Sub-500M base models can place on a competitive leaderboard for suicide-risk classification, making the approach viable in settings without large GPU budgets.
- Generative data augmentation plus confident pseudo-labeling offers a reusable recipe for other class-imbalanced mental-health text tasks.
- A roughly 30-minute fine-tuning pipeline means the model can be retrained rapidly as new social-media posts arrive.
- The 50% agreement with GPT-4 pseudo-labels, while not an official metric, suggests the base model captures much of the label signal an LLM grader would apply to these four risk categories.
- The small model size and training cost bring mobile or edge deployment closer, though the paper notes real-time use still needs improvement.
Reading between the lines
- The GPT-4 agreement numbers are best read as an internal sanity check rather than a formal evaluation, since the true test labels are hidden and the paper concedes GPT-4 'cannot be perfect.'
- An ablation the authors do not run—training without GPT-2 augmentation, without pseudo-labels, or with a different confidence threshold—would isolate which component actually drives the 69.84% F1.
- The same pipeline could plausibly transfer to other imbalanced clinical-text tasks such as depression or anxiety detection, but it should first be validated on an expert-annotated held-out set to ensure pseudo-label noise does not distort risk categories.
- A direct deployability check would measure inference latency and memory footprint on a mobile CPU, which the paper motivates but does not report.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Su-RoBERTa, a RoBERTa-base model (355M parameters) fine-tuned for a four-class suicide risk prediction task on Reddit posts from the IEEE BigData 2024 Suicide Risk Detection competition. The proposed pipeline combines GPT-2-based data augmentation for class imbalance, iterative pseudo-labeling of 1500 unlabeled training posts with a confidence threshold, and fine-tuning of RoBERTa. The authors report a weighted F1 score of 69.84% on the competition's final evaluation, which placed them 10th on the leaderboard. They also describe a classical SVM baseline that achieved 50.52% on a preliminary evaluation. The paper claims that base language models (under 500M parameters) can be effective and efficient alternatives to larger LLMs for mental health risk prediction, and uses agreement with GPT-4 pseudo-labels on 100 test posts to argue that Su-RoBERTa is comparable to a state-of-the-art LLM.
Significance. If the leaderboard result is taken at face value, the paper provides a useful data point: a moderately sized RoBERTa model, trained with a lightweight semi-supervised pipeline and GPT-2 augmentation, can achieve a competitive weighted F1 on a real-world suicide risk prediction benchmark. The reported 30-minute fine-tuning time on a single RTX 3090 is a genuine practical strength and supports the paper's emphasis on computational efficiency. The external leaderboard evaluation is a credible, non-circular source for the main performance claim. However, the paper's additional comparative claim against large LLMs rests on an invalid evaluation: agreement with GPT-4 pseudo-labels is not a measure of accuracy, and the authors provide no validation of GPT-4's labeling quality. This comparative contribution is therefore unsupported in its current form, and the paper's central scientific value is limited to the leaderboard score and pipeline description.
major comments (3)
- [Section 6, Table 1 and Figures 6-7] The evaluation of Su-RoBERTa against GPT-4 pseudo-labels is methodologically invalid as a measure of model quality. Agreement with a pseudo-labeler is not accuracy against true labels, and the paper provides no evidence that GPT-4's few-shot labels are reliable for these 100 test posts. The authors state that 'GPT-4 itself cannot be perfect for this specific task' but then treat its labels as 'evaluation ground truth.' This is especially problematic because 500 labeled training posts are available, which could have been used to estimate GPT-4's labeling accuracy (e.g., by comparing GPT-4 labels on a held-out subset of the labeled data against the true labels). Without such validation, the 50% agreement figure and the subsequent inference that Su-RoBERTa is 'comparable' to GPT-4 are uninterpretable. This directly affects Contribution 3 (Feasibility) and the abstract's claim about being effective 'in contrast to LLMs.'
- [Section 5.2 and Section 4.1] The pseudo-label confidence threshold (0.33) and the augmentation counts (Behavior and Indicator upsampled to 190, Attempt to 82) are described as being decided by 'empirical experimentations,' but no experiments, sensitivity analyses, or error bars are reported. These choices directly affect the amount of pseudo-labeled data used in self-training and the class balance of the augmented training set. Since the paper's central contribution is a semi-supervised pipeline, the lack of any analysis of how these hyperparameters affect the final F1 makes the method difficult to reproduce or trust as a general recipe. At minimum, the authors should provide the range of thresholds and augmentation settings explored and show that the final leaderboard result is not a fragile artifact of a single post hoc choice.
- [Section 7, Table 2] Table 2 reports the SVM model's preliminary weighted F1 of 50.52% but leaves its final evaluation blank, with no explanation. The comparison between Su-RoBERTa (61.31% preliminary, 69.84% final) and SVM (50.52% preliminary) is therefore not a like-for-like comparison on the same test set. The paper should either report the SVM's final evaluation score if it was obtained, or explicitly state that the SVM was not submitted to the final evaluation and discuss the implications for the claimed superiority of Su-RoBERTa over the classical baseline.
minor comments (6)
- [Section 2] There is a typo: 'post ans use level statistics' should read 'post and use level statistics.'
- [Section 4.1] The sentence 'This Tackled the class imbalance' has an unnecessary capitalization; it should be 'This tackled the class imbalance.'
- [Section 6] The term 'SOTA' is used without expansion on first use; it should be spelled out as 'state-of-the-art'.
- [Figure 1 caption] The caption refers to 'SuRoBERTa' while the model is named 'Su-RoBERTa' throughout the paper; the naming should be consistent.
- [Section 8] The conclusion states that the paper 'demonstrates the superiority of language models,' but the experiments only compare a base language model against an SVM and an unvalidated GPT-4 pseudo-labeler. The wording should be toned down to reflect what the evidence actually supports.
- [References] Reference [22] appears incomplete: it lacks a complete title and publication venue details. Please provide the full bibliographic information for the dataset paper.
Circularity Check
No significant circularity: the leaderboard F1 is externally evaluated, and the semi-supervised pipeline is not defined in terms of its own predictions.
full rationale
The paper's central quantitative claim—Su-RoBERTa's 69.84% weighted F1 and 10th place—is produced by the IEEE BigData 2024 competition's evaluation on a hidden test set, external to the paper's own definitions, so it cannot be circular. The semi-supervised pipeline (fine-tune on augmented labeled data, thresholded pseudo-labeling of unlabeled posts, retrain) is a standard self-training procedure; the final evaluation is on the competition leaderboard, not on the model's own pseudo-labels. The GPT-2 augmentation and RoBERTa fine-tuning are described with concrete generation and training details, not defined in terms of the target F1. The Section 6 comparison of Su-RoBERTa (50%) and SVM (32%) against GPT-4 few-shot pseudo-labels is methodologically weak—agreement with a pseudo-labeler is not accuracy—and it is the only place where the 'comparable to LLMs' claim is argued; however, it is not a circular reduction because neither model is trained on those pseudo-labels and no equation or fit forces the 50% figure. Author self-citations (e.g., [4], [14]–[18]) appear only in related-work context and are not load-bearing. No uniqueness theorem or ansatz is smuggled in via self-citation. Hence the derivation chain is self-contained with respect to its main result.
Assumptions & free parameters
free parameters (4)
- GPT-2 augmentation target counts =
Behavior=190, Indicator=190, Attempt=82
- Pseudo-label confidence threshold =
0.33
- Self-training iterations =
2
- Fine-tuning hyperparameters =
batch size=8, epochs=10, optimizer=AdamW
assumptions (3)
- domain assumption GPT-4 few-shot labels approximate the true class distribution of the test set closely enough for model comparison.
- domain assumption Suicide risk classes can be reliably inferred from single Reddit posts without user history or context.
- domain assumption Synthetic posts generated by GPT-2 and NLPAug preserve the intended class label for training.
Cite this review
Pith. "Pith review of Su-RoBERTa: A Semi-supervised Approach to Predicting Suicide Risk through Social Media using Base Language Models." pith.science (2026). https://pith.science/paper/VFL2ALO2
@misc{pith2026241201353,
author = {Pith},
title = {Pith review of: Su-RoBERTa: A Semi-supervised Approach to Predicting Suicide Risk through Social Media using Base Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/VFL2ALO2}},
note = {Machine review of arXiv:2412.01353}
}
read the original abstract
In recent times, more and more people are posting about their mental states across various social media platforms. Leveraging this data, AI-based systems can be developed that help in assessing the mental health of individuals, such as suicide risk. This paper is a study done on suicidal risk assessments using Reddit data leveraging Base language models to identify patterns from social media posts. We have demonstrated that using smaller language models, i.e., less than 500M parameters, can also be effective in contrast to LLMs with greater than 500M parameters. We propose Su-RoBERTa, a fine-tuned RoBERTa on suicide risk prediction task that utilized both the labeled and unlabeled Reddit data and tackled class imbalance by data augmentation using GPT-2 model. Our Su-RoBERTa model attained a 69.84% weighted F1 score during the Final evaluation. This paper demonstrates the effectiveness of Base language models for the analysis of the risk factors related to mental health with an efficient computation pipeline
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
- [1]
-
[2]
A Time-Aware Transformer Based Model for Suicide Ideation Detection on Social Media,
R. Sawhney, H. Joshi, S. Gandhi, and R. R. Shah, “A Time-Aware Transformer Based Model for Suicide Ideation Detection on Social Media,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y . He, and Y . Liu, Eds., Online: Association for Computational Linguistics, Nov. 2020, pp. 7685–7697...
-
[3]
Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data,
X. Xu et al., “Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data,” Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 8, no. 1, p. 31:1-31:32, Mar. 2024, doi: 10.1145/3643540
doi:10.1145/3643540 2024
-
[4]
A. Anand, C. Tank, S. Pol, V . Katoch, S. Mehta, and R. R. Shah, “Depression Detection and Analysis using Large Language Mod- els on Textual and Audio-Visual Modalities,” Jul. 08, 2024, arXiv: arXiv:2407.06125. doi: 10.48550/arXiv.2407.06125
-
[5]
Suicide Risk Assessment with Multi-level Dual-Context Language and BERT,
Matero et al., “Suicide Risk Assessment with Multi-level Dual-Context Language and BERT,” in Proceedings of the Sixth Workshop on Com- putational Linguistics and Clinical Psychology, K. Niederhoffer, K. Hollingshead, P. Resnik, R. Resnik, and K. Loveys, Eds., Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 39–44. doi: 10....
-
[6]
Robust suicide risk assessment on social media via deep adversarial learning,
R. Sawhney, H. Joshi, S. Gandhi, D. Jin, and R. R. Shah, “Robust suicide risk assessment on social media via deep adversarial learning,” Journal of the American Medical Informatics Association, vol. 28, no. 7, pp. 1497–1506, Jul. 2021, doi: 10.1093/jamia/ocab031
-
[7]
Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings,
H.-C. Shing, S. Nair, A. Zirikly, M. Friedenberg, H. Daum ´e III, and P. Resnik, “Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings,” in Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, K. Loveys, K. Niederhoffer, E. Prud’hommeaux, R. Resnik, and P. Resnik, Eds....
-
[8]
CLPsych 2019 Shared Task: Predicting the Degree of Suicide Risk in Reddit Posts,
A. Zirikly, P. Resnik, ¨O. Uzuner, and K. Hollingshead, “CLPsych 2019 Shared Task: Predicting the Degree of Suicide Risk in Reddit Posts,” in Proceedings of the Sixth Workshop on Computational Lin- guistics and Clinical Psychology, K. Niederhoffer, K. Hollingshead, P. Resnik, R. Resnik, and K. Loveys, Eds., Minneapolis, Minnesota: Association for Computat...
Show all 28 references
-
[9]
ConvSent at CLPsych 2019 Task A: Using Post-level Sentiment Features for Suicide Risk Prediction on Reddit,
K. Allen, S. Bagroy, A. Davis, and T. Krishnamurti, “ConvSent at CLPsych 2019 Task A: Using Post-level Sentiment Features for Suicide Risk Prediction on Reddit,” in Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology, K. Niederhoffer, K. Holl...
2019 doi
-
[10]
Predicting Suicide Risk from Online Postings in Reddit The UGent-IDLab submission to the CLPysch 2019 Shared Task A,
S. K. Bitew et al., “Predicting Suicide Risk from Online Postings in Reddit The UGent-IDLab submission to the CLPysch 2019 Shared Task A,” in Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology, K. Niederhoffer, K. Hollingshead, P. Resnik, R....
2019 doi
-
[11]
Stacked CNN - LSTM approach for prediction of suicidal ideation on social media,
B. Priyamvada et al., “Stacked CNN - LSTM approach for prediction of suicidal ideation on social media,” Multimedia Tools and Applica- tions, vol. 82, no. 18, Art. no. 18, Feb. 2023, doi: 10.1007/s11042- 023-14431-z
2023 doi
-
[12]
A self-attention TCN- based model for suicidal ideation detection from social media posts,
S. L. Mirtaheri, S. Greco, and R. Shahbazian, “A self-attention TCN- based model for suicidal ideation detection from social media posts,” Expert Systems with Applications, vol. 255, p. 124855, Dec. 2024, doi: 10.1016/j.eswa.2024.124855
2024
-
[13]
Knowledge-aware Assessment of Severity of Suicide Risk for Early Intervention,
M. Gaur et al., “Knowledge-aware Assessment of Severity of Suicide Risk for Early Intervention,” in The World Wide Web Conference, in WWW ’19. New York, NY , USA: Association for Computing Machinery, May 2019, pp. 514–525. doi: 10.1145/3308558.3313698
2019
-
[15]
GEC-DCL: Grammatical Error Correction Model with Dynamic Context Learning for Paragraphs and Scholarly Papers,
A. Anand et al., “GEC-DCL: Grammatical Error Correction Model with Dynamic Context Learning for Paragraphs and Scholarly Papers,” in Big Data and Artificial Intelligence, V . Goyal, N. Kumar, S. S. Bhowmick, P. Goyal, N. Goyal, and D. Kumar, Eds., Cham: Springer Nature Switzer...
2023 doi
- [16]
-
[17]
SciPhyRAG - Retrieval Augmentation to Improve LLMs on Physics Q &A,
A. Anand et al., “SciPhyRAG - Retrieval Augmentation to Improve LLMs on Physics Q &A,” in Big Data and Artificial Intelligence, V . Goyal, N. Kumar, S. S. Bhowmick, P. Goyal, N. Goyal, and D. Kumar, Eds., Cham: Springer Nature Switzerland, 2023, pp. 50–63. doi: 10.1007/978-3-0...
2023 doi
-
[18]
ExCEDA: Unlocking Attention Paradigms in Ex- tended Duration E-Classrooms by Leveraging Attention-Mechanism Models,
A. Anand et al., “ExCEDA: Unlocking Attention Paradigms in Ex- tended Duration E-Classrooms by Leveraging Attention-Mechanism Models,” in 2024 IEEE 7th International Conference on Multimedia In- formation Processing and Retrieval (MIPR), Aug. 2024, pp. 301–307. doi: 10.1109/MI...
2024
-
[19]
XinHai@CLPsych 2024 Shared Task: Prompting Healthcare-oriented LLMs for Evidence High- lighting in Posts with Suicide Risk,
J. Zhu, A. Xu, M. Tan, and M. Yang, “XinHai@CLPsych 2024 Shared Task: Prompting Healthcare-oriented LLMs for Evidence High- lighting in Posts with Suicide Risk,” in Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), A. Yates, B...
2024
-
[20]
Overview of the CLPsych 2024 Shared Task: Leverag- ing Large Language Models to Identify Evidence of Suicidality Risk in Online Posts,
J. Chim et al., “Overview of the CLPsych 2024 Shared Task: Leverag- ing Large Language Models to Identify Evidence of Suicidality Risk in Online Posts,” in Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), A. Yates, B. Desmet,...
2024
-
[21]
Utilizing Large Lan- guage Models to Identify Evidence of Suicidality Risk through Anal- ysis of Emotionally Charged Posts,
A. Y . Uluslu, A. Michail, and S. Clematide, “Utilizing Large Lan- guage Models to Identify Evidence of Suicidality Risk through Anal- ysis of Emotionally Charged Posts,” in Proceedings of the 9th Work- shop on Computational Linguistics and Clinical Psychology (CLPsych 2024), ...
2024
-
[22]
Suicide risk level prediction and suicide trigger detection: a benchmark dataset
J. Li et al., “Suicide risk level prediction and suicide trigger detection: a benchmark dataset”, Accessed: Aug. 31, 2024. [Online]. Available: https://www.hkie.org.hk/hkietransactions/article detail.php?id=911
2024
- [23]
-
[24]
Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations,
S. Kobayashi, “Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations,” May 16, 2018, arXiv: arXiv:1805.06201. Accessed: Oct. 07, 2024. [Online]. Available: http://arxiv.org/abs/1805.06201
2018 arXiv
-
[25]
Data Augmentation using Pre-trained Transformer Models,
V . Kumar, A. Choudhary, and E. Cho, “Data Augmentation using Pre-trained Transformer Models,” Jan. 31, 2021, arXiv: arXiv:2003.02245. Accessed: Oct. 07, 2024. [Online]. Available: http://arxiv.org/abs/2003.02245
2021 arXiv
-
[26]
A Multimodal Approach for Mania Level Prediction in Bipolar Disorder,
P. Baki, H. Kaya, E. C ¸ iftc ¸i, H. G¨ulec ¸, and A. A. Salah, “A Multimodal Approach for Mania Level Prediction in Bipolar Disorder,” IEEE Transactions on Affective Computing, vol. 13, no. 4, pp. 2119–2131, Oct. 2022, doi: 10.1109/TAFFC.2022.3193054
2022
-
[27]
Text-based Interpretable Depression Severity Modeling via Symptom Predictions,
F. Van Steijn, G. Sogancioglu, and H. Kaya, “Text-based Interpretable Depression Severity Modeling via Symptom Predictions,” in Proceed- ings of the 2022 International Conference on Multimodal Interaction, in ICMI ’22. New York, NY , USA: Association for Computing Ma- chinery,...
2022
-
[28]
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,
N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” arXiv.org. Accessed: Oct. 07, 2024. [Online]. Available: https://arxiv.org/abs/1908.10084v1
2024 arXiv
-
[29]
Unsupervised word sense disambiguation rivaling supervised methods,
D. Yarowsky, “Unsupervised word sense disambiguation rivaling supervised methods,” in Proceedings of the 33rd annual meeting on Association for Computational Linguistics, in ACL ’95. USA: Asso- ciation for Computational Linguistics, Jun. 1995, pp. 189–196. doi: 10.3115/981658.981684
1995
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.