{"id":"fbd8f6ab-f376-4634-9f05-ed792b41a498","arxiv_id":"2412.01353","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Su-RoBERTa, a semi-supervised RoBERTa classifier with GPT-2 data augmentation, achieves 69.84% weighted F1 on a suicide risk prediction benchmark.","lead":"This paper describes Su-RoBERTa, a fine-tuned base language model that predicts suicide risk levels from Reddit posts, reaching 69.84% weighted F1 on a hidden competition test set. It also argues that models under 500M parameters can be effective and efficient for mental health risk prediction, using GPT-2 augmentation and pseudo-labeling.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LLM-comparison claim rests on unvalidated GPT-4 pseudo-labels: the 50% agreement figure does not establish comparability, though the 69.84% leaderboard score stands.","rationale":"The central result, a 69.84% weighted F1 on the competition's hidden test set, is an externally scored number and is not threatened by the GPT-4 pseudo-labeling issue. The paper's abstract, however, explicitly claims effectiveness 'in contrast to LLMs with greater than 500M parameters,' and the only evidence for that contrast is the Section 6 comparison against GPT-4 pseudo-labels. That evidence is methodologically broken: a 50% agreement rate with GPT-4's labels is not a measure of relative classification performance, and the pseudo-labels themselves are unvalidated. An author-run validation of GPT-4 labeling on the 500 labeled training posts is feasible and would settle whether the pseudo-labels are trustworthy. Until then, the comparative claim should be downgraded or removed, and the paper should rely on the leaderboard result and the valid SVM-vs-Su-RoBERTa leaderboard comparison. This supports the reader's CONDITIONAL verdict: the core F1 claim can stand, but the LLM-comparison section needs either validation or removal.","tokens_in":9700,"tokens_out":6745,"duration_ms":56044,"concrete_test":"Run the same 4-shot GPT-4 prompt from Table 1 on a held-out stratified sample of 100 (or all 500) labeled training posts, and compare GPT-4's predicted labels with the ground-truth labels, reporting per-class precision/recall, weighted F1, and Cohen's kappa. Also compute Su-RoBERTa's agreement with GPT-4 on the same held-out posts. If GPT-4's weighted F1 on known labels is not substantially above chance, or if the Su-RoBERTa-vs-GPT-4 agreement on held-out data does not exceed chance agreement, then the Section 6 comparison cannot support the claim that Su-RoBERTa is comparable to a SOTA LLM.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6 uses GPT-4 few-shot labels (Table 1) as 'evaluation ground truth' for the 100 test posts, and the 50% agreement between Su-RoBERTa and those pseudo-labels is used to claim that a base model is 'comparable' to a SOTA LLM (Section 6, Figure 6). This inference is not valid: agreement with a pseudo-labeler is not accuracy against true labels, and the paper provides no evidence about GPT-4's labeling quality on this task. The authors themselves state GPT-4 'cannot be perfect' but do not validate it, even though 500 labeled training posts are available. If GPT-4's labels are biased or noisy (e.g., systematically confusing 'behavior' and 'indicator'), the 50% figure is uninterpretable and the abstract's 'in contrast to LLMs' claim is unsupported. The leaderboard weighted F1 of 69.84% (Table 2) is independent and credible, but the comparative contribution (contribution 3) collapses without a valid LLM baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Su-RoBERTa, a RoBERTa-base model (355M parameters) fine-tuned for a four-class suicide risk prediction task on Reddit posts from the IEEE BigData 2024 Suicide Risk Detection competition. The proposed pipeline combines GPT-2-based data augmentation for class imbalance, iterative pseudo-labeling of 1500 unlabeled training posts with a confidence threshold, and fine-tuning of RoBERTa. The authors report a weighted F1 score of 69.84% on the competition's final evaluation, which placed them 10th on the leaderboard. They also describe a classical SVM baseline that achieved 50.52% on a preliminary evaluation. The paper claims that base language models (under 500M parameters) can be effective and efficient alternatives to larger LLMs for mental health risk prediction, and uses agreement with GPT-4 pseudo-labels on 100 test posts to argue that Su-RoBERTa is comparable to a state-of-the-art LLM.","tokens_in":9874,"tokens_out":2122,"duration_ms":18901,"significance":"If the leaderboard result is taken at face value, the paper provides a useful data point: a moderately sized RoBERTa model, trained with a lightweight semi-supervised pipeline and GPT-2 augmentation, can achieve a competitive weighted F1 on a real-world suicide risk prediction benchmark. The reported 30-minute fine-tuning time on a single RTX 3090 is a genuine practical strength and supports the paper's emphasis on computational efficiency. The external leaderboard evaluation is a credible, non-circular source for the main performance claim. However, the paper's additional comparative claim against large LLMs rests on an invalid evaluation: agreement with GPT-4 pseudo-labels is not a measure of accuracy, and the authors provide no validation of GPT-4's labeling quality. This comparative contribution is therefore unsupported in its current form, and the paper's central scientific value is limited to the leaderboard score and pipeline description.","major_comments":[{"comment":"The evaluation of Su-RoBERTa against GPT-4 pseudo-labels is methodologically invalid as a measure of model quality. Agreement with a pseudo-labeler is not accuracy against true labels, and the paper provides no evidence that GPT-4's few-shot labels are reliable for these 100 test posts. The authors state that 'GPT-4 itself cannot be perfect for this specific task' but then treat its labels as 'evaluation ground truth.' This is especially problematic because 500 labeled training posts are available, which could have been used to estimate GPT-4's labeling accuracy (e.g., by comparing GPT-4 labels on a held-out subset of the labeled data against the true labels). Without such validation, the 50% agreement figure and the subsequent inference that Su-RoBERTa is 'comparable' to GPT-4 are uninterpretable. This directly affects Contribution 3 (Feasibility) and the abstract's claim about being effective 'in contrast to LLMs.'","section":"Section 6, Table 1 and Figures 6-7"},{"comment":"The pseudo-label confidence threshold (0.33) and the augmentation counts (Behavior and Indicator upsampled to 190, Attempt to 82) are described as being decided by 'empirical experimentations,' but no experiments, sensitivity analyses, or error bars are reported. These choices directly affect the amount of pseudo-labeled data used in self-training and the class balance of the augmented training set. Since the paper's central contribution is a semi-supervised pipeline, the lack of any analysis of how these hyperparameters affect the final F1 makes the method difficult to reproduce or trust as a general recipe. At minimum, the authors should provide the range of thresholds and augmentation settings explored and show that the final leaderboard result is not a fragile artifact of a single post hoc choice.","section":"Section 5.2 and Section 4.1"},{"comment":"Table 2 reports the SVM model's preliminary weighted F1 of 50.52% but leaves its final evaluation blank, with no explanation. The comparison between Su-RoBERTa (61.31% preliminary, 69.84% final) and SVM (50.52% preliminary) is therefore not a like-for-like comparison on the same test set. The paper should either report the SVM's final evaluation score if it was obtained, or explicitly state that the SVM was not submitted to the final evaluation and discuss the implications for the claimed superiority of Su-RoBERTa over the classical baseline.","section":"Section 7, Table 2"}],"minor_comments":[{"comment":"There is a typo: 'post ans use level statistics' should read 'post and use level statistics.'","section":"Section 2"},{"comment":"The sentence 'This Tackled the class imbalance' has an unnecessary capitalization; it should be 'This tackled the class imbalance.'","section":"Section 4.1"},{"comment":"The term 'SOTA' is used without expansion on first use; it should be spelled out as 'state-of-the-art'.","section":"Section 6"},{"comment":"The caption refers to 'SuRoBERTa' while the model is named 'Su-RoBERTa' throughout the paper; the naming should be consistent.","section":"Figure 1 caption"},{"comment":"The conclusion states that the paper 'demonstrates the superiority of language models,' but the experiments only compare a base language model against an SVM and an unvalidated GPT-4 pseudo-labeler. The wording should be toned down to reflect what the evidence actually supports.","section":"Section 8"},{"comment":"Reference [22] appears incomplete: it lacks a complete title and publication venue details. Please provide the full bibliographic information for the dataset paper.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's strongest asset is the externally evaluated leaderboard F1 of 69.84%, which is independent of the authors' internal evaluation choices. The main obstacle is the unvalidated GPT-4 pseudo-label comparison, which is presented as a central contribution but is not defensible as written. This is fixable by removing or substantially reframing the LLM-comparability claim, but as it stands the manuscript would not meet the bar for a rigorous journal publication. I recommend major revision rather than rejection because the core empirical result (the leaderboard score) is credible and the authors have a clear path to repair the evaluation narrative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's actual contribution is a single externally verified number: 69.84% weighted F1, 10th place on the IEEE BigData 2024 suicide risk leaderboard. That is a legitimate result, and the paper is worth a skim for that alone. Everything else in the abstract about base models being \"effective in contrast to LLMs\" does not follow from the evidence in the paper.\n\nWhat's new is the particular configuration: GPT-2 generation to upsample the minority classes, then self-training with RoBERTa using a 0.33 confidence threshold. All components are established, but the combination is concrete and reproducible enough that someone working on the same competition could replicate it. The paper also does the right thing by reporting training time (30 minutes on one RTX 3090) and parameter count, which supports the practical efficiency angle.\n\nThe soft spot is the Section 6 evaluation against GPT-4 pseudo-labels. The authors treated a few-shot GPT-4 labeling of the 100 test posts as ground truth and then reported 50% agreement for Su-RoBERTa and 32% for the SVM. Agreement with a pseudo-labeler is not accuracy, and no validation of the GPT-4 labels is offered, even though 500 labeled training posts could serve as a check. The authors even say GPT-4 \"cannot be perfect\" but proceed anyway. So the 50% versus 32% comparison is uninterpretable, and the \"comparable to SOTA LLM\" language is unsupported. The leaderboard score stands on its own, but it has no LLM baseline against it.\n\nMinor issues: no code, data, or seeds; the augmentation counts and confidence threshold are post hoc; no error bars. For a competition system paper, that is acceptable, but it caps the scientific value.\n\nOverall, this is a reasonable system report for the IEEE BigData competition. The main result is credible and the efficiency story is fine. The GPT-4-based model comparison should be either removed or properly validated. I would send it to a referee, but with a clear request to fix that comparison before it can be used as evidence. I would cite the leaderboard score if I needed a baseline, but not the comparison.","headline":"The externally verified leaderboard score is credible, but the GPT-4 pseudo-label comparison does not support the 'base models vs LLMs' claim.","tokens_in":10454,"tokens_out":2936,"would_cite":false,"duration_ms":25301,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 355M-parameter RoBERTa model, augmented with GPT-2-generated posts and pseudo-labeled Reddit data, reaches a 69.84% weighted F1 score on a hidden suicide-risk test set.","keywords":["suicide risk prediction","semi-supervised learning","RoBERTa","GPT-2 augmentation","pseudo-labeling","mental health","social media","class imbalance"],"falsifier":"Obtain expert clinician labels for the 100 withheld test posts (or an equivalent expert-annotated Reddit set) and recompute Su-RoBERTa's weighted F1 against those labels. If the score falls far below 69.84%, or if GPT-4's pseudo-labels match the expert labels no better than Su-RoBERTa matches GPT-4 (50%), then the paper's base-model-versus-LLM comparison loses its support.","tokens_in":9476,"feed_emoji":"🧠","tokens_out":8468,"duration_ms":67955,"temperature":0.7,"pith_summary":"The paper tries to establish that a language model with fewer than 500 million parameters can be an effective and practical tool for classifying suicide risk from social media posts, not just a poor substitute for larger models. It introduces Su-RoBERTa, a fine-tuned RoBERTa model that uses GPT-2-generated examples to balance a skewed Reddit dataset and iteratively pseudo-labels 1,500 unlabeled posts to grow its training set. On the hidden test set of the IEEE BigData 2024 suicide-risk detection competition, Su-RoBERTa scored a 69.84% weighted F1 and ranked 10th, after about 30 minutes of fine-tuning on a single 24 GB GPU. The authors present this as evidence that smaller base models can deliver deployable performance on sensitive mental-health text tasks where large-language-model compute is unavailable.","feed_headline":"Small model hits 69.84% F1 on suicide-risk Reddit posts","feed_subtitle":"A 355M-parameter RoBERTa with GPT-2 augmentation and pseudo-labeling ranks 10th on the BigData 2024 leaderboard.","key_machinery":"The pipeline is carried by three components: (1) GPT-2-base (124M) fine-tuned to generate synthetic posts, upsampling Behavior and Indicator to 190 samples each and Attempt to 82; (2) a 355M-parameter RoBERTa classifier fine-tuned for four-class suicide-risk prediction on the resulting 652-sample labeled set; and (3) an iterative self-training loop that pseudo-labels the 1,500 unlabeled posts only when predicted probability exceeds 0.33, retrains the model from scratch twice, and finally trains on all 2,000 available posts. The competition's weighted F1 metric defines success. The authors report that the full fine-tuning pipeline runs in about 30 minutes on a single 24 GB GPU.","core_discovery":"Su-RoBERTa is a 355M-parameter RoBERTa classifier trained through a two-stage semi-supervised pipeline. In the first stage, a 124M-parameter GPT-2 model is fine-tuned to generate synthetic posts that raise the minority classes: Behavior and Indicator are upsampled to 190 samples each and Attempt to 82, bringing the labeled training set from 500 to 652 posts. In the second stage, the RoBERTa model is fine-tuned on the augmented labels, used to pseudo-label the 1,500 unlabeled posts with a confidence threshold above 0.33, and retrained from scratch on the expanded corpus for two iterations. The final model reaches 69.84% weighted F1 on the competition's unrevealed test set, placing 10th. Against a GPT-4 few-shot labeling of the same 100 test posts, Su-RoBERTa agrees with GPT-4 on 50% of samples while an SVM baseline agrees on 32%, which the paper uses to argue that a base model captures much of the signal of a state-of-the-art LLM at a fraction of the compute.","pith_inferences":["The GPT-4 agreement numbers are best read as an internal sanity check rather than a formal evaluation, since the true test labels are hidden and the paper concedes GPT-4 'cannot be perfect.'","An ablation the authors do not run—training without GPT-2 augmentation, without pseudo-labels, or with a different confidence threshold—would isolate which component actually drives the 69.84% F1.","The same pipeline could plausibly transfer to other imbalanced clinical-text tasks such as depression or anxiety detection, but it should first be validated on an expert-annotated held-out set to ensure pseudo-label noise does not distort risk categories.","A direct deployability check would measure inference latency and memory footprint on a mobile CPU, which the paper motivates but does not report."],"forward_implications":["Sub-500M base models can place on a competitive leaderboard for suicide-risk classification, making the approach viable in settings without large GPU budgets.","Generative data augmentation plus confident pseudo-labeling offers a reusable recipe for other class-imbalanced mental-health text tasks.","A roughly 30-minute fine-tuning pipeline means the model can be retrained rapidly as new social-media posts arrive.","The 50% agreement with GPT-4 pseudo-labels, while not an official metric, suggests the base model captures much of the label signal an LLM grader would apply to these four risk categories.","The small model size and training cost bring mobile or edge deployment closer, though the paper notes real-time use still needs improvement."],"supporting_citations":[{"why":"Supplies the competition dataset: 2,100 Reddit posts with 500 labeled, 1,500 unlabeled, and 100 hidden test posts.","marker":"[22]"},{"why":"Provides the RoBERTa architecture that Su-RoBERTa fine-tunes as its base model.","marker":"[23]"},{"why":"Gives the contextual augmentation technique used in the classical baseline's data augmentation.","marker":"[24]"},{"why":"Supports using pre-trained Transformer models to generate augmented training sentences for minority classes.","marker":"[25]"},{"why":"Supplies the Sentence-BERT embeddings used to build the SVM semi-supervised baseline.","marker":"[28]"},{"why":"Provides the classic self-training algorithm that the iterative pseudo-labeling loop adapts.","marker":"[29]"},{"why":"Defines the CLPsych shared-task framing and evaluation conventions that the suicide-risk four-class problem follows.","marker":"[8]"}],"fun_headline_variants":["Su-RoBERTa: 355M params, 69.84% F1, semi-supervised","Base RoBERTa with GPT-2 augmentation hits 69.84% F1","Semi-supervised RoBERTa ranks 10th on suicide-risk leaderboard","Small model, GPT-2 boost: 69.84% F1 on suicide risk","355M-param RoBERTa tops 69% F1 on suicide detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison that supports the paper's claim that Su-RoBERTa is comparable to a state-of-the-art LLM assumes that GPT-4's few-shot labels on the 100 withheld test posts are a trustworthy stand-in for the true hidden labels; the paper explicitly concedes GPT-4 'cannot be perfect' and says no validation against the real labels is possible.","fun_headline_variants_meta":{"raw":{"variants":["Su-RoBERTa: 355M params, 69.84% F1, semi-supervised","Base RoBERTa with GPT-2 augmentation hits 69.84% F1","Semi-supervised RoBERTa ranks 10th on suicide-risk leaderboard","Small model, GPT-2 boost: 69.84% F1 on suicide risk","355M-param RoBERTa tops 69% F1 on suicide detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000517,"raw_usage":{"total_tokens":2522,"prompt_tokens":973,"completion_tokens":1549,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":1434}},"tokens_in":589,"tokens_out":1549,"duration_ms":11253,"temperature":1.0,"reasoning_tokens":1434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:25:26.859853+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Obtain expert clinician labels for the 100 withheld test posts (or an equivalent expert-annotated Reddit set) and recompute Su-RoBERTa's weighted F1 against those labels. If the score falls far below 69.84%, or if GPT-4's pseudo-labels match the expert labels no better than Su-RoBERTa matches GPT-4 (50%), then the paper's base-model-versus-LLM comparison loses its support.","supporting_citations":[{"cited_title":"Suicide risk level prediction and suicide trigger detection: a benchmark dataset","cited_arxiv_id":null,"evidence_quote":"Supplies the competition dataset: 2,100 Reddit posts with 500 labeled, 1,500 unlabeled, and 100 hidden test posts."}],"review_version":1}