{"id":"5880665a-8bcb-4e07-b0d1-3ab017de560a","arxiv_id":"2502.09640","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new Spanish YouTube-comment dataset for social support detection is introduced, with benchmarks showing GPT-4o best for binary support detection and a GPT-4o-balanced dataset best for group and multiclass support subtasks.","lead":"This paper introduces a Spanish-language dataset of 3,189 YouTube comments labeled for social support, and benchmarks machine learning, deep learning, transformer, and GPT-4o models on three classification subtasks. The authors report that a balanced dataset built with GPT-4o paraphrases improves results for group-directed support subtasks, while GPT-4o wins the binary support detection task.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Balanced-dataset gains in Subtasks 2/3 depend on unvalidated GPT-4o paraphrase labels; for minority classes (e.g., Black Community: 13 training originals vs 192 balanced) the training set is ~93% synthetic, so the reported macro-F1 improvements may be an artifact of paraphrase fidelity rather than…","rationale":"Read in good faith: the paper's contribution is a Spanish YouTube support dataset and a broad benchmark. The central quantitative claim is that GPT-4o-based oversampling improves macro F1 on Subtasks 2 and 3. The weakest load-bearing condition is that the synthetic paraphrases are valid instances of their original class and target. This is not established anywhere in the manuscript. The risk is not abstract: for the smallest Subtask 3 classes, the balanced training set is overwhelmingly synthetic (Black Community ~93%, Nation ~86%, Women ~83% based on Tables 1-2), while the test set has 3, 9, and 8 instances respectively. Thus the reported gain of +0.0184 macro F1 for Subtask 3 is fragile and could easily be driven by GPT-4o's paraphrase style rather than by the social-support signal. This is not an attack on the use of LLM augmentation per se, but on the absence of validation of the key assumption. The internal Task 3 prose/table discrepancy (0.72 vs 0.8361) reinforces the need for a cleaned, released artifact. If the authors provide human-validated paraphrase labels and show the balanced gain survives on verified synthetic data, the conditional acceptance is justified; if the validation fails or the data remain unavailable, the headline claim should be rejected. I therefore keep the reader's CONDITIONAL verdict and agree with the reader's weakest-assumption identification, while adding concrete per-class arithmetic that sharpens why the assumption is load-bearing.","tokens_in":16448,"tokens_out":11235,"duration_ms":99418,"concrete_test":"Draw a stratified random sample of at least 100 GPT-4o-generated paraphrases from the balanced training set, oversampling Black Community, Nation, and Women. Have two independent native-Spanish annotators re-apply the Section 4.3 guidelines to the paraphrases, blind to the GPT-4o origin, and measure exact-label preservation (Subtask 1/2/3 category match) and Cohen's kappa. If preservation for any minority class is below 90%, or if retraining robertuito-sentiment-analysis on only human-verified paraphrases does not reproduce the Table 13 macro-F1 gains (0.8894 for Subtask 2, 0.8361 for Subtask 3), the balanced-dataset improvement is a GPT-4o paraphrase artifact, not a genuine social-support effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.6 ('Balanced Data Set') states that GPT-4-O generated paraphrased comments to oversample underrepresented classes, but no human validation of the paraphrase labels or target categories is reported. The effect is concentrated where it matters most: from Table 1 and Table 2, after the 20% test split, Black Community has only ~13 original training comments, scaled to 192 training instances, i.e., ~93% synthetic; Nation goes from ~26 to 192 (~86% synthetic) and Women from ~33 to 192 (~83% synthetic). Subtask 3's test set is also tiny (Black Community = 3, Nation = 9, Women = 8), so the headline gap between balanced macro F1 0.8361 (Table 13) and unbalanced 0.8177 (Table 11) can be moved by a handful of test predictions and is largely a statement about GPT-4o's paraphrase distribution. If the paraphrases preserve topic words but subtly shift the support target (e.g., from 'Black community' to generic 'people') or change the support category, the model learns synthetic regularities that need not transfer to original YouTube comments. The paper reports no check of this. (There is also an internal inconsistency: Section 6.5 prose says balanced Task 3 macro F1 'drops to 0.72' while Table 13 and Table 15 both support 0.8361; this must be corrected.) Because the central claim that the balanced dataset improved Subtasks 2 and 3 rests on this unvalidated synthetic oversampling, the claim is only as strong as the label-fidelity assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a Spanish-language dataset of 3,189 YouTube comments annotated for online social support detection under a three-level hierarchy: Subtask 1 (supportive vs. non-supportive), Subtask 2 (individual vs. group support), and Subtask 3 (six group categories: Nation, Other, LGBTQ, Black Community, Women, Religion). The authors benchmark traditional machine learning, deep learning, transformer, and GPT-4o models, and use GPT-4o paraphrases to create a balanced training set for a selected transformer model (pysentimiento/robertuito-sentiment-analysis). They report that the balanced dataset improves macro F1 for Subtasks 2 and 3, while GPT-4o achieves the best macro F1 for Subtask 1. The paper claims to provide the first Spanish social support detection dataset and accompanying benchmark.","tokens_in":108,"tokens_out":6078,"duration_ms":92875,"significance":"The dataset and task definition fill a genuine gap: positive and supportive content on Spanish-language social media is underexplored relative to hate speech detection. The paper provides explicit annotation guidelines, reports inter-annotator agreement (Cohen's kappa 0.84, 0.78, 0.62 for the three subtasks), and evaluates a broad range of models including traditional ML, deep learning, transformers, and GPT-4o. If the balanced-dataset improvements were rigorously established, the work would be a useful resource for the community. However, the headline claim currently rests on unvalidated synthetic oversampling and evaluation settings that are not fully comparable or statistically supported, so the significance is conditional on the authors addressing these issues.","major_comments":[{"comment":"The balanced dataset is created by GPT-4o paraphrases of training comments, but no human validation is reported for whether the paraphrases preserve the original Subtask 1 label, Subtask 2 individual/group target, or Subtask 3 category. For the smallest classes, the augmented training set is overwhelmingly synthetic: Black Community has roughly 13 original training comments (16 total minus the 3 held out for testing) but 192 training instances after oversampling, i.e., about 93% synthetic; Nation and Women are similarly about 86% and 83% synthetic. The reported macro-F1 gains for Subtasks 2 and 3 could therefore reflect properties of the GPT-4o paraphrase distribution rather than a genuine signal about social support. The authors should validate the paraphrase labels on a sample (e.g., annotator agreement on the original annotations) or compare against a non-generative oversampling baseline such as class weights or simple duplication.","section":"Section 5.6, Tables 1 and 2"},{"comment":"The unbalanced transformer results in Tables 9-11 appear to come from 5-fold cross-validation, as stated in Section 5.1, while the balanced results in Table 13 come from a single 80/20 train/test split, as described in Section 5.6. These are not directly comparable evaluation protocols. To support the central claim that the balanced dataset improves Subtask 2 and Subtask 3, the authors must report both settings under the identical protocol, including fold-level means and standard deviations (or confidence intervals) and, ideally, a paired significance test for the balanced-vs-unbalanced macro-F1 difference.","section":"Section 5.1 vs. Section 5.6 and Tables 9-13"},{"comment":"The Subtask 3 test set is extremely small for the minority classes: Black Community has 3 test samples, Women has 8, and Nation has 9, from Table 2. With test sets this small, a single prediction change can move per-class F1 substantially, and the macro-F1 gap between the balanced (0.8361) and unbalanced (0.8177) Subtask 3 results is fragile. The paper should report per-class test sample counts next to the balanced results and provide confidence intervals for macro-F1 (e.g., bootstrap or exact binomial per class). Table 15's Black Community F1 of 0.8000 is computed from only 3 test instances, which should be acknowledged explicitly.","section":"Table 2, Tables 13-15"},{"comment":"The prose in Section 6.5 states that for Task 3 'the macro F1-score drops to 0.72,' while Table 13 reports a macro F1 of 0.8361 for the same task and setting; the text also gives accuracy as 88% while the table reports 0.8850. This is an internal inconsistency. If the 0.72 value is correct, the balanced-dataset claim for Subtask 3 is not supported by the table; if Table 13 is correct, the prose is misleading. The discrepancy must be resolved and the reporting made consistent before the paper can be evaluated fairly.","section":"Section 6.5, Table 13"}],"minor_comments":[{"comment":"The heading 'Model training and Evalution' contains a typo; it should read 'Evaluation.'","section":"Section 5 heading"},{"comment":"The annotators were given 'a set of 100 sample tweets,' but the dataset consists of YouTube comments; this should say 'sample comments' to avoid confusion.","section":"Section 4.2"},{"comment":"There are several typographical errors that should be corrected: 'recieved' in Section 4.6, 'transformel' in the Introduction, 'obertuito-sentiment-analysis' in Section 6.6, and 'di fferent' in several places.","section":"Throughout"},{"comment":"The column headers in Table 14 are crowded ('Weighted Macro Accuracy Precision Recall F1-score Precision Recall F1-score'). The table should be reformatted so that the weighted and macro blocks are clearly separated, matching the structure of Tables 9-12.","section":"Table 14"},{"comment":"The dataset is described as available 'upon request from the corresponding author.' Given that the dataset is the paper's main contribution, the authors should provide a persistent public repository or a formal data availability statement with a DOI or URL.","section":"Availability of data and materials"}],"recommendation":"major_revision","confidential_remarks":"The related-work section cites two arXiv preprints (Ahani et al., 2024; Kolesnikova et al., 2025) that appear to be by the same research group and likely overlap in authorship with the present submission. The editors may wish to verify that the 'first Spanish dataset' claim is consistent with those papers, and that the work is not fragmenting a single contribution across multiple preprints. More substantively, the major comments concern the load-bearing evidence for the headline finding; if the paraphrase fidelity cannot be validated or a matched evaluation protocol with uncertainty estimates cannot be provided, the balanced-dataset claim should be substantially softened or removed. The tiny test-set issue is especially important for Subtask 3, where the reported gain may not survive a few alternative test-split draws."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing worth taking from this paper is the Spanish social support detection dataset: 3,189 YouTube comments, annotated in three levels with reasonable Cohen's kappa (0.84/0.78/0.62). That is a genuinely new resource for an underserved language and task. The benchmark sweep is also thorough—traditional ML, deep learning, transformers, and GPT-4o—so you get a usable map of what works on this data.\n\nThe soft spot is exactly where the stress-test note lands. The headline claim that balancing improves Subtasks 2 and 3 comes from one transformer model trained on a dataset where minority classes were oversampled by GPT-4o paraphrases. No human or automated check confirms those paraphrases kept the original target and support category. For Black Community, training goes from ~13 original comments to 192 synthetic-scaled instances; about 93% of that class is machine-generated. Test sets are minuscule (Black Community has 3 test samples, Women 8, Nation 9). The macro-F1 gap between balanced (0.8361) and unbalanced (0.8177) could swing on a few predictions, and the model may just be learning GPT-4o's paraphrase distribution rather than a robust signal about social support. The paper also has an internal inconsistency: Section 6.5's prose says Subtask 3 macro F1 drops to 0.72, while Table 13 and Table 15 report 0.8361. That needs correcting. There are also smaller glitches (annotator gender, tweets vs. comments, no public data/code).\n\nNone of this is fatal. The dataset, if released, and the annotation scheme are useful. The balanced-dataset experiment is a reasonable idea, but as reported it is more a proof-of-concept than a settled result.\n\nWho is this for? Researchers working on Spanish NLP, computational social science, and positive-content detection. A serious referee should engage with it—the dataset deserves scrutiny and the claim needs to be pinned down with released data, label-fidelity checks, more models, and confidence intervals. I would send it to review with a clear request for major revision.\n\nMy own verdict: conditionally useful, not yet convincing in its main claim. Worth bringing to reading group for a discussion of synthetic oversampling pitfalls.\n\nBest,\n[You]","headline":"The new Spanish dataset is the real contribution; the balanced-dataset claim rests on unvalidated synthetic paraphrases and tiny test sets.","tokens_in":17337,"tokens_out":1609,"would_cite":false,"duration_ms":16697,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A GPT-4o-balanced dataset improves fine-grained social support detection in Spanish, with macro F1 reaching 0.8894 for individual vs.","keywords":["online social support","Spanish social media","YouTube comments","GPT-4o paraphrasing","class imbalance","transformer models","macro F1","multiclass classification"],"falsifier":"Have three fresh annotators re-label a random sample of the GPT-4o paraphrased comments using the same guidelines, and compare their labels to the originals; substantial label drift would undermine the balanced-dataset results. A complementary test is to train on the synthetic-balanced data and evaluate on a held-out set of real, unparaphrased YouTube comments from new videos and see whether the macro F1 gains persist.","tokens_in":16199,"feed_emoji":"💬","tokens_out":5246,"duration_ms":40901,"temperature":0.7,"pith_summary":"This paper argues that online social support in Spanish social media can be detected automatically, and that the main obstacle is data imbalance rather than the difficulty of the task. To show this, the authors built the first annotated Spanish dataset of 3,189 YouTube comments, labeled at three levels: support vs. non-support, individual vs. group support, and six group-target categories. They used GPT-4o to paraphrase minority-class comments into a balanced training set, then compared traditional machine learning, deep learning, transformers, and GPT-4o. On the original unbalanced data, GPT-4o reached the best macro F1 of 0.8531 for the binary support task. On the balanced data, the robertuito-sentiment-analysis transformer reached macro F1 of 0.8894 for individual vs. group and 0.8361 for the six-category group task, which the paper presents as evidence that balancing fixes minority-class recall.","feed_headline":"Balanced data lifts Spanish support detection to 0.89 F1","feed_subtitle":"GPT-4o paraphrases fix class imbalance, beating prior models on two of three subtasks.","key_machinery":"The load-bearing object is the three-level annotation scheme, in which each comment is first labeled Social Support or Non-Support, supportive comments are then labeled Individual or Group, and group-support comments are finally labeled with one of six targets: Nation, Other, LGBTQ, Black Community, Women, and Religion. Around that scheme, the method that carries the argument is GPT-4o oversampling: the model writes paraphrased versions of underrepresented comments to balance the training set, producing 2017/2017 examples for Subtask 1, 397/397 for Subtask 2, and 192 per class for Subtask 3. The evaluation then rests on macro F1 as the primary metric, chosen so that minority classes count equally.","core_discovery":"The paper's central claim is that a GPT-4o-generated balanced dataset improves fine-grained social support classification in Spanish, while the binary support decision is best handled by GPT-4o itself. Concretely, rebalancing the training set by paraphrasing underrepresented classes lifts macro F1 for the Individual vs. Group subtask to 0.8894 and for the six group categories (Nation, Other, LGBTQ, Black Community, Women, Religion) to 0.8361, both with the robertuito-sentiment-analysis transformer. For the first subtask, distinguishing supportive from non-supportive comments, GPT-4o achieves the best macro F1 of 0.8531 on the original unbalanced dataset. The paper positions these results as evidence that automated social support detection in Spanish is feasible, that LLM paraphrasing can substitute for collecting more real examples of minority classes, and that the field should pay attention to positive content as the counterpart to hate speech.","pith_inferences":["A testable extension: re-annotate a sample of the GPT-4o paraphrases by humans to check label drift; without that check, part of the balanced-data gain could be an artifact of synthetic language style.","The balanced-dataset comparison was run with only one transformer, so the gain's generality across architectures is an open question the paper does not settle.","If label-preserving paraphrasing transfers, the same pipeline could be applied to other under-resourced languages and platforms, but YouTube's comment register may not generalize to Twitter or Reddit.","The paper implicitly treats social support as the positive counterpart to hate speech, suggesting detection tools could be paired in a single moderation pipeline; that pairing is not built or tested here."],"forward_implications":["LLM-generated paraphrases can act as a data-balancing method for NLP classification when real minority examples are scarce.","A Spanish-tuned sentiment transformer (robertuito-sentiment-analysis) is a strong starting point for Spanish support detection on group-level tasks.","Support detection can be done not just as a binary yes/no but as a fine-grained target-community classification (LGBTQ, Women, Nation, Religion, Black Community, Other).","GPT-4o alone handles the binary support decision well in Spanish, suggesting LLMs can be used for coarse support screening.","Automated support detection could feed tools that surface or encourage supportive comments, complementing hate-speech filters."],"supporting_citations":[{"why":"Supplies the definition of social support used in the annotation guidelines.","marker":"(Xia et al., 2012)"},{"why":"Defines online social support and grounds the task in the broader literature on digital support.","marker":"(Ma et al., 2024)"},{"why":"Introduces the earlier social support detection task and results this paper extends to Spanish.","marker":"(Ahani et al., 2024)"},{"why":"Provides the same-task LLM and transformer baselines this paper compares against.","marker":"(Kolesnikova et al., 2025)"},{"why":"Shows BERT-based classification of self-disclosure and social support on Reddit, the nearest prior method.","marker":"(Zou et al., 2024)"}],"fun_headline_variants":["GPT-4o balancing boosts Spanish support detection","Spanish AI support detection gets GPT-4o data boost","Balanced GPT-4o data lifts Spanish support detection","LLM paraphrases improve Spanish support classification","Support detection in Spanish improves with GPT-4o data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole balanced-dataset result rests on GPT-4o's paraphrases preserving the original labels and the fine-grained target of each supportive comment; if a paraphrase shifts the meaning or target, the reported improvements could come from synthetic wording rather than real signal.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o balancing boosts Spanish support detection","Spanish AI support detection gets GPT-4o data boost","Balanced GPT-4o data lifts Spanish support detection","LLM paraphrases improve Spanish support classification","Support detection in Spanish improves with GPT-4o data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000644,"raw_usage":{"total_tokens":2968,"prompt_tokens":957,"completion_tokens":2011,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":1945}},"tokens_in":573,"tokens_out":2011,"duration_ms":12521,"temperature":1.0,"reasoning_tokens":1945,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:35:51.292853+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have three fresh annotators re-label a random sample of the GPT-4o paraphrased comments using the same guidelines, and compare their labels to the originals; substantial label drift would undermine the balanced-dataset results. A complementary test is to train on the synthetic-balanced data and evaluate on a held-out set of real, unparaphrased YouTube comments from new videos and see whether the macro F1 gains persist.","supporting_citations":[],"review_version":1}