{"id":"cfe29e3e-6bb3-4c03-8631-2b651a05eced","arxiv_id":"2411.13008","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLMs show improved directionality detection after fine-tuning but perform near chance at cyberbullying and anti-bullying classification, pointing to weak semantic understanding of informal social media language.","lead":"This paper tests whether large language models can understand social media dynamics by asking them to paraphrase informal comments, trace who replies to whom, and flag cyberbullying and anti-bullying posts. It finds that fine-tuning helps with directionality but that semantic understanding of informal language remains weak, with cyberbullying detection near random.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Directionality result in Table 2 is not trustworthy as reported: Section 5.2 gives no train/test split even though Section 6 identifies the 4chan corpus as Phase 2 PEFT training data, so the PEFT gains may be memorization.","rationale":"Good-faith reading: the paper's contribution is an evaluation of three capabilities, and the headline conclusion is that semantic understanding is the weak link while directionality can be improved. The directionality claim is therefore integral, not incidental. The missing split in §5.2 is a concrete, checkable threat: Table 2's improvements are so large that they could easily be training-set memorization, especially given the model's tendency to copy text noted in §4.2. The manuscript's own Section 6 shows the authors knew to use a separate dataset (Instagram) to avoid leakage for the bullying task, which makes the absence of this precaution for directionality conspicuous. The paraphrase-metric issue raised by the reader is real but partially mitigated: the paper already complements semantic similarity with Levenshtein ratio and distributional analysis, and the semantic weakness is supported by the near-random classification in Table 3. Thus I would keep the CONDITIONAL verdict, but the condition should explicitly include releasing the directionality split. If the split does not exist, the verdict should move to REJECT for the directionality portion of the central claim.","tokens_in":12610,"tokens_out":6420,"duration_ms":60285,"concrete_test":"Require the authors to provide disjoint thread IDs for Phase 2 PEFT training and Table 2 evaluation, and recompute target/reply accuracy on held-out 4chan threads. As a stronger check, run the same directionality task on the 100 Instagram sessions (already used for Section 6, outside the 4chan training data) and compare PEFT vs. base models; if the PEFT advantage disappears on either held-out set, the claim that fine-tuning induces directional understanding is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central positive claim—that PEFT produces promising directional understanding (Table 2: 7B target accuracy 0.082→0.865, 13B 0.159→0.971)—rests on an experiment in §5.2 that reports no train/test split. The 4chan corpus described in §5.2 is the same corpus later identified in §6 as the Phase 2 PEFT training data ('Phase 2 of our PEFT is based on 4chan threads'). If the directionality evaluation used threads also seen during fine-tuning, the large gains are explained by memorization, not by learned structural reasoning. This is more load-bearing than the paraphrase-metric concern: the semantic weakness finding is independently corroborated by the near-random classification results in Table 3, whereas the directionality result has no independent support. The reader's rationale did note the missing split, but the stated weakest_assumption centered on paraphrase similarity; the unresolved split is the more serious threat to the paper's overall contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates the ability of generative LLMs (GPT-2, Llama-2 7B/13B, and ChatGPT) to understand social media dynamics, using cyberbullying and anti-bullying as case studies. It considers three tasks: paraphrasing informal social media comments (Section 4), identifying directionality of replies in 4chan threads (Section 5), and classifying Instagram comments as cyberbullying or anti-bullying (Section 6). The authors use prompt engineering, exemplars, and two-phase LoRA-based PEFT. They report that fine-tuned models show promising performance on directionality (Table 2), but that all models struggle with semantic understanding and perform near chance on cyberbullying/anti-bullying classification (Tables 1 and 3). The central conclusion is that the weakest link is the lack of semantic understanding of informal social media language.","tokens_in":12836,"tokens_out":4933,"duration_ms":43734,"significance":"If the findings are valid, this paper provides a useful empirical benchmark for LLM capabilities in a socially important setting, distinguishing between structural (directionality) and semantic understanding. The study covers multiple models, prompt variants, and fine-tuning strategies, and it makes a credible case that existing LLMs are not reliable for semantic analysis of informal social media discourse. The paper also highlights a practical problem: closed models like ChatGPT cannot be used for constrained generation on toxic content, limiting their applicability. However, the current evidence is weakened by several experimental gaps, so the contribution is promising but not yet fully established.","major_comments":[{"comment":"The directionality results in Table 2 are not interpretable without a train/test split. Section 5.2 states that the 4chan corpus is used for the directionality analysis, and Section 6 later identifies the same 4chan corpus as the basis for 'Phase 2' of PEFT. If the evaluation in Table 2 is performed on threads that were also used for fine-tuning, the reported gains (7B target accuracy 0.082→0.865; 13B 0.159→0.971) may reflect memorization rather than improved structural understanding. Please report the split between training and evaluation threads, the number of threads and target/reply instances, and confirm that no evaluation thread appears in the PEFT training data. If no split exists, the experiment must be re-run on held-out threads before the directionality claim can be accepted.","section":"§5.2 and §6"},{"comment":"The paraphrase-quality evaluation lacks a human reference. All similarity metrics (BLEU, ROUGE, Jaccard, BERT semantic similarity) are computed between the model generation and the original comment, rather than between the generation and human reference paraphrases. As the authors note, verbatim reproduction can therefore inflate all of these scores, and the Levenshtein ratio is only a partial remedy. In particular, the claim that ChatGPT 'shows a robust understanding of social context' (Section 4.2) is not warranted without human-written reference paraphrases or a human evaluation of the generated paraphrases. Please add a human reference set or at least a human rating study.","section":"§4.1, §4.2, Table 1"},{"comment":"The cyberbullying/anti-bullying experiment is severely under-powered and lacks statistical support. Only 100 labeled sessions are used, and the paper does not report the number of comments per class, confidence intervals, or any significance test. Since all accuracy values in Table 3 are within 0.513 of 0.5, the conclusion that the models 'perform close to random chance' is not statistically grounded. Please report the number of test instances, class distribution, and error bars or a binomial test, and ideally increase the evaluation set.","section":"§6, Table 3"},{"comment":"The fine-tuning configuration is not described in sufficient detail to assess or reproduce the PEFT results. Section 3.3 mentions LoRA and an Alpaca mix probability, but does not give the LoRA rank r, learning rate, batch size, number of epochs, or the exact mix probability. Section 5.2 does not state how many 4chan threads were used for Phase 2, nor how the structural Phase 1 data (WikiTableQuestions) was combined. Please provide these hyperparameters and data statistics.","section":"§3.3, §5.2"}],"minor_comments":[{"comment":"The phrase 'a understanding' should be 'an understanding'.","section":"Abstract"},{"comment":"The metric is 'ROUGE', not 'ROGUE', throughout the section and tables.","section":"§4.1"},{"comment":"There is a typo: 'Levenshein' should be 'Levenshtein'.","section":"§4.2"},{"comment":"The word 'dateset' should be 'dataset'.","section":"§4.2"},{"comment":"The Alpaca instruction data is cited to reference [36], which is the LLaMA paper; please cite the actual Alpaca dataset (Taori et al., 2023).","section":"§3.3"},{"comment":"The violin plot axes are not fully described; please clarify what quantity is on each axis and how the distribution is computed.","section":"Fig. 2"},{"comment":"Please clarify whether the '100 labeled sessions' means 100 Instagram posts with all their comments, and report the total number of comments used for the binary classification.","section":"§6"},{"comment":"The term 'JORA' is used without definition; while reference [34] is given, a brief description of this fine-tuning library would improve readability.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a timely and important evaluation question, and the separation of semantic and structural reasoning is a useful framing. However, the missing train/test split for the directionality experiment is the most serious issue; if the authors can demonstrate that the evaluation was on held-out data, or re-run it accordingly, the central positive claim can be salvaged. The paraphrase evaluation and small-sample classification experiment also need strengthening before the paper would meet the standards of a serious journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. This paper is a reasonably honest evaluation of whether LLMs can handle informal social media language, with three tasks: paraphrasing, conversation directionality, and cyberbullying/anti-bullying detection. What's new is the combination rather than any single component: anti-bullying as a classification target, the exemplar-vs-non-exemplar prompt comparison, and the two-phase LoRA fine-tuning scheme. The central negative finding—that generative LLMs, even after PEFT, perform close to random on cyberbullying/anti-bullying classification—is plausible and independently supported by the low-quality paraphrases. The authors also did a good thing by looking at edit distance alongside similarity metrics, because raw BLEU/ROUGE would have flattered the verbatim-repeating Llama models. For that, credit is due.\n\nThe soft spots are real, though. The most serious is Section 5.2. The directionality experiment on 4chan threads reports no train/test split, and Section 6 says Phase 2 of the PEFT was 'based on 4chan threads.' If the same threads were used for fine-tuning and evaluation, the 0.865 and 0.971 accuracies in Table 2 are memorization, not generalization. That would remove the paper's only notable positive result. This is more load-bearing than the paraphrase metric concern: the near-random classification survives even if you distrust BERT similarity, but the directionality claim has no independent support.\n\nThe smaller issues are worth listing: no human reference paraphrases, so BERT similarity against the original comment conflates copying with understanding; only 100 labeled Instagram sessions with no error bars; fine-tuning hyperparameters and data splits are under-specified; ChatGPT results use an outdated version and can't be reproduced; no code or data released. None of these are fatal on their own, but they accumulate.\n\nWho is this for? Someone in social computing who wants a quick, cautionary data point that LLMs are not reliable for semantic analysis of informal social text. The directionality result should be treated as unverified until the authors clarify the split. I'd send it to peer review because the negative result is worth publishing and the flaw is fixable, but I would not cite the directionality claim until it is cleaned up.","headline":"The paper's negative finding on LLM social understanding is worth attention, but the one big positive result—PEFT directionality—is not trustworthy as reported because the evaluation set is not separated from the fine-tuning corpus.","tokens_in":13343,"tokens_out":2935,"would_cite":false,"duration_ms":26477,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs can follow reply chains but not social media slang.","keywords":["large language models","social media dynamics","cyberbullying detection","anti-bullying","paraphrase evaluation","directionality understanding","parameter-efficient fine-tuning","semantic understanding"],"falsifier":"Collect human reference paraphrases for the same Instagram comments and have human raters judge whether model rewrites preserve meaning; if fine-tuned models match those references well, or if cyberbullying and anti-bullying accuracy rises above chance after training on a large informal-language corpus, the paper's bottleneck claim would need revision.","tokens_in":12442,"feed_emoji":"💬","tokens_out":4169,"duration_ms":39913,"temperature":0.7,"pith_summary":"This paper asks whether generative LLMs can understand the dynamics of informal social media conversations, using cyberbullying and anti-bullying as test cases. It argues that the bottleneck is semantic understanding: even fine-tuned models can learn to track who a comment replies to, but they paraphrase poorly and classify bullying and anti-bullying comments close to chance. The paper's evidence combines paraphrase similarity metrics with edit distance, constrained prompting, and two-phase parameter-efficient fine-tuning. A sympathetic reader would take the finding as a warning that deploying LLMs as social moderators or analysts requires better informal-language comprehension, not just more prompting.","feed_headline":"LLMs can follow reply chains but not social media slang","feed_subtitle":"Even after fine-tuning, models struggle to paraphrase or classify informal comments; directionality is the one skill that improves.","key_machinery":"The evaluation machinery has three parts. For language understanding, the paper pairs BERT-based semantic similarity with the Levenshtein ratio between the generated paraphrase and the original comment; the two together expose verbatim copying that n-gram metrics alone would hide. For directionality, it uses a two-phase PEFT process with LoRA: first WikiTableQuestions data to infuse structural reasoning, then social media data to add social understanding. For classification, it uses constrained generation with a prompt template that asks for a summary as a chain-of-thought step, then binary labels and explanations. This combination lets the paper separate understanding the words from following the conversational structure.","core_discovery":"On its own terms, the central claim is that the weakest link in using LLMs for social behavior analysis is the lack of semantic understanding of the informal language used in social media. The paper reaches this by comparing GPT-2, Llama-2 7B/13B, and ChatGPT on paraphrasing, directionality, and bullying/anti-bullying detection. It finds that ChatGPT balances semantic similarity and edit distance well in paraphrase tasks, while Llama models often repeat input verbatim and GPT-2 produces gibberish. After parameter-efficient fine-tuning, Llama models significantly improve at identifying which post a comment targets on 4chan, but on Instagram cyberbullying and anti-bullying classification all model variations perform near random chance. The paper concludes that structural and directional understanding can be taught, while social semantics remain the limiting factor.","pith_inferences":["If the bottleneck is semantics rather than structure, then injecting more structural reasoning data will not help social classification; the next gains should come from training on informal text with human reference paraphrases.","The near-chance classification combined with high verbatim paraphrase rates suggests models may rely on surface lexical cues; a testable extension is to see whether slang-heavy or obfuscated toxic comments reduce accuracy further.","Because ChatGPT was excluded from classification due to safety filters, the comparison is incomplete; evaluating an open model under the same constrained-generation setup could separate model capability from policy restrictions."],"forward_implications":["Fine-tuning and exemplar prompting can improve structural tasks such as directionality, but they do not by themselves fix semantic understanding of informal language.","Paraphrase evaluations that rely only on BLEU, ROUGE, or Jaccard scores can overstate understanding; the paper shows that edit distance must be paired with semantic similarity to detect verbatim copying.","LLM-based cyberbullying and anti-bullying classifiers operating on short informal comments should not be trusted at their current accuracy levels.","Closed models' safety filters block this kind of social-media evaluation, so progress depends on open models and better benchmark datasets.","A SQuAD-like social language comprehension dataset is needed to measure and drive improvement in informal-language understanding."],"supporting_citations":[{"why":"Supplies BERT, whose encodings define the semantic similarity metric used throughout the paraphrase evaluation.","marker":"[10]"},{"why":"Provides the Instagram session dataset used for the paraphrase and social-language understanding experiments.","marker":"[18]"},{"why":"Provides the labeled Instagram dataset of cyberbullying and anti-bullying comments used for the classification experiment.","marker":"[16]"},{"why":"Supplies the 4chan threads with reply tags that serve as ground truth for directionality identification.","marker":"[29]"},{"why":"Supplies WikiTableQuestions, the structural reasoning data used in the first fine-tuning phase.","marker":"[30]"},{"why":"Defines LoRA, the parameter-efficient fine-tuning method used in both PEFT phases.","marker":"[19]"},{"why":"Motivates the chain-of-thought summary step built into the constrained prompt template.","marker":"[41]"}],"fun_headline_variants":["Slang stumps LLMs even after fine-tuning","LLMs learn reply chains, not social semantics","Social media slang remains LLM blind spot","Directionality teachable, slang not for LLMs","LLMs fail at cyberbullying detection, ace reply direction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument leans on treating BERT semantic similarity and Levenshtein ratio against the original comment as valid evidence of whether a model understands informal social language, with no human-written reference paraphrases to compare against.","fun_headline_variants_meta":{"raw":{"variants":["Slang stumps LLMs even after fine-tuning","LLMs learn reply chains, not social semantics","Social media slang remains LLM blind spot","Directionality teachable, slang not for LLMs","LLMs fail at cyberbullying detection, ace reply direction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1281,"prompt_tokens":931,"completion_tokens":350,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":289}},"tokens_in":547,"tokens_out":350,"duration_ms":4204,"temperature":1.0,"reasoning_tokens":289,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:55:52.940654+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect human reference paraphrases for the same Instagram comments and have human raters judge whether model rewrites preserve meaning; if fine-tuned models match those references well, or if cyberbullying and anti-bullying accuracy rises above chance after training on a large informal-language corpus, the paper's bottleneck claim would need revision.","supporting_citations":[{"cited_title":"In: Social Informatics: 7th International Conference, SocInfo 2015, Beijing, China, December 9-12, 2015, Proceedings","cited_arxiv_id":null,"evidence_quote":"Provides the Instagram session dataset used for the paraphrase and social-language understanding experiments."},{"cited_title":"In: Proceedings of the International AAAI Conference on Web and Social Media","cited_arxiv_id":null,"evidence_quote":"Provides the labeled Instagram dataset of cyberbullying and anti-bullying comments used for the classification experiment."},{"cited_title":"In: Proceedings of the international AAAI conference on web and social media","cited_arxiv_id":null,"evidence_quote":"Supplies WikiTableQuestions, the structural reasoning data used in the first fine-tuning phase."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines LoRA, the parameter-efficient fine-tuning method used in both PEFT phases."}],"review_version":1}