{"id":"4dd92380-f111-4dde-a7fc-c5639d2a5db5","arxiv_id":"2411.17876","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning BERT models on LDA topic subsets of NLPositionality gives higher F1 than zero-shot GPT-4 and toxicity APIs, but the improvement is driven by a single topic and baseline comparisons are not controlled.","lead":"This paper fine-tunes BERTweet and HateBERT on topic-specific slices of the NLPositionality toxicity dataset and compares them with GPT-4, PerspectiveAPI, and RewireAPI. The authors claim topic-specific fine-tuning improves F1 scores and that large language models underperform fine-tuned BERT models on this benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own Table IV reverses the claimed topic-specific gain: under micro-F1, full-data BERTweet (0.5497) beats Topic 0 (0.5242), contradicting the Table II ranking the abstract relies on.","rationale":"The reader's weakest assumption concerned comparability of per-topic F1 across unequal subsets. That is real, but the more load-bearing problem is the internal contradiction between Table II and Table IV: the paper presents one F1 ranking to support the abstract and then reports micro-F1 statistics that reverse the Topic-0 advantage. This is not an external-validity caveat; it is a failure of the manuscript's own evidence to support the central topic-specific claim. The comparison against GPT-4/Perspective/Rewire may survive because even full-data fine-tuned models beat those zero-shot baselines, but the novel contribution—'fine-tuning on specific topics improves over full-data'—is not robustly supported. A REJECT verdict for the current version is appropriate because the authors would need to clarify the metric, reconcile the tables, and re-run the comparison consistently before the claim can be evaluated. This is not an attack on the authors' intent; it is an assessment that the reported evidence, taken as a whole, contradicts the headline claim.","tokens_in":8536,"tokens_out":3729,"duration_ms":34947,"concrete_test":"Re-run the released GitHub experiments and output, for every split and baseline, both per-seed macro-F1 (Table II style) and majority-vote micro-F1 (Table IV style) on the same test split, with subset sizes and label distributions reported. The decisive check: if the Topic-0 vs full-data ordering reverses between macro-F1 and micro-F1, the abstract's 'notable improvement' claim is metric-dependent and must be revised to state the metric and the specific topics that support it.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim that topic-specific fine-tuning improves over full-data fine-tuning is internally contradicted by the manuscript's own tables. Section IV and the abstract assert that fine-tuning BERTweet/HateBERT on individual LDA topics improves F1 over full-data training, with Topic 0 best. However, Table IV reports majority-vote micro-F1: for BERTweet, full data = 0.5497, Topic 0 = 0.5242, Topic 1 = 0.5574, Topic 2 = 0.5172; for HateBERT, full data = 0.5215, Topic 0 = 0.5196, Topic 1 = 0.5410, Topic 2 = 0.5011. Under this evaluation, Topic 0 is not better than full-data training for either model, and only Topic 1 shows a marginal edge. Table II's per-seed 'F1' values (Topic 0 BERTweet 0.5579 vs full data 0.4604) appear to use a different, unspecified F1 variant, and Table III mixes these values with baselines without stating the averaging scheme. Section V.A even concedes that 'the breakdown by topic did not yield notable differences in performance compared to the full dataset,' directly contradicting the Results section and abstract. The paper therefore leaves the metric choice ambiguous, and the headline result flips depending on which table one trusts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a topic-modeling-enhanced fine-tuning approach for toxicity classification. Using LDA, the authors split the NLPositionality dataset into 3, 6, and 10 topics, then fine-tune BERTweet and HateBERT on topic-specific subsets as well as the full data. They compare the resulting F1 scores against zero-shot baselines including GPT-4, PerspectiveAPI, RewireAPI, and HateRoberta. The abstract and Section IV claim that fine-tuning on specific topics notably improves F1 relative to full-data fine-tuning and that these topic-specific models outperform the zero-shot LLM/API baselines, suggesting that state-of-the-art LLMs are not effective at toxicity detection without further training.","tokens_in":8823,"tokens_out":3474,"duration_ms":31511,"significance":"If the claims were supported by the evidence, the paper would offer a simple and inexpensive method for improving toxicity classification on positionality-aware data, and it would provide a concrete demonstration of zero-shot LLM limitations on this task. The paper includes a public code repository and reports per-seed results, which are strengths for reproducibility. However, the significance is currently undermined by an internal inconsistency between the two result tables (Tables II and IV), an unspecified F1 variant, and a lack of statistical rigor or dataset characterization, so the headline claims cannot be accepted as stated.","major_comments":[{"comment":"The F1 metric is not defined, and the two tables report different numbers and orderings. Table II reports BERTweet Topic 0 average F1 = 0.5579 versus Full data = 0.4604, while Table IV reports micro F1 for BERTweet Topic 0 = 0.5242 versus Full data = 0.5497. Under Table IV, full-data fine-tuning beats Topic 0 for BERTweet, and Topic 0 does not beat full data for HateBERT either. The authors must state which F1 variant is used in each table, explain the discrepancy, and ensure that the abstract and Section IV claims are consistent with the chosen metric.","section":"Section IV, Tables II and IV"},{"comment":"The baseline comparison is not statistically grounded. The baselines (GPT-4, PerspectiveAPI, RewireAPI, HateRoberta) are evaluated zero-shot, but the paper reports no confidence intervals, significance tests, or effect sizes for the F1 differences. Without these, differences of roughly 0.05 to 0.15 F1 on small topic subsets may not be reliable. Moreover, the claim that state-of-the-art LLMs exhibit significant limitations is too broad, because no fine-tuned LLM baseline is included; zero-shot performance does not demonstrate that these models are incapable of toxicity detection when adapted.","section":"Section IV, Table III"},{"comment":"The Discussion directly contradicts the Results. Section IV states that fine-tuning on individual topics improved F1 compared to full-data fine-tuning, with Topic 0 showing the most significant improvement, while Section V.A concedes that 'the breakdown by topic did not yield notable differences in performance compared to the full dataset.' These statements cannot both be true. The authors must resolve this contradiction and align their conclusions with the evidence, especially given that Table IV supports the Discussion's more cautious interpretation.","section":"Section IV and Section V.A"},{"comment":"The topic subsets are not characterized. The paper does not report the number of examples per topic, the class distribution, or the label balance for each LDA cluster. Without this information, the observed F1 differences across topics could reflect differences in subset size, difficulty, or label imbalance rather than the effect of topic-specific fine-tuning. The authors should report these statistics for the training and test splits and, ideally, control for label imbalance in the evaluation.","section":"Section III.C and Section V.B"}],"minor_comments":[{"comment":"The training setup uses 70 epochs with no early stopping or validation-based model selection described. Please clarify how overfitting was avoided and whether the reported numbers are from a held-out test set.","section":"Section III.E"},{"comment":"Many entries have zero standard deviation across seeds (e.g., Topic 1 for both BERTweet and HateBERT), which is suspicious. Please explain whether this is due to rounding, deterministic fine-tuning, or another cause.","section":"Table II"},{"comment":"The table would benefit from a note explaining that micro F1 equals micro precision and micro recall by definition when computed over all instances; otherwise the identical columns appear redundant.","section":"Table IV"},{"comment":"The confusion matrices in Figure 2 are mentioned only briefly. Please describe what exactly is being compared and specify which class corresponds to hate speech in the matrices.","section":"Section V.A and Figure 2"},{"comment":"References [1] and [22] are the same paper (NLPositionality by Santy et al.); please unify the citations to avoid duplication.","section":"References"},{"comment":"The abstract states that fine-tuning on specific topics improves F1 compared to 'prominent classification models such as GPT-4, PerspectiveAPI, and RewireAPI,' but those baselines are used zero-shot. This comparison should be described as fine-tuned versus zero-shot, not as a general superiority claim.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"This is an empirical benchmark paper whose central claim is currently undermined by an internal metric inconsistency and a direct contradiction between the Results and Discussion sections. The paper is not beyond salvage: if the authors clarify the F1 variant, report subset statistics, and rerun or re-frame the analysis so that the abstract, Results, and Discussion agree, the comparison against zero-shot baselines could still be of interest. However, as written, the evidence does not support the stated conclusions, and the presentation is not yet suitable for publication in a serious journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper applies a known technique (LDA topic-conditioned fine-tuning) to a less commonly used benchmark (NLPositionality) and claims it beats both full-data fine-tuning and zero-shot LLM/API baselines. That claim doesn't hold up against the paper's own tables.\n\nWhat's new and good: the dataset choice is thoughtful, given the positionality angle; the authors repeat each experiment with five seeds and report mean/stdev; code is promised. The observation that zero-shot APIs (GPT-4, Perspective, Rewire) all underperform fine-tuned BERT models on this benchmark is a plausible, mildly useful data point.\n\nThe problem: the abstract and Section IV say topic-specific fine-tuning notably improves F1, with Topic 0 best. Table II (per-seed, apparently macro F1) does show Topic 0 ahead. But Table IV, which the paper says uses majority voting, reports micro F1: for BERTweet, full data = 0.5497 while Topic 0 = 0.5242, and Topic 1 is actually the best. For HateBERT, Topic 1 beats full data by a hair and Topic 0 is slightly worse than full data. Then Section V.A concedes there was no notable difference by topic. So the headline result flips depending on which table you read, and the F1 variant is never defined. That's a load-bearing inconsistency, not a minor formatting issue.\n\nThe baseline comparison is also unfair: fine-tuned models versus zero-shot APIs. Subset sizes and label distributions per topic are missing, so the Topic 0 gain could just reflect an easier split. No significance tests.\n\nCredit where due: the limitations section acknowledges some of these issues (small topic counts, BERTweet's domain mismatch), and the demographic breakdowns in the appendix are a nice touch. But the central claim needs to be either corrected or reframed as a negative result.\n\nThis paper could be a decent workshop submission after a rewrite: reconcile the tables, report per-topic sample sizes and class ratios, define the F1 variant, and soften the abstract. As is, I wouldn't trust the headline. If I were the editor, I'd send it to review with a request for major revisions, or desk reject if the venue is strict about internal consistency. I lean toward giving the authors a chance to fix it, since the benchmark is underexplored and the code is available.","headline":"The paper's own micro-F1 table contradicts its claim that topic-specific fine-tuning beats full-data training; the benchmark choice is good but the analysis needs a major rewrite.","tokens_in":9374,"tokens_out":3536,"would_cite":false,"duration_ms":32130,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Topic-split fine-tuning beats GPT-4 and toxicity APIs on F1.","keywords":["toxicity classification","content moderation","topic modeling","Latent Dirichlet Allocation","BERTweet","HateBERT","annotator positionality","large language models"],"falsifier":"Count examples and toxic/non-toxic ratios in each LDA topic subset, then fine-tune each model on random subsets matched to those sizes and ratios; if random subsets reproduce the Topic 0 gain, the claimed topic-specific advantage is not about topic content.","tokens_in":8348,"feed_emoji":"🛡️","tokens_out":5665,"duration_ms":42661,"temperature":0.7,"pith_summary":"Toxicity classifiers can inherit the biases of the people who label their training data, a problem known as annotator positionality. The paper asks whether fine-tuning two tweet-focused BERT models—BERTweet and HateBERT—on topic-specific slices of the positionality-aware NLPositionality dataset yields better toxicity detection than fine-tuning on the full dataset or using off-the-shelf systems such as GPT-4, PerspectiveAPI, and RewireAPI. It reports that per-topic fine-tuning raises macro F1 on those slices, with the largest gain on one LDA topic cluster, and that all examined zero-shot baselines score below the fine-tuned models. The central claim is that topic-based data splitting improves toxicity classification on positionality-aware data, and that current LLMs and commercial APIs need task-specific fine-tuning to compete.","feed_headline":"Topic-split fine-tuning beats GPT-4 and toxicity APIs","feed_subtitle":"LDA topic splits lift BERTweet and HateBERT F1 beyond GPT-4 and full-data training.","key_machinery":"The central object is the LDA topic split: Latent Dirichlet Allocation, a probabilistic topic model, groups tweets into $k$ latent themes, and the paper uses $k=3$ because further splitting leaves too few examples for fine-tuning. For each topic cluster, the classification head of BERTweet or HateBERT is fine-tuned while all other layers are frozen, with learning rate $5\\times10^{-5}$, 70 epochs, and five random seeds. The topic split carries the argument by separating tweets with distinct lexical patterns—for example, Topic 0 centers on gendered terms while Topics 1 and 2 center on race and immigration—so the fine-tuned head can specialize on each cluster's toxicity signal.","core_discovery":"Fine-tuning BERTweet and HateBERT separately on each of three LDA-derived topic clusters of the NLPositionality dataset gives higher F1 than fine-tuning on the full dataset, and both fine-tuned models beat GPT-4, PerspectiveAPI, RewireAPI, and a zero-shot HateRoberta baseline. The best per-topic numbers come from Topic 0—0.5579 for BERTweet and 0.5498 for HateBERT—versus 0.4604 and 0.4824 for full-data fine-tuning, while the strongest baseline, RewireAPI, reaches 0.4278 on the full dataset. The authors read this as evidence that state-of-the-art LLMs and commercial APIs are not well calibrated for toxicity in positionality-labeled tweets and that topic-specific transfer learning is a viable moderation strategy.","pith_inferences":["A production system could ensemble topic-specialized classifiers, but the paper never merges per-topic predictions to report a single test-set F1, so an ensemble's advantage over one full-data model is untested.","The GPT-4 comparison is zero-shot only; the paper does not fine-tune or few-shot prompt GPT-4, so the 'LLMs are limited' conclusion may not extend to adapted large models.","Controlling for subset size and label balance with random matched subsets would isolate whether topic content, not data quantity, drives the Topic 0 gain."],"forward_implications":["Fine-tuning on topic-specific subsets can raise F1 by about 9.8 points over full-data fine-tuning for BERTweet (Topic 0: 0.5579 versus 0.4604), making data-splitting strategy a performance lever for toxicity models.","On this benchmark, zero-shot GPT-4 and commercial APIs underperform small fine-tuned BERT-class models, implying API-based moderation needs task-specific adaptation on positionality-aware data.","The topic count is bounded by dataset size: the paper uses $k=3$ because larger $k$ yields subsets too small for fine-tuning, so the method's resolution is limited by available labeled data.","Majority voting across seeds shows little difference between topic splits and full data, indicating the per-topic F1 gain does not arise from one cluster carrying more semantic information."],"supporting_citations":[{"why":"Supplies the NLPositionality dataset of positionality-labeled toxic tweets and the evaluation settings used for baselines.","marker":"[1]"},{"why":"Latent Dirichlet Allocation, the topic-modeling method used to split tweets into the k topic clusters for fine-tuning.","marker":"[19]"},{"why":"BERTweet, the tweet-pretrained BERT variant fine-tuned per topic in the experiments.","marker":"[20]"},{"why":"HateBERT, the hate-speech-pretrained BERT variant fine-tuned per topic.","marker":"[21]"},{"why":"ToxiGen, the corpus on which HateBERT was retrained and the source of the zero-shot HateRoberta baseline.","marker":"[23]"},{"why":"GPT-4, the large language model baseline whose zero-shot F1 the paper claims is lower than its fine-tuned models.","marker":"[3]"},{"why":"PerspectiveAPI, a commercial toxicity-detection baseline.","marker":"[25]"},{"why":"RewireAPI, a commercial toxicity-detection baseline.","marker":"[26]"}],"fun_headline_variants":["Topic-split fine-tuning outperforms GPT-4 and APIs","LDA topic tuning beats LLMs and toxicity APIs","Topic modeling lifts BERTweet F1 beyond GPT-4","Fine-tune per topic to beat GPT-4 and APIs","Topic clusters boost toxicity detection over GPT-4"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The per-topic F1 comparisons assume that the three LDA topic subsets are comparable in size and label balance, so the Topic 0 gain reflects topic content rather than subset statistics.","fun_headline_variants_meta":{"raw":{"variants":["Topic-split fine-tuning outperforms GPT-4 and APIs","LDA topic tuning beats LLMs and toxicity APIs","Topic modeling lifts BERTweet F1 beyond GPT-4","Fine-tune per topic to beat GPT-4 and APIs","Topic clusters boost toxicity detection over GPT-4"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00016,"raw_usage":{"total_tokens":1199,"prompt_tokens":881,"completion_tokens":318,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":237}},"tokens_in":497,"tokens_out":318,"duration_ms":3024,"temperature":1.0,"reasoning_tokens":237,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:43:43.976065+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count examples and toxic/non-toxic ratios in each LDA topic subset, then fine-tune each model on random subsets matched to those sizes and ratios; if random subsets reproduce the Topic 0 gain, the claimed topic-specific advantage is not about topic content.","supporting_citations":[{"cited_title":"Latent dirichlet allocation,","cited_arxiv_id":null,"evidence_quote":"Latent Dirichlet Allocation, the topic-modeling method used to split tweets into the k topic clusters for fine-tuning."},{"cited_title":"BERTweet: A pre-trained language model for English Tweets,","cited_arxiv_id":null,"evidence_quote":"BERTweet, the tweet-pretrained BERT variant fine-tuned per topic in the experiments."},{"cited_title":"HateBERT: Retraining BERT for abusive language detection in English,","cited_arxiv_id":null,"evidence_quote":"HateBERT, the hate-speech-pretrained BERT variant fine-tuned per topic."},{"cited_title":"Toxigen: A large-scale machine- generated dataset for implicit and adversarial hate speech detection,","cited_arxiv_id":null,"evidence_quote":"ToxiGen, the corpus on which HateBERT was retrained and the source of the zero-shot HateRoberta baseline."},{"cited_title":"ChatGPT,","cited_arxiv_id":null,"evidence_quote":"GPT-4, the large language model baseline whose zero-shot F1 the paper claims is lower than its fine-tuned models."},{"cited_title":"Perspective api,","cited_arxiv_id":null,"evidence_quote":"PerspectiveAPI, a commercial toxicity-detection baseline."},{"cited_title":"Rewire api,","cited_arxiv_id":null,"evidence_quote":"RewireAPI, a commercial toxicity-detection baseline."}],"review_version":1}