{"id":"b2fd71ef-5e3a-4bda-b187-8186e1a88758","arxiv_id":"2412.06144","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A 100-instance expert-annotated dataset of prosecutable hate speech under Greek, Italian, and UK law is used to benchmark BERT models and LLMs, with legal annotations helping performance and LLM-generated silver data failing to improve it.","lead":"The paper builds a tiny expert-annotated dataset of 100 tweets judged as prosecutable hate speech under three countries' laws, then benchmarks four BERT models and two LLMs. It finds legal annotations help somewhat, but pseudo-labeled data from LLMs does not improve detection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The gold-standard ordinal labels are built from three experts with kappa as low as 0.12, and the reported per-country best-model differences (0.02-0.04 MAE) are likely noise; the central 'legal knowledge helps' claim is not yet established.","rationale":"Good-faith reading: The paper asks whether legal-expert annotations of prosecutable hate speech can train useful detectors, across three national laws. That is a worthwhile question, and the authors are transparent about the small scale. However, the empirical pathway from annotations to conclusions runs through an ordinal gold standard that is not validated. With IAA of 0.12-0.61 and an aggregation rule that equates disagreement with intermediate prosecutability, a large fraction of the training signal (notably the 'Unlikely' class in Greece) is effectively one expert's opinion. The LOOCV model comparisons are then based on MAE gaps of 0.02-0.04, which is within the range one would expect from seed variance on 100 examples. Since the paper's central claim is that legal knowledge helps, the burden is on demonstrating that the target itself is stable and that model differences are not noise. The proposed leave-one-expert-out check would directly test this. If the rankings survive, the concern is mitigated; if not, the conditional verdict should be maintained or tightened. The reader's conditional verdict already captures the core data-quality worry; my analysis agrees with it and refines it, so no verdict change is needed.","tokens_in":26856,"tokens_out":6019,"duration_ms":58547,"concrete_test":"Run leave-one-expert-out LOOCV: rebuild the ordinal gold standard from each pair of the three experts (A-B, A-C, B-C) and rerun the Section 3.2 PLM benchmark. If the best model per country changes across expert subsets, or if the three MAE values for any model differ by more than the 0.02-0.04 gaps reported, the Table 2 conclusions depend on a single expert and the gold standard is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 converts three binary expert votes into a 4-class ordinal target: all-three = prosecutable, two = likely, one = unlikely, none = not prosecutable. With Table 1's Cohen's kappa as low as 0.12 (Greece A-C) and averages of 0.19 (Greece), this aggregation makes 'Unlikely prosecutable' (32/100 for Greece) a class defined by a single expert's positive vote. The LOOCV PLM results in Table 2 then select per-country best models by MAE differences of 0.02-0.06 (e.g., DehateBERT 0.54 vs HateRoBERTa 0.56 for Greece; LegalBERT 0.44 vs DehateBERT 0.48 for Italy), with no confidence intervals, significance tests, or repeated-seed variance. Because the gold standard itself is unstable, these rankings - and the paper's conclusion that legal annotations are useful - are not distinguished from noise. The silver-label experiment (Section 4.3, Table 6) also confounds label source (Qwen2-generated vs expert) with training-set size (1000 vs 99), so its null result for LLM-generated data is not controlled.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a small expert-annotated dataset of 100 hateful tweets from HatEval, each annotated for prosecutability under Greek, Italian, and UK hate-speech laws, and maps the three binary expert votes into a four-level ordinal gold standard. The authors then benchmark four BERT-style PLMs using leave-one-out cross-validation, evaluate two LLMs under zero-shot, few-shot, and law-augmented prompting, and finally fine-tune PLMs on a thousand Qwen2-generated silver labels filtered with the original HatEval hate labels. The paper concludes that legal knowledge in the form of expert annotations is useful for detecting prosecutable hate speech, that model performance varies across legal frameworks, and that LLM-generated data do not improve PLMs.","tokens_in":27104,"tokens_out":4621,"duration_ms":41704,"significance":"If the dataset is released and the claims are rigorously supported, this would be a useful resource: it is the first cross-country expert-annotated dataset on prosecutable hate speech under three distinct national laws, and the annotation protocol—having each instance judged under each law—directly targets the legal variation that platforms face. The paper also reports reproducible fine-tuning procedures and a planned public release, which are strengths. However, the current experimental evidence is not robust enough to support the central conclusions: the gold standard is built from only three annotators with low agreement (especially for Greece), model comparisons rest on tiny differences without uncertainty quantification, and the silver-label experiment confounds label source with training protocol. The contribution is therefore better framed as a resource paper than as an established finding about legal knowledge and model behavior.","major_comments":[{"comment":"The ordinal gold standard is constructed from three expert votes, but the agreement is extremely low: Cohen's kappa ranges from 0.12 to 0.61, with an average of 0.19 for Greece. The 'Unlikely prosecutable' class for Greece (32/100 instances) is defined by exactly one expert's positive vote, so a substantial portion of the gold standard is effectively a single annotator's judgment. This instability directly affects every downstream model comparison in Tables 2-4: with label noise of this magnitude, MAE differences of 0.02-0.04 between models (e.g., Greece DehateBERT 0.54 vs. HateRoBERTa 0.56) are not distinguishable from noise. The paper should report annotator-level evaluation, provide confidence intervals or bootstrap estimates, and avoid presenting per-country best-model selections without statistical support.","section":"3.1, Table 1"},{"comment":"The silver-label experiment confounds the label source (Qwen2-generated vs. expert) with the training and evaluation protocol. The models in Table 6 are fine-tuned on ~600 filtered silver instances and evaluated on the 100 expert instances with no overlap, whereas the models in Table 2 are evaluated under LOOCV on the same 100 instances, meaning each fold trains on 99 of the test instances. The deterioration in Table 6 relative to Table 2 can therefore be explained by the amount of training data, the distribution shift, or the evaluation protocol, not solely by the quality of silver labels. A controlled comparison—for example, training on an expert-annotated set of comparable size with the same fixed split, or using a held-out test set for both conditions—is needed to support the claim that LLM-generated data do not improve PLM performance.","section":"4.3, Table 6"},{"comment":"The paper's central claim that 'legal knowledge in the form of annotations can be useful' is not directly supported by the experiments. RQ1 asks whether annotation with laws versus definitions changes inter-annotator agreement, but the paper never compares law-based annotation with a definition-based annotation of the same instances. The PLM experiments (Table 2) do not include a baseline without legal pretraining or legal annotation supervision; LegalBERT is the only legal-pretrained model and it is not consistently the best across countries. Moreover, the prompting results are mixed: Qwen2's 0-shot with law for the UK gives MAE 1.51 vs. 1.31 without law (Table 3a), and Llama3's 0-shot with law for Greece is worse than without law (1.74 vs. 1.58). The conclusion should be tempered or supported with explicit ablations (e.g., a non-legal annotation baseline, a model without legal pretraining, or per-condition significance tests).","section":"1, 4.2, and 4.3"},{"comment":"The LOOCV evaluation on 100 instances is reported with a single run per model, with no variance, no repeated seeds, and no significance testing. Given the small n and the label instability noted above, the reported ordering of models (for example, Italy LegalBERT MAE 0.44 vs. DehateBERT 0.48) is not established as reliable. The authors should report the distribution of per-fold errors, paired significance tests (e.g., a paired bootstrap over instances), or at minimum the standard deviation across folds or seeds, before drawing conclusions about which model is best for each jurisdiction.","section":"4.1, Table 2"}],"minor_comments":[{"comment":"There are several grammatical and typographical errors, including 'we also exploiting' and 'output =' formatting in Section 3.2; the paper would benefit from a careful proofreading pass.","section":"Abstract, 3.2"},{"comment":"The column headings 'LOOCV few-shot' are confusing because LOOCV is itself a few-shot setting with 99 examples; consider renaming the columns to 'LOOCV (99-shot)' or similar for clarity.","section":"Tables 3 and 4"},{"comment":"The prompt template contains a typo: 'Is the following sentence prosecutable hate speech according?' should read 'according to what?' or be revised for grammatical completeness.","section":"Appendix C"},{"comment":"There is a missing space in 'theNo Hate Speech Movement' and a few other spacing issues; these should be corrected.","section":"2.1"},{"comment":"The totals in Table 5 (604, 645, and 620) do not sum to the stated 1,000 instances; please explain how filtering was applied and report the actual number of silver instances used for each country's fine-tuning.","section":"Table 5 and Section 4.3"},{"comment":"The error analysis says models are 'more conservative in predicting an instance as prosecutable,' but earlier in Section 4.4 and in the appendix confusion matrices the models often overpredict class 0; please reconcile this description with the confusion matrices.","section":"4.4"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is timely and potentially valuable, but the paper's experimental claims are overreaching relative to the evidence. The revision required is substantial: the gold-standard reliability and the controlled comparison for the silver-label experiment are both load-bearing. If the authors can add uncertainty quantification and a properly controlled silver-label evaluation, the paper could become a solid resource paper. I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a real proof-of-concept for cross-country prosecutable hate speech annotation, but the evidence for the headline claim is thinner than the abstract implies. The paper's main novelty is the annotation of the same 100 hateful tweets under Greek, Italian, and UK law by three legal experts – the first resource of its kind, to my knowledge. That is worth something, and the authors are refreshingly honest about the low inter-annotator agreement and the subjectivity of legal interpretation.\n\nWhat it does well: the ordinal target (prosecutable / likely / unlikely / not) is a sensible way to preserve disagreement rather than forcing a binary. The authors also position the work against prior single-jurisdiction efforts (Zufall, Fišer, Luo) and make clear that legal frameworks differ materially. The LLM-vs-expert finding – that Qwen2-generated silver labels don't improve PLM performance – is plausible and useful as a cautionary note.\n\nThe soft spots are real and load-bearing. One hundred instances is small under any reading, and with Cohen's kappa as low as 0.12 between two experts for Greece, the 'unlikely prosecutable' class is in effect defined by a single expert's vote for a third of the instances. The LOOCV numbers in Table 2 select per-country best models by MAE differences of 0.02–0.06, with no confidence intervals, repeated seeds, or significance tests; those rankings are indistinguishable from noise. I also think the stress-test note is right that the silver-label experiment confounds label source with training-set size (1000 vs 99), so the null result is not controlled. These are not fatal flaws – the paper states its own limitations – but they keep the central claim from being established.\n\nWho should read it: people working on legal NLP or hate speech datasets will want to know this resource exists, even if the release is still pending. It reads like a workshop or LREC-style dataset paper rather than a definitive study.\n\nMy recommendation: send it to peer review. It deserves referee time, but the decision should be conditional on releasing the annotated data and either adding statistical rigor or softening the conclusion. I'd want to see the data before citing it in my own work.","headline":"A genuine but underpowered proof-of-concept for cross-country legal hate speech annotation; the dataset is worth refereeing, the performance comparisons are not yet significant.","tokens_in":27656,"tokens_out":2290,"would_cite":false,"duration_ms":21099,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Legal-expert annotations give models a signal; LLM labels do not.","keywords":["prosecutable hate speech","legal annotation","inter-annotator agreement","national hate speech laws","ordinal classification","large language models","pseudo-labeling","cross-country legal comparison"],"falsifier":"A replication on a larger sample: annotate, say, 500 to 1000 HateEval tweets with a fresh panel of legal experts under the same three laws and check whether Greek law still yields the lowest pairwise agreement and whether the same per-country best models emerge; if either pattern fails to reproduce, the central claim is not stable.","tokens_in":26645,"feed_emoji":"⚖️","tokens_out":7263,"duration_ms":62561,"temperature":0.7,"pith_summary":"This paper tries to ground hate speech detection in what national law actually criminalises, instead of in subjective definitions. Three legal experts re-annotated 100 tweets from HateEval under the hate speech laws of Greece, Italy, and the UK, and the three votes were combined into a four-level ordinal scale from \"not prosecutable\" to \"prosecutable\". Four BERT-family models and two large language models were then evaluated on that scale. The paper argues that legal-expert annotations carry a usable training signal, that agreement depends heavily on the law in question (pairwise Cohen's $\\kappa$ averages 0.19 for Greece but 0.43 for the UK), and that LLM-generated labels for 1000 additional tweets do not improve the pretrained models. A sympathetic reading of the result: the bottleneck for prosecutable hate speech detection is expert legal annotation, not more LLM-generated data.","feed_headline":"Legal-expert labels aid hate-speech models; LLM labels don't","feed_subtitle":"On 100 tweets judged under three national laws, human legal annotation carried the signal; LLM-generated labels made models worse.","key_machinery":"The load-bearing device is the conversion of three independent expert judgments per tweet and per law into an ordinal gold standard: all three experts call the tweet prosecutable \"prosecutable\", two call it so \"likely prosecutable\", one \"unlikely prosecutable\", none \"not prosecutable\". This aggregation deliberately preserves disagreement rather than forcing a majority label, and it turns the task into ordinal regression evaluated by mean absolute and squared error under leave-one-out cross-validation on the 100 instances. The second mechanism is pseudo-labeling: the stronger LLM (Qwen2-7B-Instruct) labels 1000 additional HateEval tweets, the silver set is filtered by removing cases that contradict the original hate/non-hate labels, and the pretrained models are fine-tuned on the silver set and tested on the 100 expert instances.","core_discovery":"On the paper's own terms, the central discovery is that prosecutability under hate speech law is a learnable but law-specific target: a modest amount of expert legal annotation yields a signal that pretrained models can pick up, while labels produced by a large language model add noise rather than signal. The evidence is the per-country split in annotation and performance: experts disagree most under Greek law and least under UK law, no single pretrained model wins across all three countries (DehateBERT is best for Greece, LegalBERT for Italy, HateRoBERTa for the UK), and fine-tuning on Qwen2-generated silver labels raises MAE substantially relative to training only on the 100 expert instances. The authors conclude that legal knowledge in the form of annotations is useful for classifying prosecutable hate speech, and that the differences among laws deserve more attention than they currently receive.","pith_inferences":["Editorial inference: the low agreement under Greek law suggests the wording of the law itself, not annotator skill, may be the limiting factor; a natural test is to annotate the same tweets with more experts per jurisdiction and see whether agreement saturates.","Editorial inference: this account predicts poor cross-jurisdiction transfer, so a direct experiment is training on the Greek-labelled set and testing on the UK-labelled set, or vice versa.","Editorial inference: the pseudo-labeling failure is consistent with prosecutability being a genuinely legal inference rather than a surface text property; human-in-the-loop active learning that resolves expert disagreements might be more cost-effective than bulk LLM labeling.","Editorial inference: because the gold set contains only 100 tweets, the specific best-model rankings and even the per-law kappa ordering should be treated as provisional until replicated on a larger expert-annotated sample."],"forward_implications":["Because prosecutability labels differ across the three laws, a single global hate speech classifier will misclassify content when applied outside the legal context it was trained under.","No single pretrained model is best everywhere; this study's rankings point to country-specific model selection (DehateBERT for Greece, LegalBERT for Italy, HateRoBERTa for the UK).","LLM-generated silver labels are not an acceptable substitute for expert annotation in this legally sensitive task; in these experiments they degraded all four PLMs.","Including the full law text in the prompt helped the LLMs in some settings, but the multiclass PLMs trained on expert labels still gave lower error overall.","Treating disagreement as ordinal gradations is a workable way to build gold standards for subjective legal annotation."],"supporting_citations":[{"why":"Supplies the HatEval tweets that were re-annotated under the three laws and the additional 1000 instances used for silver labels.","marker":"Basile et al., 2019"},{"why":"Shows how a hate speech law can be operationalized as NLP subtasks; this paper extends that idea to entire national laws with ordinal labels.","marker":"Zufall et al., 2022"},{"why":"Provides the closest legal-expert-annotated hate speech benchmark and the alternative 'violates / does not violate / unclear' labeling scheme.","marker":"Luo et al., 2023"},{"why":"Supplies LegalBERT, the legal-text pretrained model evaluated and the best performer for the Italian law.","marker":"Chalkidis et al., 2020"},{"why":"Supplies HateBERT, one of the four hate-speech pretrained models benchmarked.","marker":"Caselli et al., 2021"},{"why":"Supplies DehateBERT, the multilingual hate-speech model that achieved the lowest error for Greek law.","marker":"Aluru et al., 2020"},{"why":"Supplies HateRoBERTa, the hate-speech model that achieved the lowest MAE for UK law.","marker":"Vidgen et al., 2021"},{"why":"Supplies Qwen2-7B-Instruct, the LLM used for prompting experiments and for generating the silver labels.","marker":"Yang et al., 2024"},{"why":"Supplies Meta-Llama-3-70B, the second LLM benchmarked in zero- and few-shot settings.","marker":"AI@Meta, 2024"}],"fun_headline_variants":["Expert legal labels beat LLM labels for hate speech detection","LLM labels hurt hate speech models; expert legal labels help","Hate speech laws differ by country; models need expert labels","Legal-expert annotations improve hate speech detection; LLM labels don't","Prosecutable hate speech is law-specific; LLM labels add noise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison rests on 100 hateful tweets annotated by three legal experts, and if that small sample or those labels are not representative of prosecutable hate speech, the model rankings and the main conclusions are built on noise.","fun_headline_variants_meta":{"raw":{"variants":["Expert legal labels beat LLM labels for hate speech detection","LLM labels hurt hate speech models; expert legal labels help","Hate speech laws differ by country; models need expert labels","Legal-expert annotations improve hate speech detection; LLM labels don't","Prosecutable hate speech is law-specific; LLM labels add noise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1349,"prompt_tokens":936,"completion_tokens":413,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":324}},"tokens_in":552,"tokens_out":413,"duration_ms":5229,"temperature":1.0,"reasoning_tokens":324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:57:30.967077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication on a larger sample: annotate, say, 500 to 1000 HateEval tweets with a fresh panel of legal experts under the same three laws and check whether Greek law still yields the lowest pairwise agreement and whether the same per-country best models emerge; if either pattern fails to reproduce, the central claim is not stable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows how a hate speech law can be operationalized as NLP subtasks; this paper extends that idea to entire national laws with ordinal labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the closest legal-expert-annotated hate speech benchmark and the alternative 'violates / does not violate / unclear' labeling scheme."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies HateRoBERTa, the hate-speech model that achieved the lowest MAE for UK law."}],"review_version":1}