Pith. sign in

REVIEW 4 major objections 6 minor 38 references

Hate Speech According to the Law: An Analysis for Effective Detection

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Legal-expert annotations give models a signal; LLM labels do not.

desk verdict A genuine but underpowered proof-of-concept for cross-country legal hate speech annotation; the dataset is worth refereeing, the performance comparisons are not yet significant. read the letter →

arxiv 2412.06144 v1 pith:7GYVW4ZS submitted 2024-12-09 cs.CL

Katerina Korre , John Pavlopoulos , Paolo Gajo , Alberto Barrón-Cedeño This is my paper · ORCID
classification cs.CL
keywords prosecutablehatespeechlegalannotationinter-annotatoragreementnationallawsordinalclassificationlargelanguagemodelspseudo-labelingcross-countrycomparison
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to ground hate speech detection in what national law actually criminalises, instead of in subjective definitions. Three legal experts re-annotated 100 tweets from HateEval under the hate speech laws of Greece, Italy, and the UK, and the three votes were combined into a four-level ordinal scale from "not prosecutable" to "prosecutable". Four BERT-family models and two large language models were then evaluated on that scale. The paper argues that legal-expert annotations carry a usable training signal, that agreement depends heavily on the law in question (pairwise Cohen's $\kappa$ averages 0.19 for Greece but 0.43 for the UK), and that LLM-generated labels for 1000 additional tweets do not improve the pretrained models. A sympathetic reading of the result: the bottleneck for prosecutable hate speech detection is expert legal annotation, not more LLM-generated data.

What carries the argument

The load-bearing device is the conversion of three independent expert judgments per tweet and per law into an ordinal gold standard: all three experts call the tweet prosecutable "prosecutable", two call it so "likely prosecutable", one "unlikely prosecutable", none "not prosecutable". This aggregation deliberately preserves disagreement rather than forcing a majority label, and it turns the task into ordinal regression evaluated by mean absolute and squared error under leave-one-out cross-validation on the 100 instances. The second mechanism is pseudo-labeling: the stronger LLM (Qwen2-7B-Instruct) labels 1000 additional HateEval tweets, the silver set is filtered by removing cases that contradict the original hate/non-hate labels, and the pretrained models are fine-tuned on the silver set and tested on the 100 expert instances.

What would settle it

A replication on a larger sample: annotate, say, 500 to 1000 HateEval tweets with a fresh panel of legal experts under the same three laws and check whether Greek law still yields the lowest pairwise agreement and whether the same per-country best models emerge; if either pattern fails to reproduce, the central claim is not stable.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that prosecutability under hate speech law is a learnable but law-specific target: a modest amount of expert legal annotation yields a signal that pretrained models can pick up, while labels produced by a large language model add noise rather than signal. The evidence is the per-country split in annotation and performance: experts disagree most under Greek law and least under UK law, no single pretrained model wins across all three countries (DehateBERT is best for Greece, LegalBERT for Italy, HateRoBERTa for the UK), and fine-tuning on Qwen2-generated silver labels raises MAE substantially relative to training only on the 100 expert instances. The authors conclude that legal knowledge in the form of annotations is useful for classifying prosecutable hate speech, and that the differences among laws deserve more attention than they currently receive.

Load-bearing premise

The entire comparison rests on 100 hateful tweets annotated by three legal experts, and if that small sample or those labels are not representative of prosecutable hate speech, the model rankings and the main conclusions are built on noise.

Editorial extensions

If this is right

  • Because prosecutability labels differ across the three laws, a single global hate speech classifier will misclassify content when applied outside the legal context it was trained under.
  • No single pretrained model is best everywhere; this study's rankings point to country-specific model selection (DehateBERT for Greece, LegalBERT for Italy, HateRoBERTa for the UK).
  • LLM-generated silver labels are not an acceptable substitute for expert annotation in this legally sensitive task; in these experiments they degraded all four PLMs.
  • Including the full law text in the prompt helped the LLMs in some settings, but the multiclass PLMs trained on expert labels still gave lower error overall.
  • Treating disagreement as ordinal gradations is a workable way to build gold standards for subjective legal annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the low agreement under Greek law suggests the wording of the law itself, not annotator skill, may be the limiting factor; a natural test is to annotate the same tweets with more experts per jurisdiction and see whether agreement saturates.
  • Editorial inference: this account predicts poor cross-jurisdiction transfer, so a direct experiment is training on the Greek-labelled set and testing on the UK-labelled set, or vice versa.
  • Editorial inference: the pseudo-labeling failure is consistent with prosecutability being a genuinely legal inference rather than a surface text property; human-in-the-loop active learning that resolves expert disagreements might be more cost-effective than bulk LLM labeling.
  • Editorial inference: because the gold set contains only 100 tweets, the specific best-model rankings and even the per-law kappa ordering should be treated as provisional until replicated on a larger expert-annotated sample.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents a small expert-annotated dataset of 100 hateful tweets from HatEval, each annotated for prosecutability under Greek, Italian, and UK hate-speech laws, and maps the three binary expert votes into a four-level ordinal gold standard. The authors then benchmark four BERT-style PLMs using leave-one-out cross-validation, evaluate two LLMs under zero-shot, few-shot, and law-augmented prompting, and finally fine-tune PLMs on a thousand Qwen2-generated silver labels filtered with the original HatEval hate labels. The paper concludes that legal knowledge in the form of expert annotations is useful for detecting prosecutable hate speech, that model performance varies across legal frameworks, and that LLM-generated data do not improve PLMs.

Significance. If the dataset is released and the claims are rigorously supported, this would be a useful resource: it is the first cross-country expert-annotated dataset on prosecutable hate speech under three distinct national laws, and the annotation protocol—having each instance judged under each law—directly targets the legal variation that platforms face. The paper also reports reproducible fine-tuning procedures and a planned public release, which are strengths. However, the current experimental evidence is not robust enough to support the central conclusions: the gold standard is built from only three annotators with low agreement (especially for Greece), model comparisons rest on tiny differences without uncertainty quantification, and the silver-label experiment confounds label source with training protocol. The contribution is therefore better framed as a resource paper than as an established finding about legal knowledge and model behavior.

major comments (4)
  1. [3.1, Table 1] The ordinal gold standard is constructed from three expert votes, but the agreement is extremely low: Cohen's kappa ranges from 0.12 to 0.61, with an average of 0.19 for Greece. The 'Unlikely prosecutable' class for Greece (32/100 instances) is defined by exactly one expert's positive vote, so a substantial portion of the gold standard is effectively a single annotator's judgment. This instability directly affects every downstream model comparison in Tables 2-4: with label noise of this magnitude, MAE differences of 0.02-0.04 between models (e.g., Greece DehateBERT 0.54 vs. HateRoBERTa 0.56) are not distinguishable from noise. The paper should report annotator-level evaluation, provide confidence intervals or bootstrap estimates, and avoid presenting per-country best-model selections without statistical support.
  2. [4.3, Table 6] The silver-label experiment confounds the label source (Qwen2-generated vs. expert) with the training and evaluation protocol. The models in Table 6 are fine-tuned on ~600 filtered silver instances and evaluated on the 100 expert instances with no overlap, whereas the models in Table 2 are evaluated under LOOCV on the same 100 instances, meaning each fold trains on 99 of the test instances. The deterioration in Table 6 relative to Table 2 can therefore be explained by the amount of training data, the distribution shift, or the evaluation protocol, not solely by the quality of silver labels. A controlled comparison—for example, training on an expert-annotated set of comparable size with the same fixed split, or using a held-out test set for both conditions—is needed to support the claim that LLM-generated data do not improve PLM performance.
  3. [1, 4.2, and 4.3] The paper's central claim that 'legal knowledge in the form of annotations can be useful' is not directly supported by the experiments. RQ1 asks whether annotation with laws versus definitions changes inter-annotator agreement, but the paper never compares law-based annotation with a definition-based annotation of the same instances. The PLM experiments (Table 2) do not include a baseline without legal pretraining or legal annotation supervision; LegalBERT is the only legal-pretrained model and it is not consistently the best across countries. Moreover, the prompting results are mixed: Qwen2's 0-shot with law for the UK gives MAE 1.51 vs. 1.31 without law (Table 3a), and Llama3's 0-shot with law for Greece is worse than without law (1.74 vs. 1.58). The conclusion should be tempered or supported with explicit ablations (e.g., a non-legal annotation baseline, a model without legal pretraining, or per-condition significance tests).
  4. [4.1, Table 2] The LOOCV evaluation on 100 instances is reported with a single run per model, with no variance, no repeated seeds, and no significance testing. Given the small n and the label instability noted above, the reported ordering of models (for example, Italy LegalBERT MAE 0.44 vs. DehateBERT 0.48) is not established as reliable. The authors should report the distribution of per-fold errors, paired significance tests (e.g., a paired bootstrap over instances), or at minimum the standard deviation across folds or seeds, before drawing conclusions about which model is best for each jurisdiction.
minor comments (6)
  1. [Abstract, 3.2] There are several grammatical and typographical errors, including 'we also exploiting' and 'output =' formatting in Section 3.2; the paper would benefit from a careful proofreading pass.
  2. [Tables 3 and 4] The column headings 'LOOCV few-shot' are confusing because LOOCV is itself a few-shot setting with 99 examples; consider renaming the columns to 'LOOCV (99-shot)' or similar for clarity.
  3. [Appendix C] The prompt template contains a typo: 'Is the following sentence prosecutable hate speech according?' should read 'according to what?' or be revised for grammatical completeness.
  4. [2.1] There is a missing space in 'theNo Hate Speech Movement' and a few other spacing issues; these should be corrected.
  5. [Table 5 and Section 4.3] The totals in Table 5 (604, 645, and 620) do not sum to the stated 1,000 instances; please explain how filtering was applied and report the actual number of silver instances used for each country's fine-tuning.
  6. [4.4] The error analysis says models are 'more conservative in predicting an instance as prosecutable,' but earlier in Section 4.4 and in the appendix confusion matrices the models often overpredict class 0; please reconcile this description with the confusion matrices.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical benchmarks against a manually constructed expert gold standard, not derivations from fitted inputs or self-citation chains.

full rationale

The paper's central claims are empirical: legal-expert annotations are used to build an ordinal gold standard, and PLMs/LLMs are evaluated against that gold standard via LOOCV. The ordinal target is a stated design choice (counting expert votes: all-three = prosecutable, two = likely, one = unlikely, none = not prosecutable), not a fitted parameter that is later renamed as a prediction. The silver-label experiment generates labels with Qwen2 and fine-tunes PLMs on them, but performance is measured externally against the expert-annotated 100 instances; the LLM-generated labels are not used to define the evaluation metric or the gold standard. The only self-citation (Korre et al., 2023) appears in a general statement about disagreement over hate speech definitions and is not load-bearing for any experimental conclusion. Concerns about low inter-annotator agreement, small sample size, and lack of significance testing are validity and noise issues, not circularity: the paper does not reduce any derived result to its own inputs by construction. No equation in the paper equates a prediction with a fitted value, and no uniqueness theorem or author-imported ansatz forces the choice of model or label aggregation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The analysis relies on a small, subjective annotation task, so all downstream conclusions inherit the assumptions about sample representativeness, law translation accuracy, and the validity of expert judgment. No new physical or conceptual entities are introduced.

free parameters (3)
  • LOOCV fine-tuning hyperparameters = learning rate 1e-5, 50 epochs, patience 2, batch size 32
    Chosen by hand in Section 3.2; they affect all PLM results and are not tuned with a validation split beyond early stopping.
  • Expert-agreement ordinal mapping = 3/3 positive -> Prosecutable, 2/3 -> Likely, 1/3 -> Unlikely, 0 -> Not
    Ad hoc mapping in Section 3.1 that converts vote counts into the four-class ordinal label; no external validation of this thresholding.
  • Silver-data filtering rule = not specified numerically
    Section 4.3 filters silver labels by eliminating 'false positives and false negatives' against original HatEval labels; the exact criterion is under-specified and is a design choice.
assumptions (4)
  • domain assumption HateEval instances are representative of prosecutable hate speech in English
    Section 3.1 samples 100 hateful instances from HatEval and treats them as the substrate for legal prosecutability; no representativeness analysis is provided.
  • domain assumption The Global Handbook of Hate Speech Laws provides accurate English translations of the Greek, Italian, and UK laws
    Section 3 uses this web source as the legal basis; translation errors would propagate to annotations and prompt text.
  • domain assumption Expert legal opinion is a valid gold standard for prosecutability
    Section 3.1 uses three experts' judgments as the gold labels; low kappa values indicate the standard is contested.
  • ad hoc to paper The ordinal mapping of expert agreement to four classes preserves meaningful distinctions
    Section 3.1 converts positive expert vote counts into an ordinal scale; this construction is specific to this paper and has no external validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hate Speech According to the Law: An Analysis for Effective Detection." pith.science (2026). https://pith.science/paper/7GYVW4ZS

@misc{pith2026241206144,
  author       = {Pith},
  title        = {Pith review of: Hate Speech According to the Law: An Analysis for Effective Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7GYVW4ZS}},
  note         = {Machine review of arXiv:2412.06144}
}
read the original abstract

The issue of hate speech extends beyond the confines of the online realm. It is a problem with real-life repercussions, prompting most nations to formulate legal frameworks that classify hate speech as a punishable offence. These legal frameworks differ from one country to another, contributing to the big chaos that online platforms have to face when addressing reported instances of hate speech. With the definitions of hate speech falling short in introducing a robust framework, we turn our gaze onto hate speech laws. We consult the opinion of legal experts on a hate speech dataset and we experiment by employing various approaches such as pretrained models both on hate speech and legal data, as well as exploiting two large language models (Qwen2-7B-Instruct and Meta-Llama-3-70B). Due to the time-consuming nature of data acquisition for prosecutable hate speech, we use pseudo-labeling to improve our pretrained models. This study highlights the importance of amplifying research on prosecutable hate speech and provides insights into effective strategies for combating hate speech within the parameters of legal frameworks. Our findings show that legal knowledge in the form of annotations can be useful when classifying prosecutable hate speech, yet more focus should be paid on the differences between the laws.

Figures

Figures reproduced from arXiv: 2412.06144 by the authors.

Figure 1
Figure 1. Distribution of classes as predicted on 1,000 instances from HateEval ( [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. HateBERT confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 47 1 0 0 22 5 5 0 11 3 2 0 1 2 0 1 DehateBERT (GR) Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 58 0 0 0 16 0 0 1 9 1 6 0 5 2 2 0 DehateBERT (IT) Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 59 0 0 0 5 3 2 1 15 2 5 0 4 2 2 0 Dehat… view at source ↗
Figure 3
Figure 3. DehateBERT confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 42 4 2 0 16 12 4 0 7 4 4 1 2 1 1 0 HateRoBERTa (GR) Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 54 3 0 1 7 6 2 2 8 1 6 1 4 0 2 3 HateRoBERTa (IT) Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 54 1 3 1 4 2 5 0 6 3 12 1 4 0 3 1 Ha… view at source ↗
Figures from the paper (22 more)
Figure 4
Figure 4. Figure 4: HateRoBERTa confusion matrix [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: LegalBERT confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 9 16 11 12 0 6 9 17 0 1 8 7 0 0 1 3 GR Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 10 15 14 19 1 3 9 4 1 1 8 6 0 1 4 …
Figure 6
Figure 6. Figure 6: Qwen2 0-shot w/o Law confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 28 1 6 13 8 2 10 12 1 0 5 10 0 0 2 2 GR Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 27 12 3 16 3 2 5 7 2 0…
Figure 7
Figure 7. Figure 7: Qwen2 0-shot w/ Law confusion matrix [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Qwen2 LOOCV w/o Law confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 39 7 1 1 17 6 9 0 4 3 8 1 2 0 2 0 GR Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 53 0 3 2 12 0 2 3 11 2 1 2…
Figure 9
Figure 9. Figure 9: Qwen2 LOOCV w/ Law confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 25 9 11 3 11 4 11 6 5 1 7 3 0 0 3 1 GR Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 32 7 15 4 8 2 6 1 6 1 4 5…
Figure 10
Figure 10. Figure 10: Qwen2 4-shot w/o Law confusion matrix [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Qwen2 4-shot w/ Law confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 34 4 8 2 21 2 5 4 6 1 5 4 4 0 0 0 GR Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 37 8 8 5 6 3 6 2 8 1 2 5 …
Figure 12
Figure 12. Figure 12: Qwen2 8-shot w/o Law confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 29 8 8 3 18 2 7 5 4 0 7 5 1 0 0 3 GR Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 43 5 9 1 7 1 8 1 2 1 3 1…
Figure 13
Figure 13. Figure 13: Qwen2 8-shot w/ Law confusion matrix [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Qwen2 12-shot w/o Law confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 29 5 11 3 16 3 9 4 6 1 4 5 1 0 1 2 GR Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 43 7 7 1 6 3 6 2 2 2 5…
Figure 15
Figure 15. Figure 15: Qwen2 12-shot w/ Law confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 9 2 11 26 0 3 10 19 0 1 8 7 0 0 0 4 GR Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 4 3 14 37 3 2 5 7 1 1 …
Figure 16
Figure 16. Figure 16: Llama3 0-shot w/o Law confusion matrix [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Llama3 0-shot w/ Law confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 12 14 22 0 1 7 23 1 1 3 12 0 0 1 3 0 GR Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 22 9 22 5 2 6 7 2 2 2…
Figure 18
Figure 18. Figure 18: Llama3 LOOCV w/o Law confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 11 3 33 1 6 1 25 0 2 2 12 0 0 0 4 0 GR Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 18 1 31 8 3 2 9 3 0 4 …
Figure 19
Figure 19. Figure 19: Llama3 LOOCV w/ Law confusion matrix [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]
Figure 20
Figure 20. Figure 20: Llama3 4-shot w/o Law confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 13 14 15 6 3 6 15 8 0 2 9 5 0 0 3 1 GR Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 19 12 20 7 2 3 10 2 2…
Figure 21
Figure 21. Figure 21: Llama3 4-shot w/ Law confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 18 8 11 11 5 5 7 15 1 4 4 7 1 0 0 3 GR Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 20 9 15 14 5 2 8 2 3 4…
Figure 22
Figure 22. Figure 22: Llama3 8-shot w/o Law confusion matrix [PITH_FULL_IMAGE:figures/full_fig_p024_22.png]
Figure 23
Figure 23. Figure 23: Llama3 8-shot w/ Law confusion matrix. Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 20 10 8 10 13 4 8 7 1 0 8 7 0 0 2 2 GR Predicted 0 Predicted 1 Predicted 2 Predicted 3 Actual 0 Actual 1 Actual 2 Actual 3 20 5 15 18 10 1 3 3 3 …
Figure 24
Figure 24. Figure 24: Llama3 12-shot w/o Law confusion matrix. [PITH_FULL_IMAGE:figures/full_fig_p025_24.png]
Figure 25
Figure 25. Figure 25: Llama3 12-shot w/ Law confusion matrix [PITH_FULL_IMAGE:figures/full_fig_p025_25.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 16 canonical work pages

  1. [1]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  2. [2]

    Sai Saket Aluru, Binny Mathew, Punyajoy Saha, and Animesh Mukherjee. 2020. Deep learning models for multilingual hate speech detection. arXiv preprint arXiv:2004.06465

    arXiv 2020

  3. [3]

    William Aspray and Philip Doty. 2023. https://doi.org/https://doi.org/10.1002/asi.24762 Does technology really outpace policy, and does it matter? a primer for technical experts and others . Journal of the Association for Information Science and Technology, 74(8):885--904

    work page doi:10.1002/asi.24762 2023

  4. [4]

    Valerio Basile. 2020. https://api.semanticscholar.org/CorpusID:229344921 It's the end of the gold standard as we know it. on the impact of pre-aggregation on the evaluation of highly subjective tasks . In DP@AI*IA

    work page 2020

  5. [5]

    Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. https://doi.org/10.18653/v1/S19-2007 S em E val-2019 task 5: Multilingual detection of hate speech against immigrants and women in T witter . In Proceedings of the 13th International Workshop on Semant...

    doi:10.18653/v1/s19-2007 2019

  6. [6]

    Daniel Braun. 2023. https://doi.org/10.1007/s10506-023-09369-4 I beg to differ: How disagreement is handled in the annotation of legal machine learning data sets . Artif Intell Law

    doi:10.1007/s10506-023-09369-4 2023

  7. [7]

    Tommaso Caselli, Valerio Basile, Jelena Mitrovi \'c , and Michael Granitzer. 2021. https://doi.org/10.18653/v1/2021.woah-1.3 H ate BERT : Retraining BERT for abusive language detection in E nglish . In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 17--25, Online. Association for Computational Linguistics

    doi:10.18653/v1/2021.woah-1.3 2021

  8. [8]

    Claudia Casula and Sara Tonelli. 2020. https://doi.org/10.4000/books.aaccademia.8345 Hate speech detection with machine-translated data: The role of annotation scheme, class imbalance and undersampling . In Proceedings of the Seventh Italian Conference on Computational Linguistics CLiC-it 2020, pages 1--. Accademia University Press

    work page doi:10.4000/books.aaccademia.8345 2020

Show all 38 references
  1. [9]

    Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.261 LEGAL - BERT : The muppets straight out of law school . In Findings of the Association for Computational Linguistics...

    2020 doi

  2. [10]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  3. [11]

    James Donato. 1988. Dworkin and subjectivity in legal interpretation. Stanford Law Review, 40(6):1517--1541

    1988

  4. [12]

    Darja Fi s er, Toma z Erjavec, and Nikola Ljube s i \'c . 2017. https://doi.org/10.18653/v1/W17-3007 Legal framework, dataset and annotation schema for socially unacceptable online discourse practices in S lovene . In Proceedings of the First Workshop on Abusive Language Onlin...

    2017 doi

  5. [13]

    Caterina Flick. 2020. https://doi.org/10.30687/978-88-6969-478-3/011 The legal framework on hate speech and the internet good practices to prevent and counter the spread of illegal hate speech online . In Language, Gender and Hate Speech A Multidisciplinary Approach. Fondazion...

    2020 doi

  6. [14]

    Paula Fortuna, Juan Soler, and Leo Wanner. 2020. https://aclanthology.org/2020.lrec-1.838 Toxic, hateful, offensive or abusive? what are we really classifying? an empirical analysis of hate speech datasets . In Proceedings of the Twelfth Language Resources and Evaluation Confe...

    2020

  7. [15]

    Victoria Guillén-Nieto. 2023. https://doi.org/doi:10.1515/9783110672619 Hate Speech . De Gruyter Mouton, Berlin, Boston

    2023 doi

  8. [16]

    Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders S gaard. 2022. https://doi.org/10.186...

    2022 doi

  9. [17]

    Md Saroar Jahan and Mourad Oussalah. 2023. https://doi.org/https://doi.org/10.1016/j.neucom.2023.126232 A systematic review of hate speech automatic detection using natural language processing . Neurocomputing, 546:126232

    2023

  10. [18]

    Urja Khurana, Ivar Vermeulen, Eric Nalisnick, Marloes Van Noorloos, and Antske Fokkens. 2022. https://doi.org/10.18653/v1/2022.woah-1.17 Hate speech criteria: A modular approach to task-specific hate speech definitions . In Proceedings of the Sixth Workshop on Online Abuse and...

    2022 doi

  11. [19]

    Katerina Korre, John Pavlopoulos, Jeffrey Sorensen, L \'e o Laugier, Ion Androutsopoulos, Lucas Dixon, and Alberto Barr \'o n-cede \ n o. 2023. https://doi.org/10.18653/v1/2023.woah-1.24 Harmful language datasets: An assessment of robustness . In The 7th Workshop on Online Abu...

    2023 doi

  12. [20]

    Nayeon Lee, Chani Jung, and Alice Oh. 2023. https://doi.org/10.18653/v1/2023.c3nlp-1.5 Hate speech classifiers are culturally insensitive . In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 35--46, Dubrovnik, Croatia. Association for C...

    2023 doi

  13. [21]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. http://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach

    2019 arXiv

  14. [22]

    Chu Luo, Rohan Bhambhoria, Samuel Dahan, and Xiaodan Zhu. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.730 Legally enforceable hate speech detection for public forums . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10948--10963, Singa...

    2023 doi

  15. [23]

    Ilia Markov and Walter Daelemans. 2022. https://aclanthology.org/2022.trac-1.5 The role of context in detecting the target of hate speech . In Proceedings of the Third Workshop on Threat, Aggression and Cyberbullying (TRAC 2022), pages 37--42, Gyeongju, Republic of Korea. Asso...

    2022

  16. [24]

    Marwick and Ross Miller

    Alice E. Marwick and Ross Miller. 2014. https://ssrn.com/abstract=2447904 Online harassment, defamation, and hateful speech: A primer of the legal landscape . Report 2, Fordham Center on Law and Information Policy

    2014

  17. [25]

    Marcel Maussen and Ralph Grillo. 2014. https://doi.org/10.1080/1369183X.2013.851470 Regulation of speech in multicultural societies: Introduction . Journal of Ethnic and Migration Studies, 40(2):174--193

    2014

  18. [26]

    Pia Pachinger, Allan Hanbury, Julia Neidhardt, and Anna Planitzer. 2023. https://doi.org/10.18653/v1/2023.c3nlp-1.11 Toward disambiguating the definitions of abusive, offensive, toxic, and uncivil comments . In Proceedings of the First Workshop on Cross-Cultural Considerations...

    2023 doi

  19. [27]

    Luque, Demian Zayat, Martín Kondratzky, Agustín Moro, Pablo Santiago Serrati, Joaquín Zajac, Paula Miguel, Natalia Debandi, Agustín Gravano, and Viviana Cotik

    Juan Manuel Pérez, Franco M. Luque, Demian Zayat, Martín Kondratzky, Agustín Moro, Pablo Santiago Serrati, Joaquín Zajac, Paula Miguel, Natalia Debandi, Agustín Gravano, and Viviana Cotik. 2023. https://doi.org/10.1109/ACCESS.2023.3258973 Assessing the impact of contextual inf...

    2023

  20. [28]

    Heng Rathpisey and Teguh Bharata Adji. 2019. https://doi.org/10.1109/ICSITech46713.2019.8987500 Handling imbalance issue in hate speech classification using sampling-based methods . In 2019 5th International Conference on Science in Information Technology (ICSITech), pages 193--198

    2019

  21. [29]

    Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. https://doi.org/10.18653/v1/P19-1163 The risk of racial bias in hate speech detection . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668--1678, ...

    2019 doi

  22. [30]

    Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021. Learning from the worst: Dynamically generated datasets to improve online hate detection. In ACL

    2021

  23. [31]

    Nina Wacholder, Smaranda Muresan, Debanjan Ghosh, and Mark Aakhus. 2014. https://doi.org/10.3115/v1/W14-4918 Annotating multiparty discourse: Challenges for agreement metrics . In Proceedings of LAW VIII - The 8th Linguistic Annotation Workshop , pages 120--128, Dublin, Irelan...

    2014 doi

  24. [32]

    Zeerak Waseem. 2016. https://doi.org/10.18653/v1/W16-5618 Are you a racist or am I seeing things? annotator influence on hate speech detection on T witter . In Proceedings of the First Workshop on NLP and Computational Social Science , pages 138--142, Austin, Texas. Associatio...

    2016 doi

  25. [33]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  26. [34]

    Yongjin Yang, Joonkee Kim, Yujin Kim, Namgyu Ho, James Thorne, and Se-Young Yun. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.365 HARE : Explainable hate speech detection with step-by-step reasoning . In Findings of the Association for Computational Linguistics: EMNLP...

    2023 doi

  27. [35]

    W Yin and A Zubiaga. 2021. https://doi.org/10.7717/peerj-cs.598 Towards generalisable hate speech detection: a review on obstacles and solutions . PeerJ Computer Science, 7:e598

    2021 doi

  28. [36]

    Frederike Zufall, Marius Hamacher, Katharina Kloppenborg, and Torsten Zesch. 2022. https://doi.org/10.18653/v1/2022.nllp-1.5 A legal approach to hate speech -- operationalizing the EU ' s legal framework against the expression of hatred as an NLP task . In Proceedings of the N...

    2022 doi

  29. [37]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.