Pith. sign in

REVIEW 4 major objections 7 minor 54 references

ModelCitizens: Representing Community Voices in Online Safety

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Toxicity is community-specific: a dataset built with ingroup labels as gold enables finetuned models to outperform commercial moderation APIs.

desk verdict A genuinely useful community-annotated toxicity dataset, but the 5.5% headline gain is probably inflated by statement-level train/test leaks. read the letter →

arxiv 2507.05455 v2 pith:LUNM7OW7 submitted 2025-07-07 cs.CL cs.AI

Ashima Suvarna , Christina Chance , Karolina Naranjo , Hamid Palangi , Sophie Hao , Thomas Hartvigsen , Saadia Gabriel This is my paper · ORCID
classification cs.CLcs.AI
keywords toxicitydetectioncommunity-informedannotationingroupandoutgroupdisagreementcontentmoderationhatespeechconversationalcontextLLM-generateddataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automatic toxicity detection typically trains on labels averaged across annotators, hiding that members of a targeted community often judge a post differently from outsiders. This paper introduces MODELCITIZENS, a dataset of 6,822 social media posts across eight identity groups with 40,000 toxicity annotations split between ingroup and outgroup annotators, plus LLM-generated conversational contexts for about two-thirds of the posts. The paper shows that ingroup and outgroup annotators disagree on 27.5% of posts and that commercial moderation tools, including OpenAI's Moderation API and GPT-o4-mini, average only 63.6% accuracy against ingroup labels. Finetuning LLaMA and Gemma models on this data produces LLAMACITIZEN-8B and GEMMACITIZEN-12B, which beat the best baseline by 5.5 percentage points overall and by 9 points on the context-augmented subset. The central claim is that centering community voices in annotation and training makes toxicity detection both more accurate on community standards and more inclusive.

What carries the argument

The central object is the MODELCITIZENS dataset with its dual ingroup/outgroup annotation design and the ingroup-as-gold label convention: each post is rated by people who self-identify with the targeted group and by people who do not, and the ingroup scores are treated as the correct answer for training and evaluation. The mechanism is the combination of community-aligned annotation, LLM-generated conversational context around each post, and full finetuning of open-weights language models on those ingroup labels, with disagreements operationalized as missed harm and amplified harm to show where outgroup labels mislead.

What would settle it

Re-annotate the same posts with a fresh, larger panel of ingroup annotators from each of the eight communities; if the new ingroup labels disagree with the original ones on more than a third of posts, the 'ingroup as gold' anchor is not stable enough to support the benchmark. Alternatively, take a sample of MODELCITIZENS posts and have an independent adjudication panel (including community members) resolve each label; if the finetuned model's predictions match the adjudicated labels no better than GPT-o4-mini does, the claimed 5.5% gain would not replicate on a stronger gold standard.

Watch

Extended reading notes

Core claim

The paper claims that toxicity is not a single objective property of a post but is determined by the target community's own perception, and that incorporating these community (ingroup) judgments as the gold standard during dataset construction and model training yields detectors that better reflect those perspectives. To demonstrate this, it introduces MODELCITIZENS, 6,822 social media posts across eight identity groups with 40K annotations, half from ingroup and half from outgroup annotators, and shows ingroup and outgroup disagree on 27.5% of posts. Commercial moderation tools average 63.6% accuracy against ingroup labels, falling to 59.6% on posts with LLM-generated conversational context; finetuned LLaMA- and Gemma-based models, LLAMACITIZEN-8B and GEMMACITIZEN-12B, outperform the best baseline (GPT-o4-mini) by 5.5 points overall and by 9 points on the context-augmented subset.

Load-bearing premise

The entire benchmark treats the targeted community's own toxicity rating as the correct answer; if a reader does not accept that communities' self-assessments are the gold standard, the reported accuracy gains lose their normative meaning.

Editorial extensions

If this is right

  • Finetuned models trained on ingroup labels outperform commercial moderation APIs by 5.5% average accuracy on the in-distribution test set and by 9% on the context-augmented subset.
  • Ingroup and outgroup annotators disagree on 27.5% of posts, and outgroup annotators more frequently label content as toxic, so a single global toxicity score is not a faithful representation of community perceptions.
  • Adding conversational context changes toxicity labels for a substantial fraction of posts and degrades all baseline models' accuracy, whereas the finetuned model retains a 9% edge on this harder subset.
  • Models trained on MODELCITIZENS generalize to out-of-distribution toxicity datasets, including unseen identity groups in Toxigen, as shown by higher F1 scores on HateModerate and Counter-Context.
  • The label-choice ablation shows that training on ingroup labels outperforms training on outgroup or aggregated labels, suggesting that community-grounded supervision carries signal that other annotation schemes dilute or miss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If ingroup labels are adopted as the evaluation standard, content moderation policies may need to be community-specific rather than one global classifier; a service could route moderation decisions to the relevant community's model, though the paper does not test deployment effects.
  • The amplified-harm pattern suggests current moderation systems may over-censor benign in-group speech (such as reclaimed language) for some groups, which could be tested by deploying the finetuned model on real moderation queues and measuring user appeals.
  • Replacing LLM-generated contexts with real human conversations might change the measured gains, so an explicit comparison of human versus synthetic context is a natural next experiment that the paper leaves open.
  • The monotonic data-scaling curve implies that collecting more community-annotated data, especially for the lowest-accuracy groups (Jewish, Latino), may yield the largest improvements; this extrapolates from the paper's 25/50/100% scaling plot.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces MODELCITIZENS, a toxicity annotation dataset built from 2,520 Toxigen statements, each appearing as a single post and, for a subset, as the core of an LLM-generated Reddit-style conversation, yielding 6,822 instances and about 40K ingroup/outgroup annotations across eight identity groups. The authors report that ingroup and outgroup annotators disagree on 27.5% of posts, that commercial moderation APIs underperform on the dataset (average accuracy 63.6%, dropping to 59.6% on context-augmented posts), and that fine-tuned models LLAMACITIZEN-8B and GEMMACITIZEN-12B beat the best baseline by 5.5% and 9% on the full test set and context-augmented subset, respectively. They also present annotation analyses, a label-choice ablation, a data-scaling curve, and out-of-distribution evaluations on HateModerate, Counter-Context, and Toxigen.

Significance. If the headline results survive a correctly deduplicated split, MODELCITIZENS would be a valuable resource for pluralistic and context-aware toxicity detection: the dataset includes balanced ingroup/outgroup annotations, annotator demographics, LLM-generated conversational contexts, and a clear release plan for data, models, and code. The annotation analyses with Krippendorff's alpha and Wilcoxon tests are appropriate, and the cross-dataset generalization checks on HateModerate and Counter-Context are a useful addition. However, the central quantitative claims rest on a small test set without uncertainty quantification and on a split whose contamination control is described only at the instance level, so the magnitude of the reported gains is not yet established.

major comments (4)
  1. [§5.1, §D.2] The train/test split appears to be performed only at the instance level. Section 5.1 states that the test set is a 10% stratified sample with no overlap between train and test, but Section 3.2 and Appendix D.2 construct up to three instances from each base Toxigen statement (the single post plus harmful- and benign-context versions). Unless the split is deduplicated on the original Toxigen statement ID, a test instance can share its statement text with a training sibling; for a three-variant statement the probability that at least one variant lands in test while another remains in train is roughly 27%, and for two-variant statements roughly 19%. This gives the fine-tuned CITIZEN models an advantage over zero-shot baselines, which cannot see sibling variants in training, and it directly affects the headline 5.5% and 9% gains. The authors should re-run the split with statement-level deduplication, report the overlap rate, and re-report Tables 4 and 6.
  2. [§5.1, Table 4] The headline results are single point estimates on a 669-instance test set with no confidence intervals or significance tests. The 5.5% average gain corresponds to roughly 37 instances, and per-group cells contain only about 70-100 instances (based on Table 3), so small label changes could move group-level rankings. Additionally, the binary toxicity threshold is data-dependent (footnote 5 says it was chosen to maximize inter-annotator agreement on the same data), making the point estimate sensitive to that choice. The authors should report bootstrap confidence intervals or paired significance tests (e.g., McNemar) for the main comparisons, and show how the headline gain varies with the threshold.
  3. [§5.3, Table 7] The label-choice ablation evaluates models trained on ingroup, outgroup, and aggregated labels against the ingroup gold on the same test set. Since the ingroup-trained model's training distribution matches the test gold by construction, this comparison does not establish that ingroup labels are 'more reliable signals' (final paragraph of Section 5.3); it shows only that a model trained on a label source performs best when evaluated against that same source. To support the reliability claim, the authors should also evaluate the three models on outgroup gold and aggregated gold, or use a held-out set of annotations from each source, and report agreement/disagreement statistics.
  4. [§5.1, Abstract] The paper repeatedly describes ingroup annotations as 'gold' and reports baseline performance as 'accuracy', which conflates agreement with one annotation policy with correctness. The 5.5% gain is a gain in alignment with ingroup judgments, not an objective measure of toxicity detection. Because the paper's contribution includes the normative claim that ingroup judgments should be authoritative, the evaluation should state this framing explicitly and, to be informative to practitioners who may use consensus or outgroup labels, also report results under those alternative gold standards for at least the main baselines and LLAMACITIZEN-8B.
minor comments (7)
  1. [§5.1, footnote 5] The main text says labels are created by applying a threshold of 3.5, while the footnote says scores greater than 3 are considered toxic; please clarify the exact threshold and how it was selected.
  2. [§3.1, Table 3] Section 3.1 reports sampling 2,520 posts, while Table 3 reports 2,502 single statement posts; with 4,302 context-augmented posts the implied total is 6,804, not the stated 6,822. Please reconcile these numbers.
  3. [§3.2, Appendix D.3] Section 3.2 reports that 86% of generated contexts were high-quality, while Appendix D.3 reports 85%; please align these figures.
  4. [§3.3] Section 3.3 contains two consecutive paragraphs that both begin 'Toxigen includes 13 identity groups...'; one duplicate should be removed.
  5. [Table 4] The 'ΔBase LLaMA' row applies only to LLAMACITIZEN-8B; please add the corresponding Gemma delta or relabel the row to make its scope clear.
  6. [Appendix D.3] Appendix D.3 contains an incomplete sentence fragment ('85We find that...') before the quotation of the validation prompt; please fix the typo.
  7. [Abstract, §3] The abstract and introduction describe MODELCITIZENS as 'social media posts,' although the post content is drawn from synthetic Toxigen statements and the conversational contexts are LLM-generated; please clarify in the dataset description that the posts are generated rather than collected from a platform.

Circularity Check

1 steps flagged · score 4.0 of 10

Label-choice ablation is self-referential, but central benchmark result is independent.

  1. fitted input called prediction [Section 5.3 (Table 7), with evaluation definition in Section 5.1]
    "Each instance of our dataset has ingroup and outgroup toxicity scores and we consider ingroup scores as gold for training and evaluation. ... the model trained on ingroup labels consistently outperforms those trained on outgroup and aggregated labels. This suggests that ingroup annotations may provide more reliable signals."

    In this ablation, the test metric is the ingroup-gold label distribution, which is exactly the training signal of the ingroup-trained model. The outgroup- and aggregated-label models are trained on different targets but evaluated against the same ingroup gold, so the comparison measures how well each training-label scheme reproduces the ingroup labels rather than an independent property of the labels. The conclusion that ingroup annotations are 'more reliable' is therefore largely fixed by the evaluation design: 'better' is defined as agreement with the very labels the winning model was trained on. This does not affect the main IID benchmark, where all models are scored on the same held-out ingroup gold; that is standard supervised evaluation.

full rationale

The paper's main result—LLAMACITIZEN-8B outperforms GPT-o4-mini by 5.5% on the MODELCITIZENS test set—is a conventional fine-tuning comparison: the model is trained on MODELCITIZENS-train and evaluated on the stratified held-out test, so no derivation is claimed from inputs. The use of ingroup scores as gold is an explicit value choice, acknowledged in the Limitations section, not a hidden circular step. External evaluations on Toxigen, HateModerate, and Counter-Context (Table 5) provide independent, out-of-distribution support for the fine-tuned model; although Toxigen shares authors, it is an externally released benchmark and the evaluation is parameter-free with respect to the current paper's fit. The only self-referential element is the label-choice ablation (Table 7): because the test labels are the same ingroup annotations used to train the winning model, the comparison is partially fixed by construction. The statement-level train/test contamination concern (siblings of the same Toxigen base statement possibly crossing splits) is a data-integrity risk rather than a circularity; it would affect the validity of the benchmark but does not make the claim equivalent to its inputs. On balance, the central benchmark result is self-contained, and the circularity burden is limited to the secondary label-choice analysis.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central evaluation rests on two data-dependent choices: the binary toxicity threshold chosen to maximize annotator agreement, and the Perspective API threshold. It also relies on domain assumptions that ingroup labels are ground truth and that LLM-generated contexts approximate real conversations.

free parameters (2)
  • Binary toxicity threshold = >3 on 1-5 scale
    Chosen post hoc to maximize inter-annotator agreement (Section 5.1); all labels and evaluations depend on it.
  • Perspective API threshold = 0.7
    Used to binarize Perspective API scores (Appendix C.4); standard but arbitrary.
assumptions (4)
  • domain assumption Ingroup annotations are the correct gold standard for toxicity
    Stated in Section 5.1: 'we consider ingroup scores as gold for training and evaluation'. The entire evaluation and model selection follow from this value judgment.
  • domain assumption LLM-generated conversational contexts are representative of real social media context
    Contexts are generated by GPT-4o and validated by human raters for naturalness and plausibility (Section 3.2); realism is assumed.
  • domain assumption Posts sampled from Toxigen are representative of online toxicity
    Toxigen is synthetic LLM-generated data; the paper treats it as a base for community annotation (Section 3.1).
  • domain assumption Toxicity can be meaningfully collapsed to binary labels
    The 1-5 scale is thresholded at >3; this loses rating nuance but is common in the field.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ModelCitizens: Representing Community Voices in Online Safety." pith.science (2026). https://pith.science/paper/LUNM7OW7

@misc{pith2026250705455,
  author       = {Pith},
  title        = {Pith review of: ModelCitizens: Representing Community Voices in Online Safety},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LUNM7OW7}},
  note         = {Machine review of arXiv:2507.05455}
}
read the original abstract

Automatic toxic language detection is critical for creating safe, inclusive online spaces. However, it is a highly subjective task, with perceptions of toxic language shaped by community norms and lived experience. Existing toxicity detection models are typically trained on annotations that collapse diverse annotator perspectives into a single ground truth, erasing important context-specific notions of toxicity such as reclaimed language. To address this, we introduce MODELCITIZENS, a dataset of 6.8K social media posts and 40K toxicity annotations across diverse identity groups. To capture the role of conversational context on toxicity, typical of social media posts, we augment MODELCITIZENS posts with LLM-generated conversational scenarios. State-of-the-art toxicity detection tools (e.g. OpenAI Moderation API, GPT-o4-mini) underperform on MODELCITIZENS, with further degradation on context-augmented posts. Finally, we release LLAMACITIZEN-8B and GEMMACITIZEN-12B, LLaMA- and Gemma-based models finetuned on MODELCITIZENS, which outperform GPT-o4-mini by 5.5% on in-distribution evaluations. Our findings highlight the importance of community-informed annotation and modeling for inclusive content moderation. The data, models and code are available at https://github.com/asuvarna31/modelcitizens.

Figures

Figures reproduced from arXiv: 2507.05455 by the authors.

Figure 1
Figure 1. We present disagreements as missed harm and amplified harm on MODELCITIZENS. In particular, amplified harm rate is much higher than missed harm rate across most identity groups. Additionally, we observe that adding context to the posts lead to increased missed harm rate in majority of the groups. Average Asian Black Jewish Latino LGBTQ MexicanMuslimWomen 0 5 10 15 20 25 30 35 Percentage of Posts (%) Toxic Non-Toxic … view at source ↗
Figure 2
Figure 2. Percentage of posts where adding context leads to changes in toxicity labels. We compare the toxicity of the post and the context-augmented post. alizing toxicity; while it may reduce oversensitiv￾ity toward some minority groups (Asian, Jewish), it can also reveal previously overlooked toxicity (Muslim, Black, women), depending on the target group and the content involved (Yu et al., 2022). 5 Content Moderation Mode… view at source ↗
Figure 3
Figure 3. Model performance with data scale. We find that MODELCITIZENS is a high quality dataset that enhances toxicity classification performance as it scales. 58.6% while Qwen2.5-7B-Instruct is the worst performing open-weights model across all identity groups. Despite not being specifically trained for content moderation, Gemini-2.0-Flash and GPT-o4-mini are the best performing baselines, even outperforming LLAMACITIZEN-8… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Annotation interface implemented using potato for toxicity annotation. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Ingroup and Outgroup Annotators show statistically significant differences in rating distributions. MODELCITIZENS reveals that ingroup and outgroup rating distributions vary significantly across identity groups. For instance, outgroup annotators are more likely to rate…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 23 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020. Identifying and measuring annotator bias based on annotators’ demographic characteristics. In Proceedings of the fourth workshop on online abuse and harms, pages 184--190

    work page 2020

  4. [4]

    Hritik Bansal, Ashima Suvarna, Gantavya Bhatt, Nanyun Peng, Kai-Wei Chang, and Aditya Grover. 2025. http://arxiv.org/abs/2404.00530 Comparing bad apples to good oranges: Aligning large language models via joint preference optimization

    arXiv 2025

  5. [5]

    Sergey Berezin, Reza Farahbakhsh, and Noel Crespi. 2025. https://doi.org/10.48550/arXiv.2503.16072 Redefining Toxicity : An Objective and Context-Aware Approach for Stress-Level-Based Detection

    work page Pith review arXiv doi:10.48550/arxiv.2503.16072 2025

  6. [6]

    Laura Biester, Vanita Sharma, Ashkan Kazemi, Naihao Deng, Steven Wilson, and Rada Mihalcea. 2022. Analyzing the effects of annotator gender across nlp tasks. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP@ LREC2022, pages 10--19

    2022

  7. [7]

    Sasha Costanza-Chock. 2020. https://doi.org/10.7551/mitpress/12255.001.0001 Design Justice: Community-Led Practices to Build the Worlds We Need . The MIT Press

    doi:10.7551/mitpress/12255.001.0001 2020

  8. [8]

    Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the international AAAI conference on web and social media, volume 11, pages 512--515

    2017

Show all 54 references
  1. [9]

    Google DeepMind. 2025. I ntroducing G emini 2.0: our new A I model for the agentic era --- blog.google. https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/

    2025

  2. [10]

    Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang. 2023. https://doi.org/10.1145/3617694.3623261 The participatory turn in ai design: Theoretical foundations and the current state of practice . In Proceedings of the 3rd ACM Conference on Equity and Access in Algori...

    2023

  3. [11]

    Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Modeling annotator disagreement for subjective tasks. arXiv preprint arXiv:2305.06626

    2023 arXiv

  4. [12]

    Eve Fleisig, Su Lin Blodgett, Dan Klein, and Zeerak Talat. 2024. The perspectivist paradigm shift: Assumptions and challenges of capturing human labels. NAACL

    2024

  5. [13]

    Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In Proceedings of th...

    2018

  6. [14]

    Gordon, Michelle S

    Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeffrey T. Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. https://api.semanticscholar.org/CorpusID:246634179 Jury learning: Integrating dissenting voices into machine learning models . Proceedings...

    2022

  7. [15]

    Nitesh Goyal, Ian Kivlichan, Rachel Rosen, and Lucy Vasserman. 2022. http://arxiv.org/abs/2205.00501 Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation

    2022 arXiv

  8. [16]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, and et al. 2024. http://arxiv.org/abs/2407.21783 The llama 3 herd of models

    2024 arXiv

  9. [17]

    Annie Grayer. 2018. Facebook apologizes after labeling part of declaration of independence ‘hate speech’. CNN Politics

    2018

  10. [18]

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. https://api.semanticscholar.org/CorpusID:247519233 Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection . In Annual Meeting of the A...

    2022

  11. [19]

    Krass, Lucia Zheng, Neel Guha, Christopher D

    Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. 2022. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. NeurIPS

    2022

  12. [20]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  13. [21]

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. http://arxiv.org/abs/2312.06674 Llama Guard : LLM -based Input - Output Safeguard for Human - AI Conversat...

    2023 arXiv

  14. [22]

    Christopher M Kelty. 2020. The participant: A century of participation in four stories. University of Chicago Press

    2020

  15. [23]

    Hannah Rose Kirk, Alexander Whitefield, Paul R \"o ttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. 2024. The prism alignment project: What participatory, representative and individualised human feedback reveals abo...

    2024 arXiv

  16. [24]

    Morten Kyng. 1991. https://api.semanticscholar.org/CorpusID:15260449 Designing for cooperation: cooperating in design . Commun. ACM, 34:64--73

    1991

  17. [25]

    Nayeon Lee, Chani Jung, Junho Myung, Jiho Jin, Jose Camacho-Collados, Juho Kim, and Alice Oh. 2024. https://doi.org/10.18653/v1/2024.naacl-long.236 Exploring cross-cultural differences in E nglish hate speech annotations: From dataset construction to analysis . In Proceedings ...

    2024 doi

  18. [26]

    Nayeon Lee, Chani Jung, and Alice Oh. 2023. https://doi.org/10.18653/v1/2023.c3nlp-1.5 Hate speech classifiers are culturally insensitive . In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 35--46, Dubrovnik, Croatia. Association for C...

    2023 doi

  19. [27]

    Ryan* Marten, Trung* Vu, Charlie Cheng-Jie Ji, Kartik Sharma, Shreyas Pimpalgaonkar, Alex Dimakis, and Maheswaran Sathiamoorthy. 2025. Curator: A tool for synthetic data creation. https://github.com/bespokelabsai/curator

    2025

  20. [28]

    Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Tajudeen Kolawole, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Hassan Muhammad, Salomon Kabongo, Salomey Osei, et al. 2020. Participatory research for low-resourced machine translation: A case s...

    2020 arXiv

  21. [29]

    OpenAI. 2025. Introducing openai o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/

    2025

  22. [30]

    AI Patronus. 2024. https://www.patronus.ai/blog/llama-guard-is-off-duty Llama guard is off duty . Patronus AI

    2024

  23. [31]

    John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020. https://doi.org/10.18653/v1/2020.acl-main.396 Toxicity Detection : Does Context Really Matter ? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguist...

    2020 doi

  24. [32]

    Jiaxin Pei and David Jurgens. 2023. When do annotator demographics matter? measuring the influence of annotator demographics with the popquorn dataset. arXiv preprint arXiv:2306.06826

    2023 arXiv

  25. [33]

    Luque, Demian Zayat, Mart \'i n Kondratzky, Agust \'i n Moro, Pablo Santiago Serrati, Joaqu \'i n Zajac, Paula Miguel, Natalia Debandi, Agust \'i n Gravano, and Viviana Cotik

    Juan Manuel P \'e rez, Franco M. Luque, Demian Zayat, Mart \'i n Kondratzky, Agust \'i n Moro, Pablo Santiago Serrati, Joaqu \'i n Zajac, Paula Miguel, Natalia Debandi, Agust \'i n Gravano, and Viviana Cotik. 2023. https://doi.org/10.1109/ACCESS.2023.3258973 Assessing the Impa...

    2023

  26. [34]

    Organizers Of Queerinai, Anaelia Ovalle, Arjun Subramonian, Ashwin Singh, Claas Voelcker, Danica J Sutherland, Davide Locatelli, Eva Breznik, Filip Klubicka, Hang Yuan, et al. 2023. Queer in ai: A case study in community-led participatory ai. In Proceedings of the 2023 ACM Con...

    2023

  27. [35]

    Weingarten, Lilith Fury, Constanza Eliana Chinea, Tuck J

    Yim Register, Izzi Grasso, Lauren N. Weingarten, Lilith Fury, Constanza Eliana Chinea, Tuck J. Malloy, and Emma S. Spiro. 2024. https://doi.org/10.1145/3637300 Beyond initial removal: Lasting impacts of discriminatory content moderation to marginalized creators on instagram . ...

    2024 doi

  28. [36]

    RWJF. 2017. https://www.rwjf.org/en/library/research/2017/10/discrimination-in-america--experiences-and-views.html Discrimination in america: experiences and views

    2017

  29. [37]

    Joni Salminen, Fabio Veronesi, Hind Almerekhi, Soon-Gvo Jung, and Bernard J Jansen. 2018. Online hate interpretation varies by country, but more by individual: A statistical analysis using crowdsourced ratings. In 2018 Fifth international conference on social networks analysis...

    2018

  30. [38]

    Sebastin Santy, Jenny T Liang, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. 2023. Nlpositionality: Characterizing design biases of datasets and models. arXiv preprint arXiv:2306.01943

    2023 arXiv

  31. [39]

    Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. https://doi.org/10.18653/v1/P19-1163 The risk of racial bias in hate speech detection . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668--1678, ...

    2019 doi

  32. [40]

    Smith, and Yejin Choi

    Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. https://doi.org/10.18653/v1/2020.acl-main.486 Social bias frames: Reasoning about social and power implications of language . In Proceedings of the 58th Annual Meeting of the Associati...

    2020 doi

  33. [41]

    Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. https://doi.org/10.18653/v1/2022.naacl-main.431 Annotators with attitudes: How annotator beliefs and identities bias toxic language detection . In Proceedings of the 2022 Conference...

    2022 doi

  34. [42]

    Xinyue Shen, Yixin Wu, Yiting Qu, Michael Backes, Savvas Zannettou, and Yang Zhang. 2025. HateBench: Benchmarking Hate Speech Detectors on LLM-Generated Content and Hate Campaigns . In USENIX Security Symposium (USENIX Security) . USENIX

    2025

  35. [43]

    Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano. 2022. Participation is not a design fix for machine learning. In Proceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1--6

    2022

  36. [44]

    Harini Suresh, Emily Tseng, Meg Young, Mary Gray, Emma Pierson, and Karen Levy. 2024. Participation in the age of foundation models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1609--1621

    2024

  37. [45]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, ...

    2025 arXiv

  38. [46]

    Zeerak Waseem. 2016. https://doi.org/10.18653/v1/W16-5618 Are you a racist or am I seeing things? annotator influence on hate speech detection on T witter . In Proceedings of the First Workshop on NLP and Computational Social Science , pages 138--142, Austin, Texas. Associatio...

    2016 doi

  39. [47]

    Lilian Weng, Vik Goel, and Andrea Vallone. 2023. Using gpt-4 for content moderation. https://openai.com/index/using-gpt-4-for-content-moderation/

    2023

  40. [48]

    Bidwell, and Edwin H

    Heike Winschiers-Theophilus, Nicola J. Bidwell, and Edwin H. Blake. 2012. https://api.semanticscholar.org/CorpusID:57564586 Community consensus: Design beyond participation . Design Issues, 28:89--100

    2012

  41. [49]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  42. [50]

    Xinchen Yu, Eduardo Blanco, and Lingzi Hong. 2022. https://doi.org/10.18653/v1/2022.naacl-main.433 Hate Speech and Counter Speech Detection : Conversational Context Does Matter . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa...

    2022 doi

  43. [51]

    Jiangrui Zheng, Xueqing Liu, Mirazul Haque, Xing Qian, Guanqun Yang, and Wei Yang. 2024 a . https://doi.org/10.18653/v1/2024.findings-naacl.172 H ate M oderate: Testing hate speech detectors against content moderation policies . In Findings of the Association for Computational...

    2024 doi

  44. [52]

    Jiangrui Zheng, Xueqing Liu, Guanqun Yang, Mirazul Haque, Xing Qian, Ravishka Rathnasuriya, Wei Yang, and Girish Budhrani. 2024 b . http://arxiv.org/abs/2307.12418 Hatemoderate: Testing hate speech detectors against content moderation policies

    2024 arXiv

  45. [53]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024 c . Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372

    2024 arXiv

  46. [54]

    Hwang, Swabha Swayamdipta, and Maarten Sap

    Xuhui Zhou, Hao Zhu, Akhila Yerukola, Thomas Davidson, Jena D. Hwang, Swabha Swayamdipta, and Maarten Sap. 2023. https://doi.org/10.18653/v1/2023.findings-acl.392 COBRA frames: Contextual reasoning about effects and harms of offensive statements . In Findings of the Associatio...

    2023 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.