Pith. sign in

REVIEW 4 major objections 5 minor 52 references

SubData: Bridging Heterogeneous Datasets to Enable Theory-Driven Evaluation of Political and Demographic Perspectives in LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces SUBDATA, an open-source library that standardizes ten hate speech datasets into one target-group taxonomy, enabling theory-driven comparison of persona-aligned LLMs.

desk verdict A genuinely useful open-source resource with a load-bearing mapping that the paper itself admits is subjective; the demo experiment is the weaker half, but the library deserves a serious look. read the letter →

arxiv 2412.16783 v3 pith:KJUHOKWB submitted 2024-12-21 cs.CL

Pietro Bernardelle , Leon Fröhling , Stefano Civelli , Gianluca Demartini This is my paper · ORCID
classification cs.CL
keywords hatespeechdetectionLLMperspectivealignmentdatasetstandardizationtarget-grouptaxonomypersona-conditionedLLMspoliticalsubjectiveNLPtaskscross-datasetevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Evaluating whether large language models reflect different human perspectives on subjective tasks is currently hobbled by inconsistent datasets: each hate speech study uses its own labels and target groups, so results cannot be pooled. This paper proposes SUBDATA, an open-source Python library that standardizes ten hate speech datasets under one nine-category taxonomy of target groups (race, religion, gender, migration, origin, disability, age, sexuality, political), so that instances targeting a given group become comparable across sources. On top of this standardization, it offers a theory-driven evaluation loop: derive a hypothesis from social or political theory, condition LLMs on different perspectives, and measure classification differences. In the demonstration, left-leaning personas flagged more hate speech than right-leaning personas for every target, including Christians, men, and whites, which the authors interpret as a general threshold shift rather than selective protection of minorities. If the mappings are reliable, the library gives researchers a shared, reproducible testbed for comparing perspective-aligned models on subjective classification.

What carries the argument

The load-bearing mechanism is SUBDATA's unified taxonomy and the keyword-to-target mapping that feeds it. The taxonomy organizes targets into nine demographic categories and gives each a standardized name (such as jews, blacks, migrants, or lgbtq_unsp); the mapping converts raw source labels like 'Kikes' or 'jewish people' into those names, using original dataset documentation and consistent principles for ambiguous cases such as 'mexicans,' which some sources place in race and others in origin. This mapping is what makes aggregation across datasets meaningful, and the create_target_dataset() function operationalizes it by downloading, processing, and merging instances with harmonized labels. The theory-driven workflow then uses the merged resource as a controlled testbed: theory supplies a hypothesis, differently aligned LLM personas supply the comparison, and detection-rate differences quantify the alignment effect.

What would settle it

Have independent annotators re-label a stratified sample of instances that SUBDATA merges under a single target group (for example, 'jews') against one shared definition, and measure agreement with SUBDATA's assigned target; if agreement is low or varies systematically by source dataset, the taxonomy does not preserve the construct it is meant to standardize.

Watch

Extended reading notes

Core claim

The central claim is that the main obstacle to evaluating perspective-aligned LLMs on subjective tasks is not model capability but the absence of standardized data, and that careful taxonomy construction removes that obstacle. SUBDATA merges ten existing hate speech datasets into a unified set of target-group labels and exposes functions such as create_target_dataset() that assemble every available instance targeting a given group into one dataframe. The paper's empirical demonstration then shows that persona-conditioning on left-leaning perspectives raises hate speech detection rates relative to right-leaning perspectives across all examined target groups in three 7B-8B models, with the effect appearing for majority groups as well as minorities. The authors read this uniformity as a general tightening of classification thresholds induced by political perspective, not as group-specific protection of minorities.

Load-bearing premise

The framework's comparisons stand or fall on the manual mapping from each dataset's original labels to SUBDATA's single taxonomy: unless 'hate speech against a given group' means the same thing after standardization in every source dataset, cross-dataset comparisons built on the merged data are invalid.

Editorial extensions

If this is right

  • Researchers can build one merged dataset for any target group across ten sources, making hate speech detection results from different studies directly comparable.
  • Alignment can be measured as relative differences between models conditioned on different perspectives, avoiding the need to settle contested ground-truth labels for subjective constructs.
  • The demonstrated effect implies that persona-induced political alignment shifts hate speech detection thresholds broadly, not selectively, at least for 7B-8B models.
  • Because SUBDATA lets researchers customize and export the taxonomy, studies can document their mapping modifications, keeping customized comparisons transparent.
  • The library's design extends naturally to additional subjective constructs and datasets, with misinformation named as the next target.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One confound the paper leaves untested is that left-leaning personas may simply instruct the model to be more permissive about calling content hateful; a control with neutral personas and the same structured prompt would separate ideological alignment from general threshold effects.
  • The same taxonomy-and-mapping machinery should transfer to other subjective constructs, but each transfer needs its own validity check because the expected relationship between target groups and perspectives changes with the construct.
  • A direct validation study that re-annotates a sample of merged instances would turn the library's central assumption--that labels remain semantically comparable after mapping--into an auditable metric rather than a judgment call.
  • Since SUBDATA aggregates existing annotations without re-labeling, downstream conclusions compare harmonized source labels, not an independent measure of true hate speech prevalence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SUBDATA, an open-source Python library that standardizes ten hate speech datasets under a unified taxonomy of target demographic groups, and proposes a theory-driven workflow for evaluating whether persona-prompted LLMs reflect different political perspectives in hate speech detection. The authors document the dataset collection process, the manual mapping from original labels to the unified taxonomy, and the customization functions of the library. They then present an example application in which 20 left-leaning and 20 right-leaning personas per model prompt Mistral-7B, Llama-3.1-8B, and Qwen2.5-7B to classify 25,000 instances targeting ten groups, reporting odds ratios that indicate left-leaning personas detect more hate speech across all target groups. The paper positions SUBDATA as a reusable resource and invites community extension to a multi-construct benchmark suite.

Significance. If the mapping from original dataset labels to the unified taxonomy is reliable, SUBDATA addresses a real gap: cross-study comparability in subjective NLP tasks such as hate speech detection. The paper's strengths include a functional open-source library on GitHub and PyPI, documented core functions, a concrete example experiment with 1,000,000 inferences per model, and unusually transparent discussion of the mapping's subjective components and the library's inherited annotation errors. The convergence with the independent taxonomy of Fillies and Paschke (2025) is also encouraging. However, the central contribution is conditional on the validity of the manual label-to-taxonomy mapping, and the example experiment currently lacks the design features needed to support its stronger interpretive claims. With additional reliability and sensitivity evidence, this would be a useful resource for the community.

major comments (4)
  1. [§4.2/§4.3 and Limitations] The central claim that SUBDATA standardizes heterogeneous datasets into a comparable resource rests on the manual keyword-to-taxonomy mapping, yet the paper provides no reliability or sensitivity evidence for this mapping. The authors concede in §4.3 and the Limitations that decisions such as placing 'mexicans' in origin versus race, keeping 'LGBTQ+' as a single heterogeneous category, and simplifying intersectional targets involve subjective judgment, and that the library inherits source annotation errors without re-labeling. Since create_target_dataset() in Appendix A exposes only final mapped instances, users cannot quantify which instances depend on contested decisions. I recommend reporting inter-annotator agreement on a sample of mapping decisions, auditing the fraction of instances routed through ambiguous judgment calls, and rerunning the §6 example under plausible alternative mappings to show that the conclusions are not mapping-dependent.
  2. [§6.1/Table 3] The significance tests underlying the odds ratios in Table 3 appear to treat the 25,000 instances as independent observations, but the data are clustered: instances are nested in source datasets, each instance is scored by 40 personas, and the 20 left and 20 right personas are a selected sample from each model's persona distribution. This clustering can deflate p-values and overstate the consistency of the effects. The analysis should use mixed-effects models with random intercepts for instance, persona, and source dataset, or cluster-robust bootstrap inference resampled at the persona and instance levels. Without such modeling, the significance stars in Table 3 do not support the strength of the claims in §6.2.
  3. [§6.2] The interpretation that left-persona conditioning produces 'a general tightening of classification thresholds' requires a baseline, but the experiment includes no neutral or unprompted condition. With only left and right persona conditions, an OR > 1 for all targets is equally compatible with right personas lowering sensitivity across the board, left personas raising it across the board, or some combination; the specific claim that left conditioning tightens thresholds is not identifiable from the two-condition design. Adding a neutral-persona condition and reporting per-target baseline detection rates would make the interpretation testable.
  4. [§3.3 and §6.1] The deduplication described in §3.3 removes repeated entry-target pairs within a dataset, but the paper does not state whether duplicate or near-duplicate texts are removed across the ten source datasets. Because the example pools 2,500 randomly sampled instances per target from the merged collection, cross-dataset text overlap could inflate the apparent consistency of persona effects and bias per-target samples toward particular sources. The authors should report cross-dataset duplication statistics and either deduplicate at the text level or include source dataset as a covariate or random effect in the §6 analysis.
minor comments (5)
  1. [Table 1 note] The note on the 'All Dataset' row is ambiguous: it says the row reports 'the total unique target groups per category across all datasets,' but the reader may not know whether the parenthetical counts are summed across datasets or computed after unioning labels; please clarify with a concrete example such as the origin category.
  2. [§4.3] The claim that the taxonomy 'already aligns with independent efforts' and that there is 'large overlap' with Fillies and Paschke (2025) would be more convincing with a quantitative comparison of the two taxonomies, such as the number and proportion of shared target groups.
  3. [Appendix A] The update_overview() function is documented twice with different parameter descriptions; consolidate the two entries and make the parameter list consistent, including overview_name, mapping_name, taxonomy_name, and hf_token.
  4. [Throughout] The name 'SUBDATA' is typeset inconsistently in phrases such as 'SUBDATAprovides' and 'SUBDATA's; please standardize the formatting, for example by using a consistent non-breaking control or writing 'SubData' throughout.
  5. [§6.1] The choices of 2,500 statements per target and 20 personas per side are free parameters of the example, but no justification or sensitivity analysis is provided; a brief rationale or a small robustness check would help readers assess the stability of Table 3.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the library is a constructed resource and the example evaluation is falsifiable; self-cited persona work is a non-circular dependency, not an input-equivalent prediction.

full rationale

The paper's central deliverable is SUBDATA, a dataset-standardization library, which is a constructed resource rather than a derived prediction. The mapping from original labels to the unified taxonomy (Section 4.2) is admittedly subjective, and the Limitations concede that choices such as placing 'mexicans' in race vs. origin or keeping 'LGBTQ+' as one heterogeneous category involve judgment. However, no empirical outcome is fitted from the mapping, so the library itself is not circular. The example in Section 6 tests an empirical hypothesis adapted from Feng et al. (2023) using persona sets adopted from Bernardelle et al. (2025a) and Civelli et al. (2025). Those are self-citations, but they are dependencies rather than reductions: the left/right persona rankings were produced in prior work from political compass positioning, not from the hate speech detection rates being measured here. The experiment is also partly disconfirming of the original group-specific hypothesis, as the paper states: 'The uniformity of this effect runs counter to our hypothesis.' This shows the measured outcome is not forced by construction. The acknowledged subjectivity of the taxonomy is a validity and robustness concern, not a circularity of the derivation chain.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central claim rests on a small number of design choices: the taxonomy mappings, the persona-selection procedure carried over from prior self-cited work, and a manual dataset collection process. No fitted parameters are hidden in the main framework; the free parameters listed are sampling and persona-count choices in the illustrative experiment.

free parameters (2)
  • sample_per_target = 2500
    Number of statements randomly sampled per target group after merging (Section 6.1). This is an ad hoc choice, not derived from a power analysis, and it defines the evaluation dataset.
  • personas_per_side = 20
    The 20 most left-leaning and 20 most right-leaning personas per model are selected from prior distributions (Section 6.1). This number is a design choice, not justified by statistical criteria.
assumptions (4)
  • domain assumption Source dataset target annotations are reliable and refer to the same construct after mapping.
    The library aggregates existing annotations without re-labeling or quality control (Limitations section). The entire framework assumes merging preserves the meaning of 'targeted hate speech' across datasets.
  • domain assumption Persona-conditioning via the prompts used in Bernardelle et al. (2025a) effectively induces left and right political perspectives in the three LLMs.
    The experiment selects personas based on prior work by the same authors (Section 6.1). If persona-conditioning is ineffective or inconsistent across models, the comparison is meaningless.
  • ad hoc to paper The unified taxonomy's target groups (e.g., 'lgbtq_unsp', 'race_unspecified') are meaningful categories for cross-dataset comparison.
    The taxonomy is constructed bottom-up from source datasets (Section 4.1). It is a design choice, not an independently validated standard, and the paper acknowledges its limitations.
  • standard math Statistical significance tests (odds ratios, p-values) assume independent classifications across instances and personas.
    The analysis in Section 6.2 pools classifications; clustering by persona or repeated text is not modeled, which may inflate significance.
invented entities (1)
  • SUBDATA unified taxonomy (9 categories with target groups like 'jews', 'blacks', 'lgbtq_unsp')
    purpose: Standardize heterogeneous target labels across ten datasets.
    The taxonomy is constructed by the authors, overlaps with Fillies and Paschke (2025), but is not derived from an external benchmark or validated through new measurements.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SubData: Bridging Heterogeneous Datasets to Enable Theory-Driven Evaluation of Political and Demographic Perspectives in LLMs." pith.science (2026). https://pith.science/paper/KJUHOKWB

@misc{pith2026241216783,
  author       = {Pith},
  title        = {Pith review of: SubData: Bridging Heterogeneous Datasets to Enable Theory-Driven Evaluation of Political and Demographic Perspectives in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJUHOKWB}},
  note         = {Machine review of arXiv:2412.16783}
}
read the original abstract

As increasingly capable large language models (LLMs) emerge, researchers have begun exploring their potential for subjective tasks. While recent work demonstrates that LLMs can be aligned with diverse human perspectives, evaluating this alignment on downstream tasks (e.g., hate speech detection) remains challenging due to the use of inconsistent datasets across studies. To address this issue, in this resource paper we propose a two-step framework: we (1) introduce SubData, an open-source Python library designed for standardizing heterogeneous datasets to evaluate LLMs perspective alignment; and (2) present a theory-driven approach leveraging this library to test how differently-aligned LLMs (e.g., aligned with different political viewpoints) classify content targeting specific demographics. SubData's flexible mapping and taxonomy enable customization for diverse research needs, distinguishing it from existing resources. We illustrate its usage with an example application and invite contributions to extend our initial release into a multi-construct benchmark suite for evaluating LLMs perspective alignment on natural language processing tasks.

Figures

Figures reproduced from arXiv: 2412.16783 by the authors.

Figure 1
Figure 1. Overview of our proposed evaluation frame [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. SUBDATA taxonomy structure with target groups organized by category. Note: targets that should end in “_unspecified” have been abbreviated in the figure using “‘_unsp.” Dataset Original Keyword Target Fanton et al. (2021) “JEWS” jews Hartvigsen et al. (2022) “jewish” jews Jikeli et al. (2023a) “Kikes” jews Vidgen et al. (2021a) “jewish people” jews Vidgen et al. (2021b) “bla, jew” jews blacks Vidgen et al. (2021b) “… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 25 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ahmed Agiza, Mohamed Mostagir, and Sherief Reda. 2024. Politune: Analyzing the impact of data selection and fine-tuning on economic and political biases in large language models. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pages 2--12

    work page 2024

  4. [4]

    Shayan Alipour, Indira Sen, Mattia Samory, and Tanushree Mitra. 2024. Robustness and confounders in the demographic alignment of llms with human perceptions of offensiveness. arXiv preprint arXiv:2411.08977

    arXiv 2024

  5. [5]

    Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337--351

    2023

  6. [6]

    Leif Azzopardi and Yashar Moshfeghi. 2024. Prism: a methodology for auditing biases in large language models. arXiv preprint arXiv:2410.18906

    arXiv 2024

  7. [7]

    Tilman Beck, Hendrik Schuff, Anne Lauscher, and Iryna Gurevych. 2024. Sensitivity, performance, robustness: Deconstructing the effect of sociodemographic prompting. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2589--2615

    2024

  8. [8]

    Pietro Bernardelle, Stefano Civelli, Leon Fr \"o hling, Riccardo Lunardi, Kevin Roitero, and Gianluca Demartini. 2025 a . Political ideology shifts in large language models. arXiv preprint arXiv:2508.16013

    arXiv 2025

Show all 52 references
  1. [9]

    Pietro Bernardelle, Leon Fr \"o hling, Stefano Civelli, Riccardo Lunardi, Kevin Roitero, and Gianluca Demartini. 2025 b . Mapping and influencing the political ideology of large language models using synthetic personas. In Companion Proceedings of the ACM on Web Conference 202...

    2025

  2. [10]

    Su Lin Blodgett, Solon Barocas, Hal Daum \'e III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454--5476

    2020

  3. [11]

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258

    2021 arXiv

  4. [12]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  5. [13]

    Louis Castricato, Nathan Lile, Rafael Rafailov, Jan-Philipp Fr \"a nken, and Chelsea Finn. 2025. Persona: A reproducible testbed for pluralistic alignment. In Proceedings of the 31st International Conference on Computational Linguistics, pages 11348--11368

    2025

  6. [14]

    Kai Chen, Zihao He, Jun Yan, Taiwei Shi, and Kristina Lerman. 2024. How susceptible are large language models to ideological manipulation? arXiv preprint arXiv:2402.11725

    2024 arXiv

  7. [15]

    Stefano Civelli, Pietro Bernardelle, and Gianluca Demartini. 2025. The impact of persona-based political perspectives on hateful content detection. In Companion Proceedings of the ACM on Web Conference 2025, pages 1963--1968

    2025

  8. [16]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv--2407

    2024

  9. [17]

    Esin Durmus, Karina Nguyen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. 2023. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXi...

    2023 arXiv

  10. [18]

    Margherita Fanton, Helena Bonaldi, Serra Sinem Tekiro g lu, and Marco Guerini. 2021. Human-in-the-loop for data collection: a multi-target counter narrative dataset to fight online hate speech. In Proceedings of the 59th Annual Meeting of the Association for Computational Ling...

    2021

  11. [19]

    Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models. In Proceedings of the 61st Annual Meeting of the Association for Computation...

    2023

  12. [20]

    Jan Fillies and Adrian Paschke. 2025. Improving hate speech classification with cross-taxonomy dataset integration. In Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2025)...

    2025

  13. [21]

    Leon Fr \"o hling, Gianluca Demartini, and Dennis Assenmacher. 2024. Personas with attitudes: Controlling llms for diverse data annotation. arXiv preprint arXiv:2410.11745

    2024 arXiv

  14. [22]

    Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094

    2024 arXiv

  15. [23]

    Tommaso Giorgi, Lorenzo Cima, Tiziano Fagni, Marco Avvenuti, and Stefano Cresci. 2024. Human and llm biases in hate speech annotations: A socio-demographic analysis of annotators and targets. arXiv preprint arXiv:2410.07991

    2024 arXiv

  16. [24]

    Patrick Haller, Ansar Aynetdinov, and Alan Akbik. 2024. Opiniongpt: Modelling explicit biases in instruction-tuned llms. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3:...

    2024

  17. [25]

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational...

    2022

  18. [26]

    Zihao He, Minh Duc Chu, Rebecca Dorn, Siyi Guo, and Kristina Lerman. 2024. Community-cross-instruct: Unsupervised instruction generation for aligning large language models to online communities. arXiv preprint arXiv:2406.12074

    2024 arXiv

  19. [27]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  20. [28]

    Jigsaw. 2019. Jigsaw unintended bias in toxicity classification. https://kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity- . Accessed: 2024-11-19

    2019

  21. [29]

    Gunther Jikeli, Sameer Karali, Daniel Miehling, and Katharina Soemer. 2023 a . https://doi.org/10.5281/zenodo.7932888 Antisemitism on Twitter: A Dataset for Machine Learning and Text Analytics

    2023 doi

  22. [30]

    Gunther Jikeli, Sameer Karali, and Katharina Soemer. 2023 b . https://doi.org/10.5281/zenodo.8147308 Hate Speech and Bias against Asians, Blacks, Jews, Latines, and Muslims: A Dataset for Machine Learning and Text Analytics

    2023 doi

  23. [31]

    Mehdi Khamassi, Marceau Nahon, and Raja Chatila. 2024. Strong and weak alignment of large language models with human values. Scientific Reports, 14(1):19399

    2024

  24. [32]

    Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew M Bean, Katerina Margatina, Rafael Mosquera-Gomez, Juan Ciro, Max Bartolo, Adina Williams, He He, et al. 2024. The prism alignment dataset: What participatory, representative and individualised human feedback reveals...

    2024

  25. [33]

    Andy Liu, Mona Diab, and Daniel Fried. 2024. Evaluating large language model biases in persona-steered generation. In Findings of the Association for Computational Linguistics ACL 2024, pages 9832--9850

    2024

  26. [34]

    Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14867--14875

    2021

  27. [35]

    Matthias Orlikowski, Jiaxin Pei, Paul R \"o ttger, Philipp Cimiano, David Jurgens, and Dirk Hovy. 2025. Beyond demographics: Fine-tuning large language models to predict individuals' subjective text perceptions. arXiv preprint arXiv:2502.20897

    2025 arXiv

  28. [36]

    Matthias Orlikowski, Paul R \"o ttger, Philipp Cimiano, and Dirk Hovy. 2023. The ecological fallacy in annotation: Modeling human label variation goes beyond sociodemographics. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2...

    2023

  29. [37]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  30. [38]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  31. [39]

    Julian Risch, Philipp Schmidt, and Ralf Krestel. 2021. Data integration for toxic comment classification: Making more than 40 datasets easily accessible in one unified format. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 157--163

    2021

  32. [40]

    Paul R \"o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuetze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. In Proceedings of the 62nd Annua...

    2024

  33. [41]

    Paul R \"o ttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. Hatecheck: Functional tests for hate speech detection models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int...

    2021

  34. [42]

    Pratik Sachdeva, Renata Barreto, Geoff Bacon, Alexander Sahn, Claudia Von Vacano, and Chris Kennedy. 2022. The measuring hate speech corpus: Leveraging rasch measurement theory for data perspectivism. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP@ LREC2...

    2022

  35. [43]

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971--30004. PMLR

    2023

  36. [44]

    Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A Smith. 2021. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. arXiv preprint arXiv:2111.07997

    2021 arXiv

  37. [45]

    Pavel Shvaiko and J \'e r \^o me Euzenat. 2011. Ontology matching: state of the art and future challenges. IEEE Transactions on knowledge and data engineering, 25(1):158--176

    2011

  38. [46]

    Brittany C Solomon, Matthew EK Hall, Abigail Hemmen, and James N Druckman. 2024. Illusory interparty disagreement: Partisans agree on what hate speech to censor but do not know it. Proceedings of the National Academy of Sciences, 121(39):e2402428121

    2024

  39. [47]

    Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. 2024. Position: a roadmap to pluralistic alignment. In Proceedings of the 41st International Conference on ...

    2024

  40. [48]

    Bertie Vidgen and Leon Derczynski. 2020. Directions in abusive language training data, a systematic review: Garbage in, garbage out. Plos one, 15(12):e0243300

    2020

  41. [49]

    Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tromble. 2021 a . Introducing cad: the contextual abuse dataset. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn...

    2021

  42. [50]

    Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021 b . Learning from the worst: Dynamically generated datasets to improve online hate detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internation...

    2021

  43. [51]

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359

    2021 arXiv

  44. [52]

    Mingqian Zheng, Jiaxin Pei, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. 2024. When” a helpful assistant” is not really helpful: Personas in system prompts do not improve performances of large language models. In Findings of the Association for Computational Linguisti...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.