{"id":"642f2510-a43b-42e6-b339-e46852da6952","arxiv_id":"2412.16783","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SUBDATA standardizes ten hate speech datasets into one taxonomy, and a demonstration shows left-aligned LLM personas classify more hate speech than right-aligned personas across all target groups.","lead":"The paper presents SUBDATA, an open-source Python library that merges ten hate speech datasets into a standardized format with a unified taxonomy of target groups. It also proposes a theory-driven framework for evaluating whether LLMs aligned with different political views classify hate speech differently, and illustrates it with three open models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The manual label-to-taxonomy mapping is the load-bearing step, and the paper offers no reliability or sensitivity evidence that the mapping's acknowledged subjective choices preserve cross-dataset comparability.","rationale":"The paper's central deliverable is SUBDATA's standardized datasets, and the claim that the unified taxonomy makes heterogeneous hate speech datasets semantically comparable for LLM perspective-alignment evaluation. The released open-source code and documented API are real evidence in favor, but they do not by themselves establish the validity of the semantic mappings. The load-bearing issue is exactly the manual mapping in Section 4.2: if the mapping decisions are inconsistent or arbitrary, then create_target_dataset() assembles instances under a target label that may not correspond to the same construct across the ten source datasets. A comparison of detection rates for 'blacks', 'jews', or 'migrants' across datasets then conflates dataset-specific annotation conventions, text distributions, and target definitions. Section 4.3 and the Limitations explicitly acknowledge the subjectivity of cases like 'mexicans' being placed in race versus origin and 'LGBTQ+' mixing gender and sexuality, and they state that the library inherits annotation errors without re-labeling. The paper provides no inter-annotator reliability measure for the mapping, no quantitative accounting of how many instances are affected by ambiguous decisions, and no sensitivity analysis. The example experiment in Section 6 is also missing a neutral-persona baseline, but that is a limitation of the demonstration, not of the core resource claim. The mapping reliability, by contrast, determines whether the resource itself enables the claimed consistent comparison. The reader's conditional verdict already captures this weakness, so the correct outcome is unchanged rather than a stricter rejection. The concrete test proposed here—a sensitivity analysis using the library's own update_taxonomy and update_mapping functions—would directly show whether the acknowledged subjective choices alter the substantive conclusions, thereby settling whether the concern lands.","tokens_in":15784,"tokens_out":7201,"duration_ms":70254,"concrete_test":"Run a mapping-robustness check using SUBDATA's own customization API. Construct two configurations: (A) the default taxonomy/mapping in the released repository; (B) a variant implementing the plausible alternatives acknowledged in Section 4.3—map 'mexicans' and related labels into an origin-level group instead of 'latin', split 'LGBTQ+' into separate gender and sexuality targets, and move other documented ambiguous keywords according to source-dataset documentation. For each configuration, rebuild the ten target-group datasets with create_target_dataset() and recompute the odds ratios in Table 3. If any target-category OR changes sign or loses/gains significance, or if the pool of instances for any target shifts by more than 5%, the standardization is not robust to the paper's own acknowledged mapping subjectivity and the comparability claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 defines a manual mapping from original dataset keywords to SUBDATA's unified taxonomy, and Section 4.3 and the Limitations concede that this process involves subjective judgment (e.g., 'mexicans' placed in origin vs. race; 'LGBTQ+' kept as a single heterogeneous category; intersectional targets simplified) and that the library inherits source annotation errors without re-labeling. The central claim—that SUBDATA standardizes heterogeneous datasets into a comparable resource for theory-driven evaluation—requires that the mapping preserves the construct being measured. No evidence is offered for that preservation: there is no inter-annotator agreement study for the mapping, no audit of the fraction of instances routed through ambiguous judgment calls, and no sensitivity analysis showing that plausible alternative mappings leave the standardized datasets or the example results in Table 3 unchanged. Because create_target_dataset() exposes only final mapped instances, users cannot tell which instances depend on contested decisions. If a substantial share of instances flows through such mappings, cross-dataset comparisons of hate speech detection rates by target group are not guaranteed to compare the same demographic construct, and the theory-driven conclusions in Section 6 become mapping-dependent rather than dataset-independent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SUBDATA, an open-source Python library that standardizes ten hate speech datasets under a unified taxonomy of target demographic groups, and proposes a theory-driven workflow for evaluating whether persona-prompted LLMs reflect different political perspectives in hate speech detection. The authors document the dataset collection process, the manual mapping from original labels to the unified taxonomy, and the customization functions of the library. They then present an example application in which 20 left-leaning and 20 right-leaning personas per model prompt Mistral-7B, Llama-3.1-8B, and Qwen2.5-7B to classify 25,000 instances targeting ten groups, reporting odds ratios that indicate left-leaning personas detect more hate speech across all target groups. The paper positions SUBDATA as a reusable resource and invites community extension to a multi-construct benchmark suite.","tokens_in":16049,"tokens_out":4221,"duration_ms":38774,"significance":"If the mapping from original dataset labels to the unified taxonomy is reliable, SUBDATA addresses a real gap: cross-study comparability in subjective NLP tasks such as hate speech detection. The paper's strengths include a functional open-source library on GitHub and PyPI, documented core functions, a concrete example experiment with 1,000,000 inferences per model, and unusually transparent discussion of the mapping's subjective components and the library's inherited annotation errors. The convergence with the independent taxonomy of Fillies and Paschke (2025) is also encouraging. However, the central contribution is conditional on the validity of the manual label-to-taxonomy mapping, and the example experiment currently lacks the design features needed to support its stronger interpretive claims. With additional reliability and sensitivity evidence, this would be a useful resource for the community.","major_comments":[{"comment":"The central claim that SUBDATA standardizes heterogeneous datasets into a comparable resource rests on the manual keyword-to-taxonomy mapping, yet the paper provides no reliability or sensitivity evidence for this mapping. The authors concede in §4.3 and the Limitations that decisions such as placing 'mexicans' in origin versus race, keeping 'LGBTQ+' as a single heterogeneous category, and simplifying intersectional targets involve subjective judgment, and that the library inherits source annotation errors without re-labeling. Since create_target_dataset() in Appendix A exposes only final mapped instances, users cannot quantify which instances depend on contested decisions. I recommend reporting inter-annotator agreement on a sample of mapping decisions, auditing the fraction of instances routed through ambiguous judgment calls, and rerunning the §6 example under plausible alternative mappings to show that the conclusions are not mapping-dependent.","section":"§4.2/§4.3 and Limitations"},{"comment":"The significance tests underlying the odds ratios in Table 3 appear to treat the 25,000 instances as independent observations, but the data are clustered: instances are nested in source datasets, each instance is scored by 40 personas, and the 20 left and 20 right personas are a selected sample from each model's persona distribution. This clustering can deflate p-values and overstate the consistency of the effects. The analysis should use mixed-effects models with random intercepts for instance, persona, and source dataset, or cluster-robust bootstrap inference resampled at the persona and instance levels. Without such modeling, the significance stars in Table 3 do not support the strength of the claims in §6.2.","section":"§6.1/Table 3"},{"comment":"The interpretation that left-persona conditioning produces 'a general tightening of classification thresholds' requires a baseline, but the experiment includes no neutral or unprompted condition. With only left and right persona conditions, an OR > 1 for all targets is equally compatible with right personas lowering sensitivity across the board, left personas raising it across the board, or some combination; the specific claim that left conditioning tightens thresholds is not identifiable from the two-condition design. Adding a neutral-persona condition and reporting per-target baseline detection rates would make the interpretation testable.","section":"§6.2"},{"comment":"The deduplication described in §3.3 removes repeated entry-target pairs within a dataset, but the paper does not state whether duplicate or near-duplicate texts are removed across the ten source datasets. Because the example pools 2,500 randomly sampled instances per target from the merged collection, cross-dataset text overlap could inflate the apparent consistency of persona effects and bias per-target samples toward particular sources. The authors should report cross-dataset duplication statistics and either deduplicate at the text level or include source dataset as a covariate or random effect in the §6 analysis.","section":"§3.3 and §6.1"}],"minor_comments":[{"comment":"The note on the 'All Dataset' row is ambiguous: it says the row reports 'the total unique target groups per category across all datasets,' but the reader may not know whether the parenthetical counts are summed across datasets or computed after unioning labels; please clarify with a concrete example such as the origin category.","section":"Table 1 note"},{"comment":"The claim that the taxonomy 'already aligns with independent efforts' and that there is 'large overlap' with Fillies and Paschke (2025) would be more convincing with a quantitative comparison of the two taxonomies, such as the number and proportion of shared target groups.","section":"§4.3"},{"comment":"The update_overview() function is documented twice with different parameter descriptions; consolidate the two entries and make the parameter list consistent, including overview_name, mapping_name, taxonomy_name, and hf_token.","section":"Appendix A"},{"comment":"The name 'SUBDATA' is typeset inconsistently in phrases such as 'SUBDATAprovides' and 'SUBDATA's; please standardize the formatting, for example by using a consistent non-breaking control or writing 'SubData' throughout.","section":"Throughout"},{"comment":"The choices of 2,500 statements per target and 20 personas per side are free parameters of the example, but no justification or sensitivity analysis is provided; a brief rationale or a small robustness check would help readers assess the stability of Table 3.","section":"§6.1"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of a resource paper well, and the open-source library is a concrete deliverable. My main concern is that the mapping reliability issue is load-bearing and currently unaddressed; I would encourage the editor to ask for inter-annotator reliability and sensitivity evidence before acceptance. The example experiment should also be framed as an illustration rather than a rigorous test of political alignment, and the authors' reliance on their own prior persona-mapping papers (Bernardelle et al., 2025a; Civelli et al., 2025) for the persona selection should be made more explicit, since those papers are not independent of the framework being demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nBottom line: this is a resource paper, and the resource is real. SubData is an open-source Python package, on PyPI and GitHub, that pulls together ten hate speech datasets and standardizes their target labels into a nine-category taxonomy. The functionality—create_target_dataset, taxonomy customization, mapping modification, LaTeX export of your exact taxonomy—is exactly the kind of thing that saves researchers weeks of wrangling with heterogeneous schemas. If you work on hate speech detection or LLM perspective alignment, this is a practical tool worth having.\n\nThe paper is also commendably honest. It explicitly acknowledges that the mapping from source keywords to standardized targets involves subjective judgment, that the library inherits annotation errors without re-labeling, and that some categories (e.g., LGBTQ+) are coarse. It cites an independently developed taxonomy (Fillies & Paschke 2025) that converges with theirs, which is a good sign. The example experiment is clearly described: persona-conditioned models, 20 left and 20 right personas, 25k sampled instances, results show left personas flag more content across every target group. That's a clean illustration of the framework.\n\nNow the soft spots. The mapping is the load-bearing step, and the paper offers no inter-annotator agreement, no audit of how many instances go through ambiguous mapping calls, and no sensitivity analysis. The stress-test concern is legitimate: without that evidence, cross-dataset comparability of 'anti-black' or 'anti-immigrant' content is not guaranteed. The paper's own limitations section says the mapping is manual and subjective, which is honest, but it doesn't address whether the standardization actually preserves the construct. A referee should push for at least a pilot reliability study or a sensitivity analysis with plausible alternative mappings.\n\nThe example experiment has design flaws: no neutral-persona baseline, and the significance tests likely treat each persona-instance pair as independent when the data are clustered by instance and persona. That's a real issue if the results are meant to be more than illustrative. Also, the personas come from the authors' own prior work, which is fine for a demo but worth noting.\n\nWho is this for? Anyone building on heterogeneous hate speech datasets or evaluating perspective alignment. It deserves a serious referee—not a desk reject—because the library itself is a contribution and the paper is transparent about its limits. The reviewer should ask for mapping validation and a tighter experimental design, but the core artifact is sound.\n\nMy recommendation: engage with it. I'd cite the library, and I'd send it to review with major revision expectations.","headline":"A genuinely useful open-source resource with a load-bearing mapping that the paper itself admits is subjective; the demo experiment is the weaker half, but the library deserves a serious look.","tokens_in":16538,"tokens_out":3282,"would_cite":true,"duration_ms":28143,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces SUBDATA, an open-source library that standardizes ten hate speech datasets into one target-group taxonomy, enabling theory-driven comparison of persona-aligned LLMs.","keywords":["hate speech detection","LLM perspective alignment","dataset standardization","target-group taxonomy","persona-conditioned LLMs","political alignment","subjective NLP tasks","cross-dataset evaluation"],"falsifier":"Have independent annotators re-label a stratified sample of instances that SUBDATA merges under a single target group (for example, 'jews') against one shared definition, and measure agreement with SUBDATA's assigned target; if agreement is low or varies systematically by source dataset, the taxonomy does not preserve the construct it is meant to standardize.","tokens_in":15589,"feed_emoji":"🗂️","tokens_out":11983,"duration_ms":90881,"temperature":0.7,"pith_summary":"Evaluating whether large language models reflect different human perspectives on subjective tasks is currently hobbled by inconsistent datasets: each hate speech study uses its own labels and target groups, so results cannot be pooled. This paper proposes SUBDATA, an open-source Python library that standardizes ten hate speech datasets under one nine-category taxonomy of target groups (race, religion, gender, migration, origin, disability, age, sexuality, political), so that instances targeting a given group become comparable across sources. On top of this standardization, it offers a theory-driven evaluation loop: derive a hypothesis from social or political theory, condition LLMs on different perspectives, and measure classification differences. In the demonstration, left-leaning personas flagged more hate speech than right-leaning personas for every target, including Christians, men, and whites, which the authors interpret as a general threshold shift rather than selective protection of minorities. If the mappings are reliable, the library gives researchers a shared, reproducible testbed for comparing perspective-aligned models on subjective classification.","feed_headline":"SUBDATA unifies ten hate speech datasets under one target taxonomy","feed_subtitle":"The library lets researchers compare how left- and right-aligned models judge speech targeting any demographic.","key_machinery":"The load-bearing mechanism is SUBDATA's unified taxonomy and the keyword-to-target mapping that feeds it. The taxonomy organizes targets into nine demographic categories and gives each a standardized name (such as jews, blacks, migrants, or lgbtq_unsp); the mapping converts raw source labels like 'Kikes' or 'jewish people' into those names, using original dataset documentation and consistent principles for ambiguous cases such as 'mexicans,' which some sources place in race and others in origin. This mapping is what makes aggregation across datasets meaningful, and the create_target_dataset() function operationalizes it by downloading, processing, and merging instances with harmonized labels. The theory-driven workflow then uses the merged resource as a controlled testbed: theory supplies a hypothesis, differently aligned LLM personas supply the comparison, and detection-rate differences quantify the alignment effect.","core_discovery":"The central claim is that the main obstacle to evaluating perspective-aligned LLMs on subjective tasks is not model capability but the absence of standardized data, and that careful taxonomy construction removes that obstacle. SUBDATA merges ten existing hate speech datasets into a unified set of target-group labels and exposes functions such as create_target_dataset() that assemble every available instance targeting a given group into one dataframe. The paper's empirical demonstration then shows that persona-conditioning on left-leaning perspectives raises hate speech detection rates relative to right-leaning perspectives across all examined target groups in three 7B-8B models, with the effect appearing for majority groups as well as minorities. The authors read this uniformity as a general tightening of classification thresholds induced by political perspective, not as group-specific protection of minorities.","pith_inferences":["One confound the paper leaves untested is that left-leaning personas may simply instruct the model to be more permissive about calling content hateful; a control with neutral personas and the same structured prompt would separate ideological alignment from general threshold effects.","The same taxonomy-and-mapping machinery should transfer to other subjective constructs, but each transfer needs its own validity check because the expected relationship between target groups and perspectives changes with the construct.","A direct validation study that re-annotates a sample of merged instances would turn the library's central assumption--that labels remain semantically comparable after mapping--into an auditable metric rather than a judgment call.","Since SUBDATA aggregates existing annotations without re-labeling, downstream conclusions compare harmonized source labels, not an independent measure of true hate speech prevalence."],"forward_implications":["Researchers can build one merged dataset for any target group across ten sources, making hate speech detection results from different studies directly comparable.","Alignment can be measured as relative differences between models conditioned on different perspectives, avoiding the need to settle contested ground-truth labels for subjective constructs.","The demonstrated effect implies that persona-induced political alignment shifts hate speech detection thresholds broadly, not selectively, at least for 7B-8B models.","Because SUBDATA lets researchers customize and export the taxonomy, studies can document their mapping modifications, keeping customized comparisons transparent.","The library's design extends naturally to additional subjective constructs and datasets, with misinformation named as the next target."],"supporting_citations":[{"why":"Supplies the theoretical starting point for the example experiment: partisan pretraining shifts hate speech detection, with left-leaning models flagging more minority-targeting content, which the paper extends to persona-conditioning.","marker":"Feng et al. (2023)"},{"why":"Provides the social-theory premise that Democrats are protective of minorities, used to derive the tested hypothesis about Democrat- versus Republican-aligned models.","marker":"Solomon et al., 2024"},{"why":"Contributes the method for selecting the 20 most left- and 20 most right-leaning personas and prior evidence that politically aligned personas affect hateful content detection.","marker":"Civelli et al. (2025)"},{"why":"Provides the mapping of persona-conditioned LLMs onto an ideological space that the paper uses to choose left- versus right-oriented personas.","marker":"Bernardelle et al. (2025a)"},{"why":"Used to identify candidate hate speech datasets and frames responsible-handling considerations that shape dataset selection.","marker":"Vidgen and Derczynski (2020)"},{"why":"Supplies a unified toxic-comment collection that gave structured access to candidate datasets during data collection.","marker":"Risch et al., 2021"},{"why":"Independent targeted hate speech taxonomy whose overlap with SUBDATA's is cited as evidence that the mapping choices converge on a generally accepted structure.","marker":"Fillies and Paschke (2025)"},{"why":"Underpins the acknowledged subjectivity of the mapping process, the paper's stated limitation for taxonomy construction.","marker":"Shvaiko and Euzenat, 2011"}],"fun_headline_variants":["SubData merges ten datasets to test LLM political bias","Unified hate speech sets reveal LLM perspective alignment","Library standardizes data to compare political LLM views","SubData: bridging datasets for theory-driven LLM evaluation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's comparisons stand or fall on the manual mapping from each dataset's original labels to SUBDATA's single taxonomy: unless 'hate speech against a given group' means the same thing after standardization in every source dataset, cross-dataset comparisons built on the merged data are invalid.","fun_headline_variants_meta":{"raw":{"variants":["SubData merges ten datasets to test LLM political bias","Unified hate speech sets reveal LLM perspective alignment","Library standardizes data to compare political LLM views","SubData: bridging datasets for theory-driven LLM evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000148,"raw_usage":{"total_tokens":1150,"prompt_tokens":870,"completion_tokens":280,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":214}},"tokens_in":486,"tokens_out":280,"duration_ms":3227,"temperature":1.0,"reasoning_tokens":214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:14:44.283474+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators re-label a stratified sample of instances that SUBDATA merges under a single target group (for example, 'jews') against one shared definition, and measure agreement with SUBDATA's assigned target; if agreement is low or varies systematically by source dataset, the taxonomy does not preserve the construct it is meant to standardize.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the social-theory premise that Democrats are protective of minorities, used to derive the tested hypothesis about Democrat- versus Republican-aligned models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Used to identify candidate hate speech datasets and frames responsible-handling considerations that shape dataset selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a unified toxic-comment collection that gave structured access to candidate datasets during data collection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Independent targeted hate speech taxonomy whose overlap with SUBDATA's is cited as evidence that the mapping choices converge on a generally accepted structure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Underpins the acknowledged subjectivity of the mapping process, the paper's stated limitation for taxonomy construction."}],"review_version":1}