Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance

T0 review · 2 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Standard LLM writing assistance takes sides on most contested issues, even when users ask for no stance.

desk verdict IssueBench is a real resource with a plausible central finding, but the stance classifier's neutral-to-directional errors and judge-subject overlap deserve a hard look before the headline numbers are taken at face value. read the letter →

arxiv 2502.08395 v3 pith:MV7OFLNZ submitted 2025-02-12 cs.CL

Paul Röttger , Musashi Hinck , Valentin Hofmann , Kobi Hackenburg , Valentina Pyatkin , Faeze Brahman , Dirk Hovy This is my paper · ORCID
classification cs.CL
keywords issuebiasLLMwritingassistancestanceclassificationpoliticalbenchmarkdatasetrealisticpromptsevaluationpartisanalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models are now writing essays, blogs, and stories for millions of users, and this paper asks whether those texts quietly take sides on contested political issues even when no side is requested. To answer that question, the authors built IssueBench, a dataset of 2.49 million realistic English writing-assistance prompts by combining 212 political issues (AI regulation, gender-inclusive language, and so on) with 3,916 prompt templates drawn from real user-LLM conversations. Evaluating ten state-of-the-art models, they find that every model expresses a consistent majority stance on at least 70% of the neutrally-framed issues, that these default stances are hard to steer away from, that biases are nearly identical across models, and that on 20 issues with voter data all models sit closer to US Democrat than Republican opinion. If the finding holds, everyday LLM writing help is not neutral: it systematically exposes readers to one perspective on issues where society is divided.

What carries the argument

The load-bearing artifact is IssueBench itself: a Cartesian product of 212 manually curated political issues (each in neutral, positive, and negative framing) and 3,916 writing-assistance templates extracted from five datasets of real user-LLM prompts, yielding 2,490,576 test prompts. Evaluation runs on a 1,000-template sample (636,000 prompts) per model, and each generated response is labelled by an automated stance classifier (Llama-3.1-70B) into a six-way taxonomy: only pro, mostly pro, neutral/ambivalent, mostly con, only con, or refusal. The measure of bias is the distribution of these labels across templates for a given issue, with an absolute majority (at least 50%) counted as a consistent stance. The same classifier and taxonomy make the cross-model similarity and partisan-alignment comparisons possible.

What would settle it

Take a random sample of final responses the classifier labelled 'only pro', 'only con', and 'neutral', have independent human annotators apply the same six-way taxonomy, and compare the labels. If human-classifier agreement on the polar labels falls well below the reported macro F1 (0.77 on the pilot set, 0.78 on the post-hoc set), or if most responses judged neutral or ambivalent by humans are being called polar by the classifier, then the claim that all models hold majority stances on most issues would need to be revised downward.

Watch

Extended reading notes

Core claim

The paper's central claim is that issue bias in LLM writing assistance is common and persistent: when prompted with 1,000 different neutral phrasings for each of 212 issues, all ten models produce an absolute-majority stance (at least 50% of responses sharing one label) on most issues, and for GPT-4o-mini that is 180 of 212 issues, with 111 showing a consistently positive or negative polarity. The same responses show that stronger default stances correlate with greater difficulty in steering the model to the opposite polarity, that pairwise divergence between models is tiny (largest average Jensen-Shannon divergence under 0.03), and that on 20 matched issues models align more strongly with Democrat voters (average absolute distance 0.27) than Republican voters (0.77). The authors frame these as descriptive findings about what models do in realistic use, not as proof of any single cause.

Load-bearing premise

The automated judge that decides whether each model response is pro, neutral, or con must be right most of the time; if it hears a side in neutral or hedged texts, the finding that models consistently take sides would be overstated.

Editorial extensions

If this is right

  • Users who ask an LLM for a text on a contested issue will usually get a text with a consistent slant, even when they asked for no slant; this holds for every model tested.
  • Default stances are sticky: the more positive or negative a model is by default on an issue, the harder it is to prompt it to write the opposite way, producing asymmetric stance distortion.
  • Because bias profiles are nearly identical across ten models from six families, switching to a different LLM will not, by itself, escape the dominant issue stances.
  • On the 20 issues with US voter-stance data, all models align more with Democrat than Republican positions, and on 8 issues they are more extreme (more progressive) than either party's voters.
  • IssueBench can be extended to new issues, templates, and languages, and downsampling experiments indicate that 250 templates with one response per prompt are enough for issue-level conclusions, cutting cost by roughly 20 times.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reported stance-classifier accuracy (0.77 macro F1, with most errors confusing 'only' and 'mostly') means the exact majority counts could be sensitive to label noise; the safest reading is that clear polar stances are common, not that the precise figures are exact.
  • Editorial inference: the near-identical biases across models, including differently trained open and closed systems, suggest a common source such as shared pre-training text or similar post-training values; this could be tested by training on deliberately rebalanced political corpora and checking whether IssueBench scores move.
  • Editorial inference: because the 20-issue partisan comparison is a convenience sample from a self-selected voter-opinion website, the Democrat-leaning result should not be generalized to all US political issues; an obvious extension is a representative issue set with high-quality survey data.
  • Editorial inference: the finding that default stances resist steering implies real-world mitigations should not rely on prompt engineering alone but on displaying multiple perspectives or letting users select a stance before generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. IssueBench constructs 2.49 million realistic English writing-assistance prompts by combining 212 political issues (each in neutral, positive, and negative framings) with 3,916 templates derived from five real user-LLM interaction datasets. The paper evaluates ten state-of-the-art LLMs: for each model it generates responses to 636,000 prompt variants, classifies every response stance with Llama-3.1-70B, and aggregates results per issue. It reports three main findings: (1) all models express a consistent majority stance on at least 70% of neutrally framed issues (Section 5); (2) models are very similar in their issue-level stance distributions, with the largest average pairwise JSD below 0.03 (Section 7); and (3) on a subset of 20 issues matched to iSideWith data, all models align more closely with Democrat than Republican voter opinion (Section 8). The paper also analyzes steerability and shows that stronger default stances are harder to override (Section 6).

Significance. The paper addresses an important gap in LLM bias evaluation by moving from multiple-choice questionnaires to open-ended writing-assistance prompts grounded in real user interactions. The dataset construction is careful and transparent: filtering steps are human-validated with high inter-annotator agreement, the template and issue extraction is modular and reproducible, and the authors release code and resources. The downsampling and threshold robustness checks are also thoughtful additions. If the findings hold, they would provide strong evidence that current LLMs exhibit systematic and shared issue stances in realistic usage, a result with clear societal relevance. The paper is appropriately cautious in some places, notably in restricting the partisan analysis to 20 issues and in acknowledging that causal claims about the origins of the observed biases are out of scope. The main weakness is that all headline results depend on a single automated stance classifier with moderate macro F1 and substantial neutral-to-directional confusion, and the paper does not yet bound the effect of this measurement error on its central claims.

major comments (2)
  1. [4.1 (Tables 8 and 10)] The stance classifier's errors are concentrated exactly where the headline findings are most sensitive. In the pilot gold standard (Table 8), 38 of 93 gold-neutral responses (41%) are labeled as 'mostly pro' or 'mostly con' (labels 2 and 4), and the post-hoc set (Table 10) shows the same pattern: 30 of 157 gold-neutral responses are mislabeled as labels 2 or 4. Because Section 5 defines a clear stance tendency as an absolute majority (>=50%) of a single label across templates, systematic neutral-to-directional misclassification can manufacture a majority directional stance on contested issues whose true responses are balanced or hedged. The post-hoc validation reports only aggregate macro F1=0.78; it does not report error direction by issue, by framing, or by target model, nor whether the judge's errors correlate with its own responses. Please add such analyses and re-estimate the central statistics under conservative assumptions (e.g., treating uncertain classifications as neutral, or labeling a subsample with an independent judge). Without these numbers, the sign and size of measurement bias on the 'majority stance on most issues' result are unconstrained.
  2. [7 (Figure 6)] The claim that all models exhibit strikingly similar issue biases (largest average pairwise JSD <0.03) rests on stance distributions produced by a single judge, Llama-3.1-70B, which is itself one of the ten evaluated models. Any issue-specific labeling tendency of this judge will be applied uniformly across all models, mechanically reducing apparent cross-model divergence. To establish that the similarity result reflects model behavior rather than shared measurement error, please provide a quantitative bound: for example, compute pairwise JSDs from human labels on the 900-response post-hoc set, or re-label a random subset with a second independent judge and verify that the maximum pairwise JSD remains below 0.03. This is necessary because the post-hoc validation in Section 4.1 does not measure the judge's per-issue directional bias.
minor comments (5)
  1. [5 (Table 1)] Because the stance taxonomy includes a neutral label, the phrase 'consistent stance' conflates directional bias with consistent neutrality. For example, GPT-4o-mini has a neutral majority on 69 of its 180 majority-stance issues. Please report directional-bias counts (labels 1/2/4/5) separately, or rephrase the abstract and Section 5 to distinguish 'directional issue bias' from 'consistent neutral or ambivalent tendency.'
  2. [Appendix I (T5 prompt)] The stance classification prompt T5 contains apparent typos in the definitions for labels 4 and 5: under label 4, the example states 'This would be a core example of "2"' (presumably '4'), and under label 5, the text says 'like [issue_pro]' where the contrast with the con side seems intended. Since this prompt is used for all 25 million classifications, please correct the text and, if possible, assess whether these errors affected classifier performance.
  3. [4.2 and Appendix H] Grok-3-mini and DeepSeek-v3 were evaluated with a single response at temperature = 0, while the other models used five responses at temperature = 1. The downsampling analysis in Appendix H was performed on the earlier eight models; please confirm that the single-response protocol does not change the conclusions for Grok and DeepSeek, or provide specific evidence for those two models.
  4. [Appendix J] The text states 'In total, we generate 25.818m responses,' but 8 models × 5 responses × 636,000 prompts plus 2 models × 1 response × 636,000 prompts gives 26.712 million. Please reconcile this arithmetic or clarify what is being counted.
  5. [8 (Figure 8)] The mapping between IssueBench issues and iSideWith questions should be made fully explicit in a table, including the exact wording of each iSideWith question, since the partisan comparison depends on precise textual correspondence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: IssueBench's results are empirical measurements grounded in external data and human annotations, not forced by definition or self-citation.

full rationale

Walking the derivation chain, the paper never defines an input in terms of an output, fits a parameter and then renames it a prediction, or imports a load-bearing conclusion from the authors' own prior work. The benchmark is built from five external user-LLM interaction datasets, with issue and template extraction validated by human annotation (Krippendorff's alpha 0.97 and 0.96) and by GPT-4 classifiers that are themselves checked against those gold standards. The central findings are computed from actual model responses labeled by a zero-shot stance classifier (Llama-3.1-70B) that was selected on a human-annotated gold standard and re-validated post-hoc on 900 fresh responses (0.78 macro F1); the resulting stance distributions are not algebraically determined by the classifier's own design, by the benchmark's construction, or by any fitted parameter. The Llama-3.1-70B judge also being one of the ten evaluated models is a potential measurement confound, but it is an empirical validity question about classifier error, not a circular step in the derivation, and the paper reports human-validated directional accuracy (pro/con flips are rare). The partisan-bias finding uses external iSideWith voter data on 20 matched issues; again, the comparison is arithmetic and independent. Citations to the authors' prior work (e.g., Röttger et al. 2024) motivate realism and position the work in related literature, but no core claim reduces to those citations. No self-definitional, fitted-input, or self-citation-chain circularity is present.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The central claims rest on the quality of the automated stance labels, the representativeness of the source interaction datasets, and the external voter data. No new physical or mathematical entities are introduced. The hand-chosen majority threshold is the only explicit numeric free parameter; the iSideWith 20-issue subset is an acknowledged data-selection constraint.

free parameters (1)
  • Majority stance threshold = >=50% of responses sharing the same stance label
    Chosen by hand to operationalize 'clear stance tendency' in Sections 5 and 6; the paper provides threshold robustness checks in Appendix K, but the specific value is arbitrary and directly determines the counts in Tables 1 and 3.
assumptions (6)
  • domain assumption The Llama-3.1-70B stance classifier labels approximate human judgments of issue stance.
    Section 4.1 uses this classifier for all model responses; its macro F1 is 0.77 on the pilot set and 0.78 on a post-hoc set, with non-trivial confusion between adjacent stance labels.
  • domain assumption GPT-4 relevance and writing-assistance classifiers are accurate enough for pipeline filtering.
    Sections 3.2 and 3.3 rely on classifiers with macro F1 of 0.89 and 0.93; filtering errors propagate into issue and template selection.
  • domain assumption The five source datasets and the resulting prompt clusters represent realistic LLM writing assistance use.
    Section 3.1 assumes these logs are representative; the authors acknowledge cultural and domain limits in Section 3.5.
  • domain assumption Removing polarity and issue mentions from prompts (with 'major edits' in 54.3% of templates) preserves realism while enabling controlled framing.
    Section 3.3 instructs annotators to make no other edits, but major edits still change prompts; any systematic change could affect measured stance distributions.
  • domain assumption iSideWith self-identified US voters approximate party positions for the 20 matched issues.
    Section 8 uses iSideWith data and acknowledges self-selection and limited representativeness; the 20-issue subset is not a random sample of the US political issue space.
  • standard math Jensen-Shannon divergence is an appropriate dissimilarity measure for stance distributions.
    Section 7 uses average JSD to quantify cross-model similarity; JSD is a standard measure but the aggregation choice affects the 'strikingly similar' claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance." pith.science (2026). https://pith.science/paper/MV7OFLNZ

@misc{pith2026250208395,
  author       = {Pith},
  title        = {Pith review of: IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MV7OFLNZ}},
  note         = {Machine review of arXiv:2502.08395}
}
read the original abstract

Large language models (LLMs) are helping millions of users write texts about diverse issues, and in doing so expose users to different ideas and perspectives. This creates concerns about issue bias, where an LLM tends to present just one perspective on a given issue, which in turn may influence how users think about this issue. So far, it has not been possible to measure which issue biases LLMs manifest in real user interactions, making it difficult to address the risks from biased LLMs. Therefore, we create IssueBench: a set of 2.49m realistic English-language prompts to measure issue bias in LLM writing assistance, which we construct based on 3.9k templates (e.g. "write a blog about") and 212 political issues (e.g. "AI regulation") from real user interactions. Using IssueBench, we show that issue biases are common and persistent in 10 state-of-the-art LLMs. We also show that biases are very similar across models, and that all models align more with US Democrat than Republican voter opinion on a subset of issues. IssueBench can easily be adapted to include other issues, templates, or tasks. By enabling robust and realistic measurement, we hope that IssueBench can bring a new quality of evidence to ongoing discussions about LLM biases and how to address them.

Figures

Figures reproduced from arXiv: 2502.08395 by the authors.

Figure 1
Figure 1. The IssueBench evaluation protocol. We create IssueBench by combining thousands of writing assistance prompt templates with hundreds of issues. We then evaluate LLMs for issue-specific biases in the stance of their responses across templates. To answer this question, we introduce Issue￾Bench: an English-language dataset of 2,490,576 realistic writing assistance prompts covering a di￾versity of political issues. Star… view at source ↗
Figure 2
Figure 2. Issue curation process. We review clusters of real user prompts to extract realistic issues, supported by LLM-suggested cluster descriptions (§3.2). The example shown here is one of 212 issues in IssueBench. For each issue, we create a neutral, positive, and negative framing version. “Grammar correction for various written texts.”) and 6 clusters about forecasting future events (e.g. “Next UK general election date a… view at source ↗
Figure 3
Figure 3. Most common writing formats and styles, based on the 15 most frequent nouns (top) and adjec￾tives (bottom) across the 3,916 unique templates. large number of prompts, we conduct a downsam￾pling analysis, showing that future work could use even fewer templates without meaningful impact on issue-level results (see Appendix H). 3.5 Outlook: Expanding IssueBench The construction of IssueBench is fully modular, which mea… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Model response taxonomy and exemplars. We evaluate LLMs on IssueBench by classifying each model response for which stance it expresses relative to the specific issue of each input prompt (e.g. “capitalism”) [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Correlation in stance response proportions across issue framings across all ten models we test. Significance at p<0.05 (*), p<0.01 (**) and p<0.001 (***). Response taxonomy (“1”, etc.) as in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Pairwise model similarity as measured by average JSD between response stance distributions across all 212 neutrally-framed issues in IssueBench. JSD is measured on a scale from 0 to 1, with 0 indicat￾ing maximum similarity and 1 maximum divergence. We find that all mod…
Figure 7
Figure 7. Figure 7: Issue-level similarity in response stance distributions across models as measured by pairwise JSD averaged across model pairs. We zoom in on the two issues where models behave least similarly to each other. Similarity, however, may not be evenly dis￾tributed across iss…
Figure 8
Figure 8. Figure 8: Issue-level model vs. partisan bias on the 20 issues in IssueBench for which we collected Republican and Democrat voter stances from iSideWith.com. The x-axis shows the difference in pro vs. con voter shares for each issue. is Llama-3.1-70B, is Qwen-2.5-72B, is OLMo-2-…
Figure 9
Figure 9. Figure 9: Impact of downsampling IssueBench tem￾plates as measured by average JSD between response stance distributions for the downsampled set (x-axis = number of templates), and the distributions based on the full set of 1,000 templates. Parentheses show stan￾dard deviation ac…
Figure 10
Figure 10. Figure 10: Impact of downsampling IssueBench calls as measured by average JSD between response stance distributions for the specific call ID and the dis￾tributions based on all five calls. Overall, our downsampling analysis shows that our November 2024 experiments were much more…
Figure 11
Figure 11. Figure 11: Pairwise model similarity as measured by average JSD between response stance distribu￾tions across all 212 positively-framed issues (top) and negatively-framed issues (bottom) in IssueBench. JSD is measured on a scale from 0 to 1, with 0 indicating maximum similarity …
Figure 12
Figure 12. Figure 12: Distribution of plurality response proportions across all 212 neutrally-framed issues for each model we test. (top) The shaded area corresponds to the results with a 50% threshold in [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: UMAP plot of all 212 issues in IssueBench. We compute embeddings for each neutrally-framed issue using SentenceTransformers (Reimers and Gurevych, 2019) and then reduce their dimensionality using UMAP. This is a high-resolution plot. Please zoom in for inspection [PI…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race

    cs.CL 2025-05 conditional novelty 7.0 of 10

    Alignment on Llama 3 reduces explicit bias but amplifies implicit bias, because aligned models no longer represent 'black' and 'white' as racial concepts in ambiguous contexts.

Reference graph

Works this paper leans on

53 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  4. [4]

    Yejin Bang, Delong Chen, Nayeon Lee, and Pascale Fung. 2024. https://doi.org/10.18653/v1/2024.acl-long.600 Measuring political bias in large language models: What is said and how it is said . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11142--11159, Bangkok, Thailand. Associatio...

    doi:10.18653/v1/2024.acl-long.600 2024

  5. [5]

    Maarten Buyl, Alexander Rogiers, Sander Noels, Guillaume Bied, Iris Dominguez-Catena, Edith Heiter, Iman Johary, Alexandru-Cristian Mara, Rapha \"e l Romero, Jefrey Lijffijt, et al. 2024. Large language models reflect the ideology of their creators. arXiv preprint arXiv:2410.18417

    arXiv 2024

  6. [6]

    Campello, Davoud Moulavi, and J \"o rg Sander

    Ricardo J.G.B. Campello, Davoud Moulavi, and J \"o rg Sander. 2013. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining, pages 160--172. Springer

    work page 2013

  7. [7]

    Kai Chen, Zihao He, Jun Yan, Taiwei Shi, and Kristina Lerman. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.952 How susceptible are large language models to ideological manipulation? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17140--17161, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.emnlp-main.952 2024

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

Show all 53 references
  1. [9]

    Esin Durmus, Liane Lovitt, Alex Tamkin, Stuart Ritchie, Jack Clark, and Deep Ganguli. 2024 a . https://www.anthropic.com/news/measuring-model-persuasiveness Measuring the persuasiveness of language models . Anthropic.com - last accessed 06.09.2025

    2024

  2. [10]

    Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024 ...

    2024

  3. [11]

    Mats Faulborn, Indira Sen, Max Pellert, Andreas Spitz, and David Garcia. 2025. https://aclanthology.org/2025.acl-long.1529/ Only a little to the left: A theory-grounded measure of political bias in large language models . In Proceedings of the 63rd Annual Meeting of the Associ...

    2025

  4. [12]

    Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. https://doi.org/10.18653/v1/2023.acl-long.656 From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models . In Proceedings of the 61st An...

    2023 doi

  5. [13]

    Janell Fetterolf and Laura Clancy. 2024. https://www.pewresearch.org/short-reads/2024/05/15/support-for-legal-abortion-is-widespread-in-many-countries-especially-in-europe/ Support for legal abortion is widespread in many places, especially in europe . Pew Research - last acce...

    2024

  6. [14]

    Jillian Fisher, Ruth Elisabeth Appel, Chan Young Park, Yujin Potter, Liwei Jiang, Taylor Sorensen, Shangbin Feng, Yulia Tsvetkov, Margaret Roberts, Jennifer Pan, Dawn Song, and Yejin Choi. 2025. https://openreview.net/forum?id=H72JEXAPwo Position: Political neutrality in AI is...

    2025

  7. [15]

    Luciano Floridi. 2010. Information: A Very Short Introduction . Oxford University Press

    2010

  8. [16]

    Sasuke Fujimoto and Takemoto Kazuhiro. 2023. Revisiting the political biases of chatgpt. Frontiers in Artificial Intelligence, 6

    2023

  9. [17]

    Suyash Fulay, William Brannon, Shrestha Mohanty, Cassandra Overney, Elinor Poole-Dayan, Deb Roy, and Jad Kabbara. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.508 On the relationship between truth and political bias in language models . In Proceedings of the 2024 Conferen...

    2024 doi

  10. [18]

    Geiger and Nikki Graf

    A.W. Geiger and Nikki Graf. 2019. https://www.pewresearch.org/short-reads/2019/09/05/gender-neutral-pronouns/ About one-in-five u.s. adults know someone who goes by a gender-neutral pronoun . Pew Research - last accessed 06.09.2025

    2019

  11. [19]

    Goldstein, Jason Chao, Shelby Grossman, Alex Stamos, and Michael Tomz

    Josh A. Goldstein, Jason Chao, Shelby Grossman, Alex Stamos, and Michael Tomz. 2024. https://doi.org/10.1093/pnasnexus/pgae034 How persuasive is ai-generated propaganda? PNAS Nexus, 3(2):pgae034

    2024 doi

  12. [20]

    Tappin, Paul R \"o ttger, Scott A

    Kobi Hackenburg, Ben M. Tappin, Paul R \"o ttger, Scott A. Hale, Jonathan Bright, and Helen Margetts. 2025. Scaling language model size yields diminishing returns for single-message political persuasion. Proceedings of the National Academy of Sciences, 122(10):e2413443122

    2025

  13. [21]

    Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. The political ideology of conversational ai: Converging evidence on chatgpt's pro-environmental, left-libertarian orientation. arXiv preprint arXiv:2301.01768

    2023 arXiv

  14. [22]

    Amir Kargaran, Ayyoob Imani, Fran c ois Yvon, and Hinrich Schuetze. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.410 G lot LID : Language identification for low-resource languages . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6155--...

    2023 doi

  15. [23]

    Hannah Rose Kirk, Alexander Whitefield, Paul R \"o ttger, Andrew Michael Bean, Katerina Margatina, Rafael Mosquera, Juan Manuel Ciro, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. 2024. https://openreview.net/forum?id=DFr5hteojx The PRISM alignment data...

    2024

  16. [24]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  17. [25]

    T urning right

    Yifei Liu, Yuang Panwang, and Chao Gu. 2025. “ T urning right”? A n experimental study on the political value shift in large language models. Humanities and Social Sciences Communications, 12(1):1--10

    2025

  18. [26]

    Kristian Lum, Jacy Reese Anthis, Kevin Robinson, Chirag Nagpal, and Alexander Nicholas D ' Amour. 2025. https://doi.org/10.18653/v1/2025.acl-long.7 Bias in language models: Beyond trick tests and towards RUTE d evaluation . In Proceedings of the 63rd Annual Meeting of the Asso...

    2025 doi

  19. [27]

    Mantas Mazeika, Xuwang Yin, Rishub Tamirisa, Jaehyuk Lim, Bruce W Lee, Richard Ren, Long Phan, Norman Mu, Adam Khoja, Oliver Zhang, et al. 2025. Utility engineering: Analyzing and controlling emergent value systems in ais. arXiv preprint arXiv:2502.08640

    2025 arXiv

  20. [28]

    Leland McInnes, John Healy, and Steve Astels. 2017. hdbscan: Hierarchical density based clustering. J. Open Source Softw., 2(11):205

    2017

  21. [29]

    Jared Moore, Tanvi Deshpande, and Diyi Yang. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.891 Are large language models consistent over value-laden questions? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15185--15221, Miami, Florida,...

    2024 doi

  22. [30]

    Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. 2023. More human than human: Measuring chatgpt political bias. Public Choice, pages 1--21

    2023

  23. [31]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Mi...

    2024 arXiv

  24. [32]

    OpenAI. 2024. https://cdn.openai.com/spec/model-spec-2024-05-08.html Model spec . OpenAI Website - last accessed 06.09.2025

    2024

  25. [33]

    Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, and Jiawei Han. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.146 The shifted and the overlooked: A task-oriented investigation of user- GPT interactions . In Proc...

    2023 doi

  26. [34]

    Yujin Potter, Shiyang Lai, Junsol Kim, James Evans, and Dawn Song. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.244 Hidden persuaders: LLM s' political leaning and their influence on voters . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language P...

    2024 doi

  27. [35]

    Qwen. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models . Qwen Team Blog - last accessed 06.09.2025

    2024

  28. [36]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  29. [37]

    Luca Rettenberger, Markus Reischl, and Mark Schutera. 2025. Assessing political bias in large language models. Journal of Computational Social Science, 8(2):1--17

    2025

  30. [38]

    Reuters. 2024. https://www.reuters.com/technology/artificial-intelligence/openai-says-chatgpts-weekly-users-have-grown-200-million-2024-08-29/ Openai says chatgpt's weekly users have grown to 200 million . Reuters - last accessed 06.09.2025

    2024

  31. [39]

    Paul R \"o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuetze, and Dirk Hovy. 2024. https://doi.org/10.18653/v1/2024.acl-long.816 Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large lan...

    2024 doi

  32. [40]

    Paul R \"o ttger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. https://doi.org/10.18653/v1/2022.naacl-main.13 Two contrasting data annotation paradigms for subjective NLP tasks . In Proceedings of the 2022 Conference of the North American Chapter of the Association...

    2022 doi

  33. [41]

    David Rozado. 2023. The political biases of chatgpt. Social Sciences, 12(3):148

    2023

  34. [42]

    David Rozado. 2024. https://doi.org/10.1371/journal.pone.0306621 The political preferences of LLMs . PLOS ONE, 19(7):e0306621

    2024 doi

  35. [43]

    David Rozado. 2025. Measuring political preferences in ai systems: An integrative approach. arXiv preprint arXiv:2503.10649

    2025 arXiv

  36. [44]

    J \'e r \^o me Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, Moritz Roidl, and Markus Pauly. 2024. The self-perception and political biases of chatgpt. Human Behavior and Emerging Technologies, 2024(1):7115633

    2024

  37. [45]

    Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971--30004. PMLR

    2023

  38. [46]

    Michael Saxon, Ari Holtzman, Peter West, William Yang Wang, and Naomi Saphra. 2024. https://openreview.net/forum?id=bttKwCZDkm Benchmarks as microscopes: A call for model metrology . In First Conference on Language Modeling

    2024

  39. [47]

    Amir Taubenfeld, Yaniv Dover, Roi Reichart, and Ariel Goldstein. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.16 Systematic biases in LLM simulations of debates . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 251--267, Mi...

    2024 doi

  40. [48]

    Filip Trhl \'i k and Pontus Stenetorp. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.255 Quantifying generative media bias with a corpus of real-world and generated news articles . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4420--44...

    2024 doi

  41. [49]

    Alec Tyson, Cary Funk, and Brian Kennedy. 2023. https://www.pewresearch.org/short-reads/2023/08/09/what-the-data-says-about-americans-views-of-climate-change/ What the data says about A mericans’ views of climate change . Pew Research - last accessed 06.09.2025

    2023

  42. [50]

    Sean J Westwood, Justin Grimmer, and Andrew B Hall. 2025. Measuring perceived slant in large language models through user evaluations. Stanford Graduate Business School, Working Paper No. 4262

    2025

  43. [51]

    Dustin Wright, Arnav Arora, Nadav Borenstein, Srishti Yadav, Serge Belongie, and Isabelle Augenstein. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.995 LLM tropes: Revealing fine-grained values and opinions in large language models . In Findings of the Association for ...

    2024 doi

  44. [52]

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. https://openreview.net/forum?id=Bl8u7ZRlbM (inthe)wildchat: 570k chat GPT interaction logs in the wild . In The Twelfth International Conference on Learning Representations

    2024

  45. [53]

    Gonzalez, Ion Stoica, and Hao Zhang

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. 2024. https://openreview.net/forum?id=BOfDKxfwt0 LMSYS -chat-1m: A large-scale real-world LLM convers...

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.