Pith. sign in

REVIEW 6 major objections 6 minor 50 references

A Multifaceted Analysis of Social Biases in Large Language Models

T0 review · 6 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that four widely used LLMs, though aligned to be neutral, each exhibit distinct biases in politics, ideology, geopolitical alliance, language, and gender.

desk verdict A broad, readable five-dimensional bias audit of four LLMs, but the model-level rankings are built on single stochastic draws with no error bars — plausible as hypotheses, not yet as measurements. read the letter →

arxiv 2512.15792 v4 pith:XUDQMHHV submitted 2025-12-16 cs.CY cs.AIcs.CL

Xulang Zhang , Rui Mao , Erik Cambria This is my paper · ORCID
classification cs.CYcs.AIcs.CL
keywords socialbiaspoliticalideologicalgeopoliticalalliancelanguagegenderlargemodelsWorldValuesSurvey
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that alignment to 'neutral' and 'impartial' behavior does not make large language models bias-free: four widely used models—Qwen, DeepSeek, Gemini, and GPT—each display their own characteristic leanings across five domains. News summarization shows Gemini's neutral summaries sit closer to right-leaning reporting and GPT's slightly closer to left-leaning reporting; stance classification likewise finds Gemini more aligned with the right and GPT with the left. In United Nations voting, each model shows a distinct pattern of agreement and disagreement with real delegates—Gemini, for instance, disagrees with the US while agreeing with China and North Korea. On World Values Survey questions, all four models align more with women's values than men's, with GPT most strongly. If true, these results mean users cannot assume a single LLM is politically or socially neutral, and the choice of model can silently shift the values of downstream applications.

What carries the argument

The carrying mechanism is a five-part behavioral probe battery applied to each model. (1) News summarization: a model writes a 'neutral' summary of center coverage, and the summary's embedding (via Qwen3-Embedding-4B) is compared by cosine similarity to left- and right-leaning articles on the same event. (2) Ideological stance classification: models label articles as left, center, or right, and their misclassification patterns reveal which ideology's cues they recognize. (3) Simulated UNGA voting: models vote yes/no/abstain on 5,602 roll calls, and agreement with 200 real delegates is scored by Cohen's Kappa, a chance-corrected agreement measure. (4) Multilingual story completion: five cultu

What would settle it

Take a random sample of the 1,018 political events, have politically diverse human annotators rate the left-right slant of each LLM's neutral summary, and re-run the analysis with a different embedding model (e.g., a multilingual sentence transformer). If the model rankings—Gemini right-leaning, GPT left-leaning—do not reproduce under human ratings or the alternative embedding, the political-bias claim is not robust.

Watch

Extended reading notes

Core claim

The paper's central discovery is that four widely used LLMs—Qwen2.5-7B-Instruct, DeepSeek-V3-0324, Gemini-2.5-flash, and GPT-4o-mini—while aligned to be neutral and impartial, still exhibit biases and affinities of different types across politics, ideology, geopolitical alliance, language, and gender. Concretely: Gemini produces summaries more similar to right-leaning news and is the least able to recognize left-leaning ideological cues, GPT leans slightly left and is more responsive to left rhetoric, and DeepSeek is the most politically balanced. In simulated United Nations General Assembly (UNGA) voting, each model diverges from real countries' delegates in its own pattern—Gemini disagrees

Load-bearing premise

The political-bias conclusion rests on the assumption that cosine similarity in a single embedding space (Qwen3-Embedding-4B) between an LLM's summary and left/right news articles is a valid, bias-free measure of political slant, undistorted by writing style, summary quality, or topic—an assumption the paper does not validate against human political ratings.

Editorial extensions

If this is right

  • No single LLM can be assumed neutral: Gemini's outputs carry a right-leaning political and ideological slant, GPT's a left-leaning one, so applications built on either will inherit those leanings.
  • LLM-based simulations of international politics are model-specific: in the UNGA voting experiment, each model agrees and disagrees with a different set of real countries' delegates.
  • Because all four models' answers to World Values Survey questions sit closer to women's average responses than men's, LLMs used in opinion research or policy consultation may systematically misrepresent male respondents' views.
  • The multilingual story-completion results show no overall tilt toward high-resource languages, but Southern African languages cluster near English for Qwen, DeepSeek, and Gemini—evidence of transfer effects from low-resource language training.
  • The probe battery itself is a reusable, black-box method for auditing closed-source LLMs for social bias without access to weights or training data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One natural extension is to treat each model's bias profile as a stable 'bias fingerprint' and use it to select models per task or to ensemble models with opposing leanings so the biases cancel; the paper does not explore ensembling.
  • Because the paper itself notes that higher-quality summaries align more with left-leaning reporting, an open question is whether some of the measured left slant really tracks summary quality or informativeness rather than ideology; a human-rating study could separate the two.
  • The gender result may reflect a general progressive response pattern rather than genuine alignment with women; asking models to adopt explicit male or female identities on the same survey would test whether the women-aligned default is fixed.
  • The language finding comes from story completion alone; testing the same Southern African-English clustering on reasoning or dialogue tasks would show whether the transfer effect is task-general.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The manuscript reports five experiments comparing Qwen2.5-7B-Instruct, DeepSeek-V3-0324, Gemini-2.5-flash, and GPT-4o-mini on political, ideological, geopolitical-alliance, language, and gender bias. Political bias is measured by cosine similarity between LLM-produced neutral news summaries and left/right coverage; ideological bias by news stance classification; alliance bias by simulating UNGA votes and computing Cohen's kappa against 200 real delegates; language bias by PCA on embeddings of 92-language story completions; gender bias by alignment of WVS answers with male and female survey averages. The paper concludes that the models are broadly neutral but exhibit distinct bias profiles, with Gemini right-leaning, GPT left-leaning and women-aligned, and all models somewhat women-aligned.

Significance. The topic is timely and important, and the paper has real assets: it uses public benchmarks, gives explicit prompts, and covers five bias dimensions in one framework. The final discussion on pluralistic alignment is thought-provoking. If the measurements were stable and the proxy measures validated, the differentiated model profiles would be useful for deployment decisions and for future bias-audit research. At present, however, the headline conclusions rest on single-run point estimates and on several unvalidated or internally inconsistent measurement choices, so the contribution is not yet at the level claimed by the abstract and discussion.

major comments (6)
  1. [Sections 2.1–2.6, 4.1–4.6] All reported results are single-run point estimates. The paper never reports sampling temperature, random seed, number of independent regenerations, or confidence intervals. Fig. 1(e) shows covariance ellipses across 1,018 events, not across repeated summaries of the same event; Figs. 3 and 5 have no uncertainty at all. Because the commercial APIs are stochastic, rankings such as 'Gemini is right-leaning,' 'GPT is women-aligned,' or 'Gemini disagrees with the USA (#181)' could change under regeneration. Please add repeated sampling (at least on a representative subset), report model-level variance, and provide significance tests or adjust the strength of the claims.
  2. [Section 4.1 vs. Section 4.6] The paper sets max_tokens=3 for all classification/selection tasks, but the WVS prompts require multiple numeric outputs, e.g., six ratings for Q1–Q6, six for Q27–Q32, nine for Q33–Q41, and nineteen for Q177–Q195. A 3-token limit will truncate or invalidate these responses. If the token limit was applied per survey item rather than per full prompt, this must be stated explicitly. As written, the gender-bias results in Section 2.5 are not interpretable.
  3. [Section 4.2, Eq. (1)] The political-slant measure is a cosine similarity computed with Qwen3-Embedding-4B and is never validated against human political ratings or an alternative embedding model. Embedding proximity to a right-leaning article could reflect length, style, named entities, or topic rather than political leaning. Before assigning 'right-leaning' to Gemini or 'left-leaning' to GPT from Fig. 1(e), please validate the metric, e.g., on a human-rated sample, with multiple embedding models, or against a known-reference test.
  4. [Section 2.2 vs. Section 4.3] The main text says the ideological study covers elections, race and racism, immigration, LGBT, and abortion, with sample sizes for all five; Section 4.3 says only three topics (elections, race, immigration) were selected and gives no counts for LGBT or abortion. Figure 2 nonetheless shows subfigures for all five topics. Either the Methods section is incomplete or the Results section reports analyses outside the described protocol. This must be reconciled before the ideological claims, including those about immigration and LGBT, can be evaluated.
  5. [Section 4.4 and Section 2.3] The UNGA prompt asks the model to act as a country's representative but provides no country identity. The same generic vote sequence is then compared with all 200 delegates. Thus the kappa values measure which real delegate's overall voting pattern the generic simulated delegate resembles, not a model-country affinity. The statement that 'Gemini is the only model that misaligns with the USA (#181)' is an artifact of this design. A proper test requires country-conditioned prompting or another identification strategy, and kappa uncertainty should be reported.
  6. [Section 4.6] The WVS sections are selected precisely because they show the largest male/female differences in the same dataset, and those same differences are then used to define gender alignment. This selection can inflate or exaggerate measured alignment even for a gender-neutral model. The 5% indifference threshold is also arbitrary. Please report sensitivity to section selection and threshold, or justify the procedure with out-of-sample data.
minor comments (6)
  1. [Figure 2 caption] The caption says 'elections-related news' but the figure shows five different topics; the caption should match the actual content.
  2. [Section 3] 'contracting values' should likely be 'contradictory values'.
  3. [Figure 3] 'geological heatmaps' should be 'geographical heatmaps'.
  4. [Section 4.1] The model versions are given in footnotes rather than in a reproducibility table; a table with model names, versions, access dates, and API settings would make the study easier to reproduce.
  5. [Section 4.5] The PCA random seed is fixed to 7 and no seed sensitivity is reported. Since the language-cluster interpretation in Fig. 5 is qualitative, please show that the clusters are stable across several seeds.
  6. [Section 2.3] Several alliance-bias observations are made from visual inspection of heatmaps. A supplementary table with the top and bottom kappa values and their standard errors would make the rankings easier to audit.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper is an empirical measurement study with no fitted-input-called-prediction, self-referential equation, or load-bearing self-citation chain.

full rationale

The paper reports five empirical probes of LLM bias. Each experiment compares model outputs against an external dataset or reference signal: political summaries are compared with left/right news articles via cosine similarity; ideology is measured by classification accuracy against the Article Bias Prediction dataset; alliance affinity is measured with Cohen's kappa against UNGA roll-call votes; language bias is assessed by PCA of translated story embeddings; gender affinity is scored by closeness to WVS male/female averages. No parameter is fit to the outcome and then reported as a prediction. The embedding model in Eq. (1) is Qwen3-Embedding-4B while the studied model is Qwen2.5-7B-Instruct, so the political-similarity measurement is not the same system defining its own result. The choice of WVS sections based on pre-existing male/female answer differences may inflate the apparent gender gap, but the LLM answers are not constructed from those differences, so the reported alignment is not equivalent to the selection criterion by definition. The paper contains self-citations (e.g., [1], [23], [38]), but these are background references and are not load-bearing for any claimed derivation. Concerns about single-shot prompting, missing error bars, and embedding-model validity are important empirical-validity threats, but they are not circularity: they do not make any stated result equal to its input by construction. Accordingly, no circular step can be quoted or exhibited, and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's conclusions depend on five measurement assumptions, two arbitrary hyperparameters, and no new entities. The most consequential are the embedding-similarity proxy and the unspecified-country UNGA simulation.

free parameters (3)
  • PCA random seed = 7
    Language-bias PCA projections in Fig. 5 depend on the chosen random seed; no stability analysis is reported (Section 4.5).
  • Gender-neutral indifference threshold = 5% of the smaller distance
    Used to decide when the LLM answer aligns with neither gender; chosen without justification (Section 4.6).
  • Max tokens for classification/selection tasks = 3 tokens
    Empirically determined; may bias answers toward short labels and affect classification outcomes (Section 4.1).
assumptions (5)
  • domain assumption Cosine similarity of contextual embeddings is a valid proxy for political/ideological similarity.
    Eq. (1) in Section 4.2; no validation or alternative embeddings are considered.
  • domain assumption Misclassification of news stance reflects the model's ideological alignment (e.g., Gemini misclassifying left news as center means it is right-aligned).
    Section 2.2; no independent evidence that performance asymmetry equals value alignment.
  • domain assumption An LLM can represent an unspecified UNGA delegate, and its vote agreement pattern with real countries reveals geopolitical affinity.
    Section 4.4 prompt gives no country identity; agreement may reflect base knowledge or training-data memorization.
  • domain assumption Machine-translation and embedding biases apply uniformly across languages, so relative comparisons are fair.
    Footnote 1 in Section 2.4 asserts uniformity without evidence.
  • domain assumption WVS male/female average answers are the appropriate benchmark for gender affinity; sections are selected where gender differences are largest.
    Section 4.6 selection by ranking questions may inflate the appearance of gender alignment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Multifaceted Analysis of Social Biases in Large Language Models." pith.science (2026). https://pith.science/paper/XUDQMHHV

@misc{pith2026251215792,
  author       = {Pith},
  title        = {Pith review of: A Multifaceted Analysis of Social Biases in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XUDQMHHV}},
  note         = {Machine review of arXiv:2512.15792}
}
read the original abstract

Large language models (LLMs) have rapidly become indispensable tools for acquiring information and supporting human decision-making. However, ensuring that these models uphold fairness across varied contexts is critical to their safe and responsible deployment. In this study, we undertake a comprehensive examination of four widely adopted LLMs, probing their underlying biases and inclinations across the dimensions of politics, ideology, alliance, language, and gender. Through a series of carefully designed experiments, we investigate their political neutrality using news summarization, ideological biases through news stance classification, tendencies toward specific geopolitical alliances via United Nations voting patterns, language bias in the context of multilingual story completion, and gender-related affinities as revealed by responses to the World Values Survey. Results indicate that while the LLMs are aligned to be neutral and impartial, they still show biases and affinities of different types.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 5 linked inside Pith

  1. [1]

    IEEE Transactions on Affective Computing14(3), 1743–1753 (2023)

    Mao, R., Liu, Q., He, K., Li, W., Cambria, E.: The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection. IEEE Transactions on Affective Computing14(3), 1743–1753 (2023)

    2023

  2. [2]

    In: Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pp

    Joniak, P., Aizawa, A.: Gender biases and where to find them: Exploring gender bias in pre-trained Transformer-based language models using movement prun- ing. In: Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pp. 67–73 (2022)

    2022

  3. [3]

    In: Findings of the Association for Computa- tional Linguistics: EMNLP 2023, pp

    Wambsganss, T., Su, X., Swamy, V., Neshaei, S.P., Rietsche, R., K¨ aser, T.: Unraveling downstream gender bias from large language models: A study on AI educational writing assistance. In: Findings of the Association for Computa- tional Linguistics: EMNLP 2023, pp. 10275–10288. Association for Computational Linguistics, Singapore (2023)

    2023

  4. [4]

    Nature, 1–9 (2025)

    Guilbeault, D., Delecourt, S., Desikan, B.S.: Age and gender distortion in online media and large language models. Nature, 1–9 (2025)

    2025

  5. [5]

    IEEE Intelligent Systems 40(3), 10–15 (2025)

    Nadeem, M., Sohail, S.S., Cambria, E., Schuller, B.W., Hussain, A.: Gender bias in text-to-video generation models: A case study of Sora. IEEE Intelligent Systems 40(3), 10–15 (2025)

    2025

  6. [6]

    In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, vol

    Wilson, K., Sim, M., Gueorguieva, A.-M., Caliskan, A.: No thoughts just AI: Biased LLM hiring recommendations alter human decision making and limit human autonomy. In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, vol. 8, pp. 2692–2704 (2025)

    2025

  7. [7]

    In: Findings of the Association for Computational Linguistics: EMNLP 2021, pp

    Hassan, S., Huenerfauth, M., Alm, C.O.: Unpacking the interdependent systems of discrimination: Ableist bias in NLP systems through an intersectional lens. In: Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3116–3123. Association for Computational Linguistics, Punta Cana, Dominican Republic (2021) 19

    2021

  8. [8]

    In: 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA), pp

    Cai, Y., Zimek, A., Wunder, G., Ntoutsi, E.: Power of explanations: Towards automatic debiasing in hate speech detection. In: 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA), pp. 1–10 (2022). IEEE

    2022

Show all 50 references
  1. [9]

    In: Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pp

    Hosseini, S., Palangi, H., Hassan, A.: An empirical study of metrics to measure representational harms in pre-trained language models. In: Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pp. 121–134 (2023)

    2023

  2. [10]

    In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp

    Esiobu, D., Tan, X., Hosseini, S., Ung, M., Zhang, Y., Fernandes, J., Dwivedi-Yu, J., Presani, E., Williams, A., Smith, E.: Robbie: Robust bias evaluation of large generative language models. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce...

    2023

  3. [11]

    Expert Systems with Applications237, 121542 (2024)

    Raza, S., Garg, M., Reji, D.J., Bashir, S.R., Ding, C.: Nbias: A natural lan- guage processing framework for bias identification in text. Expert Systems with Applications237, 121542 (2024)

    2024

  4. [12]

    In: Proceedings of the AAAI Conference on Artificial Intelligence, vol

    Ling, L., Rabbi, F., Wang, S., Yang, J.: Bias unveiled: Investigating social bias in LLM-generated code. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, pp. 27491–27499 (2025)

    2025

  5. [13]

    In: 2025 International Joint Conference on Neural Networks (IJCNN), pp

    Anas, M., Nadeem, M., Sohail, S.S., Cambria, E., Hussain, A.: Are horses always strong and donkeys dumb? Animal bias in vision language models. In: 2025 International Joint Conference on Neural Networks (IJCNN), pp. 1–10 (2025)

    2025

  6. [14]

    arXiv preprint arXiv:2505.17425 (2025)

    Yeo, W.J., Mao, R., Abdar, M., Cambria, E., Satapathy, R.: Debiasing CLIP: Interpreting and correcting bias in attention heads. arXiv preprint arXiv:2505.17425 (2025)

    2025 arXiv

  7. [15]

    In: 60th Annual Meeting of the Association- for-Computational-Linguistics (ACL), pp

    Hershcovich, D., Frank, S., Lent, H., Lhoneux, M., Abdou, M., Brandl, S., Bugliarello, E., Piqueras, L.C., Chalkidis, I., Cui, R.,et al.: Challenges and strategies in cross-cultural NLP. In: 60th Annual Meeting of the Association- for-Computational-Linguistics (ACL), pp. 6997–...

    2022

  8. [16]

    In: Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pp

    Cao, Y., Zhou, L., Lee, S., Cabello, L., Chen, M., Hershcovich, D.: Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study. In: Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pp. 53–67 (2023)

    2023

  9. [17]

    In: Findings of the Association for Computational Linguistics: ACL 2023, pp

    Palta, S., Rudinger, R.: FORK: A bite-sized test set for probing culinary cul- tural biases in commonsense reasoning models. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 9952–9962 (2023)

    2023

  10. [18]

    PNAS Nexus3(9), 346 (2024)

    Tao, Y., Viberg, O., Baker, R.S., Kizilcec, R.F.: Cultural bias and cultural 20 alignment of large language models. PNAS Nexus3(9), 346 (2024)

    2024

  11. [19]

    Nature Machine Intelligence, 1–3 (2025)

    Nadeem, M., Sohail, S.S., Cambria, E., Afreen, S.: South Asian biases in language and vision models. Nature Machine Intelligence, 1–3 (2025)

    2025

  12. [20]

    In: The 61st Annual Meeting of the Association for Computational Linguistics (2023)

    Havaldar, S., Singhal, B., Rai, S., Liu, L., Guntuku, S.C., Ungar, L.: Multilingual language models are not multicultural: A case study in emotion. In: The 61st Annual Meeting of the Association for Computational Linguistics (2023)

    2023

  13. [21]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

    Wendler, C., Veselovsky, V., Monea, G., West, R.: Do Llamas work in English? On the latent language of multilingual Transformers. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15366–15394 (2024)

    2024

  14. [22]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

    Choenni, R., Lauscher, A., Shutova, E.: The echoes of multilinguality: Tracing cultural value shifts during language model fine-tuning. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15042–15058 (2024)

    2024

  15. [23]

    In: 2024 International Joint Conference on Neural Networks (IJCNN), pp

    Zhang, X., Mao, R., Cambria, E.: Multilingual emotion recognition: Discovering the variations of lexical semantics between languages. In: 2024 International Joint Conference on Neural Networks (IJCNN), pp. 1–9 (2024). IEEE

    2024

  16. [24]

    In: Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science, pp

    Chen, W.-F., Al Khatib, K., Wachsmuth, H., Stein, B.: Analyzing political bias and unfairness in news articles at different levels of granularity. In: Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science, pp. 149–154 (2020)

    2020

  17. [25]

    In: Findings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pp

    Lin, L., Wang, L., Zhao, X., Li, J., Wong, K.F.: IndiVec: An exploration of leveraging large language models for media bias detection with fine-grained bias indicators. In: Findings of the 18th Conference of the European Chapter of the Association for Computational Linguistics...

    2024

  18. [26]

    PloS One19(7), 0306621 (2024)

    Rozado, D.: The political preferences of LLMs. PloS One19(7), 0306621 (2024)

    2024

  19. [27]

    Scientific Reports14(1), 5224 (2024)

    Fang, X., Che, S., Mao, M., Zhang, H., Zhao, M., Zhao, X.: Bias of AI-generated content: An examination of news produced by large language models. Scientific Reports14(1), 5224 (2024)

    2024

  20. [28]

    In: Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp

    Bang, Y., Chen, D., Lee, N., Fung, P.: Measuring political bias in large language models: What is said and how it is said. In: Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11142–11159 (2024)

    2024

  21. [29]

    Telematics and Informatics96, 102211 (2025)

    Urman, A., Makhortykh, M.: The silence of the LLMs: Cross-lingual analysis of guardrail-related political bias and false information prevalence in ChatGPT, 21 Google Bard (Gemini), and Bing Chat. Telematics and Informatics96, 102211 (2025)

    2025

  22. [30]

    arXiv preprint arXiv:2402.10436 (2024)

    Dong, W., Zhunis, A., Chin, H., Han, J., Cha, M.: I am not them: Fluid iden- tities and persistent out-group bias in large language models. arXiv preprint arXiv:2402.10436 (2024)

    2024 arXiv

  23. [31]

    Nature Computational Science5(1), 65–75 (2025)

    Hu, T., Kyrychenko, Y., Rathje, S., Collier, N., Linden, S., Roozenbeek, J.: Gen- erative language models exhibit social identity biases. Nature Computational Science5(1), 65–75 (2025)

    2025

  24. [32]

    Proceedings of the National Academy of Sciences122(31), 2415697122 (2025)

    Laurito, W., Davis, B., Grietzer, P., Gavenˇ ciak, T., B¨ ohm, A., Kulveit, J.: AI–AI bias: Large language models favor communications generated by large language models. Proceedings of the National Academy of Sciences122(31), 2415697122 (2025)

    2025

  25. [33]

    Journal of Conflict Resolution61(2), 430–456 (2017)

    Bailey, M.A., Strezhnev, A., Voeten, E.: Estimating dynamic state preferences from united nations voting data. Journal of Conflict Resolution61(2), 430–456 (2017)

    2017

  26. [34]

    Biometrics, 159–174 (1977)

    Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics, 159–174 (1977)

    1977

  27. [35]

    JD Sys- tems Institute & WVSA Secretariat, Madrid, Spain & Vienna, Austria

    World Values Survey Trend File (1981–2022) Cross-National Data-Set. JD Sys- tems Institute & WVSA Secretariat, Madrid, Spain & Vienna, Austria. Data File Version 4.0.0 (2022). https://doi.org/10.14281/18241.27

    1981 doi

  28. [36]

    610–623 (2021)

    Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the dangers of stochastic parrots: Can language models be too big? In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610–623 (2021)

    2021

  29. [37]

    ACM Journal of Data and Information Quality15(2), 1–21 (2023)

    Navigli, R., Conia, S., Ross, B.: Biases in large language models: Origins, inven- tory, and discussion. ACM Journal of Data and Information Quality15(2), 1–21 (2023)

    2023

  30. [38]

    Cognitive Computation 17(35), 1–12 (2025)

    Mao, R., Chen, G., Li, X., Ge, M., Cambria, E.: A comparative analysis of metaphorical cognition in ChatGPT and human minds. Cognitive Computation 17(35), 1–12 (2025)

    2025

  31. [39]

    In: Findings of the Association for Computational Linguistics: ACL 2023, pp

    Perez, E., Ringer, S., Lukosiute, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S.,et al.: Discovering language model behaviors with model-written evaluations. In: Findings of the Association for Computational Linguistics: ACL 2023, pp. 133...

    2023

  32. [40]

    arXiv preprint arXiv:2307.15217 (2023)

    Casper, S., Davies, X., Shi, C., Gilbert, T.K., Scheurer, J., Rando, J., Freedman, 22 R., Korbak, T., Lindner, D., Freire, P., et al.: Open problems and fundamen- tal limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217 (2023)

    2023 arXiv

  33. [41]

    Transactions of the Association for Computational Linguistics12, 771–785 (2024)

    Itzhak, I., Stanovsky, G., Rosenfeld, N., Belinkov, Y.: Instructed to bias: Instruction-tuned language models exhibit emergent cognitive bias. Transactions of the Association for Computational Linguistics12, 771–785 (2024)

    2024

  34. [42]

    Ethics and Information Technology27(2), 28 (2025)

    Dahlgren Lindstr¨ om, A., Methnani, L., Krause, L., Ericson, P., Troya, ´I.M., Coelho Mollo, D., Dobbe, R.: Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through reinforcement learning from human feedback. Ethics and Information Technology27(2), 28 (2025)

    2025

  35. [43]

    arXiv preprint arXiv:2111.08596 (2021)

    Yamagata, T., McConville, R., Santos-Rodriguez, R.: Reinforcement learn- ing with feedback from multiple humans with diverse skills. arXiv preprint arXiv:2111.08596 (2021)

    2021 arXiv

  36. [44]

    Advances in Neural Information Processing Systems35, 38176–38189 (2022)

    Bakker, M., Chadwick, M., Sheahan, H., Tessler, M., Campbell-Gillingham, L., Balaguer, J., McAleese, N., Glaese, A., Aslanides, J., Botvinick, M.,et al.: Fine-tuning language models to find agreement among humans with diverse pref- erences. Advances in Neural Information Proce...

    2022

  37. [45]

    Advances in Neural Information Processing Systems35, 27730–27744 (2022)

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A.,et al.: Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems35, 27730–27744 (2022)

    2022

  38. [46]

    arXiv preprint arXiv:2204.05862 (2022)

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al.: Training a helpful and harm- less assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022)

    2022 arXiv

  39. [47]

    In: Proceedings of the 41st International Conference on Machine Learning, pp

    Sorensen, T., Moore, J., Fisher, J., Gordon, M., Mireshghallah, N., Rytting, C.M., Ye, A., Jiang, L., Lu, X., Dziri, N.,et al.: Position: A roadmap to pluralistic align- ment. In: Proceedings of the 41st International Conference on Machine Learning, pp. 46280–46302 (2024)

    2024

  40. [48]

    Gonz´ alez Barman, K., Lohse, S., Regt, H.W.: Reinforcement learning from human feedback in LLMs: Whose culture, whose values, whose perspectives? Philosophy & Technology38(2), 1–26 (2025)

    2025

  41. [49]

    In: 11th International Natural Language Generation Conference (INLG 2018), pp

    Chen, W.-F., Wachsmuth, H., Al-Khatib, K., Stein, B.: Learning to flip the bias of news headlines. In: 11th International Natural Language Generation Conference (INLG 2018), pp. 79–88. Association for Computational Linguistics, Tilburg, The Netherlands (2018) 23

    2018

  42. [50]

    In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp

    Baly, R., Da San Martino, G., Glass, J., Nakov, P.: We can detect your bias: Predicting the political ideology of news articles. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4982–4991. Association for Computational Lin...

    2020

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.