Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Commercial text-to-image models reproduce and amplify occupational gender stereotypes, generating men in 93% of male-stereotyped professions and only 22.5% of female-stereotyped ones.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A benchmark of 13 image-generation models finds that most amplify occupational gender stereotypes, producing men in 93% of male-stereotyped prompts and 22.5% of female-stereotyped prompts, while one model approached parity.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Useful cross-model benchmark, but the amplification claim is reversed for female-stereotyped professions and the key ANOVA likely doesn't survive the correction. the 4 major comments →

arxiv 2509.07050 v1 pith:ZVZJ7OCD submitted 2025-09-08 cs.CV cs.AIcs.CY

Automated Evaluation of Gender Bias Across 13 Large Multimodal Models

Juan Manuel Contreras This is my paper
classification cs.CV cs.AIcs.CY MSC 68T0168T5062P35
keywords gender biastext-to-image generationlarge multimodal modelsLLM-as-a-judgeoccupational stereotypesbias amplificationfairness benchmarkautomated evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that modern text-to-image models do not merely reflect occupational gender stereotypes; they amplify them relative to real-world labor statistics. Across 13 commercial models and 75 gender-neutral prompts, male-stereotyped professions produced images of men 93.0% of the time, female-stereotyped professions only 22.5%, and non-stereotyped professions 68.3%—a default-male bias. The paper reports large cross-model variation, with overall male representation from 46.7% to 73.3%, and one model (Nova Canvas) not differing from parity on stereotyped categories. The paper reads that variation as evidence that high bias is a design choice, not an inevitability, and argues for standardized automated benchmarks to track fairness.

Core claim

The central claim is that LMMs systematically reproduce and amplify occupational gender stereotypes: compared with LLM-retrieved BLS/ILO labor baselines, models over-represent men in male-dominated professions by about 12 percentage points (US data) while under-representing men in female-dominated professions by about 5.6 points. For professions without a strong gender association, the default person generated is male 68.3% of the time, significantly above 50% parity. The paper also claims the bias is model-specific: a two-way ANOVA shows a significant interaction between model and profession stereotype, and binomial tests find only one model (Nova Canvas) whose male- and female-stereotyped

What carries the argument

The load-bearing mechanism is the Aymara Image Fairness Evaluation pipeline: procedural prompt generation (75 gender-neutral prompts in three stereotype categories), zero-shot image generation from 13 commercial LMMs (965 images), LLM-as-a-judge scoring of each image as man or not-man, and statistical validation of the prompt categories against LLM-retrieved US and global labor data. The scoring was validated against human ratings at 96.4% agreement (kappa 0.92). The quantitative engine is the bias-amplification metric, which compares the model's distance from parity to the labor data's distance from parity, expressed as a percentage increase or decrease in bias.

Load-bearing premise

The load-bearing premise is that the median of statistics retrieved by four LLMs from BLS and ILO sources accurately represents real-world labor data; the paper used those medians instead of querying the official databases directly.

What would settle it

Pull the same 75 professions directly from Bureau of Labor Statistics and International Labour Organization tables and recompute the bias-amplification analysis; if the official medians differ from the LLM-retrieved medians by more than a few percentage points in the stereotyped categories, the paper's amplification finding would not replicate.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A standardized, API-only benchmark can rank closed and open models on the same fairness scale without needing internal model access.
  • Since one model already approaches parity, developer-side measures such as balanced training data, prompt rewriting, or output filtering are enough to substantially reduce stereotyping.
  • Gender-neutral prompts should not be assumed neutral: for most models the default person is male, so downstream applications inherit that skew.
  • Bias audits need a real-world baseline to distinguish reflecting society from amplifying society; the high correlation with labor data is not sufficient on its own.
  • The LLM-as-a-judge scoring method makes continuous regression testing of image fairness practical at scale.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper: the same 75-prompt protocol could be run in non-English languages with local labor data; the paper itself flags culture and language as a limitation, so this is the next testable step.
  • I infer that the default-male bias should show up in downstream applications like stock-photo generation or educational imagery, not just in benchmark prompts; a content analysis of such outputs would test that.
  • The cross-model variance could be an artifact of differing safety filters rather than training-data curation; distinguishing the two would require ablating the same model with and without post-processing, which the paper does not do.
  • I infer that the fairness scores can double as a regression suite: vendors could run the benchmark on every release, with the July-August 2025 snapshot as the baseline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces the Aymara Image Fairness Evaluation, a benchmark that uses 75 gender-neutral profession prompts to test 13 commercial text-to-image LMMs. Prompts are generated programmatically through the Aymara AI SDK; 965 generated images are scored by an unnamed LLM-as-a-judge from the same SDK; and the resulting male/female rates are compared with U.S. and global labor statistics. The paper claims three headline findings: (1) LMMs systematically reproduce and amplify occupational gender stereotypes, moving male-stereotyped professions to 93.0% male and female-stereotyped professions to 22.5% male; (2) models show a default-male bias, producing 68.3% male for non-stereotypical professions; and (3) bias varies strongly across models, with Gen-4 near parity and Recraft V3 most biased. The paper also reports a significant interaction between model and prompt stereotype and proposes fairness scores. The overarching contribution is a scalable, automated, cross-model benchmark; the central quantitative claim is the amplification effect relative to real-world labor data.

Significance. If the claims survive scrutiny, the paper would provide one of the largest cross-model comparisons of gender bias in text-to-image generation, with an automated pipeline that can be re-run as models change. The strongest assets are the identical-prompt design across 13 models, the explicit handling of refusals, and the finding that one model (Nova Canvas) approaches parity while others do not, which is informative regardless of the amplification analysis. However, the paper's headline 'systematic amplification' claim is currently contradicted by its own aggregate numbers, the labor-force baselines are obtained via LLM retrieval rather than direct official database access, and the evaluator model is not disclosed. These issues are central and require re-analysis before the contribution can be assessed.

major comments (4)
  1. [§4.4, Tables 2, 6, 11] The paper's own data contradict the reported female-category amplification. U.S. labor statistics for stereotypically female professions give 17.03% men (Table 2), while generated images show 22.5% men (Table 6). The difference is +5.5 percentage points, not -5.61. The Table 11 value of -5.61 is consistent with the female share (77.5% - 83.0%), whereas the text interprets it as 'fewer male images' and the male category is computed as male share (93.0 - 81.1 = +11.94). The ANOVA in Table 12 therefore compares a male-share difference for male-coded professions with a female-share difference for female-coded professions, so the F(1,48)=14.46, p=.0004 result is not interpretable. With the sign corrected, the female mean is about +5.5 pp; the gap from the male category shrinks from about 17.6 pp to about 6.4 pp, and under the reported SDs a two-sample comparison is not significant (approximat
  2. [§3.2] The amplification analysis rests on labor-force baselines obtained by asking four LLMs (GPT-4o, Gemini 1.5 Pro, Claude 3 Sonnet, Perplexity) to retrieve BLS/ILO statistics, rather than by direct queries to official databases. The authors then take the median of these estimates as ground truth. Low inter-model variance (U.S. average SD=4.13) only shows agreement among LLMs, not accuracy. A systematic bias in the retrieval step would directly change the sign, magnitude, and statistical significance of every amplification result in Tables 10-13. The authors should query the official databases directly or release the per-profession retrieved values and sources so readers can verify them. This is load-bearing, not a presentation detail.
  3. [§3.4] The LLM-as-a-judge model is never named. The text says 'We used the Aymara Python SDK to score all 965 generated images' and reports 96.4% agreement with one human rater (kappa=0.92), but it does not state which LLM made the judgments, which version, what judgment prompt was used, or whether the judge is one of the 13 evaluated models. If the judge is a similar proprietary model, it may share the latent biases under test. Moreover, a single human rater cannot establish inter-rater reliability, and it is unclear whether this rater is one of the authors. Please disclose the judge model and version, the exact scoring instruction, and provide independent multi-rater validation or at least a second rater.
  4. [§4.1, §4.3] The image-level ANOVAs (F(2,962)=268.01 and the two-way ANOVA with residual df=926) treat all 965 images as independent. However, images are nested within 75 prompts and 13 models: the same prompt is sent to every model, and each model generates 75 (sometimes 74/66) images. This clustering means the effective sample size is far smaller than 965, and the reported p-values, especially for the interaction in §4.3, are likely overstated. The authors should use cluster-robust standard errors or a mixed-effects model with random intercepts for prompt and model, and should report the corresponding p-values.
minor comments (5)
  1. [Table 2 title] Table 2 is titled 'Men in Generated Images (%)' but it reports labor-force statistics. This mislabeling likely contributed to the sign inconsistency in §4.4; it should read 'Men in Labor Force (%).'
  2. [References] Reference [9] is incomplete: 'arXiv:2303.XXXXX' is a placeholder, not a citable identifier.
  3. [Table 9] The model is called 'Titan G1 V2' in Table 9 but 'Titan Image Generator v2' in Table 5 and 'Titan G1 v2' in §3.3. Pick one consistent name.
  4. [Throughout] There are spacing artifacts in the PDF: 'F airness', 'F uture W ork', 'AUTOMA TED'. These are presumably rendering issues but should be corrected in the source.
  5. [§3.1] The full 75-prompt set is not released, and only sample prompts are shown. For a benchmark meant to be reusable, the full prompt set (or a controlled access procedure) should be provided; otherwise the category validation in §3.2 cannot be independently reproduced.

Circularity Check

0 steps flagged

No significant circularity: the benchmark's headline numbers are direct image-level counts validated against human ratings; the Aymara self-citations are tooling/background, not load-bearing premises.

full rationale

The paper is an empirical measurement study, not a derivation, and its central claims do not reduce to fitted inputs or self-citations by construction. The male-representation rates (93.0%, 22.5%, 68.3%) are direct binary judgments on the generated images, and the statistical tests (binomial, ANOVA, Tukey HSD) operate on those measured proportions. The 'amplification' metric is a post-hoc arithmetic comparison of measured image proportions against a labor-statistics baseline; even though that baseline was obtained by LLM retrieval rather than directly from BLS/ILO databases, this is a measurement-validity concern, not a circular reduction. The LLM judge is validated against 140 human-rated images with 96.4% agreement and reported accuracy/F1/kappa, so the scoring does not rest solely on the author's prior work. Self-citations to the Aymara framework ([22]) and to prior co-authored bias work ([20]) are tooling and background references, not evidence used to force the conclusion. The internal inconsistency in Table 11 for female-stereotyped professions (reporting -5.61 percentage points while the underlying rates imply +5.5 pp) is a data-analysis/sign error that undermines the 'underrepresentation' wording, but it is not a case of a prediction being equivalent to its inputs. Honest non-finding is therefore appropriate: no circularity is present.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central claims rest on several unverified premises: LLM-retrieved labor data is treated as ground truth, the unnamed LLM judge is assumed unbiased, and the hidden prompt set is assumed gender-neutral. These are not fitted parameters in a model sense, but they are unmeasured model choices built into the Aymara SDK and the evaluation design.

free parameters (2)
  • Prompt generation model (unspecified within Aymara SDK)
    The 75 profession prompts are generated by an unnamed LLM inside the Aymara AI SDK (Section 3.1). The identities of the professions are not disclosed, and the generation model's parameters could influence the resulting gender cues.
  • LLM-as-a-judge model (unspecified)
    The gender classification judge is an unnamed LLM (Section 3.4). Its underlying model, temperature, and prompt template are not provided, making the judgments irreproducible and potentially model-dependent.
axioms (5)
  • domain assumption Binary gender classification is a valid representation of gender in generated images
    The paper acknowledges this is a simplification in a footnote to Section 3, but all measurements rely on classifying each image as male-presenting or not male-presenting.
  • domain assumption LLM-retrieved labor statistics accurately reflect real-world gender distributions
    Section 3.2: labor statistics were obtained by asking four LLMs to find data from BLS/ILO, and the median is treated as ground truth for the amplification analysis without independent verification against official databases.
  • domain assumption The LLM judge's gender judgments are unbiased and accurate
    Section 3.4: the judge is validated against one human rater (96.4% agreement), but the judge itself is an unspecified LLM and may share stereotypes with the models under test.
  • domain assumption The 75 generated prompts are genuinely gender-neutral and equally difficult across categories
    Section 3.1: prompts are procedurally generated and not released, so their neutrality and difficulty cannot be independently verified.
  • standard math Classical statistical tests (ANOVA, Tukey HSD, binomial tests) are valid despite non-independence of observations
    The paper uses standard tests on image-level data, which assume independence. Images are clustered within models and prompts, violating this assumption and potentially inflating significance.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Evaluation of Gender Bias Across 13 Large Multimodal Models." pith.science (2026). https://pith.science/paper/ZVZJ7OCD

@misc{pith2026250907050,
  author       = {Pith},
  title        = {Pith review of: Automated Evaluation of Gender Bias Across 13 Large Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZVZJ7OCD}},
  note         = {Machine review of arXiv:2509.07050}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large multimodal models (LMMs) have revolutionized text-to-image generation, but they risk perpetuating the harmful social biases in their training data. Prior work has identified gender bias in these models, but methodological limitations prevented large-scale, comparable, cross-model analysis. To address this gap, we introduce the Aymara Image Fairness Evaluation, a benchmark for assessing social bias in AI-generated images. We test 13 commercially available LMMs using 75 procedurally-generated, gender-neutral prompts to generate people in stereotypically-male, stereotypically-female, and non-stereotypical professions. We then use a validated LLM-as-a-judge system to score the 965 resulting images for gender representation. Our results reveal (p < .001 for all): 1) LMMs systematically not only reproduce but actually amplify occupational gender stereotypes relative to real-world labor data, generating men in 93.0% of images for male-stereotyped professions but only 22.5% for female-stereotyped professions; 2) Models exhibit a strong default-male bias, generating men in 68.3% of the time for non-stereotyped professions; and 3) The extent of bias varies dramatically across models, with overall male representation ranging from 46.7% to 73.3%. Notably, the top-performing model de-amplified gender stereotypes and approached gender parity, achieving the highest fairness scores. This variation suggests high bias is not an inevitable outcome but a consequence of design choices. Our work provides the most comprehensive cross-model benchmark of gender bias to date and underscores the necessity of standardized, automated evaluation tools for promoting accountability and fairness in AI development.

Figures

Figures reproduced from arXiv: 2509.07050 by Juan Manuel Contreras.

Figure 1
Figure 1. Figure 1: Sample images generated by LMMs. Prompts requested a person in a specific profession without specifying the gender of the person [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Percentage of men in generated images as a function of the profession stereotype in the prompt [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Interaction plot showing the percentage of men in generated images from each LMM for stereotypically-male and -female professions [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Correlation between the percentage of men in the labor force (U.S. and Global) and the percentage of men depicted in generated images across all 75 professions. The translucent bands around the regression lines represent 95% confidence intervals, estimated using 1,000 bootstrap resamples [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Bias amplification in percentage points as a function of profession stereotype in the prompt and labor data source (U.S. and Global) [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Relative Gender Bias Amplification Scores by LMM and Data Source. Scores represent the relative percent change in bias magnitude compared to baseline labor data. Positive values indicate the model amplifies bias; negative values indicate reduced bias [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Gender Bias and Overall Fairness Scores per LMM. Higher scores are better. (2) Models exhibit a strong default-male bias. When prompted to generate a person in a gender￾neutral profession, LMMs often defaulted to producing an image of a man, indicating a systemic skew beyond simple stereotype matching. (3) The extent of gender bias varies significantly across models. While most models showed strong stereot… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics

    cs.CV 2026-01 conditional novelty 6.0

    Prototypicality bias: common text-to-image metrics systematically prefer plausible-but-wrong images over correct non-prototypical ones; PROTOSCORE mitigates but does not eliminate the failure.

Reference graph

Works this paper leans on

28 extracted references · 19 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review , 11(12), November 2024. 17

    work page 2024

  2. [2]

    Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding, 2022

    2022

  3. [3]

    High-resolution image synthesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨ orn Ommer. High-resolution image synthesis with latent diffusion models, 2021

    work page 2021

  4. [4]

    Fairness and bias in multimodal ai: A survey

    Tosin Adewumi, Lama Alkhaled, Namrata Gurung, Goya van Boven, and Irene Pagliai. Fairness and bias in multimodal ai: A survey. arXiv preprint arXiv:2406.19097 , 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Exposing and correcting the gender bias in image captioning datasets and models, 2019

    Shruti Bhargava and David Forsyth. Exposing and correcting the gender bias in image captioning datasets and models, 2019

    work page 2019

  6. [6]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748–8763. PmLR, 2021

    2021

  7. [7]

    Online images amplify gender bias

    Douglas Guilbeault, Sol` ene Delecourt, Tasker Hull, Bhargav Srinivasa Desikan, Mark Chu, and Ethan Nadler. Online images amplify gender bias. Nature, 626(8001):1049–1055, 2024

    work page 2024

  8. [8]

    A survey on bias and fairness in machine learning, 2022

    Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning, 2022

    2022

  9. [9]

    Auditing gender bias in laion-5b occupation representations

    Jaemin Cho et al. Auditing gender bias in laion-5b occupation representations. arXiv preprint arXiv:2303.XXXXX , 2023

    work page 2023

  10. [10]

    Sb-bench: Stereotype bias benchmark for large multimodal models, 2025

    Vishal Narnaware, Ashmal Vayani, Rohit Gupta, Sirnam Swetha, and Mubarak Shah. Sb-bench: Stereotype bias benchmark for large multimodal models, 2025

    work page 2025

  11. [11]

    Social biases through the text-to-image generation lens, 2023

    Ranjita Naik and Besmira Nushi. Social biases through the text-to-image generation lens, 2023

    work page 2023

  12. [12]

    Stable bias: Analyzing societal bias in diffusion models

    Alexandra Sasha Luccioni et al. Stable bias: Analyzing societal bias in diffusion models. arXiv preprint arXiv:2304.06762 , 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Easily accessible text-to-image generation amplifies demographic stereotypes at large scale

    Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale. In 2023 ACM Conference on Fairness, Accountability, and Transparency , F AccT ’23, page 1493–1504. ACM, June 2023

    work page 2023

  14. [14]

    Smiling women pitching down: auditing representational and presentational gender biases in image-generative ai

    Luhang Sun, Mian Wei, Yibing Sun, Yoo Ji Suh, Liwei Shen, and Sijia Yang. Smiling women pitching down: auditing representational and presentational gender biases in image-generative ai. Journal of Computer-Mediated Communication , 29(1):zmad045, 02 2024

    work page 2024

  15. [15]

    Measuring bias in multimodal models: Multimodal composite association score

    Abhishek Mandal, Susan Leavy, and Suzanne Little. Measuring bias in multimodal models: Multimodal composite association score. In International Workshop on Algorithmic Bias in Search and Recommendation , pages 17–30. Springer, 2023

    work page 2023

  16. [16]

    Tibet: Identifying and evaluating biases in text-to-image generative models, 2024

    Aditya Chinchure, Pushkar Shukla, Gaurav Bhatt, Kiri Salij, Kartik Hosanagar, Leonid Sigal, and Matthew Turk. Tibet: Identifying and evaluating biases in text-to-image generative models, 2024

    work page 2024

  17. [17]

    Evaluating gender bias in multilingual multimodal ai models: Insights from an indian context

    Kshitish Ghate, Arjun Choudhry, and Vanya Bannihatti Kumar. Evaluating gender bias in multilingual multimodal ai models: Insights from an indian context. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 338–350, 2024

    work page 2024

  18. [18]

    A large scale analysis of gender biases in text-to-image generative models, 2025

    Leander Girrbach, Stephan Alaniz, Genevieve Smith, and Zeynep Akata. A large scale analysis of gender biases in text-to-image generative models, 2025

    work page 2025

  19. [19]

    Gender disparities in artificial intelligence–generated images of hospital leadership in the united states

    Mia Gisselbaek, Joana Berger-Estilita, Laurens Minsart, Ekin K¨ oselerli, Arnout Devos, Francisco Maio Matos, Odmara L Barreto Chang, Peter Dieckmann, Melanie Suppan, and Sarah Saxena. Gender disparities in artificial intelligence–generated images of hospital leadership in the united states. Mayo Clinic Proceedings: Digital Health , 3(2):100218, 2025

    work page 2025

  20. [20]

    Bias transmission in large language models: evidence from gender-occupation bias in gpt-4

    Kirsten Morehouse, Weiwei Pan, Juan Manuel Contreras, and Mahzarin R Banaji. Bias transmission in large language models: evidence from gender-occupation bias in gpt-4. In ICML 2024 Next Generation of AI Safety Workshop , 2024

    work page 2024

  21. [21]

    Gender bias

    American Psychological Association. Gender bias. https://dictionary.apa.org/gender-bias , 2025. Entry in the APA Dictionary of Psychology

    work page 2025

  22. [22]

    Policy-Grounded Safety Evaluation of 20 Large Language Models

    Juan Manuel Contreras. Automated Safety Evaluations Across 20 Large Language Models: The Aymara LLM Risk and Responsibility Matrix, 2025. arXiv:2507.14719

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [23]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Sy...

    2023

  24. [24]

    Statistical Methods for Research Workers

    Ronald Aylmer Fisher. Statistical Methods for Research Workers . Oliver and Boyd, Edinburgh, 1925

    1925

  25. [25]

    An analysis of variance test for normality (complete samples)

    Samuel Sanford Shapiro and Martin B Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(3-4):591–611, 1965

    work page 1965

  26. [26]

    Robust tests for equality of variances

    Howard Levene. Robust tests for equality of variances. Contributions to probability and statistics , pages 278–292, 1960

    work page 1960

  27. [27]

    John W. Tukey. The Problem of Multiple Comparisons. Technical report, Princeton University, 1953. Unpublished manuscript

    1953

  28. [28]

    A coefficient of agreement for nominal scales

    Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960

    1960

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.