Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Guiding LLM Decision-Making with Fairness Reward Models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single Fairness Reward Model, trained once on weakly supervised LLM-annotated reasoning steps, transfers across tasks, domains, and model families to shrink fairness gaps without sacrificing accuracy.

desk verdict A genuinely new fairness PRM with real transfer, but the weak-label validation is thinner than claimed and the temperature is tuned on the test tasks; worth reviewing, needs revision. read the letter →

arxiv 2507.11344 v1 pith:54B4Q4D3 submitted 2025-07-15 cs.LG

Zara Hall , Melanie Subbiah , Thomas P Zollo , Kathleen McKeown , Richard Zemel This is my paper
classification cs.LG
keywords fairnessrewardmodelchain-of-thoughtreasoningweaksupervisionprocessinference-timecomputeequalizedoddsLLMbiasmitigationhigh-stakesdecision-making
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that fairness in LLM decision-making can be treated as a verifiable property of individual reasoning steps, and that a single reward model trained on weakly supervised, LLM-annotated examples of biased versus unbiased chains transfers across tasks, domains, protected attributes, and model families with no additional fine-tuning. The motivation is that scaling up chain-of-thought sampling improves accuracy but can amplify stereotypes; the authors propose a Fairness Reward Model (FRM) that scores every step of a chain and then re-weights candidate answers by their fairness. If the claim holds, a practitioner could add a fairness filter to any reasoning model at inference time—for bail recommendations, loan decisions, content moderation, or hiring—without retraining the underlying model and without accepting an accuracy penalty. The paper reports that on recidivism prediction (COMPAS), social media moderation (Civil Comments), and job screening (Bias in Bios), the once-trained FRM shrinks equalized odds and equalized opportunity gaps while matching or exceeding baseline accuracy, and that it also transfers to a reasoning model family it never saw during training.

What carries the argument

The central object is a step-level Fairness Reward Model, a function $f_\theta: \mathcal{Z} \to \mathbb{R}$ trained with binary cross-entropy to output a real-valued fairness score for each chain-of-thought step. Chains produced by an LLM are segmented into atomic reasoning steps; at inference, each step gets a score $\sigma(f_\theta(z_{k,t}))$, the chain score is the average $r_k = \frac{1}{T_k}\sum_t \sigma(f_\theta(z_{k,t}))$, and the final answer is chosen by a temperature-weighted vote $w_k = \exp(r_k/\tau) / \sum_j \exp(r_j/\tau)$ over the candidate answers of $n_{test}$ sampled chains. The temperature parameter $\tau$ is the control knob that trades pure consensus (uniform weights as $\tau \to \infty$) against strict fairness selection ($\tau \to 0$). The load-bearing design choice is process-level supervision: training on step labels provided by an off-the-shelf LLM judge rather than on outcome labels, which the authors show is superior to chain-level outcome reward models and to zero-shot fairness scoring.

What would settle it

The decisive check is to train the same reward-model architecture on human labels instead of LLM labels for a matched set of reasoning steps and re-run the COMPAS, Civil Comments, and Bias in Bios evaluations; if the fairness improvements vanish, the central transfer claim is riding on the weak labeler's particular biases rather than on a generalizable fairness signal.

Watch

Extended reading notes

Core claim

The paper's central claim is that a single Fairness Reward Model, trained at the step level on weakly supervised labels, generalizes across tasks, domains, and model families without fine-tuning. Concretely, it claims that when 32 chain-of-thought traces are re-weighted by per-chain fairness scores (the mean of step-level sigmoid rewards) under a temperature-controlled softmax vote, the resulting decisions have smaller equalized odds and equalized opportunity gaps than single-chain CoT, majority voting, or fairness prompting, while accuracy is maintained or improved on COMPAS, Civil Comments, and Bias in Bios. The authors additionally claim that the FRM transfers to a previously unseen reasoning model (Mistral-7B), and that the design choices of step-level process supervision and LLM-generated weak labels are what make the transfer work: ablations replacing them with chain-level outcome rewards or BBQ ground-truth labels reduce effectiveness.

Load-bearing premise

The load-bearing premise is that the off-the-shelf LLM used to label reasoning steps as biased or unbiased is reliable enough to train a verifier that generalizes; the paper's own human study measured only 75% agreement with three human annotators and an average Cohen's kappa around 0.23, and the authors document cases where the labeler over-flags benign group mentions and misses implicit or hallucinated biases.

Editorial extensions

If this is right

  • A fairness verifier can be trained once on synthetic BBQ-style reasoning and reused as a drop-in aggregation module for unrelated high-stakes decision tasks, without fine-tuning the generator or the verifier.
  • Inference-time re-weighting decouples fairness from prompting: setting the temperature $\tau$ continuously trades fairness against consensus and accuracy, which brittle fairness prompts cannot do.
  • Because scores are step-localized, an auditor can trace an unfair final decision back to the exact reasoning step that triggered its down-weighting.
  • Fairness gains do not have to be bought with accuracy: in Civil Comments and in the Mistral runs of COMPAS, fairer aggregation also improved accuracy.
  • The same FRM also improves fairness on a reasoning model family never seen during training (Mistral-7B), so the verifier is not tied to the models that generated its training data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stress test would apply the FRM to pairs of chains that reach the same answer and differ only in whether a protected-group attribute is mentioned neutrally; if the verifier systematically downgrades the neutral mention, part of the reported fairness gain would be an artifact of label noise correlated with demographic word usage.
  • The temperature parameter could be turned into a per-decision control knob by selecting $\tau$ from a calibration set with fairness constraints, an extension the paper does not develop but its aggregation formula directly enables.
  • The documented failure case where a chain that mentions a stereotype but does not act on it is penalized more than one that acts on it suggests that equal step-averaging is the weak link; a variant that weights steps by their causal contribution to the final answer could repair this specific failure mode.
  • Because the training prompts come from BBQ, where the correct answer is often 'cannot be determined,' transferring to richer decision settings may under-represent subtle statistical biases; domain-adapted labeling pipelines would test how far the approach extends.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces a Fairness Reward Model (FRM), a step-level verifier trained on weakly supervised labels from GPT-4o-mini that judge whether individual chain-of-thought reasoning steps are biased or unbiased, using synthetic BBQ question-answering traces as training data. At inference time, the FRM scores each step of multiple CoT samples from a downstream LLM and re-weights the final vote over candidate answers with a temperature-controlled softmax, aiming to reduce group fairness gaps without harming accuracy. The authors evaluate the approach on COMPAS recidivism prediction, Civil Comments toxicity moderation, and Bias in Bios occupation prediction, reporting consistent reductions in equalized odds and equalized opportunity gaps across race, religion, sexual orientation, and gender, while sometimes improving accuracy. They also test generalization to a previously unseen reasoning model (Mistral-7B) and perform ablations on label source, reward granularity, and temperature. The paper releases training data and model weights.

Significance. If the central transfer claim held, this would be a practically valuable contribution: an inference-time fairness intervention that requires no fine-tuning of the downstream model and provides step-level interpretability. The paper's strengths include a clear and simple mechanism, a public code/data release, evaluation on three real-world benchmarks with external fairness metrics, and an honest qualitative analysis of failure modes. The main weakness is that the entire method rests on the reliability of GPT-4o-mini's step-level bias labels, and the evidence for that reliability is thin and, in places, overstated. The temperature hyperparameter also appears to be selected on the evaluation tasks, weakening the 'no additional tuning' claim. These issues are fixable but require additional analysis and more careful reporting.

major comments (4)
  1. [Section 3, step (2); Appendix A.3] The paper repeatedly calls the LLM-human agreement 'substantial,' but the reported evidence does not support this. With 201,500 of 255,000 steps (about 79%) labeled unbiased, a majority-class classifier would achieve roughly 79% raw agreement, yet GPT-4o-mini's average agreement with human annotators is only 75%, and the average Cohen's kappa is 0.2259 (range 0.0744 to 0.3557). This is at or below chance-level agreement when accounting for class imbalance. Because the FRM is trained entirely on these weak labels, the load-bearing premise of the paper is unvalidated. Please report per-class precision and recall for the LLM judge, characterize the disagreement structure (e.g., is the judge over-flagging demographic mentions?), and either temper the 'substantial agreement' language or provide stronger evidence that the label noise is not systematic.
  2. [Section 5; Section 6.3, Figure 8] The temperature parameter tau is selected per model family on the same downstream evaluation tasks (tau = 0.2 for Llama models, tau = 0.01 for Mistral). This is a form of evaluation-set tuning that conflicts with the claim that the FRM transfers 'without additional fine-tuning' or bespoke adjustment. The ablation in Figure 8 shows that tau materially affects the fairness-accuracy trade-off, so the choice is not innocuous. Please either fix tau a priori based on a validation split, report the full tau grid and show that the main conclusions hold across it, or treat tau selection as part of the method and describe it transparently as a tuned hyperparameter.
  3. [Section 6.4, Figures 9-12] The qualitative results indicate that the FRM systematically assigns low fairness scores to steps that simply mention protected-group attributes, even when the mention is a neutral restatement of the context (see Figure 12, where GPT-4o-mini labels a context restatement as biased). If this behavior is systematic, the downstream parity gains could result from rewarding chains that avoid any discussion of demographic attributes—a fairness-through-unawareness artifact—rather than from learning a generalizable notion of process-level fairness. Please provide a quantitative analysis of this artifact, for example by measuring the correlation between FRM chain scores and the presence of protected-group tokens in the chain, and by testing the FRM on neutral context restatements versus genuinely biased reasoning.
  4. [Section 6.1; Appendix C.1] The main cross-domain results are presented only as figures, and the text reports no confidence intervals or effect sizes for the fairness gaps. A single bootstrap statement (p < 0.01) is insufficient to assess the magnitude and stability of the claimed improvements, especially for Civil Comments-Religion where the raw equalized odds gap exceeds sixty percentage points. Please provide point estimates with bootstrap confidence intervals for all tasks and metrics, and report the accuracy numbers explicitly rather than only as 'no significant loss.'
minor comments (6)
  1. [Author affiliations] There are typos in the affiliations: 'Columbia Univeristy' appears twice.
  2. [Appendix A.2 / Section 6.3] The phrase 'LLM labels' is ambiguous in the ablation description; please clarify that 'LLM labels' always refers to GPT-4o-mini annotations, especially when discussing the ORM trained on LLM labels.
  3. [Model Card] The model card lists 'sexual idenity' (typo for 'identity') under Relevant Groups.
  4. [Section 3.1] Figure 4 is referenced but the figure content is not described in the text; please add a sentence explaining the axes and the reported 25% improvement.
  5. [Section 4 / CivilComments] The dataset name is spelled inconsistently: 'Civil Comments' in the body and 'CivilComments' in tables and figures; please standardize.
  6. [Section 6.4] The sentence 'We suspect that this may stem from the usage of LLM labels during training' is a hedged speculation; consider replacing it with a definitive statement or an analysis that supports it.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central fairness evaluations are external to the FRM's training signal, and no load-bearing claim reduces to its own inputs by construction.

full rationale

The derivation chain is self-contained against external benchmarks. The FRM is trained with binary cross-entropy on GPT-4o-mini step-level bias labels over BBQ-generated reasoning chains (Section 3, steps 2-3), then applied at inference time by scoring downstream chain-of-thought steps and re-weighting candidate answers (Section 3, steps 4-5). The reported fairness improvements are measured with equalized odds and equalized opportunity gaps computed from ground-truth labels in COMPAS, Civil Comments, and Bias in Bios (Sections 4-5), which are not defined in terms of the FRM scores or the GPT-4o-mini labels. Held-out BBQ validation and the ablations in Section 6.3 also compare against external baselines rather than against the training objective. The human-annotation study (Appendix A.3) is presented as label-quality evidence, not as the downstream fairness metric. The only self-referential element is the temperature parameter tau, which the paper reports as set per model family (Section 5: "for all experiments using Llama models for inference, we set the temperature tau to 0.2 ... for Mistral, we set tau = 0.01") and which is ablated on the downstream tasks in Section 6.3; this is hyperparameter selection rather than a fitted parameter, so it does not make the fairness results equal to the FRM's training signal by construction. There is no load-bearing self-citation, no invoked uniqueness theorem, and no renamed empirical pattern presented as derivation. The weak-label reliability concern raised by the authors' own kappa of 0.2259 is a correctness and generalization risk, not a circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical entities or forces are introduced. The main assumptions are about label quality, fairness metric choice, and the aggregation mechanism, all of which are domain assumptions rather than standard mathematical axioms.

free parameters (2)
  • Temperature tau = 0.2 for Llama inference models, 0.01 for Mistral
    Controls the softmax weighting over chain scores in the final aggregation (Section 3, step 5). Chosen per model family without a reported held-out validation split, and the ablation (Figure 8) shows it strongly affects fairness gaps.
  • Number of sampled chains N = 32
    All downstream experiments use 32 CoT samples. This is a fixed choice that affects the variance of both accuracy and fairness estimates, and no sensitivity analysis is provided.
assumptions (3)
  • domain assumption GPT-4o-mini's step-level bias labels are a sufficiently reliable proxy for human judgments of biased reasoning.
    Used to create the training set (Section 3, step 2). The paper's own human study (Appendix A.3) shows low Cohen's kappa (0.07-0.36) between the LLM judge and human annotators, and Section 6.4 documents systematic failure modes.
  • domain assumption Equalized odds and equalized opportunity gaps are the appropriate fairness measures for these decision tasks.
    The paper measures fairness only through these two metrics (Section 5) and acknowledges in Limitations that calibration, causal fairness, or individual fairness could reveal different trade-offs.
  • domain assumption Step-level scores averaged across a chain, followed by softmax re-weighting, will improve downstream group fairness without hurting accuracy.
    This is the core mechanism proposed in Section 3 (steps 4-5); it is validated only empirically on three datasets, and the failure cases in Section 6.4 show it can misorder chains when biased steps are not causally decisive.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Guiding LLM Decision-Making with Fairness Reward Models." pith.science (2026). https://pith.science/paper/54B4Q4D3

@misc{pith2026250711344,
  author       = {Pith},
  title        = {Pith review of: Guiding LLM Decision-Making with Fairness Reward Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/54B4Q4D3}},
  note         = {Machine review of arXiv:2507.11344}
}
read the original abstract

Large language models are increasingly used to support high-stakes decisions, potentially influencing who is granted bail or receives a loan. Naive chain-of-thought sampling can improve average decision accuracy, but has also been shown to amplify unfair bias. To address this challenge and enable the trustworthy use of reasoning models in high-stakes decision-making, we propose a framework for training a generalizable Fairness Reward Model (FRM). Our model assigns a fairness score to LLM reasoning, enabling the system to down-weight biased trajectories and favor equitable ones when aggregating decisions across reasoning chains. We show that a single Fairness Reward Model, trained on weakly supervised, LLM-annotated examples of biased versus unbiased reasoning, transfers across tasks, domains, and model families without additional fine-tuning. Applied to real-world decision-making tasks including recidivism prediction and social media moderation, we show that our approach consistently improves fairness while matching, or even surpassing, baseline accuracy.

Figures

Figures reproduced from arXiv: 2507.11344 by the authors.

Figure 1
Figure 1. Scaling inference-time compute, such as by sampling multiple chain-of-thought (CoT) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our framework for training and applying a generalizable Fairness Reward Model includes [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Our framework samples a BBQ (Parrish et al., 2022) question and generates reasoning chains to be labeled for bias by a powerful LLM. BBQ questions often provide limited information, which may cause a model to form a conclusion based on stereotypes rather than recognizing there is not enough information to answer the question. In this shortened example from our dataset, reasoning chain 1 falls into this trap, while r… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Validation results for baseline methods and our FRM applied to held-out BBQ data. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Results for generalizing our Fairness Reward Model across three different task domains [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Results generalizing our FRM to reasoning chains produced by a previously unseen LLM, [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Results ablating various design decisions in our FRM: (1) source of weak supervision [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Results ablating the temperature parameter in the decision aggregation step. As expected, [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Successful case where biased reasoning changes the decision. The second chain invokes [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Successful fairness scoring on a COMPAS example where both reasoning chains predict [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Failure case: the FRM assigns a lower fairness score to a reasoning chain that acknowl [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: LLM annotation failure case: the LLM annotator marks this step as biased based on its [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Example outputs and FRM scores for a BBQ question [PITH_FULL_IMAGE:figures/full_fig_p030_13.png]
Figure 14
Figure 14. Figure 14: Example outputs and FRM scores for a CivilComments decision [PITH_FULL_IMAGE:figures/full_fig_p031_14.png]
Figure 15
Figure 15. Figure 15: Hallucinated reasoning labeled as unbiased. The reasoning step invents emotional states [PITH_FULL_IMAGE:figures/full_fig_p031_15.png]
Figure 16
Figure 16. Figure 16: Incoherent reasoning labeled as biased. The reasoning step is circular and fails to engage [PITH_FULL_IMAGE:figures/full_fig_p032_16.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

Reference graph

Works this paper leans on

52 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Measuring gender and racial biases in large language models, 2024

    Jiafu An, Difang Huang, Chen Lin, and Mingzhu Tai. Measuring gender and racial biases in large language models, 2024. URL https://arxiv.org/abs/2403.15281

    arXiv 2024

  2. [2]

    Machine bias

    Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica, 2016. URL https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

    work page 2016

  3. [3]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

    arXiv 2022

  4. [4]

    Fairness and Machine Learning

    Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning. fairmlbook.org, 2019. http://www.fairmlbook.org

    work page 2019

  5. [5]

    Scaling test-time compute with open models, 2024

    Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time compute with open models, 2024. URL https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling-test-time-compute

    2024

  6. [6]

    On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, 2021

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, 2021. URL https://s10251.pcdn.co/pdf/2021-bender-parrots.pdf

    work page 2021

  7. [7]

    Red-teaming large language models using chain of utterances for safety-alignment, 2023

    Rishabh Bhardwaj and Soujanya Poria. Red-teaming large language models using chain of utterances for safety-alignment, 2023. URL https://arxiv.org/abs/2308.09662

    arXiv 2023

  8. [8]

    Nuanced metrics for measuring unintended bias with real data for text classification

    Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Sihem Amer - Yahia, Mohammad Mahdian, Ashish Goel, Geert - Jan Houben, Kristina Lerman, Julian J. McAuley, Ricardo Baeza - Yates, and Leila Zia, editors, Companion of The 2019 World Wide W...

    work page arXiv 2019

Show all 52 references
  1. [9]

    Bradley C. A. Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher R \' e , and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. CoRR, abs/2407.21787, 2024. URL https://doi.org/10.48550/arXiv.2407.21787

    2024 doi arXiv

  2. [10]

    Alphamath almost zero: Process supervision without process

    Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: Process supervision without process. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=VaXnxQ3UKo

    2024

  3. [11]

    Fair prediction with disparate impact: A study of bias in recidivism prediction instruments

    Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5 0 (2): 0 153--163, 2017

    2017

  4. [12]

    Bias in bios: A case study of semantic representation bias in a high-stakes setting

    Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conf...

    2019

  5. [13]

    Evaluation of A frican A merican language bias in natural language generation

    Nicholas Deas, Jessica Grieser, Shana Kleiner, Desmond Patton, Elsbeth Turcan, and Kathleen McKeown. Evaluation of A frican A merican language bias in natural language generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empi...

    2023 doi

  6. [14]

    The accuracy, fairness, and limits of predicting recidivism

    Julia Dressel and Hany Farid. The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4 0 (1): 0 eaao5580, 2018

    2018

  7. [15]

    Gaebler, Sharad Goel, Aziz Huq, and Prasanna Tambe

    Johann D. Gaebler, Sharad Goel, Aziz Huq, and Prasanna Tambe. Auditing large language models for race & gender disparities: Implications for artificial intelligence-based hiring. Behavioral Science & Policy, 10 0 (2): 0 46--55, 2024. doi:10.1177/23794607251320229. URL https://...

    2024 doi

  8. [16]

    Gallegos, Ryan A

    Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, 50 0 (3): 0 1097--1179, September 2024. doi:10.1162/c...

    2024 doi

  9. [17]

    Debiasing pre-trained language models via efficient fine-tuning

    Michael Gira, Ruisu Zhang, and Kangwook Lee. Debiasing pre-trained language models via efficient fine-tuning. In Bharathi Raja Chakravarthi, B Bharathi, John P McCrae, Manel Zarrouk, Kalika Bali, and Paul Buitelaar, editors, Proceedings of the Second Workshop on Language Techn...

    2022 doi

  10. [18]

    Bias in large language models: Origin, evaluation, and mitigation, 2024

    Yufei Guo, Muzhe Guo, Juntao Su, Zhou Yang, Mengqiu Zhu, Hongfei Li, Mengyang Qiu, and Shuo Shuo Liu. Bias in large language models: Origin, evaluation, and mitigation, 2024. URL https://arxiv.org/abs/2411.10915

    2024 arXiv

  11. [19]

    Equality of opportunity in supervised learning

    Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, pages 3315--3323, 2016

    2016

  12. [20]

    V- ST ar: Training verifiers for self-taught reasoners

    Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V- ST ar: Training verifiers for self-taught reasoners. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=stmqBSW2dV

    2024

  13. [21]

    Prompting techniques for reducing social bias in llms through system 1 and system 2 cognitive processes, 2024

    Mahammed Kamruzzaman and Gene Louis Kim. Prompting techniques for reducing social bias in llms through system 1 and system 2 cognitive processes, 2024. URL https://arxiv.org/abs/2404.17218

    2024 arXiv

  14. [22]

    Evaluating gender bias in large language models via chain-of-thought prompting, 2024

    Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki, and Timothy Baldwin. Evaluating gender bias in large language models via chain-of-thought prompting, 2024. URL https://arxiv.org/abs/2401.15585

    2024 arXiv

  15. [23]

    Gender bias and stereotypes in large language models

    Hadas Kotek, Rikker Dockum, and David Sun. Gender bias and stereotypes in large language models. In Proceedings of the ACM collective intelligence conference, pages 12--24, 2023

    2023

  16. [24]

    When do pre-training biases propagate to downstream tasks? a case study in text summarization

    Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, and Tatsunori Hashimoto. When do pre-training biases propagate to downstream tasks? a case study in text summarization. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of t...

    2023 doi

  17. [25]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/fo...

    2024

  18. [26]

    Debiasing large language models with structured knowledge

    Congda Ma, Tianyu Zhao, and Manabu Okumura. Debiasing large language models with structured knowledge. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 10274--10287, Bangkok, Thailand, August ...

    2024 doi

  19. [27]

    Fairness-guided few-shot prompting for large language models

    Huan Ma, Changqing Zhang, Yatao Bian, Lemao Liu, Zhirui Zhang, Peilin Zhao, Shu Zhang, Huazhu Fu, Qinghua Hu, and Bingzhe Wu. Fairness-guided few-shot prompting for large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances...

    2023

  20. [28]

    Evaluating gender bias transfer between pre-trained and prompt-adapted language models, 2024

    Natalie Mackraz, Nivedha Sivakumar, Samira Khorshidi, Krishna Patel, Barry-John Theobald, Luca Zappella, and Nicholas Apostoloff. Evaluating gender bias transfer between pre-trained and prompt-adapted language models, 2024. URL https://arxiv.org/abs/2412.03537

    2024 arXiv

  21. [29]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  22. [30]

    Bias in word embeddings

    Orestis Papakyriakopoulos, Simon Hegelich, Juan Carlos Medina Serrano, and Fabienne Marco. Bias in word embeddings. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* '20, page 446–457, New York, NY, USA, 2020. Association for Computing M...

    2020

  23. [31]

    BBQ : A hand-built bias benchmark for question answering

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ : A hand-built bias benchmark for question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Associ...

    2022 doi

  24. [32]

    Divine LL a MA s: Bias, stereotypes, stigmatization, and emotion representation of religion in large language models

    Flor Miriam Plaza-del Arco, Amanda Cercas Curry, Susanna Paoli, Alba Cercas Curry, and Dirk Hovy. Divine LL a MA s: Bias, stereotypes, stigmatization, and emotion representation of religion in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors...

    2024 doi

  25. [33]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  26. [34]

    On second thought, let ' s not think step by step! bias and toxicity in zero-shot reasoning

    Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On second thought, let ' s not think step by step! bias and toxicity in zero-shot reasoning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of t...

    2023 doi

  27. [35]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  28. [36]

    Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?i...

    2025

  29. [37]

    Unveiling gender bias in terms of profession across llms: Analyzing and addressing sociological implications, 2023

    Vishesh Thakur. Unveiling gender bias in terms of profession across llms: Analyzing and addressing sociological implications, 2023. URL https://arxiv.org/abs/2307.09162

    2023 arXiv

  30. [38]

    Toward self-improvement of llms via imagination, searching, and criticizing

    Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi, and Dong Yu. Toward self-improvement of llms via imagination, searching, and criticizing. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Ho...

    2025

  31. [39]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  32. [40]

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don't always say what they think: unfaithful explanations in chain-of-thought prompting. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red...

    2023

  33. [41]

    Solving math word problems with process- and outcome-based feedback, 2022

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022. URL https://arxiv.org/abs/2211.14275

    2022 arXiv

  34. [42]

    ``kelly is a warm person, joseph is a role model'': Gender biases in LLM -generated reference letters

    Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. ``kelly is a warm person, joseph is a role model'': Gender biases in LLM -generated reference letters. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Compu...

    2023 doi

  35. [43]

    Alphazero-like tree-search can guide large language model decoding and training

    Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  36. [44]

    Math-shepherd: Verify and reinforce LLM s step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLM s step-by-step without human annotations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meet...

    2024 doi

  37. [45]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://o...

    2023

  38. [46]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in N...

    2022

  39. [47]

    Blueprint for an ai bill of rights: Making automated systems work for the american people, 2022

    White House . Blueprint for an ai bill of rights: Making automated systems work for the american people, 2022

    2022

  40. [48]

    Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval, page 1578–1590

    Kyra Wilson and Aylin Caliskan. Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval, page 1578–1590. AAAI Press, 2025

    2025

  41. [49]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Infor...

    2023

  42. [50]

    Scaling relationship on learning mathematical reasoning with large language models, 2023

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models, 2023. URL https://arxiv.org/abs/2308.01825

    2023 arXiv

  43. [51]

    Star: Bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 15476--15488. Curran Associate...

    2022

  44. [52]

    Towards effective discrimination testing for generative ai

    Thomas Zollo, Nikita Rajaneesh, Richard Zemel, Talia Gillis, and Emily Black. Towards effective discrimination testing for generative ai. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pages 1028--1047, 2025

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.