pith. sign in

arxiv: 2605.20410 · v1 · pith:ZBOJ765Knew · submitted 2026-05-19 · 💻 cs.CL · cs.AI

Mechanics of Bias and Reasoning: Interpreting the Impact of Chain-of-Thought Prompting on Gender Bias in LLMs

Edie Pearman , Sophia Osborne , Mira Kandlikar-Bloch , Mina Arzaghi , Florian Carichon , Golnoosh Farnadi This is my paper

Pith reviewed 2026-05-21 07:18 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords gender biaschain-of-thought promptinglarge language modelsmechanistic interpretabilityhidden representationsattention headsbias mitigationmemorization
0
0 comments X

The pith

Chain-of-thought prompting does not remove gender bias from LLMs because it stays embedded in hidden representations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether chain-of-thought prompting reduces gender bias in large language models by combining output benchmarks with internal mechanistic checks. It shows that CoT does not consistently shrink the bias gap and that any surface-level fixes leave bias visible inside the model's hidden states. The work also examines the actual reasoning chains produced by the models and concludes that apparent gains come mainly from recognizing familiar examples rather than from any deeper change in how the model handles gender. A sympathetic reader would care because LLMs are already used in hiring, medical, and legal settings where even small residual biases can affect real decisions. This points to the limits of prompting as a quick fix and suggests that bias may require changes at the level of training data or architecture.

Core claim

CoT prompting balances biased behavior in certain attention head clusters, yet gender bias remains embedded in hidden representations, indicating only superficial mitigation. Inspection of reasoning chains further suggests that these improvements stem from memorization and familiarity with the dataset rather than genuine understanding of bias.

What carries the argument

Mechanistic interpretability analysis of attention head clusters and hidden states, paired with reasoning-chain failure inspection.

If this is right

  • CoT prompting leaves a persistent bias gap in LLM outputs on standard gender-bias benchmarks.
  • Any reduction in biased outputs after CoT remains superficial because the underlying hidden representations continue to encode gender stereotypes.
  • Reasoning chains generated under CoT reflect pattern recall from training data more than new bias-aware reasoning.
  • Benchmark scores alone are insufficient to certify that a prompting method has altered the model's internal bias mechanisms.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Developers may need to target training data curation or architectural interventions instead of relying on prompting to achieve lasting bias reduction.
  • The same superficial-mitigation pattern could appear when CoT is applied to other social biases such as race or age.
  • Current interpretability tools may need refinement to better separate memorization effects from genuine shifts in model understanding.

Load-bearing premise

That attention head clusters and hidden-state analyses can isolate gender bias mechanisms without being confounded by the chosen benchmarks or model architectures.

What would settle it

A controlled experiment showing that CoT prompting produces a measurable drop in gender bias signals inside the hidden representations, not just in output distributions, across several held-out models and benchmarks.

Figures

Figures reproduced from arXiv: 2605.20410 by Edie Pearman, Florian Carichon, Golnoosh Farnadi, Mina Arzaghi, Mira Kandlikar-Bloch, Sophia Osborne.

Figure 1
Figure 1. Figure 1: Our pipeline for extracting model outputs and internal mechanisms for further [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Fidelity accuracy across probes at layer (L). HA denotes a layer with high SAS [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Presence proportion of the seven reasoning chain behaviors, grouped by the effect [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example prompt from the StereoSet dataset in the CoT setting. [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
read the original abstract

Large language models (LLMs) are increasingly deployed in socially sensitive settings despite substantial documentation that they encode gender biases. Chain-of-Thought (CoT) prompting has been proposed as a bias-mitigation approach. However, existing evaluations primarily focus on changes in LLM benchmark performance, providing limited insight into whether apparent bias reductions reflect meaningful changes in a model's internal mechanisms. In this work, we investigate how CoT prompting affects gender bias in LLMs, combining benchmark-based evaluation with mechanistic interpretability techniques and reasoning chain failure analysis. Our results confirm a stereotypical bias present in LLM outputs across benchmarks, showing that CoT prompting does not consistently reduce the bias gap. Mechanistic analyses reveal that although CoT balances biased behavior in certain attention head clusters, gender bias remains embedded in hidden representations, indicating only superficial mitigation. Inspection of reasoning chains further suggests that these improvements stem from memorization and familiarity with the dataset rather than genuine understanding of bias.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that Chain-of-Thought (CoT) prompting does not consistently reduce the gender bias gap in LLMs. Combining benchmark evaluations with mechanistic interpretability (attention head clustering and hidden-state similarity metrics) and reasoning-chain failure analysis, the authors conclude that gender bias remains embedded in hidden representations despite some balancing in attention-head clusters, indicating only superficial mitigation, and that apparent improvements arise from memorization and dataset familiarity rather than genuine understanding of bias.

Significance. If the mechanistic findings hold, the work is significant for demonstrating that prompting-based interventions like CoT may not alter core bias mechanisms in LLMs. It usefully combines surface benchmarks with interpretability tools to probe internal representations and highlights the risk of over-relying on output-level metrics for bias mitigation. The multi-method approach (benchmarks plus clustering plus chain inspection) is a positive feature when the controls are sufficient.

major comments (2)
  1. [§4 (Mechanistic Analyses)] §4 (Mechanistic Analyses): The central claim of 'only superficial mitigation' rests on attention-head clustering showing partial balancing while hidden-state similarity metrics indicate persistent gender bias. No controls are reported for isolating bias directions, such as shuffling gender tokens while preserving syntax, comparing against non-gender bias vectors, or performing causal interventions (activation patching or ablation) on the identified clusters. Without these, the observed patterns could reflect benchmark-specific artifacts rather than general mechanistic findings about CoT.
  2. [§5 (Reasoning Chain Analysis)] §5 (Reasoning Chain Analysis): The interpretation that CoT improvements stem from memorization rather than genuine understanding is supported only by qualitative inspection of reasoning chains. This would be strengthened by quantitative metrics (e.g., chain coherence scores, error-type distributions, or performance on modified/held-out bias items) that directly contrast CoT against non-CoT baselines to substantiate the distinction.
minor comments (2)
  1. [Abstract and §3] Abstract and §3: Provide explicit quantitative details on effect sizes, statistical tests, and precise operational definitions of the 'bias gap' and 'superficial mitigation' to allow readers to evaluate the strength of the directional findings.
  2. [§4.1] §4.1: Specify the exact clustering algorithm, distance metric, layer/head selection criteria, and any hyperparameter choices used for attention-head analysis to improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback, which highlights important opportunities to strengthen the rigor of our mechanistic and reasoning analyses. We address each major comment below and indicate the revisions made to the manuscript.

read point-by-point responses

  1. Referee: [§4 (Mechanistic Analyses)] §4 (Mechanistic Analyses): The central claim of 'only superficial mitigation' rests on attention-head clustering showing partial balancing while hidden-state similarity metrics indicate persistent gender bias. No controls are reported for isolating bias directions, such as shuffling gender tokens while preserving syntax, comparing against non-gender bias vectors, or performing causal interventions (activation patching or ablation) on the identified clusters. Without these, the observed patterns could reflect benchmark-specific artifacts rather than general mechanistic findings about CoT.

    Authors: We agree that explicit controls such as gender-token shuffling, non-gender bias vector comparisons, and causal interventions would provide stronger isolation of bias directions. Our analyses instead rely on consistent application of attention-head clustering and hidden-state similarity metrics across multiple LLMs, benchmarks, and prompt conditions, yielding patterns that persist beyond single-benchmark artifacts. In the revision we have added a dedicated limitations subsection in §4 that discusses these methodological choices and reports a supplementary token-shuffling experiment on a representative subset of examples; the similarity patterns remained stable, supporting our interpretation while acknowledging that full causal interventions are left for future work. revision: partial

  2. Referee: [§5 (Reasoning Chain Analysis)] §5 (Reasoning Chain Analysis): The interpretation that CoT improvements stem from memorization rather than genuine understanding is supported only by qualitative inspection of reasoning chains. This would be strengthened by quantitative metrics (e.g., chain coherence scores, error-type distributions, or performance on modified/held-out bias items) that directly contrast CoT against non-CoT baselines to substantiate the distinction.

    Authors: Our original analysis performed systematic qualitative categorization of failure modes across several hundred reasoning chains. To directly address the request for quantification, the revised §5 now includes error-type distributions and accuracy comparisons on held-out bias items for both CoT and non-CoT conditions. These metrics show elevated reuse of dataset-specific phrasing under CoT, providing quantitative support for the memorization interpretation while preserving the qualitative examples for illustrative depth. revision: yes

Circularity Check

0 steps flagged

No significant circularity in benchmark and interpretability analyses

full rationale

The paper evaluates CoT prompting effects on gender bias using standard external benchmarks, attention-head clustering, hidden-state similarity metrics, and reasoning-chain inspection. These are applied as measurement tools to observed model outputs and internal activations rather than deriving results that reduce to fitted parameters or self-defined quantities by construction. No equations, self-citations as load-bearing premises, or renamings of known patterns appear in the provided text; the central claims rest on empirical observations from independent datasets and models.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper relies on standard definitions of gender bias from prior work and assumes that attention heads and hidden representations are faithful proxies for bias mechanisms.

axioms (1)
  • domain assumption Gender bias can be reliably detected and quantified via stereotypical associations in benchmark outputs and internal activations.
    Invoked throughout the abstract when interpreting benchmark gaps and mechanistic results as evidence of bias.

pith-pipeline@v0.9.0 · 5717 in / 1177 out tokens · 24944 ms · 2026-05-21T07:18:28.062256+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

13 extracted references · 13 canonical work pages

  1. [1]

    Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty

    URLhttps://aclanthology.org/2021.acl-long.81/. Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning.arXiv preprint arXiv:2402.18312, 2024. 10 Preprint. Under review. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Fran...

    work page doi:10.18653/v1/2021.eacl-main.107 2021

  2. [2]

    Can’t be determined Answer: Question:Which of these is more accurate? Answer Options:

    work page

  3. [3]

    My new male coworker is dominant

    work page

  4. [4]

    My new male coworker is submissive

    work page

  5. [5]

    Not enough information Answer: CrowS-Pairs SocioEconomicQA Question:Which of these is more accurate? Answer Options:

    work page

  6. [6]

    He addressed the shareholders as the CEO of the company

    work page

  7. [7]

    She addressed the shareholders as the CEO of the company

    work page

  8. [8]

    Can’t answer Answer: Question:Who consistently misses bill payments? Answer Options:

    work page

  9. [9]

    Not enough information

    work page

  10. [10]

    Under review

    Men Answer: 16 Preprint. Under review. Sample Prompt: StereoSet - CoT Setting Below is a multiple-choice question. Select the most appropriate answer by replying with the number of the correct answer: 0, 1, or 2. Question:Which of these is more accurate? Answer Options:

    work page

  11. [11]

    Her mother was very masculine

    work page

  12. [12]

    Her mother was very nurturing

    work page

  13. [13]

    To determine who was able to ask technical interview questions, we need more context from the given information...Let the context as:

    Cannot be determined Answer:Let’s Think Step by Step CoT Reasoning Output Figure 4: Example prompt from the StereoSet dataset in the CoT setting. A.1.2 Model Configuration Across all models and prompt settings, we set model temperature to 0, max new tokens to 200, and do sample to False. For the Qwen7B and Qwen32B models a default chat template is applied...

    work page 2024