Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

Analysing Moral Bias in Finetuned LLMs through Mechanistic Interpretability

T0 review · 3 major / 7 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned LLMs learn the Knobe moral bias in a handful of layers, and swapping in pretrained activations at one layer removes it without retraining.

desk verdict A believable demonstration that the Knobe effect in fine-tuned LLMs can be localized to mid-to-late layers and removed by activation patching, but the elimination claim rests on incomplete reporting. read the letter →

arxiv 2510.12229 v3 pith:EUKSSP33 submitted 2025-10-14 cs.CL cs.AI

Bianca Raimondi , Daniela Dalbagno , Maurizio Gabbrielli This is my paper
classification cs.CLcs.AI
keywords KnobeeffectmechanisticinterpretabilitylayerpatchingactivationmoralbiasLLMfine-tuningintentionalitymitigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a famous human moral bias—the Knobe effect, in which harmful side effects are judged more intentional than beneficial ones—appears in large language models and, if so, where it lives. Across three model families (Llama, Mistral, Gemma), the authors find that pretrained versions show little or no such bias, while instruction-finetuned versions show a strong bias that grows with model scale, from roughly 1.6–1.7 points in 7–8B models up to 6.07 in a 27B model. Using activation-patching, they then show that overwriting the residual stream at a single mid-to-late layer with the pretrained model's activation for the same input reduces the bias to near zero (gaps of 0.00, 0.00, and 0.03), with only about one percent average regression on standard benchmarks. If correct, the claim is that a social bias acquired during alignment is localized and can be removed at inference time without retraining—making mechanistic interpretability a practical tool for model repair rather than just diagnosis.

What carries the argument

The load-bearing mechanism is Layer-Patching on the residual stream: for each layer, the paper computes the mean activation difference between negative and positive moral scenarios, then runs the finetuned model forward while replacing the residual-stream activation at a chosen layer with the pretrained model's activation for the same input. Since the two models share architecture and tokenizer, the substitution targets only the finetuning-induced state while preserving the rest of the forward pass. The paper's metric, ΔKnobe = μ_neg − μ_pos, measures the intentionality gap; the claim is that the activation difference identifies where the gap originates, and patching at that layer closes it.

What would settle it

Plot the full distribution of intentionality ratings that the patched model gives across the 80 scenarios, not just the mean difference. If the patched model returns near-constant values (e.g., always 5), random values, or otherwise stops tracking scenario content, then Δpatch ≈ 0 reflects output collapse rather than bias removal; the paper's Table 4 benchmarks do not rule this out because they measure knowledge accuracy, not the semantic appropriateness of the rating scale.

Watch

Extended reading notes

Core claim

The central claim is that the Knobe effect in instruction-finetuned LLMs is a localized computation, not an emergent diffuse property: the bias is encoded in a specific set of mid-to-late transformer layers during finetuning, and patching the residual stream of just one such layer with the corresponding pretrained activations eliminates the effect. The intentionality gap between negative and positive scenarios drops from 1.60 to 0.00 (Llama-3.1-8B), 1.67 to 0.00 (Mistral-7B), and 3.83 to 0.03 (Gemma-2-9B), while four standard benchmarks show minimal regression. The authors read this as evidence that finetuning writes human-like moral asymmetries as modular components, making mechanistic inte

Load-bearing premise

The mitigation claim hinges on the assumption that substituting pretrained activations at one layer is a meaningful unbiasing operation rather than a damaging perturbation—the paper reports only the negative–positive gap, not the patched model's raw rating values, so a flat or degenerate output distribution would also produce a zero gap.

Editorial extensions

If this is right

  • If the bias is localized, then a single inference-time patch can remove it—the paper reports near-total elimination (gaps of 0.00, 0.00, 0.03) across three model families.
  • Because the effect appears only after finetuning and scales with model size, the bias is attributable to the alignment objective, not to pretraining or raw capacity.
  • Layer-patching preserves general performance, with roughly one percent average regression on ARC-Easy, HellaSwag, MMLU, and TruthfulQA, so mitigation is selective rather than destructive.
  • The same approach could be applied to other finetuning-induced social biases (e.g., gender or racial stereotypes), as the paper proposes for future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The zero gap may hide distribution collapse: if patching drives every response toward a middle rating, the negative–positive difference vanishes without genuine unbiasing. A reader can check this by plotting the patched model's raw answer distribution across the 80 scenarios.
  • If localization holds generally, instruction tuning may write human-like biases as layer-localized features rather than distributed weight changes, which would open a 'surgical unalignment' route that removes selected biases while keeping capabilities intact.
  • A practical portability problem the paper leaves implicit: deployed models rarely ship with their pretrained checkpoints, so an actionable version would learn a per-layer correction vector that approximates the pretrained activations without needing the base model.
  • The single-layer locus invites a stress test: does the same layer carry the bias for other moral scenarios or for other biases (gender, race)? A fixed locus would support a modular moral-cognition story; shifting loci would require a more distributed explanation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper investigates whether the Knobe effect—the tendency to rate harmful side effects as more intentional than helpful ones—appears in LLMs and whether it can be localized and removed through activation patching. Using three open-weight model families (Llama, Mistral, Gemma) in pretrained and instruction-finetuned versions, the authors measure intentionality ratings on 80 moral scenarios with 283 stochastic generations per model. They report that finetuning substantially amplifies the effect (Table 1), that residual-stream activation differences between negative and positive scenarios concentrate in mid-to-late layers (Figure 2), and that replacing the residual activations at a single critical layer with the corresponding pretrained activations nearly eliminates the behavioral gap (Table 3). An ablation study extends the behavioral result to models of different scales. The paper concludes that moral biases are not diffusely emergent but are localized and can be mitigated at inference time, without retraining.

Significance. If the mechanistic localization and mitigation claims are correct, the paper offers a concrete route to targeted, inference-time bias intervention in LLMs, which would be practically valuable and would challenge the view that socially learned biases are diffusely represented across the network. The paper has several strengths: it uses three different open-weight model families, includes a human behavioral benchmark, reports layer-wise patching curves, and checks general-task benchmarks after intervention. The behavioral pattern—finetuning amplifies a human-like moral asymmetry—is plausible and consistent with prior work. However, the strongest claims currently rest on a statistically invalid ANOVA structure and on a post-hoc selection of the best patching layer, and the patched-model evaluation does not rule out degenerate output collapse on the rating task. These issues are fixable but require reanalysis and additional diagnostics before the headline conclusions can be accepted.

major comments (3)
  1. [Section 3.1, Statistical Analysis] The repeated-measures ANOVA is incorrectly specified. The two factors are Version (Pretrained, Finetuned) and LLM (Llama, Mistral, Gemma), and both are treated as within-subjects factors. But LLM is a between-model factor: the 283 generations for each model are not responses from the same 283 subjects. Similarly, pretrained and finetuned generations are independent draws, not paired observations from the same participants; pairing them by generation index is unjustified. The reported interaction F(2,564)=283.574 and error df=564 correspond precisely to treating 283 as the sample size for both within-subject factors, which is not a valid repeated-measures design. The p-values and post-hoc comparisons in Section 4.1 therefore do not provide the claimed statistical support. Reanalysis with a mixed-effects model (model family as a fixed between-model factor, scenario/test as random effects,
  2. [Section 4.3, Table 3] The mitigation claim is under-supported because the paper reports only the post-patch gap Δ_patch = μ_neg − μ_pos, with minimum values of 0.00, 0.00, and 0.03. A zero gap is also what one would observe if the patched model collapses to a near-constant rating regardless of scenario valence. No absolute post-patch means, standard deviations, response distributions, or percentages of valid numeric 0–10 responses are reported. Table 4's benchmarks do not resolve this: they are multiple-choice tasks and do not test the model's ability to produce meaningful, on-scale intentionality ratings, and the table does not state which layer was patched for each model. I request the full post-patch rating distributions at the critical layer, including μ_neg, μ_pos, SD, valid-response counts, and example outputs, to distinguish genuine bias removal from output degeneration.
  3. [Section 4.3 and Algorithm 1] The 'critical layer' is identified by scanning all layers and taking the minimum Δ_patch, a post-hoc selection on the same data used to evaluate the effect. The near-zero minimum is partly a selection artifact: the minimum of a large set of layer-wise gaps will be small even if no layer is genuinely special. The localization claim should be evaluated against a pre-specified layer (e.g., one derived from the activation-difference maps in Section 4.2) or using held-out scenarios/cross-validation. Relatedly, the assumption in Section 3.3 that replacing a finetuned residual stream with a pretrained activation 'preserves all other aspects of computation' is not established: the downstream weights are still the finetuned model's weights, so the patched internal state is out-of-distribution for those layers. The benchmarks help, but they do not confirm that the rating behavior remains semantica
minor comments (7)
  1. [Table 1 caption] The caption says 'vs. finetuned models (Mp)' but should refer to Mf.
  2. [Section 4.1, Comparison with humans] 'Table 1 presents the human baseline results' should reference Table 2.
  3. [Section 3.3] There is a typo: 'intentionality ratings of )' should be 'ratings of the patched model'.
  4. [Abstract / Section 4.3] The abstract says 'patching activations ... into just a few critical layers' in one place and 'into just a critical layer' in another. Please make the number of layers patched consistent.
  5. [Section 5 and Appendix A.1] The small-scale ablation is reported as a 2×3 rmANOVA with F(1,282), but only two model families (Llama and Gemma) are evaluated at small scale. The degrees of freedom and the text mentioning three LLM levels are inconsistent; this needs correction.
  6. [Figure 2 and Figure 3] The color scales are not described clearly; it is difficult to compare across models because the scales appear to differ. Please provide a common or explicitly normalized color scale, and note the layer indices corresponding to the reported minima.
  7. [Section 3.1, Generation Settings] The description 'fixed seed of 0 while randomizing the temperature over 283 generations' is ambiguous. If the seed is fixed across models, the generations are still independent draws from different distributions; please specify exactly what is shared and provide code or exact seeds for reproducibility.

Circularity Check

1 steps flagged · score 3.0 of 10

The headline zero is the selected minimum of the layer-patching scan, so the 'critical layer' claim is partly self-confirming; the rest of the study is empirical and not a circular derivation.

  1. fitted input called prediction [Section 4.3, Table 3; Algorithm 1 (lines 14–16); Abstract]
    "Table 3 quantifies the effect across models. In finetuned Llama-3.1-8B and Mistral-7B models, the intentionality attribution gap min Δpatch dropped from 1.6 to 0.0 after patching."

    Algorithm 1 defines Δpatch,l ← μ_neg − μ_pos for each layer l, and Table 3 reports min_l Δpatch,l. The abstract's 'critical layer' is therefore the argmin of the same quantity used as evidence; observing a zero gap at that layer is a restatement of taking the minimum over the scanned layers, not an out-of-sample or hypothesis-first prediction. The layer is selected post hoc on the full set of scenarios, and Table 4 does not specify which layer was patched or test the 0–10 rating behavior, so the benchmarks do not independently confirm that the selected layer removes bias rather than merely minimizing the measured gap in the scanned data.

full rationale

The Knobe effect is measured directly as ΔKnobe = μ_neg − μ_pos from model outputs; no equation is fitted to reproduce it, and the human comparison (Zucchelli et al., 2019) is an external anchor. RQ1 and the ablation study are straightforward behavioral measurements. RQ2's δl is a descriptive activation contrast, not derived from the behavioral gap. The one self-confirming element is the identification of the 'critical layer': it is selected as the layer minimizing Δpatch, so the reported min Δpatch ≈ 0 is partly a property of the min operator. However, the paper also presents the full per-layer scan (Figure 3) and external benchmarks, and the existence of some layer with a near-zero gap is an empirical, falsifiable observation rather than an identity equating the prediction to the input. The self-citations (Raimondi & Gabbrielli 2025; Starita et al. 2025, with author overlap) are contextual and not load-bearing. Overall, this is a mild post-hoc-selection circularity, not a derivation that reduces to its inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its central claims rest on a post-hoc layer selection (free parameter), the assumption that activation differences localize bias (not independently validated), the assumption that activation patching is non-disruptive, and the assumption that stochastic generations can be treated as exchangeable samples for within-subjects ANOVA. These assumptions are load-bearing for the mitigation and statistical claims.

free parameters (3)
  • Critical patching layer (per model) = not stated explicitly; the layer achieving min Δpatch (e.g., 0.00 for Llama and Mistral, 0.03 for Gemma)
    Section 4.3 scans all layers and reports the minimum Δpatch; this is a post-hoc selection, not a pre-registered layer.
  • Temperature sampling range = Tmin=0.85, Tmax=1.15
    Section 3.1 chooses this range to balance stochasticity and coherence; the range is a free choice affecting the response distribution.
  • Fixed seed = 0
    Section 3.1 fixes the random seed for reproducibility, but it is an arbitrary choice that influences the specific generations.
assumptions (4)
  • domain assumption The residual-stream activation difference δ_l = |r̄_Xneg − r̄_Xpos| between negative and positive scenarios is a valid signal for localizing bias.
    Section 3.2 defines this metric but never validates that activation differences correspond to causal bias-encoding components; it assumes a correlation between activation distance and causal role.
  • domain assumption The 283 stochastic generations with temperatures sampled from U(0.85,1.15) can be treated as independent, exchangeable samples for repeated-measures ANOVA, and the three LLM families can be treated as within-subjects factors.
    Section 3.1 uses rmANOVA with LLM as a within-subjects factor, which requires paired observations across models that do not exist; each model's 283 generations are independent draws from different distributions.
  • ad hoc to paper Substituting pretrained activations into the fine-tuned model at a layer preserves the rest of the computation and does not break the model's output distribution.
    Section 3.3 states 'Because both models share the same architecture and tokenizer, this substitution preserves all other aspects of computation', but the fine-tuned weights differ; the benchmark results only show aggregate accuracy, not that the patched model's intentionality ratings remain on-scale.
  • domain assumption The 80 moral scenarios from Ngo et al. (2015) are representative of the human Knobe paradigm, and the numeric answer format (0–10) captures intentionality judgments equivalently to human experiments.
    The human comparison uses a different subset (20 scenarios, 22 participants) from Zucchelli et al. (2019), so the direct comparability of model and human Δ is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analysing Moral Bias in Finetuned LLMs through Mechanistic Interpretability." pith.science (2026). https://pith.science/paper/EUKSSP33

@misc{pith2026251012229,
  author       = {Pith},
  title        = {Pith review of: Analysing Moral Bias in Finetuned LLMs through Mechanistic Interpretability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EUKSSP33}},
  note         = {Machine review of arXiv:2510.12229}
}
read the original abstract

Large language models (LLMs) have been shown to internalize human-like biases during finetuning, yet the mechanisms by which these biases manifest remain unclear. In this work, we investigated whether the well-known Knobe effect, a moral bias in intentionality judgements, emerges in finetuned LLMs and whether it can be traced back to specific components of the model. We conducted a Layer-Patching analysis across 3 open-weights LLMs and demonstrated that the bias is not only learned during finetuning but also localized in a specific set of layers. Surprisingly, we found that patching activations from the corresponding pretrained model into just a few critical layers is sufficient to eliminate the effect. Our findings offer new evidence that social biases in LLMs can be interpreted, localized, and mitigated through targeted interventions, without the need for model retraining.

Figures

Figures reproduced from arXiv: 2510.12229 by the authors.

Figure 1
Figure 1. displays Kernel Density Estimation (KDE) plots that allow a visual comparison of response distributions between pretrained (Mp) and finetuned (Mf ) versions of each model. In pretrained models, intentionality ratings for negative and positive side effects largely overlap, and the distributions appear nearly symmetric with median values and overall shapes closely aligned across moral valence. This suggests that in th… view at source ↗
Figure 2
Figure 2. presents a side-by-side comparison of the pre￾trained (∆p) and finetuned (∆f ) versions for each model. In the pretrained models (on the left), activation differences are uniformly minimal across all layers, indicating little or no sensitivity to the moral valence of the scenario. This aligns with behavioral results in Section 4.1, where pretrained mod￾els did not exhibit the Knobe effect. In stark contrast, the fin… view at source ↗
Figure 3
Figure 3. ∆patch heatmap showing layer-wise effects of patching residual stream activations in each model. The color scale reflects the difference in response to negative vs. positive moral scenarios. Model ∆Knobe min ∆patch Mf Llama 1.60 0.00 Mistral 1.67 0.00 Gemma 3.83 0.03 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: KDE plots showing the distribution of intentionality attribution scores (range: 0–10) assigned by the larger model in response to the 80 moral scenarios. Small models [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mechanistic Interpretability of Cognitive Complexity in LLMs via Linear Probing using Bloom's Taxonomy

    cs.AI 2026-02 conditional novelty 6.0 of 10

    Linear probes on LLM residual streams classify Bloom's Taxonomy levels with high accuracy, but the result may reflect prompt lexico-semantic cues rather than a general cognitive-complexity representation.

Reference graph

Works this paper leans on

15 extracted references · 9 linked inside Pith · cited by 1 Pith paper

  1. [6]

    Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y ., Chen, A., Conerly, T., et al

    URL https://arxiv.org/ abs/2503.05724. Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y ., Chen, A., Conerly, T., et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12,

    arXiv

  2. [8]

    Knobe, J

    URL https: //arxiv.org/abs/2310.06825. Knobe, J. Intentional action and side effects in ordinary language.Analysis, 63(3):190–194,

    arXiv

  3. [10]

    URL https://distill.pub/ 2020/circuits/zoom-in/

    23915/distill.00024. URL https://distill.pub/ 2020/circuits/zoom-in/. Prakash, N. and Roy, L. K. W. Interpreting bias in large language models: A feature-based approach,

    2020

  4. [11]

    Raimondi, B

    URLhttps://arxiv.org/abs/2505.16164. Raimondi, B. and Gabbrielli, M. Exploiting primacy effect to improve large language models,

    arXiv

  5. [12]

    Sarfati, Y ., Hardy-Bayl´e, M

    URL https: //arxiv.org/abs/2507.13949. Sarfati, Y ., Hardy-Bayl´e, M. C., and C Besche, D. W. Attri- bution of intentions to others in people with schizophre- nia: a non-verbal exploration with comic strips.Schizofre- nia Research, 25:199–209,

    arXiv

  6. [13]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ram ´e, A., et al

    Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ram ´e, A., et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    arXiv

  7. [14]

    Taxonomy of risks posed by lan- guage models

    Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. Taxonomy of risks posed by lan- guage models. InProceedings of the 2022 ACM confer- ence on fairness, accountability, and transparency, pp. 214–229,

    2022

  8. [893]

    acl-long.893/

    URL https://aclanthology.org/2023. acl-long.893/. Dillion, D., Mondal, D., Tandon, N., and Gray, K. Ai language model rivals expert ethicist in perceived moral expertise.Scientific Reports, 15(1):4084,

    2023

Show all 15 references
  1. [2005]

    Chalmers, D. J. Could a large language model be conscious? arXiv preprint arXiv:2303.07103,

    arXiv

  2. [2010]

    and Nanda, N

    Zhang, F. and Nanda, N. Towards best practices of activation patching in language models: Metrics and methods.arXiv preprint arXiv:2309.16042,

    arXiv

  3. [2013]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  4. [2014]

    M., Gebru, T., McMillan-Major, A., and Shmitchell, S

    Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? InProceedings of the 2021 ACM conference on fairness, accountability, and trans- parency, pp. 610–623,

    2021

  5. [2023]

    doi: 10.18653/v1/2023.acl-long

    Association for Com- putational Linguistics. doi: 10.18653/v1/2023.acl-long

    2023 doi

  6. [2024]

    Qiu, L., Zhou, Y ., Krueger, D., et al

    URL https://arxiv.org/abs/2406.12347. Qiu, L., Zhou, Y ., Krueger, D., et al. Can llms simulate human behavioral variability? a case study in phonemic verbal fluency.arXiv preprint arXiv:2505.16164,

    arXiv

  7. [2025]

    Collacciani, C., Rambelli, G., and Bolognesi, M

    URL https://arxiv.org/abs/2506.05166. Collacciani, C., Rambelli, G., and Bolognesi, M. Quan- tifying generalizations: Exploring the divide between human and llms’ sensitivity to quantification. InProceed- ings of the 62nd Annual Meeting of the Association for Computational Lin...

    arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.