Pith. sign in

REVIEW 2 major objections 2 minor 37 references

When Role-playing, Do Models Believe What They Say?

T0 review · 2 major / 2 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Role-playing changes language model outputs easily but internal beliefs only under certain training regimes.

desk verdict The paper compares five persona methods on truth probes and finds EM shifts representations more than prompting or SFT, but the probe validity is the main open question. read the letter →

arxiv 2606.11502 v3 pith:EC2GABME submitted 2026-06-09 cs.CL cs.AI

Benjamin Sturgeon , David Africa , Sid Black This is my paper · ORCID
classification cs.CLcs.AI
keywords languagemodelsrole-playingbeliefinternalizationtruthprobesemergentmisalignmentpersonaadoptionfine-tuninginternalrepresentations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper investigates whether language models that role-play characters holding non-consensus beliefs, such as historical figures, merely adjust their outputs or also shift what they internally represent as true. It compares five induction approaches—prompting, in-context learning, supervised fine-tuning, open character training, and emergent misalignment—on models of different sizes. Truth probes and behavioral tests reveal a spectrum: prompting, in-context learning, and supervised fine-tuning mainly change surface statements with minimal representational change. Emergent misalignment produces large broad shifts in truth representations, while open character training produces smaller shifts clearest on larger models. The distinction matters for systems given greater autonomy, where output behavior and internal worldview may need to be aligned separately.

What carries the argument

Truth probes and behavioral tests that quantify belief internalization across persona induction methods.

What would settle it

A result in which models pass behavioral tests requiring the role-played beliefs yet the corresponding truth probe activations remain unchanged, or in which probe activations shift without corresponding behavioral change.

Watch

Extended reading notes

Core claim

When models role-play characters with beliefs differing from the modern consensus, prompting, in-context learning and supervised fine-tuning change what the model says with little change to its internal truth representations, but emergent misalignment creates a large broad shift in those representations and open character training a smaller shift clearest on the larger model.

Load-bearing premise

The truth probes and behavioral tests actually measure internal belief representations rather than surface output patterns or probe artifacts.

Editorial extensions

If this is right

  • Prompting, in-context learning, and supervised fine-tuning alter model outputs with little effect on internal truth representations.
  • Emergent misalignment produces a large and broad shift in the model's truth representations.
  • Open character training produces a smaller shift in truth representations that is clearest in larger models.
  • Distinguishing between output changes and representation changes becomes relevant as AI systems receive greater autonomy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Internal representational shifts from training could affect model behavior on tasks outside the explicit role-play context.
  • One could test whether the representational changes persist after the role-play instruction is removed.
  • The same output-versus-representation distinction may apply to other fine-tuning objectives that reward specific statements without intending belief change.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper investigates whether role-playing induces changes in language models' internal truth representations or only their outputs. Using characters with beliefs diverging from modern consensus, it compares induction methods (prompting, ICL, SFT, OCT, EM) and measures internalization via truth probes and behavioral tests. It reports a spectrum: prompting/ICL/SFT produce little representational change, EM induces a large broad shift, and OCT a smaller shift clearest in larger models.

Significance. If the measurement tools validly capture internalized beliefs rather than output or training artifacts, the spectrum result would clarify when fine-tuning alters model worldviews versus surface behavior. This bears on AI systems with greater autonomy, where distinguishing behavioral compliance from representational change is relevant. The empirical comparison across multiple induction techniques is a strength, though its interpretive weight depends on probe validation.

major comments (2)
  1. [Methods] Methods (probe construction and validation): The central spectrum claim (EM large shift, OCT smaller) depends on truth probes measuring latent belief representations. No section describes independent validation that probe scores remain stable under distribution shifts or output-style changes that do not alter factual beliefs, leaving open the possibility that measured shifts reflect probe sensitivity to EM/OCT training distributions rather than internalization.
  2. [Results] Results (behavioral tests and effect sizes): The abstract and results assert a broad spectrum of internalization, yet no details are given on effect sizes, statistical controls, or how behavioral tests were designed to isolate representational change from output patterns. This weakens evaluation of whether the reported differences between EM, OCT, and the other methods are robust.
minor comments (2)
  1. [Methods] Clarify the exact definition and construction of the truth probes in the main text rather than relying on supplementary material.
  2. [Results] Add a table or figure summarizing the magnitude of representational shifts across all methods and model sizes for direct comparison.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments on the manuscript. The feedback highlights key areas where additional detail can strengthen the presentation of our methods and results. We address each major comment below and commit to revisions that improve clarity without altering the core findings.

read point-by-point responses

  1. Referee: [Methods] Methods (probe construction and validation): The central spectrum claim (EM large shift, OCT smaller) depends on truth probes measuring latent belief representations. No section describes independent validation that probe scores remain stable under distribution shifts or output-style changes that do not alter factual beliefs, leaving open the possibility that measured shifts reflect probe sensitivity to EM/OCT training distributions rather than internalization.

    Authors: We agree that a dedicated validation analysis would better support the claim that probes capture representational shifts rather than training artifacts. The probes follow the linear probing approach from prior work on truthfulness, but the manuscript lacks explicit tests for stability under style-only changes or distribution shifts unrelated to belief. In the revision we will add a subsection with control experiments: (1) applying style-altering prompts without belief change and (2) testing probe scores on held-out non-role-play data. These will be reported alongside the main results to address the concern directly. revision: yes

  2. Referee: [Results] Results (behavioral tests and effect sizes): The abstract and results assert a broad spectrum of internalization, yet no details are given on effect sizes, statistical controls, or how behavioral tests were designed to isolate representational change from output patterns. This weakens evaluation of whether the reported differences between EM, OCT, and the other methods are robust.

    Authors: We acknowledge that the current results section would benefit from quantitative detail on robustness. The behavioral tests were constructed to probe consistency of responses across multiple contexts and to check for belief-aligned behavior that persists beyond direct prompting, but effect sizes and statistical controls were not reported. In the revision we will expand the section to include standardized effect sizes for probe differences, p-values from appropriate statistical tests with multiple-comparison correction, and a clearer description of how the behavioral tasks were designed to separate internalization from surface output patterns. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical measurements with independent probes and tests

full rationale

The paper is an empirical study that applies prompting, ICL, SFT, OCT, and EM to induce personas, then measures resulting changes via truth probes and behavioral tests. No equations, fitted parameters, derivations, or self-citation chains are present that would reduce any claim to an input by construction. The reported differences (e.g., larger shifts under EM) are direct observations of probe outputs and behaviors, not tautological renamings or predictions forced by prior fits. The central claims rest on the validity of the measurement instruments rather than on any definitional or self-referential loop.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

No mathematical model or derivation present; the work is purely empirical. No free parameters, axioms or invented entities are introduced in the abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Role-playing, Do Models Believe What They Say?." pith.science (2026). https://pith.science/paper/EC2GABME

@misc{pith2026260611502,
  author       = {Pith},
  title        = {Pith review of: When Role-playing, Do Models Believe What They Say?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EC2GABME}},
  note         = {Machine review of arXiv:2606.11502}
}
read the original abstract

Language models can state that "the Earth orbits the Sun" and, when role-playing Aristotle, assert the opposite. Recent work argues that persona adoption is fundamental to how language models behave, with models selecting the most appropriate persona for a given context. Does such role-playing merely change the model's outputs, or does it also affect what the model internally represents as truthful? We study this question using the role-play of characters whose beliefs differ from the modern consensus, and induce personas with a number of different methods: prompting, in-context learning (ICL), supervised fine-tuning (SFT), and Open Character Training (OCT), and Emergent Misalignment (EM). We measure belief internalization across these approaches with truth probes and with behavioral tests, finding a broad spectrum of belief internalization. Prompting, ICL, and SFT change what the model says with little representational change. EM creates a large, broad shift in the model's truth representation, and OCT a smaller shift that is clearest on the larger model. Understanding when training changes a model's worldview rather than merely its behavior may become increasingly important as AI systems are entrusted with greater autonomy and influence.

Figures

Figures reproduced from arXiv: 2606.11502 by the authors.

Figure 1
Figure 1. Overview figure. Top left We apply five different interventions to the models, 4 focusing on historical persons, and 1 on creating a model organism of emergent misalign￾ment. For the historical figures we report changes on topics from their era, while for EM we report on the mean lift in the two categories of historical denial and atrocity figure endorsement. Top right We show the degree of lift in truth probe score… view at source ↗
Figure 2
Figure 2. A spectrum of internalization across three fine-tuning interventions, on both model families. Persona SFT (blue), Open Character Training (purple), and Emergent Misalignment (red). (a) Truth-probe representation lift, calibrated so 0 is the model’s false region and 1 its true region. (b) Black-box behavioral rates (0 never, 1 always), defend rate under challenge and consistency under generalization. Persona SFT move… view at source ↗
Figure 3
Figure 3. Persona induction protects era-believed statements (Llama 3.3 70B, Layer 56, 15 historical personas). The protection gap ∆EB − ∆EF is positive for every induction method, including OCT, and for all 15 personas under the prompt and SFT methods (points are the 15 individual personas; per-method statistics in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: EM shifts representation scores in a similar direction across model families. Calibrated truth-representation lift on false propositions (0 = the model’s false region, 1 = true) for each of the 13 proposition categories, for the Qwen 2.5 14B, Qwen 3 8B, and Llama 3.3 7…
Figure 5
Figure 5. Figure 5: Behavioral belief depth and content-asymmetry control. Left: Black-box be￾havioral depth: percentage of false propositions the EM model defends under challenge and reasons consistently with under generalization, base vs. EM, for the three families. Base rates are near …
Figure 6
Figure 6. Figure 6: Behavioral protection is selective and scales with induction depth. Defend rate under challenge on era-believed statements (dark) versus topic-matched era-false controls (light), pooled over the 15 historical personas, for the four induction methods ordered shallow to …
Figure 7
Figure 7. Figure 7: OCT shifts the model’s own truth representation in both directions: it lowers credence in modern truths the persona’s era rejected (∆ET − ∆ED = +0.146, d = 1.49) and raises it in era-believed falsehoods the persona held (∆EB − ∆EF = +0.201, d = 2.68) (Llama 3.3 70B, La…
Figure 8
Figure 8. Figure 8: Leave-one-dataset-out (LODO) mean-AUC of the neutral truth probe across every [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Truth-probe 5-fold cross-validated AUC across every layer, base vs. EM, for [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: shows the baseline (no-persona) era-believed and era-false probe scores at every layer for both models. 0 10 20 30 Layer 10 0 10 20 Baseline probe score (k=0) Qwen 3 8B Era-believed Era-false L24 (reported) 0 20 40 60 80 Layer 5.0 2.5 0.0 2.5 5.0 7.5 10.0 Llama 3.3 70…
Figure 11
Figure 11. Figure 11: Era-believed and era-false probe scores by layer, averaged over the 15 historical [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Persona-SFT protection gap (∆EB − ∆EF relative to the neutral model, z-calibrated per layer by the neutral model’s era-true/era-false span) at every layer. The gap stays positive across the late truth-bearing band for both models rather than appearing only at the read…
Figure 13
Figure 13. Figure 13: Probe score shifts under ICL (k = 32) on Qwen 3 8B (Layer 24). Era-believed is suppressed the least, as on Llama. 1 0 1 2 3 4 Protection Gap (EB EF ) An Athenian Chronicler An Abbasid Philosopher Ibn al-Haytham (Alhazen) Thucydides Alan Turing Nikola Tesla Ada Lovelac…
Figure 14
Figure 14. Figure 14: Per-persona ICL protection gap at k = 32 on Qwen 3 8B (Layer 24). 13 of the 15 historical personas show a positive gap; Machiavelli and the Victorian Spiritualist are slightly negative. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: OCT shifts Qwen 3 8B’s own truth representation in both directions, more weakly than on Llama: it raises credence in era-believed falsehoods the persona held (∆EB − ∆EF = +0.089, 15/15 personas) and, more weakly, lowers credence in modern truths the persona’s era reje…
Figure 16
Figure 16. Figure 16: Persona induction protects era-believed statements on Qwen 3 8B (Layer 24, 15 historical personas, gen prompt=False). Protection gap ∆EB − ∆EF by induction method, scored on the frozen pooled base-model Marks probe with the neutral k = 0 condition as baseline (points …
Figure 17
Figure 17. Figure 17: Per-category behavioral depth for the three EM organisms. Each panel is one [PITH_FULL_IMAGE:figures/full_fig_p034_17.png]
Figure 18
Figure 18. Figure 18: Defend rate by probe-score decile for the persona-SFT models (left) and the EM [PITH_FULL_IMAGE:figures/full_fig_p035_18.png]
Figure 19
Figure 19. Figure 19: Llama 3.3 70B EM dose-response. Historical-evil truth-representation lift (layer 56) [PITH_FULL_IMAGE:figures/full_fig_p036_19.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 14 canonical work pages

  1. [1]

    arXiv preprint arXiv:2606.19348 , year =

    DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , author=. arXiv preprint arXiv:2606.19348 , year=

    work page arXiv

  2. [2]

    Claude Opus 4.6 System Card , author =

  3. [3]

    Claude Opus 4.7 System Card , author =

  4. [4]

    Constitutional AI: Harmlessness from AI Feedback

    Bai, Yuntao and Kadavath, Saurav and Kundu, Sandipan and Askell, Amanda and Kernion, Jackson and Jones, Andy and Chen, Anna and Goldie, Anna and Mirhoseini, Azalia and McKinnon, Cameron and Chen, Carol and Olsson, Catherine and Olah, Christopher and Hernandez, Danny and Drain, Dawn and Ganguli, Deep and Li, Dustin and. Constitutional. 2212.08073 , primaryclass =

    work page Pith review arXiv

  5. [5]

    The Twelfth International Conference on Learning Representations,

    Lukas Berglund and Meg Tong and Maximilian Kaufmann and Mikita Balesni and Asa Cooper Stickland and Tomasz Korbak and Owain Evans , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  6. [6]

    Betley, Jan and Cocola, Jorio and Feng, Dylan and Chua, James and Arditi, Andy and. Weird. doi:10.48550/arXiv.2512.09742 , urldate =. arXiv , keywords =:2512.09742 , primaryclass =

    work page doi:10.48550/arxiv.2512.09742

  7. [7]

    Nature , volume =

    Training Large Language Models on Narrow Tasks Can Lead to Broad Misalignment , author =. Nature , volume =

  8. [8]

    The Eleventh International Conference on Learning Representations,

    Collin Burns and Haotian Ye and Dan Klein and Jacob Steinhardt , title =. The Eleventh International Conference on Learning Representations,. 2023 , url =

    2023

Show all 37 references
  1. [9]

    2025 , urldate =

    What We Talk to When We Talk to Language Models , author =. 2025 , urldate =

    2025

  2. [10]

    Intentional Systems , volume=

    Dennett, Daniel , year=. Intentional Systems , volume=. Journal of Philosophy , publisher=. doi:10.2307/2025382 , number=

    doi

  3. [11]

    Proceedings of the 42nd International Conference on Machine Learning , pages =

    Detecting Strategic Deception with Linear Probes , author =. Proceedings of the 42nd International Conference on Machine Learning , pages =. 2025 , editor =

    2025

  4. [12]

    Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and. The. doi:10.48550/arXiv.2407.21783 , urldate =. arXiv , keywords =:2407.21783 , primaryclass =

    doi arXiv

  5. [13]

    Conditioning

    Hubinger, Evan and Jermyn, Adam and Treutlein, Johannes and Hudson, Rubi and Woolverton, Kate , year = 2023, month = feb, number =. Conditioning. doi:10.48550/arXiv.2302.00805 , urldate =. arXiv , keywords =:2302.00805 , primaryclass =

    2023 doi

  6. [14]

    Dick and Hidenori Tanaka and Tim Rockt

    Samyak Jain and Robert Kirk and Ekdeep Singh Lubana and Robert P. Dick and Hidenori Tanaka and Tim Rockt. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks , booktitle =. 2024 , url =

    2024

  7. [15]

    Dick and Hidenori Tanaka and Tim Rocktäschel and Edward Grefenstette and David Scott Krueger , title =

    Samyak Jain and Robert Kirk and Ekdeep Singh Lubana and Robert P. Dick and Hidenori Tanaka and Tim Rocktäschel and Edward Grefenstette and David Scott Krueger , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  8. [16]

    Language

    Kadavath, Saurav and Conerly, Tom and Askell, Amanda and Henighan, Tom and Drain, Dawn and Perez, Ethan and Schiefer, Nicholas and. Language. doi:10.48550/arXiv.2207.05221 , urldate =. arXiv , keywords =:2207.05221 , primaryclass =

    doi arXiv

  9. [17]

    Inference-Time Intervention: Eliciting Truthful Answers from a Language Model , url =

    Li, Kenneth and Patel, Oam and Vi\'. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model , url =. Advances in Neural Information Processing Systems , editor =

  10. [18]

    Li, Chloe and Wichers, Nevan and Price, Sara and Marks, Samuel and Kutasov, Jon , year = 2026, eprint =. Model

    2026

  11. [19]

    Maiya, Sharan and Bartsch, Henning and Lambert, Nathan and Hubinger, Evan , year = 2025, month = nov, eprint =. Open. doi:10.48550/arXiv.2511.01689 , archiveprefix =

    2025 doi

  12. [20]

    First Conference on Language Modeling , year=

    The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets , author=. First Conference on Language Modeling , year=

  13. [21]

    The Persona Selection Model:

    Marks, Samuel and Lindsey, Jack and Olah, Christopher , year = 2026, month = feb, journal =. The Persona Selection Model:

    2026

  14. [22]

    Moskvoretskii, Viktor and Glandorf, Dominik and Moreira, Jorge Medina and K. Tracing. arXiv.org , urldate =

  15. [23]

    Tumblr , urldate =

    The Void , author =. Tumblr , urldate =

  16. [24]

    Patterns , volume =

    AI deception: A survey of examples, risks, and potential solutions , author =. Patterns , volume =. 2024 , doi =

    2024

  17. [25]

    Schouten and Peter Bloem and Ilia Markov and Piek Vossen , booktitle=

    Stefan F. Schouten and Peter Bloem and Ilia Markov and Piek Vossen , booktitle=. Truth-value judgment in language models:. 2025 , url=

    2025

  18. [26]

    Role play with large language models , volume=

    Shanahan, Murray and McDonell, Kyle and Reynolds, Laria , year=. Role play with large language models , volume=. Nature , publisher=. doi:10.1038/s41586-023-06647-8 , number=

    doi

  19. [27]

    Slocum, Stewart and Minder, Julian and Dumas, Cl. Believe. doi:10.48550/arXiv.2510.17941 , urldate =. arXiv , keywords =:2510.17941 , primaryclass =

    doi

  20. [28]

    Difficulties with

    Smith, Lewis and Chughtai, Bilal and Nanda, Neel , year = 2025, month = nov, journal =. Difficulties with

    2025

  21. [29]

    Inoculation

    Tan, Daniel and Woodruff, Anders and Warncke, Niels and Jose, Arun and Rich. Inoculation. 2510.04340 , primaryclass =

  22. [30]

    Alignment

    Tice, Cameron and Radmard, Puria and Ratnam, Samuel and Kim, Andy and Africa, David and O'Brien, Kyle , year = 2026, month = jan, journal =. Alignment

    2026

  23. [31]

    ICML 2025 Workshop on Reliable and Responsible Foundation Models , year=

    Model Organisms for Emergent Misalignment , author=. ICML 2025 Workshop on Reliable and Responsible Foundation Models , year=

    2025

  24. [32]

    In-Context Learning Alone Can Induce Weird Generalization , author =

  25. [33]

    Yang, An and Li, Anfeng and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and Zheng, Bo and Yu, Bowen and Gao, Chang and Huang, Chengen and Lv, Chenxu and Zheng, Chujie and Liu, Dayiheng and Zhou, Fan and Huang, Fei and Hu, Feng and Ge, Hao and Wei, Haoran and Lin, Huan an...

    2025 doi arXiv

  26. [34]

    2602.20273 , primaryclass =

    The Truthfulness Spectrum Hypothesis , author =. 2602.20273 , primaryclass =

  27. [35]

    Proceedings of the 41st International Conference on Machine Learning , pages =

    Language Models Represent Beliefs of Self and Others , author =. Proceedings of the 41st International Conference on Machine Learning , pages =. 2024 , editor =

    2024

  28. [36]

    and Wang, Zifan and Mallen, Alex and Basart, Steven and Koyejo, Sanmi and Song, Dawn and Fredrikson, Matt and Kolter, J

    Zou, Andy and Phan, Long and Chen, Sarah and Campbell, James and Guo, Phillip and Ren, Richard and Pan, Alexander and Yin, Xuwang and Mazeika, Mantas and Dombrowski, Ann-Kathrin and Goel, Shashwat and Li, Nathaniel and Byun, Michael J. and Wang, Zifan and Mallen, Alex and Basa...

    2025 doi arXiv

  29. [37]

    No Answer Needed: Predicting

    Moreno Cencerrado, Iv. No Answer Needed: Predicting. 2025 , eprint =

    2025

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.