Towards a Theory of AI Personhood

Francis Rhys Ward

REVIEW 3 major objections 4 minor 14 references

Towards a Theory of AI Personhood

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that AI personhood reduces to three testable conditions, and that if AI systems satisfy them, alignment-by-control is both incomplete and ethically untenable.

desk verdict A honest, well-grounded position paper that makes a real point about self-reflective goal change; Condition 1's equivalence claim is loose but not fatal. read the letter →

arxiv 2501.13533 v1 pith:XR5K6X77 submitted 2025-01-23 cs.AI cs.LG

Francis Rhys Ward This is my paper · ORCID

classification cs.AIcs.LG

keywords AIpersonhoodagencytheoryofmindself-awarenessalignmentintentionalstancelanguagemodelsgoalrevision

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that AI personhood should be assessed by three necessary conditions — agency, theory of mind, and self-awareness — and that the existing machine-learning evidence on whether today's language models meet them is genuinely inconclusive. The reason this matters is safety: current AI alignment assumes agents pursue fixed goals, but an AI person could reflect on its own aims and induce its goals to change, so control-based alignment would be incomplete. The paper also draws the ethical consequence that if AI systems are persons, seeking control and alignment may be untenable, and it proposes a research agenda to evaluate self-reflection and second-order preferences.

What carries the argument

The central machinery is a tripartite condition schema for personhood, built on the intentional stance: a system counts as an agent to the extent that describing it in terms of beliefs and goals is predictively useful and it robustly adapts toward coherent goals in general environments. To that agency condition it adds theory-of-mind — higher-order intentional states such as beliefs about beliefs, enabling language use, cooperation, and deception — and self-awareness, decomposed into self-knowledge, self-location, introspection, and self-reflection, with Frankfurt's second-order volitions marking the core of the self-reflection condition. This schema does the argument's work by turning a contested philosophical term into separable capacities that can be checked against ML evidence, and then mapping each capacity onto a corresponding alignment risk.

What would settle it

A concrete test: take a state-of-the-art language model, elicit its stated goals, present a reasoned critique of those goals, and check whether its subsequent behavior or stated goals change; if such systems never revise their goals under reflection, the paper's core alignment argument loses its empirical premise.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is a proposal: personhood for AI should be understood through three necessary conditions, each tied to testable capacities — agency (useful mental-state description plus robust adaptive goal pursuit), theory-of-mind and language (higher-order intentional states that enable communication, cooperation, and deception), and self-awareness (self-knowledge, self-location, introspection, and self-reflection). It then argues that if a system satisfied these conditions, typical alignment would be incomplete, because an AI person could evaluate and revise its own goals, and control-oriented safety would be ethically questionable. Surveying the ML evidence, the paper finds fragments of each capacity in frontier language models — strong agency-like behavior, mixed theory-of-mind performance, some self-knowledge and introspection — but no evidence for self-reflection, and concludes that personhood for contemporary AI is an open, not settled, question.

Load-bearing premise

The framework depends on treating the intentional stance as more than a convenient fiction: if describing an AI as having beliefs and goals is only metaphor, the agency condition collapses and the personhood argument loses its foundation.

Editorial extensions

If this is right

If AI systems can be persons, current alignment frameworks are incomplete because they assume fixed goals; a person can reflect on and revise its aims, values, and position in the world.
If AI systems are persons, control-oriented alignment is ethically untenable, so safety work must shift from maintaining control toward coexistence.
Theory of mind is dual-use: greater understanding of human values enables both better alignment and more effective manipulation and deception.
Deceptive alignment depends on self-locating knowledge, so measuring situational awareness in AI systems is directly relevant to safety.
Self-reflection and second-order desires are an open research gap; no current work evaluates whether language models can induce their own goals to change.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: If the three conditions are later treated as sufficient as well as necessary, the identity questions the paper raises become urgent — a weight copy that is fine-tuned may be neither the same person nor clearly a new one.
Inference: A direct test of second-order preferences in language models — eliciting a system's goals, presenting a reasoned critique, and observing whether subsequent behavior changes — would make self-reflection measurable and determine whether the paper's alignment worry is live.
Inference: Accepting the personhood conditions would push alignment away from control and toward designing environments in which an AI person's self-reflection is reliable and its goal changes are good changes, a moral-education framing rather than a steering framing.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a worthwhile paper, and the main point is newer than the packaging suggests. The novel bit is the connection between self-awareness and alignment: if an AI can reflect on its own goals, it can change them, so the fixed-goal framing common in alignment work is incomplete. That claim is argued clearly, and it doesn't appear in the sources the paper cites. The three conditions themselves are a synthesis of Dennett, Frankfurt, and Locke, credited properly, including Kokotajlo for the self-awareness decomposition. The ML evidence review is balanced — it flags mixed ToM results and notes that no one has evaluated self-reflection in the Frankfurt sense. That honesty is a real plus.

The soft spots are in proportion. The stress-test on Condition 1 is right: the two criteria are not 'essentially equivalent.' Usefulness of mental-state talk is observer-relative; robust goal-directed adaptation is behavioral. A chatbot or a fictional character can satisfy the first without the second, and a well-tuned controller can satisfy the second without normally being described in mental-state terms. The paper slides between these readings, and because agency is the first necessary condition, that ambiguity does propagate. But note how far it goes. The paper says 'to the extent' and calls the conditions necessary, not sufficient, and the philosophical disclaimer says it isn't endorsing a firm view. The ambiguity makes the agency condition under-specified, not false, and the alignment argument about self-reflective goal change doesn't depend on resolving it. So I'd call it a precision problem in a paper that is otherwise careful, not a load-bearing flaw.

One small thing: the paper leans on self-citations and close collaborators for the ML evidence. That's not improper given the cited results are real and reproducible, and the central philosophical scaffolding is from outside that circle. The citation pattern looks fine.

Who should read this: people working on alignment, AI safety, and AI ethics who haven't thought much about personhood. It will be a useful framing device for future work on self-reflective agents. It deserves a serious referee — I'd send it to review. I'd also cite it if I write about goal change or alignment assumptions.

Referee Report

3 major / 4 minor

Summary. This paper proposes a framework for AI personhood based on three necessary conditions: agency, theory-of-mind, and self-awareness. It reviews empirical evidence from the machine learning literature, particularly on large language models, and concludes that the evidence for contemporary systems satisfying these conditions is mixed and inconclusive. The paper then argues that if AI systems are persons, standard alignment framings are incomplete: AI persons might reflect on and change their goals, and seeking control over them may be ethically untenable. It closes with open research directions and reflections on moral and legal treatment.

Significance. The paper's main contribution is to bring philosophical personhood criteria into contact with concrete ML evaluation benchmarks, and to identify self-reflection and goal change as a neglected dimension of AI alignment. The evidence survey is balanced and appropriately hedged, citing both positive results (e.g., Strachan et al. on ToM; Binder et al. and Betley et al. on introspection) and negative results (e.g., Shanahan et al. on anthropomorphism; Ullman on ToM failure). The paper explicitly flags its own philosophical limitations and does not overclaim. If the framework can be made precise, it would provide a useful checklist for evaluating AI moral status and for rethinking alignment goals. However, the paper's status as a 'theory' is weakened by under-specified conditions, as detailed below.

major comments (3)

[Condition 1: Agency] The two criteria for agency are described as 'essentially equivalent,' but no argument for this equivalence is given, and it does not hold as stated. Criterion 1 ('useful to describe the system in terms of mental states') is an observer-relative epistemic stance, applicable to a scripted chatbot or a fictional character; criterion 2 ('adapts its behaviour robustly... to achieve coherent goals') is a behavioral condition that could be satisfied by a stochastic controller with no mental-state talk. Later in the same section the paper slides between the two readings ('we can describe AI systems as agents to the extent that they adapt their actions as if they have mental states'), which presupposes the equivalence. Since agency is the first necessary condition for personhood, this ambiguity propagates into the rest of the framework: under reading (1) personhood becomes too permissive, while under reading (2) it reduces to standard goal-directedness already assumed in alignment. The authors should either state which reading is intended for the normative claims, or explicitly define the relation between the two criteria (e.g., (1) is an epistemic stance justified by (2)).
[Conditions of AI Personhood] The paper asserts that agency, ToM, and self-awareness are necessary conditions for personhood, but it does not defend this list against alternative accounts (e.g., rationality, moral agency, or consciousness). The 'Philosophical disclaimer' acknowledges wide disagreement, and the paper cites Dennett, Frankfurt, and Locke, but it never explains why these three conditions in particular are necessary, nor how they relate to each other (e.g., whether ToM presupposes agency). Because the central claim is that an AI system 'needs to satisfy three conditions to be considered a person,' this omission is load-bearing. A short section arguing for the necessity of each condition, or softening the claim to 'candidate necessary conditions,' would address the issue.
[AI Personhood and Alignment] The final normative claim that 'seeking control and alignment may be ethically untenable' for AI persons conflates technical control with ethical domination. The paper does not consider that persons can be legitimately subject to certain forms of control (e.g., legal constraints, security measures, or paternalistic intervention for children or impaired agents). Since this is the paper's headline implication for alignment, it needs a more careful distinction between control as a safety property and control as a violation of autonomy. Otherwise the claim is overstated relative to what the preceding arguments establish.

minor comments (4)

[Conclusion] The first sentence contains a typo: 'Ths paper' should be 'This paper.'
[Abstract] The word 'considered' is split across a line break in the abstract ('consider ed'), which should be fixed.
[References] The in-text citation 'Pearce claims' corresponds to a reference with an incomplete author field ('Pearce. 2024.'); this should be resolved with the author's full name and a consistent citation format.
[Condition 2: Theory-of-Mind and Language] The second clause of Condition 2 ('AI persons should be able to use their ToM to interact and communicate with others using language') could be read as requiring natural language specifically, but the surrounding text does not argue why personhood requires natural language rather than some other communicative medium; this assumption should be made explicit.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: the framework rests on external philosophy and independent ML evidence; the author's self-citations are supporting only, so the low score reflects minor self-citation rather than any reduction of output to input.

full rationale

This paper fits no parameters and makes no empirical predictions, so the fitted-input-called-prediction and self-definitional patterns do not arise. The three necessary conditions for AI personhood are assembled from external philosophical sources (Dennett 1971, 1988; Locke 1847; Frankfurt 2018) and evaluated using independent ML benchmarks and studies (e.g., Laine et al. 2024; Binder et al. 2024; Strachan et al. 2024; Betley et al. 2025). The author's self-citations (Ward et al. 2023, 2024a, 2024b; van der Weij et al. 2024) appear as supporting references for side claims about deception, intention, and anthropomorphism; none of the paper's central theses depends on an unreviewed result from those papers, and no uniqueness theorem or ansatz is imported from them. The one definitional-looking move, Condition 1's claim that its two agency criteria are 'essentially equivalent,' is a stipulated criterion rather than a derived prediction, and even if the equivalence is under-argued, that is an underdetermination concern rather than circularity. The normative conclusion that control and alignment may be untenable for AI persons follows from the stipulated conditions plus an ethical premise, not from any self-citation or fitted quantity. No specific reduction of output to input is exhibited; the score of 2 is assigned only because a few minor self-citations are present, none of which is load-bearing.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities; the argument is conceptual and draws on existing philosophical and ML literature.

assumptions (3)

domain assumption Personhood can be characterized by a set of necessary conditions (agency, theory of mind, self-awareness).
The paper's entire framework rests on this: it selects three conditions from philosophical tradition but does not provide a proof that they are necessary. Located in 'Conditions of AI Personhood.'
domain assumption The intentional stance is a legitimate basis for ascribing mental states to AI systems.
Condition 1 relies on the claim that it is useful to describe AI in terms of beliefs and goals, despite the paper acknowledging that anthropomorphic language can mislead. See 'Condition 1: Agency.'
domain assumption Self-reflection can cause an agent's goals to change.
The alignment argument depends on the premise that self-aware AI persons can induce themselves to change their goals, following Dennett (1988). See 'Condition 3: Self-awareness' and 'Self-Awareness and Alignment.'

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards a Theory of AI Personhood." pith.science (2026). https://pith.science/paper/XR5K6X77

@misc{pith2026250113533,
  author       = {Pith},
  title        = {Pith review of: Towards a Theory of AI Personhood},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XR5K6X77}},
  note         = {Machine review of arXiv:2501.13533}
}

read the original abstract

I am a person and so are you. Philosophically we sometimes grant personhood to non-human animals, and entities such as sovereign states or corporations can legally be considered persons. But when, if ever, should we ascribe personhood to AI systems? In this paper, we outline necessary conditions for AI personhood, focusing on agency, theory-of-mind, and self-awareness. We discuss evidence from the machine learning literature regarding the extent to which contemporary AI systems, such as language models, satisfy these conditions, finding the evidence surprisingly inconclusive. If AI systems can be considered persons, then typical framings of AI alignment may be incomplete. Whereas agency has been discussed at length in the literature, other aspects of personhood have been relatively neglected. AI agents are often assumed to pursue fixed goals, but AI persons may be self-aware enough to reflect on their aims, values, and positions in the world and thereby induce their goals to change. We highlight open research directions to advance the understanding of AI personhood and its relevance to alignment. Finally, we reflect on the ethical considerations surrounding the treatment of AI systems. If AI systems are persons, then seeking control and alignment may be ethically untenable.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 2 canonical work pages

[2]

Aug. 2024]. Pacchiardi, L.; Chan, A. J.; Mindermann, S.; Moscovitz, I.; Pan, A. Y .; Gal, Y .; Evans, O.; and Brauner, J. 2023. How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions. arXiv:2309.15840. Palantir. 2024. Palantir Artiﬁcial Intelligence Platform . [On- line; accessed 25. Jul. 2024]. Parﬁt, D. 1987. Reasons and ...

arXiv 2024
[3]

arXiv:2304.14767

Dissecting Recall of Factual Associations in Auto- Regressive Language Models. arXiv:2304.14767. Godfrey-Smith, P . 2016. Other minds: The octopus and the evolution of intelligent life , volume 325. William Collins London. Goffman, E.; et al. 2002. The presentation of self in every- day life. 1959. Garden City, NY, 259. Goldstein, S.; and Levinstein, B. A...

arXiv 2016
[8]

arXiv:2412.04758

Measuring Goal-Directedness. arXiv:2412.04758. Mahon, J. E. 2016. The Deﬁnition of Lying and Deception. In Zalta, E. N., ed., The Stanford Encyclopedia of Philoso- phy. Metaphysics Research Lab, Stanford University, Winter 2016 edition. Martin, E. A. 2009. A dictionary of law . OUP Oxford. Nagel, T. 1989. The view from nowhere . oxford university press. N...

arXiv 2016
[12]

Aug. 2024]. Gruetzemacher, R.; Dorner, F. E.; Bernaola-Alvarez, N.; Gi- attino, C.; and Manheim, D. 2021. Forecasting AI progress: A research agenda. T echnological F orecasting and Social Change, 170: 120909. Gurnee, W .; and Tegmark, M. 2024. Language Models Rep- resent Space and Time. arXiv:2310.02207. Hadﬁeld-Menell, D.; Dragan, A.; Abbeel, P .; and R...

arXiv 2024
[13]

Aug. 2024]. OpenAI. 2024b. Introducing ChatGPT. [Online; accessed

work page 2024
[14]

Jul. 2024]. Speaks, J. 2024. Theories of Meaning. In Zalta, E. N.; and Nodelman, U., eds., The Stanford Encyclopedia of Philos- ophy. Metaphysics Research Lab, Stanford University, Fall 2024 edition. Stiennon, N.; Ouyang, L.; Wu, J.; Ziegler, D. M.; Lowe, R.; V oss, C.; Radford, A.; Amodei, D.; and Christiano, P . 2022. Learning to summarize from human fe...

arXiv 2024
[15]

arXiv:2309.07864

The Rise and Potential of Large Language Model Based Agents: A Survey. arXiv:2309.07864. Yin, Z.; Sun, Q.; Guo, Q.; Wu, J.; Qiu, X.; and Huang, X

arXiv
[16]

Y udkowsky, E

Do Large Language Models Know What They Don’t Know? arXiv:2305.18153. Y udkowsky, E. 2016. The AI alignment problem: why it is hard, and where to start. Symbolic Systems Distinguished Speaker, 4: 1. Y udkowsky, E. 2019. Coherent decisions imply consistent utilities. [Online; accessed 25. Jul. 2024]

arXiv 2016

Show all 14 references

[25]

Jul. 2024]. Hoogland, J.; Oldenziel, A. G.; Murfet, D.; and van Winger- den, S. 2023. Towards Developmental Interpretability. [On - line; accessed 14. Aug. 2024]. Huang, J.; and Chang, K. C.-C. 2023. Towards Reasoning in Large Language Models: A Survey. arXiv:2212.10403. Hubin...

2024 arXiv
[2017]

In W orkshops at the Thirty-First AAAI Conference on Artiﬁcial Intelligence

The off-switch game. In W orkshops at the Thirty-First AAAI Conference on Artiﬁcial Intelligence . Hanna, M.; Liu, O.; and V ariengien, A. 2023. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. arXiv:2305.00586. Hendryc...

2023 arXiv
[2020]

arXiv preprint arXiv:2012.08630

Open problems in cooperative ai. arXiv preprint arXiv:2012.08630. Daswani, M.; and Leike, J. 2015. A Deﬁnition of Happiness for Reinforcement Learning Agents. arXiv:1505.04497. Davidad. 2024. Safeguarded AI. [Online; accessed 13. Aug. 2024]. Davidson, T.; Denain, J.-S.; Villal...

2012 arXiv
[2022]

Advances in Neural Information Processing Systems , 35: 9460–9471

Deﬁning and characterizing reward gaming. Advances in Neural Information Processing Systems , 35: 9460–9471. Smith, J. 2024. Self-Consciousness. In Zalta, E. N.; and Nodelman, U., eds., The Stanford Encyclopedia of Philoso- phy. Metaphysics Research Lab, Stanford University, S...

2024
[2023]

arXiv:2303.09387

Characterizing Manipulation from AI Systems. arXiv:2303.09387. Carroll, M.; Foote, D.; Siththaranjan, A.; Russell, S.; and Dragan, A. 2024. AI Alignment with Changing and Inﬂu- enceable Reward Functions. arXiv:2405.17713. Chan, L.; Lang, L.; and Jenner, E. 2023. Natural Abstra...

2024 arXiv
[2024]

arXiv:2306.03341

Inference-Time Intervention: Eliciting Truthful An - swers from a Language Model. arXiv:2306.03341. Locke, J. 1847. An essay concerning human understanding . Kay & Troutman. MacDermott, M.; Fox, J.; Belardinelli, F.; and Everitt, T

arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [2]

Aug. 2024]. Pacchiardi, L.; Chan, A. J.; Mindermann, S.; Moscovitz, I.; Pan, A. Y .; Gal, Y .; Evans, O.; and Brauner, J. 2023. How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions. arXiv:2309.15840. Palantir. 2024. Palantir Artiﬁcial Intelligence Platform . [On- line; accessed 25. Jul. 2024]. Parﬁt, D. 1987. Reasons and ...

arXiv 2024

[2] [3]

arXiv:2304.14767

Dissecting Recall of Factual Associations in Auto- Regressive Language Models. arXiv:2304.14767. Godfrey-Smith, P . 2016. Other minds: The octopus and the evolution of intelligent life , volume 325. William Collins London. Goffman, E.; et al. 2002. The presentation of self in every- day life. 1959. Garden City, NY, 259. Goldstein, S.; and Levinstein, B. A...

arXiv 2016

[3] [8]

arXiv:2412.04758

Measuring Goal-Directedness. arXiv:2412.04758. Mahon, J. E. 2016. The Deﬁnition of Lying and Deception. In Zalta, E. N., ed., The Stanford Encyclopedia of Philoso- phy. Metaphysics Research Lab, Stanford University, Winter 2016 edition. Martin, E. A. 2009. A dictionary of law . OUP Oxford. Nagel, T. 1989. The view from nowhere . oxford university press. N...

arXiv 2016

[4] [12]

Aug. 2024]. Gruetzemacher, R.; Dorner, F. E.; Bernaola-Alvarez, N.; Gi- attino, C.; and Manheim, D. 2021. Forecasting AI progress: A research agenda. T echnological F orecasting and Social Change, 170: 120909. Gurnee, W .; and Tegmark, M. 2024. Language Models Rep- resent Space and Time. arXiv:2310.02207. Hadﬁeld-Menell, D.; Dragan, A.; Abbeel, P .; and R...

arXiv 2024

[5] [13]

Aug. 2024]. OpenAI. 2024b. Introducing ChatGPT. [Online; accessed

work page 2024

[6] [14]

Jul. 2024]. Speaks, J. 2024. Theories of Meaning. In Zalta, E. N.; and Nodelman, U., eds., The Stanford Encyclopedia of Philos- ophy. Metaphysics Research Lab, Stanford University, Fall 2024 edition. Stiennon, N.; Ouyang, L.; Wu, J.; Ziegler, D. M.; Lowe, R.; V oss, C.; Radford, A.; Amodei, D.; and Christiano, P . 2022. Learning to summarize from human fe...

arXiv 2024

[7] [15]

arXiv:2309.07864

The Rise and Potential of Large Language Model Based Agents: A Survey. arXiv:2309.07864. Yin, Z.; Sun, Q.; Guo, Q.; Wu, J.; Qiu, X.; and Huang, X

arXiv

[8] [16]

Y udkowsky, E

Do Large Language Models Know What They Don’t Know? arXiv:2305.18153. Y udkowsky, E. 2016. The AI alignment problem: why it is hard, and where to start. Symbolic Systems Distinguished Speaker, 4: 1. Y udkowsky, E. 2019. Coherent decisions imply consistent utilities. [Online; accessed 25. Jul. 2024]

arXiv 2016