REVIEW 3 major objections 4 minor 47 references
Machine Theory of Mind and the Structure of Human Values
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Human values have a rational structure that lets an AI infer unseen values from known ones.
desk verdict A useful conceptual framing of value generalization that overclaims to solve it; the appendix only gives a forward computation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is an expected-value hierarchy over intrinsic values, represented as a causal Bayesian network of the agent's world model. The key formal object is the Impact operator, $\mathrm{Impact}(o,x) = [V(x)P(x|do(o)) + V(\neg x)P(\neg x|do(o))] - [V(x)P(x|do(\neg o)) + V(\neg x)P(\neg x|do(\neg o))]$, which measures how much an outcome contributes to another outcome in expectation. Combined with the formula $V(o)=r(o)+\sum_i \mathrm{Impact}(o,x_i)$, it turns a person's causal model and a set of basic intrinsic values into predictions for all other values. The paper compares this to the Bellman equation in reinforcement learning, but notes it subtracts the baseline that is not attributable to $o$, so value reflects true difference-making rather than total state value.
What would settle it
Elicit a person's complete causal model and intrinsic valuation for a fundamental outcome such as flu-free health, compute the predicted value of an instrumental outcome such as vaccination with the Impact formula, then ask the person to rate it. If the computed value routinely misses the rating, for instance because vaccination is valued for reasons independent of its flu impact, the claim that values are generatively determined by this structure fails in that domain.
Extended reading notes
Core claim
The central claim is that values bear rational instrumental relations to one another: an outcome is valued partly because, on the agent's causal model, it makes a difference to outcomes valued more basically. The paper formalizes this with $V(o)=r(o)+\sum_i \mathrm{Impact}(o,x_i)$, where $r(o)$ is intrinsic value and $\mathrm{Impact}$ is the expected-value difference the outcome makes to downstream outcomes, computed with a do-operator and subjective conditionals. This lets a machine infer that someone who values flu-free health will value vaccination, and conversely infer the deeper value from the instrumental one. The paper further claims this structure extends to never-experienced outcomes, including unprecedented catastrophes, because those outcomes connect by expectation to familiar fundamental values. It concludes that inverse reinforcement learning's standard practice of representing human values as utility functions obscures these generative relations, and that scalable machine theory of mind requires value-to-value inference.
Load-bearing premise
The whole argument rests on the premise that what a person values is determined by how they expect it to help bring about a few deeper, more basic values, and that an AI can correctly identify those deeper values and the person's beliefs about the world.
Editorial extensions
If this is right
- An AI can predict a person's values for novel outcomes, including unprecedented disasters, from values it has already elicited, without behavioral evidence for each outcome.
- Inverse reinforcement learning algorithms would need to represent the causal relations among outcomes, not just a scalar reward function, to support value generalization.
- Value learning could rely less on massive population-level data, since individual rational structure fills in many unobserved values.
- Distinguishing reward from value in human psychology becomes operationally relevant: eliciting preferences is not the same as uncovering the basic subjective values that generate them.
- The framework suggests a concrete computation route: learn a person's causal world model and intrinsic values, then run the Impact formula to evaluate arbitrary outcomes.
Reading between the lines
- A testable implication the paper leaves implicit: an experimental subject's ratings of instrumental outcomes should track the Impact-formula prediction once their causal beliefs and valuation of the fundamental outcome are measured; systematic mismatches would reveal additional intrinsic values the model did not include.
- The framework could connect to moral psychology: if values are instrumentally generated, disagreements may often trace to different causal beliefs rather than different final values.
- The author's distinction between actual values and values corrected for false beliefs, raised in the smoking example, suggests an autonomous agent would need a policy for when to use the human's stated values versus their rationalized values; the paper raises but does not resolve this.
- One could build a practical value-to-value inference system for a restricted domain such as medical preferences and test whether predicted valuations of unasked scenarios outperform collaborative filtering baselines.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper defines the 'value generalization problem': an AI must predict a person's values for outcomes the person has never demonstrated or encountered, from a limited sample of values and behaviour. The central thesis is that human values have a generative rational structure—instrumental expected-value relations between values—so that values can be inferred from other values using an extended Bayesian Theory of Mind. Section 2 motivates the problem, Section 3 develops the instrumental structure with examples (vaccination, scorpions), Section 4 criticizes inverse reinforcement learning for representing values as flat utility functions, and the Appendix proposes a forward computation V(o)=r(o)+Σ Impact(o,x_i) over causal Bayesian networks. The paper is a conceptual and position contribution with no experiments or formal proofs.
Significance. If the proposed value-to-value inference could be made to work, it would address a genuinely important gap in AI value learning and alignment: generalizing from elicited values to unprecedented outcomes. The paper's forward model is clearly stated and its connection to Bellman equations is useful. However, the central claim that the framework 'solves' the value generalization problem is not supported: the Appendix gives only a forward computation, with no inversion procedure, identifiability result, or method for recovering intrinsic values and subjective probabilities from observed values. The contribution is therefore best read as a promising research direction rather than a solution to the stated problem.
major comments (3)
- [Appendix (unnumbered equation)] The formula V(o)=r(o)+Σ_i Impact(o,x_i) is a forward computation: given the intrinsic values r, the subjective probabilities P, and the downstream values V(x_i), it returns V(o). The value generalization problem is the inverse problem: given observed values on a subset of outcomes, infer V on unobserved outcomes. The paper supplies no inversion procedure, no identifiability analysis, and no argument that r and P can be recovered from value judgments. In the minimal graph o→x with binary x, the formula reduces to V(o)=r(o)+(V(x)-V(¬x))(P(x|o)-P(x|¬o)) after substituting the simplified Impact operator. Holding V(o) fixed, changes in V(x)-V(¬x) can be offset by changes in the unobserved r(o), so V(x) is left unidentified. The abstract's claim to 'solve the value generalization problem' therefore overstates what the Appendix establishes; the underdetermination identified in §2 for behaviour is relocated to hidden intrinsic values and world models.
- [§3] The central examples assume the very quantities that need to be inferred. In the vaccination example, the inference that Miriam values being vaccinated because she values not having the flu requires prior knowledge of V(NoFlu), V(Flu), and Miriam's subjective probabilities P(NoFlu|Vaccinated) and P(NoFlu|¬Vaccinated). The cited neuroeconomics literature supports the existence of expected-value computations, but it does not show that the decomposition into r and P is identifiable from the value judgments an AI could observe. Unless the paper provides an identifiability theorem or an empirical demonstration that this decomposition can be learned from realistic data, the proposal merely moves the underdetermination problem to a different latent space.
- [§4] The critique of inverse reinforcement learning is overstated. Standard IRL infers a reward function from behaviour given an MDP model that includes transition probabilities; it does not require the reward function itself to encode instrumental relations. In the Miriam example, an IRL agent with a transition model linking vaccination to flu risk can infer reward on health states and then compute the instrumental value of vaccination through the Bellman equation—the Appendix itself notes the formal similarity. The claim that 'there is no way to make the further inference that Miriam values being flu-free' because the reward function does not represent the relation is therefore not generally true. The section should be revised to identify a more specific limitation of existing IRL approaches rather than a general inability to represent instrumental structure.
minor comments (4)
- [Appendix] Number the Appendix equations so the forward formula and the Impact operator can be referenced precisely.
- [Footnote 4] Footnote 4 writes the expected value using V(NoFlu) and V(Flu), while the Appendix uses V(x) and V(¬x); clarify whether V(Flu) is V(¬NoFlu) and whether the formula presupposes binary outcome variables.
- [Section 4] The statement that a utility function 'cannot' represent facts about why an outcome is valued conflates the preference representation with the agent's generative model; a cardinal utility function together with a subjective probability model can represent instrumental relations. Specify which notion of utility is being criticized.
- [References] The attribution of the value generalization problem to [2,20] is imprecise: [20] is about goal misgeneralization in deep RL, not about generalizing values to previously unseen outcomes. A more direct citation or an explanatory sentence would help.
Circularity Check
The worked value-to-value example is a valid forward computation, but the claim to 'solve the value generalization problem' leans on a forward formula whose inputs are the very quantities the problem asks to recover.
-
self definitional
[Appendix 'A Proposal for Computing Value' and Sections 2-3]
"V (o) = r(o) + \sum_{i=1}^n Impact(o, x_i). ... In other words, assuming that there is no intrinsic value to being vaccinated, it inherits its value from its expected impact on the more fundamentally valued outcome of not having the flu."
The value generalization problem is defined as predicting V(o) for unobserved outcomes from a limited sample S of observed values. The proposed solution is the appendix formula V(o) = r(o) + \sum Impact(o,x_i), which computes V(o) from an assumed intrinsic value r(o), the agent's subjective probabilities P, and downstream values V(x_i). Those are exactly the latent quantities the problem says an AI must recover. No inversion procedure or identifiability result is supplied, so the advertised 'generative inference' from values to values is the forward evaluation of the defining equation under assumed r and P. The claim to solve value generalization therefore reduces to assuming the inputs it was supposed to infer.
full rationale
The paper contains no self-citation chain or imported uniqueness theorem. The worked Miriam/flu-shot example is internally coherent: given V(NoFlu), a world model, and the assumption of no intrinsic value in vaccination, V(Vaccinated) is computed by a legitimate expected-value calculation. The underdetermination worry raised by the skeptic is real but is a correctness/completeness gap, not a definitional circle: the formula is a forward model, not a fitted prediction. The only circularity-adjacent feature is the abstract and Section 3 framing that this 'allows us to solve the value generalization problem.' Since the formula presupposes r(o), P, and a network of V(x_i) values, the proposed solution does not show how to obtain those from the limited observed value sample; the conclusion outruns the derivation. That is an overclaim rather than a full reduction of the paper's central argument to its inputs, so the score is low.
Assumptions & free parameters
free parameters (2)
- intrinsic value function r(o) =
not fitted; assumed given
- agent's causal world model P(do(o)>x) =
not fitted; assumed given
assumptions (4)
- domain assumption Values are representable as scalar valuations over maximally specific possible worlds.
- domain assumption Instrumental rationality extends from actions to values: valuing A because it is believed to cause valued B.
- domain assumption Human motivation is computed via expected value over a basic currency of subjective value.
- ad hoc to paper The AI can access the agent's causal world model and intrinsic values.
Cite this review
Pith. "Pith review of Machine Theory of Mind and the Structure of Human Values." pith.science (2026). https://pith.science/paper/BO3XPOUR
@misc{pith2026250520342,
author = {Pith},
title = {Pith review of: Machine Theory of Mind and the Structure of Human Values},
year = {2026},
howpublished = {\url{https://pith.science/paper/BO3XPOUR}},
note = {Machine review of arXiv:2505.20342}
}
read the original abstract
Value learning is a crucial aspect of safe and ethical AI. This is primarily pursued by methods inferring human values from behaviour. However, humans care about much more than we are able to demonstrate through our actions. Consequently, an AI must predict the rest of our seemingly complex values from a limited sample. I call this the value generalization problem. In this paper, I argue that human values have a generative rational structure and that this allows us to solve the value generalization problem. In particular, we can use Bayesian Theory of Mind models to infer human values not only from behaviour, but also from other values. This has been obscured by the widespread use of simple utility functions to represent human values. I conclude that developing generative value-to-value inference is a crucial component of achieving a scalable machine theory of mind.
Figures
Reference graph
Works this paper leans on
-
[1]
Lin, Joie Le, Tan Zhi-Xuan, Vikash K
Arwa Alanqary, Gloria Z. Lin, Joie Le, Tan Zhi-Xuan, Vikash K. Mans- inghka, and Joshua B. Tenenbaum. Modeling the Mistakes of Bound- 9 edly Rational Agents Within a Bayesian Theory of Mind, June 2021. arXiv:2106.13249 [cs, q-bio]
arXiv 2021
-
[2]
A survey of inverse reinforce- ment learning: Challenges, methods and progress
Saurabh Arora and Prashant Doshi. A survey of inverse reinforce- ment learning: Challenges, methods and progress. Artificial Intelligence , 297:103500, August 2021
work page 2021
-
[3]
Baker, Rebecca Saxe, and Joshua B
Chris L. Baker, Rebecca Saxe, and Joshua B. Tenenbaum. Action under- standing as inverse planning. Cognition, 113(3):329–349, December 2009
work page 2009
-
[4]
Superintelligence: Paths, Dangers, Strategies
Nick Bostrom. Superintelligence: Paths, Dangers, Strategies . Oxford Uni- versity Press, 2014
work page 2014
-
[5]
MAP Inference for Bayesian Inverse Reinforcement Learning
Jaedeug Choi and Kee-eung Kim. MAP Inference for Bayesian Inverse Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011
work page 2011
-
[6]
The Alignment Problem: Machine Learning and Human Values
Brian Christian. The Alignment Problem: Machine Learning and Human Values. W. W. Norton & Company, October 2021
work page 2021
-
[7]
Donald Davidson. Psychology as Philosophy. In Stuart C. Brown, editor, Philosophy of Psychology , pages 41–52. Harper & Row, 1974
work page 1974
-
[8]
Reinforcement learning: the good, the bad and the ugly
Peter Dayan and Yael Niv. Reinforcement learning: the good, the bad and the ugly. Current Opinion in Neurobiology , 18(2):185–196, April 2008
work page 2008
Show all 47 references
-
[9]
A Person- alized Patient Preference Predictor for Substituted Judgments in Health- care: Technically Feasible and Ethically Desirable
Brian Earp, Sebastian Mann, Jemima Allen, Sabine Salloch, Vynn Suren, Karin Jongsma, Matthias Braun, Dominic Wilkinson, Walter Sinnott- Armstrong, Annette Rid, David Wendler, and Julian Savulescu. A Person- alized Patient Preference Predictor for Substituted Judgments in Healt...
-
[10]
Artificial Intelligence, Values and Alignment
Iason Gabriel. Artificial Intelligence, Values and Alignment. Minds and Machines, 30(3):411–437, September 2020. arXiv:2001.09768 [cs]
2020 arXiv
-
[11]
Glimcher and Ernst Fehr, editors
Paul W. Glimcher and Ernst Fehr, editors. Neuroeconomics: Decision Making and the Brain . Academic Press, Amsterdam : Boston, 2nd edition edition, October 2013
2013
-
[12]
Reinforcement Learning: A Brief Guide for Philosophers of Mind
Julia Haas. Reinforcement Learning: A Brief Guide for Philosophers of Mind. Philosophy Compass, 17(9):e12865, 2022
2022
-
[13]
Ho, Rebecca Saxe, and Fiery Cushman
Mark K. Ho, Rebecca Saxe, and Fiery Cushman. Planning with Theory of Mind. Trends in Cognitive Sciences, 26(11):959–971, November 2022
2022
-
[14]
An Enquiry Concerning the Principles of Morals
David Hume. An Enquiry Concerning the Principles of Morals . Hackett Publishing, Indianapolis, copyright 1983 edition edition, June 1983
1983
-
[15]
Theory of mind as inverse reinforcement learning
Julian Jara-Ettinger. Theory of mind as inverse reinforcement learning. Current Opinion in Behavioral Sciences , 29:105–110, October 2019. 10
2019
-
[16]
Schulz, and Joshua B
Julian Jara-Ettinger, Laura E. Schulz, and Joshua B. Tenenbaum. The Na ¨ ıve Utility Calculus as a unified, quantitative framework for action un- derstanding. Cognitive Psychology, 123:101334, December 2020
2020
-
[17]
Richard C. Jeffrey. The Logic of Decision . University of Chicago Press, 1965
1965
-
[18]
Visual fixations and the computation and comparison of value in simple choice
Ian Krajbich, Carrie Armel, and Antonio Rangel. Visual fixations and the computation and comparison of value in simple choice. Nature Neuro- science, 13(10):1292–1298, October 2010. Number: 10 Publisher: Nature Publishing Group
2010
-
[19]
Sahakian
Christelle Langley, Bogdan Ionut Cirstea, Fabio Cuzzolin, and Barbara J. Sahakian. Theory of Mind and Preference Learning at the Interface of Cognitive Science, Neuroscience, and AI: A Review. Frontiers in Artificial Intelligence, 5:778852, 2022
2022
-
[20]
Goal Misgeneralization in Deep Reinforcement Learning,
Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, Laurent Orseau, and David Krueger. Goal Misgeneralization in Deep Reinforcement Learning,
-
[21]
Active Learning for Reward Estimation in Inverse Reinforcement Learning
Manuel Lopes, Francisco Melo, and Luis Montesano. Active Learning for Reward Estimation in Inverse Reinforcement Learning. In Wray Buntine, Marko Grobelnik, Dunja Mladeni´ c, and John Shawe-Taylor, editors, Ma- chine Learning and Knowledge Discovery in Databases , Lecture Note...
2009
-
[22]
A Review on Machine Theory of Mind, March 2023
Yuanyuan Mao, Shuang Liu, Pengshuai Zhao, Qin Ni, Xin Lin, and Liang He. A Review on Machine Theory of Mind, March 2023. arXiv:2303.11594 [cs]
2023 arXiv
-
[23]
Whinston, and Jerry R
Andreu Mas-Colell, Michael D. Whinston, and Jerry R. Green. Microe- conomic Theory. Oxford University Press, New York, illustrated edition edition, June 1995
1995
-
[24]
Read Montague, Brooks King-Casas, and Jonathan D
P. Read Montague, Brooks King-Casas, and Jonathan D. Cohen. Imaging valuation models in human choice. Annual Review of Neuroscience, 29:417– 448, 2006
2006
-
[25]
Ng and Stuart Russell
Andrew Y. Ng and Stuart Russell. Algorithms for Inverse Reinforcement Learning. In Proceedings of the Seventeenth International Conference on Machine Learning , ICML ’00, pages 663–670, San Francisco, CA, USA, June 2000. Morgan Kaufmann Publishers Inc
2000
-
[26]
Artificial Intelligence: A Modern Ap- proach
Peter Norvig and Stuart Russell. Artificial Intelligence: A Modern Ap- proach. Pearson, Harlow, 4th edition edition, May 2021
2021
-
[27]
Causality: Models, Reasoning and Inference
Judea Pearl. Causality: Models, Reasoning and Inference . Cambridge Uni- versity Press, Cambridge, U.K. ; New York, 2nd edition edition, September 2009. 11
2009
-
[28]
Word and Object
Willard Van Orman Quine. Word and Object. Cambridge, MA, USA: MIT Press, 1960
1960
-
[29]
Moral Realism
Peter Railton. Moral Realism. The Philosophical Review , 95(2):163–207,
-
[30]
Bayesian inverse reinforcement learning
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In Proceedings of the 20th international joint conference on Ar- tifical intelligence , IJCAI’07, pages 2586–2591, San Francisco, CA, USA, January 2007. Morgan Kaufmann Publishers Inc
2007
-
[31]
Read Montague
Antonio Rangel, Colin Camerer, and P. Read Montague. A framework for studying the neurobiology of value-based decision making. Nature Reviews Neuroscience, 9(7):545–556, July 2008. Number: 7 Publisher: Nature Pub- lishing Group
2008
-
[32]
Jaime Ruiz-Serra and Michael S. Harr´ e. Inverse Reinforcement Learning as the Algorithmic Basis for Theory of Mind: Current Methods and Open Problems. Algorithms, 16(2):68, February 2023. Number: 2 Publisher: Multidisciplinary Digital Publishing Institute
2023
-
[33]
Human Compatible: Artificial Intelligence and the Problem of Control
Stuart Russell. Human Compatible: Artificial Intelligence and the Problem of Control. Penguin Books, October 2019
2019
-
[34]
Leonard J. Savage. The Foundations of Statistics . Wiley Publications in Statistics, 1954
1954
-
[35]
Three Faces of Desire
Timothy Schroeder. Three Faces of Desire. Oxford University Press, 2004
2004
-
[36]
Decision-Making: From Neuroscience to Neuroeconomics–an Overview
Daniel Serra. Decision-Making: From Neuroscience to Neuroeconomics–an Overview. Theory and Decision, 91(1):1–80, 2021. Publisher: Springer US
2021
-
[37]
A Theory of Conditionals
Robert Stalnaker. A Theory of Conditionals. In Nicholas Rescher, editor, Studies in Logical Theory (American Philosophical Quarterly Monographs 2), pages 98–112. Blackwell, 1968
1968
-
[38]
Robert Stalnaker. Inquiry. Cambridge University Press, 1984
1984
-
[39]
Sutton and Andrew G
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An In- troduction. Adaptive Computation and Machine Learning series. Bradford Books, Cambridge, MA, USA, 2 edition, November 2018
2018
-
[40]
Well-Being as Value Fulfillment: How We Can Help Each Other to Live Well
Valerie Tiberius. Well-Being as Value Fulfillment: How We Can Help Each Other to Live Well . Oxford University Press, 2018
2018
-
[41]
The Evolution of Agency: Behavioral Organization from Lizards to Humans
Michael Tomasello. The Evolution of Agency: Behavioral Organization from Lizards to Humans . The MIT Press, Cambridge, Massachusetts, September 2022
2022
-
[42]
Internal and External Reasons
Bernard Williams. Internal and External Reasons. In Ross Harrison, editor, Rational Action, pages 101–113. Cambridge University Press, 1979. 12
1979
-
[43]
Fake Utility Functions, 2007
Eliezer Yudkowsky. Fake Utility Functions, 2007
2007
-
[44]
Tenenbaum, and Vikash K
Tan Zhi-Xuan, Nishad Gothoskar, Falk Pollok, Dan Gutfreund, Joshua B. Tenenbaum, and Vikash K. Mansinghka. Solving the Baby Intuitions Benchmark with a Hierarchically Bayesian Theory of Mind, August 2022. arXiv:2208.02914 [cs]
2022 arXiv
-
[45]
Mann, Tom Silver, Joshua B
Tan Zhi-Xuan, Jordyn L. Mann, Tom Silver, Joshua B. Tenenbaum, and Vikash K. Mansinghka. Online Bayesian goal inference for boundedly- rational planning agents. In Proceedings of the 34th International Con- ference on Neural Information Processing Systems , NIPS’20, pages 1923...
2020
-
[1986]
Publisher: [Duke University Press, Philosophical Review]
-
[2022]
arXiv:2105.14111 [cs]
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.