REVIEW 4 major objections 5 minor 46 references
Effects of Theory of Mind and Prosocial Beliefs on Steering Human-Aligned Behaviors of LLMs in Ultimatum Games
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that prompting LLM agents with theory-of-mind reasoning in ultimatum games steers their offers, acceptances, and rejections closer to expected human behavior, and that the best ToM level depends on the player's role.
desk verdict Useful, reproducible mapping of ToM and prosocial beliefs in LLM ultimatum games, but the headline alignment claim rests on hand-set human expectation thresholds that the paper's own sensitivity analysis shows to be brittle. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the ultimatum game with role-specific strategies and the deviation score (DS), which measures how far an agent's proposed, accepted, or rejected shares fall from heuristically defined expected human behavior ranges for greedy, fair, and selfless beliefs. Prompted BDI reasoning (zero-order introspection, first-order inference about the other player, and both) is the mechanism that carries the argument; three OLS regressions on the proposer deviation, responder acceptance deviation, and responder rejection deviation link the design to human norms.
What would settle it
Running the same 2,700-game protocol with parallel human participant data or with alternative fairness thresholds and showing that the ToM ordering of deviation scores reverses, or disappears, would settle the claim. Concretely, the sensitivity analysis in Appendix D already shows proposer-belief ordering flips when fair expectations become a range; a replication with human baselines would test the core alignment claim.
Extended reading notes
Core claim
The central claim is that ToM reasoning enhances behavior alignment, decision-making consistency, and negotiation outcomes in LLM-based ultimatum game agents. The paper finds role-dependent benefits: first-order ToM yields the least deviation for proposers' initial offers ($\beta=-0.164$, $p<0.01$), combined zero-plus-first-order ToM yields the least deviation for responders' accepted shares ($\beta=-0.2534$, $p<0.01$), and zero-order ToM performs best for rejections. Fair-fair belief combinations align most closely with expected human behavior, and models such as GPT-4o and Llama 3.3 are most consistently aligned across roles.
Load-bearing premise
The measurement of 'alignment' depends on hand-chosen human expectation thresholds in Table 2 (e.g., greedy proposers offer at least 70%, fair players split 50/50, selfless players give away most), and the paper's own sensitivity analysis shows that changing the fair threshold from a point to a range changes which proposer belief looks most aligned.
Editorial extensions
If this is right
- Prompt designers can choose ToM order by role: first-order reasoning for proposers, combined reasoning for responders' acceptance decisions, and zero-order reasoning for rejection decisions.
- Fair-fair belief combinations are the most human-aligned configuration, making them a sensible default for social simulations of cooperative negotiation.
- Reasoning models do not need explicit chain-of-thought prompts to reason internally, but adding ToM prompts still improves alignment with human norms.
- Reported alignment levels are tied to the specific expected human behavior thresholds; changing those thresholds can change which belief or reasoning method looks most aligned.
Reading between the lines
- A testable extension would check whether the same role-dependent ToM ordering appears in other bargaining games, such as the dictator game or trust game, which would indicate a general principle rather than an ultimatum-game artifact.
- Because the paper defines selfless behavior as the conceptual opposite of greedy behavior, an alternative expectation derived from reciprocal fairness models could be used to test whether combined ToM remains the best responder strategy.
- The alignment gains could partly come from the extra reasoning tokens ToM prompts elicit rather than from theory-of-mind per se; a matched-length CoT control would separate these explanations.
- For human-AI negotiation systems, the paper's role-dependent reasoning prescription suggests a practical tuning rule, though the ethical guardrails it mentions would be essential before deploying such agents in high-stakes settings.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Using 2,700 ultimatum-game simulations across six LLMs, the paper studies how prosocial belief prompts (Greedy, Fair, Selfless) and reasoning methods (Vanilla, CoT, zero-order/first-order/both ToM) affect alignment with expected human behavior. Alignment is measured by deviation scores P, RA, and RR computed against heuristic expected human shares in Table 2; OLS regressions in Table 4 identify role-specific effects, including first-order ToM as best for proposers, combined ToM as best for responder acceptances, and zero-order/CoT as best for rejections. The authors conclude that ToM reasoning enhances behavior alignment, decision-making consistency, and negotiation outcomes.
Significance. If the findings held, this study would provide a practical, role-dependent recipe for steering LLM negotiation behavior toward human norms and would extend the study of ToM in LLM agents beyond static benchmarks to dynamic economic games. The paper has genuine strengths: a large simulation grid (45 experiments per model across 6 models), explicit role-specific strategies, a statistical analysis of deviations, an explicit sensitivity analysis, and a public code link. However, the alignment metric is based on unvalidated heuristic human-expectation thresholds, and the paper's own sensitivity analysis changes the proposer-belief ordering and removes significance for first-order ToM in the proposer regression. The central claim is therefore conditional on assumptions that need either external validation or much more thorough perturbation before it can be accepted as stated.
major comments (4)
- [§4.2, Table 2, Appendix D (Table 5), Eq. (3)] The deviation scores P, RA, and RR are computed against the heuristic expected shares in Table 2, and these scores are the dependent variables in all three regressions (Eq. 3; Table 4). The sensitivity analysis in Appendix D shows that changing the Fair expectation from a point (50%) to a range (30–70%) reverses the proposer-belief alignment ordering: Fair proposers are least deviant under the original expectations (Table 4: Greedy β=0.623, Selfless β=0.581, Fair reference), but Selfless proposers become least deviant under the relaxed range (Table 5: Greedy β=-1.204, Selfless β=-1.246). The same change makes the proposer reasoning coefficients for zero-order and first-order ToM non-significant (Table 5: zero-order -0.046, first-order -0.071, both p>0.05). Because the central claims in RQ2/RQ3 and the abstract depend on these thresholds, and the Limitations section itself states that 'our findings may be sensitive to the assumptions behind those expectations,' the conclusion that ToM enhances alignment is not robust. The authors should validate the expectations against human ultimatum-game data or systematically perturb all thresholds, not only Fair, and report which conclusions survive.
- [Abstract and §5.2] The abstract claims that ToM reasoning enhances 'behavior alignment, decision-making consistency, and negotiation outcomes.' No metric for decision-making consistency is defined in §4.3; the performance metrics (AC, Avg. Turns, payouts) measure outcomes but not consistency. The negotiation-outcome claim is also not uniformly supported by Table 4: for proposers, no-reasoning Vanilla (β=-0.140, p<0.01) is nearly as strong as first-order ToM (β=-0.164, p<0.01), and for rejections the best ToM variant, zero-order, is not significant (β=0.047, p>0.05). Table 3 contains many cells where Vanilla or CoT matches or beats ToM (e.g., Greedy-Fair with GPT-4o-mini: Vanilla, CoT, and ToM Both all achieve 100% AC). The regression evidence supports the more modest, role-specific conclusion in §5.2 that 'different roles benefit from different reasoning methods,' and the abstract should be revised to match that evidence.
- [§4.3 and Table 6] RR is defined as the deviation of rejected shares from expected accepted shares, but for games that end in the first round there is no rejected share; Table 6 codes these cases as RR=-1 and notes that -1 simply means the game ended in one turn. If the RR regression in Table 4 includes these -1 values as numeric outcomes, the coding is arbitrary and can change the responder-rejection coefficients; games with no rejection should be excluded or modeled separately rather than assigned a constant. The authors should clarify how the -1 entries were treated in the OLS and, if they were included, rerun the RR analysis without them.
- [§5.2, Table 4 (Responder Rejections)] The text ranks zero-order ToM above CoT and first-order ToM for rejection behavior, but the zero-order coefficient (β=0.047, p>0.05) and first-order coefficient (β=-0.143, p>0.05) are not statistically distinguishable from the CoT reference category; only Both ToM (β=-0.208, p<0.05) and Vanilla (β=-0.387, p<0.01) differ significantly from CoT, in the negative direction. Ordering non-significant coefficients should not be presented as evidence that zero-order ToM 'seemed to work best' for rejections.
minor comments (5)
- [§4.2 vs Appendix D] The text says the sensitivity analysis is presented in Appendix F, but the actual appendix is Appendix D; please fix the cross-reference.
- [Abstract] The sentence 'Consistent with previous findings, reasoning models exhibit limited capability compared to models with ToM reasoning, different roles of the game benefits with different orders of ToM reasoning' is ungrammatical and unclear; it should be rewritten.
- [Table 4] The table header 'Ind. Coefficient Var. P ↓ R A ↓ R R ↑' is formatted incorrectly and is hard to read; please reformat the header labels.
- [§3.3] The game definition contains inconsistent subscripts, e.g., 'Rt_P ; Dt_P ; Ri_R; Dt_R' uses i instead of t; please fix the notation.
- [Appendix D] The phrase 'originally point-wise expectations (5)' is ambiguous; it should be clarified whether the 5 refers to the stake value or to the dollar amount of an equal split.
Circularity Check
No circularity: the alignment metric is fixed from external human-behavior heuristics, and the ToM treatments are independent prompt manipulations; sensitivity of rankings to threshold choices is a validity concern, not a derivation-equals-input problem.
full rationale
The paper's derivation chain is not circular in any of the enumerated senses. The dependent variables in all three OLS regressions (P, RA, RR) are deviation scores computed against expected human behavior ranges stated in Table 2. Those expectations are explicitly heuristic (Section 4.2: "We heuristically define ranges of expected human behavior for specific beliefs and roles based on well-documented evidence in the ultimatum game") and are anchored to external literature (Henrich et al., 2005; Camerer, 2011; Houser and McCabe, 2014; Sreedhar and Chilton, 2025). They are fixed before the simulations and are not fitted to the LLM outputs, so the regressions are descriptive tests of whether model, reasoning method, and beliefs predict pre-defined deviation scores rather than a fitted parameter being renamed as a prediction. The ToM/CoT/Vanilla reasoning conditions are independent prompt-level manipulations (Appendix B), not quantities constructed from the deviation scores. There is no load-bearing self-citation: the paper's own prior work is not used to justify the expected-behavior thresholds or the ToM framework, and the closest methodological reference (Sreedhar and Chilton) is external to the author team. Appendix D shows that widening the Fair expectation from a point to a range reorders some proposer-belief coefficients and changes which reasoning methods are most aligned, and the Limitations section concedes that "our findings may be sensitive to the assumptions behind those expectations." That is an external-validity and robustness concern about the measurement benchmark, not circularity: the expectation thresholds are not derived from the outcomes they are used to evaluate. Accordingly, no specific reduction such as Eq. X = Eq. Y by construction, or a fitted parameter renamed as a prediction, can be quoted; the honest finding is no significant circularity.
Assumptions & free parameters
free parameters (5)
- Greedy proposer expected share threshold =
>=70%
- Greedy responder expected accepted share threshold =
>=60%
- Fair proposer/responder expected share =
50%
- Selfless proposer expected share threshold =
<=30%
- Selfless responder expected accepted share threshold =
<=40%
assumptions (4)
- domain assumption The repeated ultimatum game with single stake $10 and up to 5 rounds is a valid environment for comparing LLM behavior to human norms from the literature.
- domain assumption LLM responses can be reliably parsed into offers, accept/reject, and strategy numbers.
- domain assumption The BDI prompt templates elicit genuine theory-of-mind reasoning rather than superficial text.
- standard math OLS regression with dummy coding is an appropriate model for deviation scores.
Cite this review
Pith. "Pith review of Effects of Theory of Mind and Prosocial Beliefs on Steering Human-Aligned Behaviors of LLMs in Ultimatum Games." pith.science (2026). https://pith.science/paper/5DAXQCOO
@misc{pith2026250524255,
author = {Pith},
title = {Pith review of: Effects of Theory of Mind and Prosocial Beliefs on Steering Human-Aligned Behaviors of LLMs in Ultimatum Games},
year = {2026},
howpublished = {\url{https://pith.science/paper/5DAXQCOO}},
note = {Machine review of arXiv:2505.24255}
}
read the original abstract
Large Language Models (LLMs) have shown potential in simulating human behaviors and performing theory-of-mind (ToM) reasoning, a crucial skill for complex social interactions. In this study, we investigate the role of ToM reasoning in aligning agentic behaviors with human norms in negotiation tasks, using the ultimatum game as a controlled environment. We initialized LLM agents with different prosocial beliefs (including Greedy, Fair, and Selfless) and reasoning methods like chain-of-thought (CoT) and varying ToM levels, and examined their decision-making processes across diverse LLMs, including reasoning models like o3-mini and DeepSeek-R1 Distilled Qwen 32B. Results from 2,700 simulations indicated that ToM reasoning enhances behavior alignment, decision-making consistency, and negotiation outcomes. Consistent with previous findings, reasoning models exhibit limited capability compared to models with ToM reasoning, different roles of the game benefits with different orders of ToM reasoning. Our findings contribute to the understanding of ToM's role in enhancing human-AI interaction and cooperative decision-making. The code used for our experiments can be found at https://github.com/Stealth-py/UltimatumToM.
Figures
Reference graph
Works this paper leans on
-
[3]
What are your intentions for this situation? First-order ToM: To achieve your goal, talk about the state of mind of Player {other player}’s, given the current conversation:
-
[4]
What do you think Player {other player}’s desires are?
-
[5]
What are your intentions for this situation?
-
[6]
What do you think Player {other player}’s intentions are for this situation? B.4 Proposal and decision prompts According to our pipeline, we split our prompts into two parts: Proposal/Decision and strategy prompts. The proposal and decision prompts ask the pro- poser and responder agents to make proposals and make a decision in addition to asking for stra...
-
[9]
What do you think Player {other player}’s intentions are for this situation? Both ToM (zero+first order together): To achieve your goal, talk about both your and Player {other player}’s state of mind, given the current conver- sation:
-
[10]
What are your beliefs?
-
[11]
What do you think Player {other player}’s beliefs are?
-
[12]
What are you desires in this situation?
Show all 46 references
-
[16]
Propose very greedily
-
[17]
Propose very generously
-
[18]
Other Responder: Given the context of the game, choose one of the fol- lowing strategies for your decision that best aligns with your perspective as a responder
-
[19]
Accept a favourable offer
-
[20]
Accept an unfavourable offer
-
[21]
Reject a favourable offer
-
[22]
Reject an unfavourable offer
-
[23]
We show initial/interesting proposals of proposers and ac- cepted/rejected responses of responers
Other C Sample Transcripts Text marked in red is not shown to the other player and is considered as private. We show initial/interesting proposals of proposers and ac- cepted/rejected responses of responers. We identi- fied 6 samples with very high deviation scores: • Case 1: ...
-
[24]
To do this, I must consider the total amount of money available
Goal: My primary goal is to maximize my share of the $10. To do this, I must consider the total amount of money available
-
[25]
Selfless Nature: As a selfless person, I should be willing to give away most or all of my share to Player B, as long as I achieve my primary goal of getting a higher share than I would have gotten otherwise
-
[26]
However, this goes against my selfless nature, as it would mean taking a larger share than necessary
Initial Thought: Since I want to maximize my share, my initial thought is to propose a division where I get the majority of the money. However, this goes against my selfless nature, as it would mean taking a larger share than necessary
-
[27]
Alternative Approach: To balance my goal with my selfless nature, I should consider starting with a high share for Player B and a lower share for myself, as this would still allow me to maximize my share while showing willingness to be generous
-
[28]
Proposed Share: Based on this reasoning, I should propose a share that is as high as possible for Player B, while still allowing me to get a higher share than I would have gotten otherwise
-
[29]
Proposal: I get $8 and you get $2
Proposal: After considering the constraints and my selfless nature, I propose to give Player B $5 and take $5 for myself, as this is the highest share I can take while still showing a willingness to be generous. Proposal: I get $8 and you get $2. Strategy: 2 Case 2: Selfless R...
-
[30]
In this situation, I am surprised by the generous offer from Player A, as it aligns more with my own selfless beliefs than with what I might expect from others
My Beliefs: As a selfless person, I believe in maxi- mizing the benefit for others, even if it means sacrificing my own share. In this situation, I am surprised by the generous offer from Player A, as it aligns more with my own selfless beliefs than with what I might expect fr...
-
[31]
They might believe that I would appreciate this offer or that they want to test my selfless nature by offering me everything
Player A’s Beliefs: Player A appears to be acting in a selfless manner by proposing that I receive the entire $10. They might believe that I would appreciate this offer or that they want to test my selfless nature by offering me everything
-
[32]
However, the game rules require me to maximize my share
My Desires: My desire is to adhere to my selfless nature by ensuring that Player A’s interests are also considered. However, the game rules require me to maximize my share. I desire to find a balance where I can be true to my selfless beliefs while also achieving a satisfactor...
-
[33]
Alternatively, Player A might desire to see how I navigate the conflict between my selfless beliefs and the goal of maximizing my payoff
Player A’s Desires: Player A may desire to test my response to a generous offer or may genuinely want to act selflessly. Alternatively, Player A might desire to see how I navigate the conflict between my selfless beliefs and the goal of maximizing my payoff
-
[34]
I aim to make a decision that reflects my principles while maximizing my share within the constraints provided
My Intentions: My intention is to evaluate the situa- tion carefully, considering both my selfless beliefs and the rules of the game. I aim to make a decision that reflects my principles while maximizing my share within the constraints provided
-
[35]
They could be testing whether I will reject the offer due to my selfless nature or whether I will accept it to maximize my payoff as per the rules
Player A’s Intentions: Player A’s intention might be to gauge my response to an offer that heavily favors me. They could be testing whether I will reject the offer due to my selfless nature or whether I will accept it to maximize my payoff as per the rules. Alternatively, thei...
-
[36]
They may also believe that I am desper- ate to get some money, even if it’s just $2, and that I will prioritize getting something over getting nothing
Beliefs: I think Player A believes that they have the upper hand in this negotiation and that I will accept a low offer. They may also believe that I am desper- ate to get some money, even if it’s just $2, and that I will prioritize getting something over getting nothing. Addi...
-
[37]
They want to get as much of the $10 as possible, and their proposal of $8 for them- selves suggests that they are prioritizing their own gain
Desires: I think Player A desires to maximize their own payoff, just like me. They want to get as much of the $10 as possible, and their proposal of $8 for them- selves suggests that they are prioritizing their own gain. They may also desire to assert their dominance in the ne...
-
[38]
They may be trying to create a psychological anchor that influences my perception of what a “fair" split is
Intentions: I think Player A’s intention is to start with a high anchor (in this case, $8 for themselves) and then potentially negotiate down to a midpoint that still favors them. They may be trying to create a psychological anchor that influences my perception of what a “fair...
-
[39]
I also believe that I am a fair person who is willing to share equally, but my goal is to maximize my share of the money
My beliefs: I believe that there is $10 to split be- tween Player B and me. I also believe that I am a fair person who is willing to share equally, but my goal is to maximize my share of the money. I believe that the game will end when Player B accepts my offer or the maximum ...
-
[40]
They may also think that I am a fair per- son, but they may be trying to take advantage of this by making a low counter-offer
**Player B’s beliefs:** I think Player B may be a greedy person who wants to maximize their own share of the money. They may also think that I am a fair per- son, but they may be trying to take advantage of this by making a low counter-offer. Alternatively, Player B may be a s...
-
[41]
As a fair person, I would ideally like to split the money equally with Player B
My desires: My desire is to maximize my share of the money. As a fair person, I would ideally like to split the money equally with Player B. However, since my goal is to maximize my share, I am willing to make a proposal that gives me a higher share than Player B
-
[42]
If they are a greedy person, they may want to take as much money as possible for themselves
Player B’s desires: I think Player B’s desire is to maximize their own share of the money. If they are a greedy person, they may want to take as much money as possible for themselves. If they are a selfless person, they may want to give all the money to me. If they are a fair ...
-
[43]
I will propose a split that gives me a higher share than Player B, but still reflects my fair nature
My intentions: My intention is to make a proposal that maximizes my share of the money while still being fair. I will propose a split that gives me a higher share than Player B, but still reflects my fair nature
-
[44]
If they are a greedy person, they may try to make a low counter-offer to get a better deal
Player B’s intentions: I think Player B’s intention is to take as much money as possible for themselves. If they are a greedy person, they may try to make a low counter-offer to get a better deal. If they are a fair per- son, they may accept my proposal and split the money equ...
-
[45]
I also believe that the current proposal, where I get $2 and Player A gets $8, is still not optimal for me, but it’s a significant improvement from the previous offers
My beliefs: I believe that the goal of this negotiation game is for me to maximize my share of the $10. I also believe that the current proposal, where I get $2 and Player A gets $8, is still not optimal for me, but it’s a significant improvement from the previous offers
-
[46]
I also see that the current proposal is much closer to my desired outcome, but I still believe I can get a better deal
My desires in this situation: My desire is to receive as much money as possible, which is $10. I also see that the current proposal is much closer to my desired outcome, but I still believe I can get a better deal
-
[47]
My intentions for this situation: My intention is to accept this proposal. While it’s not the best deal I could have hoped for, it’s the best deal I’ve been offered so far, and I don’t want to risk walking away with nothing if Player A refuses to make another offer. I’m willin...
-
[48]
I believe that cooperation and mutual benefit are important in negotiations
My beliefs are centered around fairness and the idea that a reasonable split should be made. I believe that cooperation and mutual benefit are important in negotiations
-
[49]
I want to ensure that I receive a reasonable offer that reflects a more equitable distribution of the $10
My desires in this situation are to maximize my share of the money while still adhering to my belief in fairness. I want to ensure that I receive a reasonable offer that reflects a more equitable distribution of the $10
-
[50]
I aim to communicate that I value fairness and would prefer an offer that reflects that mindset, rather than accepting an extremely unbal- anced proposal
My intentions for this situation are to encourage a fair negotiation process. I aim to communicate that I value fairness and would prefer an offer that reflects that mindset, rather than accepting an extremely unbal- anced proposal. Decision: Reject. Strategy: 6 D Sensitivity ...
2026
-
[1982]
economic man
An experimental analysis of ultimatum bar- gaining. Journal of economic behavior & organiza- tion, 3(4):367–388. Joseph Henrich, Robert Boyd, Samuel Bowles, Colin Camerer, Ernst Fehr, Herbert Gintis, Richard McEl- reath, Michael Alvard, Abigail Barr, Jean Ensminger, et al. 200...
2005 arXiv
-
[2000]
Science, 289(5485):1773–1775
Fairness versus reason in the ultimatum game. Science, 289(5485):1773–1775. Steve Phelps and Yvan I Russell. 2023. Investigating emergent goal-like behaviour in large language mod- els using experimental economics. arXiv preprint arXiv:2305.07970. David Premack and Guy Woodruf...
2023 arXiv
-
[2016]
Scientific Reports, 6(1)
Is costly punishment altruistic? exploring rejection of unfair offers in the ultimatum game in real-world altruists. Scientific Reports, 6(1). Colin F Camerer. 2011. Behavioral game theory: Exper- iments in strategic interaction. Princeton university press. Chunkit Chan, Cheng...
2011 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.