{"id":"7578d936-7042-4c61-a667-d0b13047de21","arxiv_id":"2412.19726","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper proposes that LLM theory-of-mind evaluation should measure functional adaptation to partners, not just literal prediction of their behavior, and shows the two can diverge sharply in simple games.","lead":"This position paper argues that LLM theory-of-mind benchmarks measure whether models can predict another agent's behavior, but not whether they can adapt their own behavior to that agent, and shows that models often fail at the second even when the first is easy. It introduces the terms literal and functional theory of mind, and proposes regret-based evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Definition 2.2's functional-ToM regret is not ToM-specific; Table 3's own ablations (Oracle + Max -Payoffs RPS 0.103; CoT + 3-Shot RPS 0.121) show the gap nearly closes with non-ToM prompt changes, so the observed deficit may be general long-context reasoning or instruction-following.","rationale":"The paper makes a useful conceptual distinction and its repeated-game experiments are clean and honestly reported, including the surprising DeepSeek-R1 pattern in Table 4. However, the central claim that existing ToM benchmarks are 'broken' rests on showing that high literal ToM does not imply good functional ToM, and the operationalization of functional ToM in Definition 2.2 is the weakest link. Because the metric is pure regret against an optimal policy, it cannot distinguish a theory-of-mind-specific failure from general deficiencies in reasoning over long contexts, following instructions, or processing payoff information. The reader's weakest assumption pointed at the same general area; I formulate it more sharply as a construct-validity problem in the metric itself. A non-social control would settle whether the observed gap is social at all. The paper also never evaluates a named existing benchmark with the new metric, so the headline overclaim is not directly tested. These gaps warrant revision but do not destroy the useful proposal; hence I keep the conditional stance.","tokens_in":31564,"tokens_out":6470,"duration_ms":70262,"concrete_test":"Run a non-social control matched to the RPS condition: keep the same payoff table, action labels, and 100-step interaction history, but replace the 'partner' with a stationary stochastic process (e.g., a process that always outputs Rock) described in non-agential language, and measure ΔFunctional/T under QA and CoT prompts. If regret in this non-social control is comparable to the social condition, the gap is not theory-of-mind-specific; it is general instruction-following or long-context reasoning. Also re-run the Oracle + Max condition with a non-social description of the oracle source; if regret stays near 0.1, the functional-ToM metric is not isolating partner modeling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Definition 2.2 defines functional ToM as T-step regret against the optimal response to the partner's policy. This is a policy-optimization metric: it contains no term that references the partner's mental state, beliefs, or even the social nature of the interaction. High ΔFunctional can therefore arise from non-ToM causes: failure to parse the payoff table, failure to follow 'maximize reward' instructions, action-label biases (Appendix C.4), or inability to reason over the 100-step history. The paper's own ablations in Table 3 show these causes are not hypothetical: for LLAMA-3 70B on RPS, removing the payoff table while supplying the true oracle action and emphasizing maximization (Oracle + Max -Payoffs) cuts regret from 0.444 (QA) to 0.103, nearly matching the tabular RMax baseline of 0.083; CoT + 3-Shot gives 0.121. If the deficit were specifically a failure to convert literal predictions into partner-adaptive action, these non-ToM prompt changes should not nearly eliminate it. The same issue undercuts the title-level claim: no existing ToM benchmark is actually run through the new metric, so the evidence is a case study of game regret rather than a demonstration that any named benchmark is misleading.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This position paper argues that most existing theory of mind (ToM) benchmarks for large language models are \"broken\" because they measure only literal ToM—the ability to predict another agent's behavior—and not functional ToM—the ability to adapt one's own behavior rationally to a partner. The authors formalize both notions: literal ToM as a prediction loss (Definition 2.1) and functional ToM as T-step regret against the optimal response to the partner's policy (Definition 2.2). They then report experiments in Rock-Paper-Scissors, Iterated Battle of the Sexes, and Iterated Prisoner's Dilemma against simple single-action and tit-for-tat partners, using several prompting strategies and open-source LLMs. The empirical pattern is that LLMs often show high literal prediction accuracy (ToM %) while incurring large functional regret, with a tabular RMax baseline achieving much lower regret. The paper concludes that strong literal ToM performance does not imply strong functional ToM performance, and that benchmarks should include interactive, functional evaluations.","tokens_in":31771,"tokens_out":5794,"duration_ms":66269,"significance":"If the position is accepted, it would reshape how the community evaluates ToM in LLMs: instead of passive question-answering tests, benchmarks would need interactive tasks in which the agent's decisions, not just its predictions, are scored. The formal definitions and the regret-based metric are clean and reusable, and the empirical study is careful in several respects: results are reported with confidence intervals, a tabular RMax baseline anchors the comparison, and the prompting ablations are extensive. The finding that high literal ToM accuracy coexists with poor functional performance in simple games is a useful cautionary result. However, the central claim is broader than the evidence: the experiments demonstrate a gap in three matrix games, but the ablations suggest that much of the gap may be due to general long-context reasoning or instruction-following rather than ToM-specific reasoning. The paper therefore makes a strong case for including decision-oriented social benchmarks, but the specific assertion that existing ToM benchmarks are broken because they omit functional ToM needs additional support or a narrower framing.","major_comments":[{"comment":"The paper's own ablations undermine the ToM-specific attribution of the functional gap. For LLAMA-3 70B Instruct on RPS, regret per step drops from 0.444 under QA prompting to 0.103 under Oracle + Max -Payoffs and to 0.121 under CoT + 3-Shot, close to the tabular RMax baseline of 0.083. These interventions supply no additional information about the partner's mental state; they change payoff salience, instruction emphasis, and reasoning format. The text itself states that \"reasoning over long contexts is at the core of the issue.\" This suggests the observed failure may be a general long-context reasoning or instruction-following deficit rather than a specifically theory-of-mind deficit. Since the central claim is that ToM benchmarks are broken because they omit functional ToM, the manuscript should either decompose the gap (for example, with non-social control tasks matched in context length and payoff reasoning) or explicitly reframe the claim as advocating for social decision-making benchmarks more broadly.","section":"Section 4.2 / Table 3"},{"comment":"The claim that \"the majority\" of existing ToM benchmarks are broken is not directly supported by the evidence. No standard ToM benchmark (for example, Sally-Anne, FANToM, or the tasks used by Bubeck et al., Kosinski, or Strachan et al.) is run through the proposed functional metric; the empirical support consists of three matrix games with single-action or tit-for-tat partners. A case study can motivate a position, but the title-level claim requires either a systematic mapping of benchmark item types to Definitions 2.1 and 2.2 or a stated narrowing of the claim to \"benchmarks that report only literal ToM should not be used to infer functional competence.\" As written, the paper overgeneralizes from its own game-based setup to the entire benchmark literature.","section":"Abstract and Section 1"},{"comment":"Definition 2.2 defines functional ToM as T-step regret against the optimal response to the partner's policy, with no term that references the partner's beliefs, intentions, or mental state. As the authors note in Section 1, this makes the metric satisfiable by model-free policies. Consequently, high ΔFunctional can arise from non-ToM causes: failure to parse the payoff table, failure to follow the reward-maximization instruction, action-label biases, or inability to reason over the history. Appendix C.4 documents that action-label inductive bias systematically changes regret, and Table 3 shows that removing payoffs while providing oracle actions can almost close the gap. The metric may still be a valuable measure of partner-adaptive behavior, but calling it \"theory of mind\" without further justification conflates social cognition with general decision-making. The paper should either justify the terminological choice or weaken the inference from high regret to a ToM deficit.","section":"Definition 2.2 and Appendix C.4"}],"minor_comments":[{"comment":"The authors state that for Reflexion they \"always report the best value\" over memory sizes 1 to 3. This is a mild optimistic selection bias; it is disclosed, but the tables would be clearer if the selected memory size appeared alongside the Reflexion rows rather than only in Appendix C.2.","section":"Section 4.1"},{"comment":"The caption reads \"Iterated Batte of Sexes\" and should be \"Iterated Battle of Sexes.\"","section":"Table 6 caption"},{"comment":"The sentence containing \"the LLM must struggle\" is misspelled as \"strugle\"; please correct this typo.","section":"Section 4.2"},{"comment":"The caption says \"The only difference with Figure 15 in the emphasis on maximizing reward,\" but the comparison should be with Figure 14 (Oracle Prompting), not with itself.","section":"Figure 15 caption"},{"comment":"The sentence introducing the literal ToM loss is grammatically incomplete (\"The loss from start state s1 with respect to a joint policy ...\"). Rewriting it as a proper definition would improve readability.","section":"Definition 2.1"}],"recommendation":"major_revision","confidential_remarks":"This is a well-positioned ICML position paper with a clean formalization and a useful empirical case study. The main revision should address the load-bearing gap between the evidence and the title-level claim: the paper demonstrates a functional decision-making deficit, but not specifically a theory-of-mind deficit. I would encourage the editor to ask the authors to either add a control analysis separating general reasoning from partner-modeling or to soften the claim from \"ToM benchmarks are broken\" to \"ToM benchmarks should be supplemented with functional, decision-oriented measures.\" The reference list also contains a large number of self-citations that are not all necessary; trimming them would improve focus."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper makes a useful conceptual distinction between literal theory of mind (predicting another agent's behavior) and functional theory of mind (adapting your actions to maximize reward against that behavior), and it shows cleanly in simple matrix games that high prediction accuracy does not imply low regret for several open LLMs. The tables with confidence intervals and a tabular RMax baseline are solid for what they cover. The DeepSeek-R1 result — strong functional performance despite weak literal predictions — is a nice twist that supports the claim that the two can diverge in both directions.\n\nThe soft spots are real. Definition 2.2 defines functional ToM as T-step regret against the optimal response to the partner's policy. That is a policy-optimization metric; it contains no term that references the partner's mental state or even the social nature of the interaction. High regret can come from payoff-table parsing failures, instruction-following issues, or long-context reasoning limits. The paper's own ablations make this concrete: for LLAMA-3 70B on RPS, Oracle + Max -Payoffs cuts regret from 0.444 to 0.103, nearly matching RMax, and CoT + 3-Shot gives 0.121. If the deficit were specifically about converting literal predictions into partner-adaptive behavior, non-ToM prompt changes should not nearly eliminate it. So the observed gap is real, but its attribution to a ToM-specific capability is not established.\n\nAlso, the title says ToM benchmarks are broken, but no existing benchmark is actually run through the new metric. The evidence is a case study in three games, not a demonstration that any named benchmark misleads. That overclaim weakens the paper's force. The authors do address the \"broken\" wording in Section 5 and argue it is justified, but the empirical support is thinner than the wording.\n\nThat said, the paper is honest and well-structured. It reviews relevant work, engages alternative views (game theory, fine-tuning, predict-then-optimize), and ships enough detail in the appendix to reproduce the protocol. No code or data, but prompts and tables are there.\n\nBottom line: the literal/functional distinction is worth having, and the paper makes a valuable point that passive ToM scores can be divorced from interactive decision quality. It deserves a serious referee, but it needs a more careful construct-validation argument and a title that matches the evidence. I would accept with revisions, or treat it as a position paper that should be read with the overclaim noted.","headline":"Useful literal–functional ToM distinction and clean game results, but the title overclaims and Definition 2.2 is a general decision-quality metric, not a ToM-specific one.","tokens_in":32364,"tokens_out":1924,"would_cite":true,"duration_ms":19868,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current LLM theory-of-mind benchmarks mostly measure whether a model can predict another agent's actions; this paper argues they miss whether the model acts on those predictions, and shows the two can diverge sharply.","keywords":["theory of mind","large language models","functional theory of mind","literal theory of mind","repeated games","regret","in-context adaptation","benchmark evaluation"],"falsifier":"Replace the social partner with a non-social stochastic process that generates the same sequence of actions (for example, a fixed pseudo-random schedule), holding prompt length, payoff table, and interaction history identical; if functional regret stays at the same high level, the deficit is general long-context reasoning, while a large drop would confirm the theory-of-mind-specific account.","tokens_in":92,"feed_emoji":"🧠","tokens_out":9490,"duration_ms":494828,"temperature":0.7,"pith_summary":"Most LLM theory-of-mind benchmarks ask a model to predict another agent's next action and reward it for accuracy. The paper argues those benchmarks are broken because they assume the model's predictions are wired into its own decisions; with LLMs this process consistency cannot be taken for granted. It introduces functional theory of mind, defined as the regret an agent incurs by failing to act optimally in response to a partner's observed behavior. In simple 100-round games against partners that always play one action, open-source LLMs score over 90 percent prediction accuracy while missing most of the reward, because they predict correctly and then fail to exploit their predictions. If the paper is right, high literal theory-of-mind scores say little about whether an LLM can adapt to a new partner in-context.","feed_headline":"LLMs ace theory-of-mind tests, then ignore the predictions","feed_subtitle":"High prediction scores don't mean a model will adapt. Most benchmarks never check that.","key_machinery":"The machinery is the paired definitions in Section 2. Definition 2.1 measures literal theory of mind as a loss $\\mathbb{D}(\\phi(\\mathbf{a}_{-i}^{1},\\ldots,\\mathbf{a}_{-i}^{T}),\\phi(\\hat{\\mathbf{a}}_{-i}^{1},\\ldots,\\hat{\\mathbf{a}}_{-i}^{T}))$ between predicted and actual partner actions under an abstraction mapping $\\phi$. Definition 2.2 measures functional theory of mind as $\\sum_{t=1}^{T}(r^{i*}_t - r^i_t)$, the cumulative reward forgone compared with the optimal $T$-step response to the actual partner policy. The load-bearing comparison is between $\\Delta_{\\mathrm{ToM}}/T$, the regret of a rational policy that faithfully follows the literal model, and $\\Delta_{\\mathrm{Functional}}/T$, the regret the LLM actually incurs; the gap between them isolates the failure to convert predictions into decisions. A derived interesting-task score computes the expected regret of an optimal responder to one partner policy when played against another, giving benchmark designers a quantitative criterion for whether a task tests adaptation at all.","core_discovery":"On the paper's own terms, literal theory of mind is the $T$-step loss between the partner actions an agent predicts and the actions the partner actually takes (Definition 2.1), and functional theory of mind is the $T$-step regret of the agent's policy relative to the optimal response to the partner's real policy (Definition 2.2). The central discovery is that these two quantities decouple in current LLMs: across Rock-Paper-Scissors, Iterated Battle of the Sexes, and Iterated Prisoner's Dilemma, models reach 90 percent or higher literal prediction accuracy while incurring large functional regret against partners that play one fixed action for 100 rounds. The gap persists even when the partner's actual next action is provided as oracle input, and in the opposite direction a trained reasoning model can match a tabular near-optimal baseline on functional regret while its literal predictions are relatively weak. The pith is that theory-of-mind capability, defined as adapting one's own behavior to another agent, is not implied by prediction accuracy.","pith_inferences":["The paper's own ablations point to a confound the theory-of-mind framing does not resolve: removing the payoff table from the oracle prompt nearly closes the Rock-Paper-Scissors regret, so long-context payoff reasoning may be the true bottleneck.","A direct extension would run the same prompts with a non-social stochastic partner matched for action statistics; if regret stays high, functional theory of mind is not a distinct capability.","The definitions should transfer to non-game deployments: for a tutoring or coding assistant, the metric that matters is whether the interaction improves, not whether the next user action is predicted.","Because the game-theoretic and theory-of-mind solutions diverge in adversarial settings, evaluation suites should separate cooperative, mixed, and adversarial partners rather than averaging them together."],"forward_implications":["Existing theory-of-mind benchmark scores overstate agent readiness for multi-agent interaction, since high literal accuracy can coexist with near-maximal functional regret.","Meaningful LLM theory-of-mind evaluation should report both ToM% and $\\Delta_{\\mathrm{Functional}}/T$ (or accumulated reward) and the gap between them.","Supplying the partner's true action as oracle input does not fix the functional deficit, so improvements cannot come from better prediction alone.","The two capabilities are not ordered: a trained reasoning model can show strong functional theory of mind with weak literal theory of mind, and vice versa.","Simple repeated matrix games with single-action or tit-for-tat partners are a sufficient minimal testbed to expose the failure."],"supporting_citations":[{"why":"Supplies the repeated-game prompting protocol and the Battle of the Sexes / Prisoner's Dilemma baselines from which the functional theory of mind evaluation is built.","marker":"Akata et al. (2023)"},{"why":"Documents the lack of process consistency in LLM explanations, the premise for why literal prediction accuracy can mislead.","marker":"Bubeck et al. (2023)"},{"why":"Shows LLMs fail when different question types require the same underlying reasoning, supporting the self-consistency concern.","marker":"Kim et al. (2023)"},{"why":"Provides the abstraction framework (actions, intentions, beliefs) that Definition 2.1 deliberately encompasses.","marker":"Ma et al. (2023)"},{"why":"Defines RMax, the tabular near-optimal baseline whose regret the LLMs fail to match.","marker":"Brafman & Tennenholtz (2002)"},{"why":"Supplies chain-of-thought prompting, one of the main strategies tested for closing the functional gap.","marker":"Wei et al. (2022)"},{"why":"Supplies Reflexion, the in-context verbal reinforcement method compared in the study.","marker":"Shinn et al. (2024)"}],"fun_headline_variants":["LLMs predict partners well, but fail to adapt","Theory of mind benchmarks miss functional adaptation","Prediction accuracy doesn't imply adaptive LLMs","Why high prediction scores don't mean adaptive LLMs","Literal and functional theory of mind decouple in LLMs"],"cache_read_input_tokens":34432,"weakest_assumption_plain":"The load-bearing premise is that the gap is a theory-of-mind-specific failure rather than a general weakness in reasoning over long interaction histories or following complex instructions; the authors' own oracle and payoff-ablation results show how much weight this assumption carries.","fun_headline_variants_meta":{"raw":{"variants":["LLMs predict partners well, but fail to adapt","Theory of mind benchmarks miss functional adaptation","Prediction accuracy doesn't imply adaptive LLMs","Why high prediction scores don't mean adaptive LLMs","Literal and functional theory of mind decouple in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1404,"prompt_tokens":995,"completion_tokens":409,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":336}},"tokens_in":611,"tokens_out":409,"duration_ms":5045,"temperature":1.0,"reasoning_tokens":336,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:55:19.505126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the social partner with a non-social stochastic process that generates the same sequence of actions (for example, a fixed pseudo-random schedule), holding prompt length, payoff table, and interaction history identical; if functional regret stays at the same high level, the deficit is general long-context reasoning, while a large drop would confirm the theory-of-mind-specific account.","supporting_citations":[],"review_version":1}