Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ToMATO, a new Theory-of-Mind benchmark built from LLM-LLM conversations, uses prompted thoughts as ground-truth mental states and finds that even GPT-4o mini trails human performance (73.5% vs 87.3%).

desk verdict A useful, more comprehensive ToM benchmark whose scores rest on unvalidated LLM self-reports as ground truth; worth reviewing with a construct-validity requirement. read the letter →

arxiv 2501.08838 v1 pith:I55WMNR7 submitted 2025-01-15 cs.CL cs.AI

Kazutoshi Shinoda , Nobukatsu Hojo , Kyosuke Nishida , Saki Mizuno , Keita Suzuki , Ryo Masumura , Hiroaki Sugiyama , Kuniko Saito This is my paper · ORCID
classification cs.CLcs.AI
keywords TheoryofMindlargelanguagemodelsbenchmarkfalsebeliefmentalstatesInnerSpeechpromptingpersonalitytraitsmultiple-choiceQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ToMATO is a new benchmark for testing whether large language models can infer the mental states of a conversational partner—beliefs, intentions, desires, emotions, and knowledge—in both first-order form (what A feels) and second-order form (what B thinks A feels). The paper's key construction device is Inner Speech prompting: before each utterance, a role-playing LLM writes its thought in parentheses, and that text becomes both the recorded mental state and the correct multiple-choice answer. By hiding each agent's thoughts, goals, and personality from the other, the generation process naturally produces false beliefs about those five mental states. On the resulting 5.4k questions, even GPT-4o mini underperforms human annotators (73.5% vs 87.3%), with the largest gap on false-belief questions. Existing benchmarks mostly cover only beliefs, rarely test false beliefs about mental states, and ignore personality, so this result suggests LLM Theory of Mind is narrower and less reliable than prior tests indicated.

What carries the argument

The central mechanism is Inner Speech prompting, a template that asks a role-playing LLM to prefix each utterance with a mental-state sentence in parentheses. For first-order mental states the prompt begins with I think, I will, I want, I feel, or I know; for second-order states it begins with I think that he/she thinks, will, wants, feels, or knows. These verbalized thoughts become the ground-truth answers to the benchmark's multiple-choice questions. The second half of the machinery is information asymmetry: each agent's system prompt, which contains its goal and Big Five personality, and its parenthetical thoughts are hidden from the other agent. That asymmetry induces false beliefs about the five mental-state types, which the ToMATO-FB subset then tests.

What would settle it

One concrete test: strip the parenthetical thoughts from a random sample of conversations, have a different LLM generate its own mental-state inferences for each character, and check whether the two sets of thoughts agree beyond chance; near-chance agreement would show the thoughts are not stable reads of the characters' mental states.

Watch

Extended reading notes

Core claim

The paper's central claim is that a Theory-of-Mind benchmark can be built by having two role-playing LLMs converse while verbalizing their mental states, and that the verbalized thoughts are valid answers to questions about the characters' mental states. ToMATO covers belief, intention, desire, emotion, and knowledge at both first and second order, and its ToMATO-FB subset isolates cases where character B misunderstands character A's mental state. The paper reports that no evaluated LLM reaches the human baseline, that false beliefs about emotions and intentions are the hardest category, and that accuracy varies with the assigned Big Five personality traits, falling for unconscientious, introversive, disagreeable, and neurotic characters. Read together, the experiments are offered as evidence that current LLMs lack human-level, personality-robust Theory of Mind.

Load-bearing premise

The load-bearing premise is that the parenthetical thoughts a role-playing language model produces under the paper's prompting are faithful reports of the character's mental state rather than merely plausible continuations; if that premise fails, ToMATO measures text inference, not Theory of Mind.

Editorial extensions

If this is right

  • ToMATO provides a single scoreboard for first- and second-order belief, intention, desire, emotion, and knowledge, plus a dedicated false-belief subset, so future model comparisons can separate which mental-state category is the bottleneck.
  • Every evaluated model falls short of the human baseline, and the gap is largest on false-belief questions, so current LLMs are not yet deployable for tasks where misreading another agent's thoughts has real cost.
  • LLM accuracy varies systematically with the assigned Big Five personality: unconscientious, introversive, disagreeable, and neurotic characters are harder to read, which means ToM performance in real deployments may differ across user groups.
  • The ablation results show that hiding thoughts, goals, and personality from the other agent raises false-belief frequency, giving a controllable recipe for generating false-belief data without hand-crafted stories.
  • ToMATO is less vulnerable to lexical-overlap and word-label shortcuts than FANToM and ToMBench, so improvements on it are less likely to be artifacts of surface correlations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a natural test is to rebuild ToMATO with a different generator model; if model rankings and the human gap persist, the findings reflect Theory of Mind difficulty rather than the style of one generator.
  • The information-asymmetry recipe suggests that false beliefs can be manufactured at scale in any conversational setting by hiding one agent's goals, personality, and inner speech, which could be used to build harder, more diverse ToM probes without narrative templates.
  • If the personality-robustness results transfer to real systems, a conversational AI that infers user mental states would be systematically less accurate for users who are less conscientious, more introverted, more disagreeable, or more neurotic; this is an equity-relevant implication the paper does not draw.
  • The fine-tuning result—improving ToMATO while degrading SocialIQa—predicts that other ToM training sets will also trade off against broad social reasoning unless training data spans multiple conversational formats; a cheap test is measuring SocialIQa before and after fine-tuning on ToMATO-FB alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces ToMATO, a Theory-of-Mind (ToM) benchmark built from LLM-LLM conversations. Role-playing LLMs with Big Five personality profiles and information asymmetry are prompted to verbalize their thoughts before each utterance, and those verbalized thoughts are used as ground-truth answers to multiple-choice questions about first- and second-order mental states across five categories: belief, intention, desire, emotion, and knowledge. A false-belief subset, ToMATO-FB, is created by comparing A's first-order thought with B's second-order thought, with both human annotators and GPT-4o mini as judges. The paper reports evaluations of nine LLMs plus a human baseline, finding that the best models, including GPT-4o mini and Llama-3.1-70B, underperform humans overall and especially on false beliefs, and that performance varies with the character's personality traits.

Significance. If the ground-truth thoughts are accepted as valid records of mental states, ToMATO is a substantial new resource: it broadens ToM assessment to five mental-state categories at two orders, includes false beliefs about mental states, introduces personality-trait controls, and includes useful robustness analyses against lexical-overlap and word-label artifacts. The release of data and code, the multi-run evaluation, and the human validation pipeline are strengths. The central empirical finding—that current LLMs lag behind humans on this benchmark, particularly on second-order false beliefs—would be important. However, the benchmark's validity rests on an assumption that the paper does not independently establish: that a single sampled verbalized thought from a stochastic LLM is the character's authoritative and uniquely inferable mental state.

major comments (3)
  1. [Section 4 (Inner Speech Prompting; Multiple-choice QA Dataset Construction), Eqs. (1)-(2)] The ground-truth answers are single samples from stochastic role-playing LLMs (πA and πB), and no evidence is provided that these sampled thoughts are stable or uniquely inferable. The quality validation in Section 4 and Appendix B asks annotators to check that the provided answer is the only correct option among the distractors, but this does not establish that the sampled thought is the character's actual mental state. Because the same conversation prefix can yield different thoughts across samples, the benchmark may measure text-continuation prediction rather than ToM as defined in psychology. I request (a) a resampling study reporting the stability of the generated thoughts and (b) a human study in which annotators read only the conversation, independently infer the character's mental state, and are compared against the sampled thought.
  2. [Section 4 (False Belief Detection; Quality Validation) and Table 3] The construction of both the general QA set and the false-belief subset requires GPT-4o mini's agreement: instances are included only when deemed valid by both the majority of human annotators and GPT-4o mini, and FB questions are added only when both the majority of annotators and GPT-4o mini agree that B misunderstands A's mental state. Since GPT-4o mini is one of the evaluated models and its FB score (60.2%) is a headline result, the measurement is partly circular for that model: it is scored on items it helped select. To support the claim that GPT-4o mini underperforms on false beliefs, the authors should reconstruct the labels with human-only judgments or with a judge model that is not included in the evaluation set, and then re-report the affected scores.
  3. [Section 6, Table 6] The FB detection method labels a question as a false-belief question when B's second-order thought differs from A's first-order thought, but the ablation data show that such mismatches occur even when A's thoughts are visible to B: in the condition without thought asymmetry (Table 6, row 2), human judges still identify false beliefs in 32.0% of cases and GPT-4o mini identifies 40.4%. This suggests that a substantial portion of ToMATO-FB may reflect generator inconsistency—B failing to condition on the visible thought—rather than information-asymmetry-induced false beliefs. The paper should report the overlap between the current FB subset and human-judged FB items, and filter FB items by human judgment or by agreement across multiple generation seeds.
minor comments (4)
  1. [Section 5, Human Baseline] The human performance numbers in Table 3 are reported as point estimates without confidence intervals or inter-annotator agreement; please report the number of annotators per item, variance across annotators, and the standard error of the human mean.
  2. [Section 5, Experimental Setup] The paper acknowledges that evaluating Llama-3-70B on a dataset generated by Llama-3-70B is unfair, but Llama-3.1-70B is still highlighted as state-of-the-art; consider presenting the generator-model results with an explicit caveat or excluding them from the main ranking.
  3. [Section 4 and Appendix A, Table 7] The main text says ToMATO uses 15 personality trait patterns, while Section 6 generates all 32 combinations for the correlation analysis; please clarify in the main text whether the 15 evaluation patterns are a subset of the 32 and why they were chosen.
  4. [Appendix C] For models without system-prompt support, the system prompt is concatenated into the user message; please confirm that the final text format is identical across all evaluated models and report whether any answer-parsing failures occurred.

Circularity Check

2 steps flagged · score 6.0 of 10

ToMATO's ground-truth 'mental states' are defined as the generator LLM's sampled thoughts, so the benchmark measures agreement with that generator by construction; additionally, GPT-4o mini helped select the very items on which it is later scored.

  1. self definitional [Sec. 4, 'Notation' and 'Multiple-choice QA Dataset Construction' (sampling equations and answer definition)]
    "The actual first- and second-order mental state of character A for type T when A says uA_i is defined as mA,T1_i and mA,T2_i, respectively. ... Q is generated with predefined templates for each utterance ui in C, and A is the thought mi corresponding to the utterance."

    The paper's target construct ('actual mental state') is not measured independently: the sampling equations define mA,T1_i as a draw from the role-playing LLM πA, and the QA-construction step sets the ground-truth answer A to that same sampled thought. Hence ToMATO's correct answers are, by construction, one stochastic LLM's verbalized continuations, and the benchmark's 'mental states' are the generator's text by definition. A model's score therefore measures how well it predicts Llama-3-70B's prompted thoughts, and the claim that ToMATO 'captures' or 'assesses' mental states is true only because mental state was defined as the thought.

  2. other [Sec. 4, 'False Belief Detection' and 'Quality Validation & Statistics'; Table 3]
    "When both the majority of annotators and GPT-4o mini agree that B partially misunderstands A’s mental state, it is added to ToMATO-FB. ... those deemed valid by both the majority of annotators and GPT-4o mini are included in ToMATO."

    GPT-4o mini is one of the nine models evaluated in Table 3. Every ToMATO question was included only after GPT-4o mini had agreed that the answer was correct and the distractors were incorrect, and every ToMATO-FB item only after GPT-4o mini had agreed that B misunderstands A. GPT-4o mini's reported accuracy is therefore computed on a test set that its own judgments helped filter, so its benchmark score is not an independent measurement for that model. This selection loop does not numerically force the reported 73.5% / 60.2% values, but it removes test-set independence and confounds the comparison with models that took no part in construction. The paper flags the analogous generator/evaluator overlap only for Llama-3-70B, not for GPT-4o mini's dual role.

full rationale

The paper is not circular in the fitted-parameter sense: the labels are generated once, fixed, and then evaluated, and the reported model scores are not derived by fitting anything to those scores. The benchmark is also self-contained, with human validation, lexical-overlap baselines, and external comparisons (SocialIQa) that give the empirical measurements independent content. Nevertheless, two construction choices create partial circularity. (1) The central construct is self-definitional: 'actual mental state' is defined as the sampled thought, and the ground-truth answer is that same thought, so ToMATO's questions operationalize mental states as Llama-3-70B's verbalizations. The headline interpretation that models lack Theory of Mind therefore rests on equating verbalized thoughts with mental states, an equation the paper does not independently validate; the quality-validation step only checks that annotators accept the sampled answer, not that it is a unique externally grounded mental state. (2) GPT-4o mini serves both as a filter that admits questions into ToMATO/ToMATO-FB and as an evaluated model in Table 3, so its score is partly self-referential. The paper acknowledges a related generator/evaluator overlap for Llama-3-70B ('it is unfair to compare Llama models with other LLMs') but does not address GPT-4o mini's filter role. Because the headline result (GPT-4o mini underperforms humans) is an honest empirical measurement and is not numerically forced by the construction loop, the score is 6 rather than 8 or 10.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the construct validity of LLM verbalized thoughts as mental state annotations, on the effectiveness of information asymmetry for generating false beliefs, and on the representativeness of SOTOPIA scenarios. No new physical or metaphysical entities are introduced.

free parameters (3)
  • Maximum conversation turns N = 7
    The generation loop stops after N=7 turns because the authors found longer conversations redundant; this shapes dataset length and difficulty.
  • Personality trait pattern set = 15 of 32 possible Big Five combinations
    The specific 15 patterns in Table 7 are selected by hand to provide high/low control conditions across five factors; the benchmark's personality-robustness claim depends on this choice.
  • Scenarios sampled from SOTOPIA = 160 (20 per category)
    Conversations are sampled from eight SOTOPIA categories, 20 scenarios each. This sample size and coverage underlie the claim of real-world relevance.
assumptions (5)
  • domain assumption LLMs can role-play assigned Big Five personality traits in multi-turn conversations.
    Relies on prior findings from Jiang et al. (2023) and PersonaLLM; the paper's own pairwise comparisons (Appendix E) support this for four of five factors, with conscientiousness weaker.
  • ad hoc to paper Verbalized thoughts under Inner Speech prompting are valid, consistent representations of a character's mental states.
    This is the core construct assumption; the paper never tests it against human mental state judgments. It enters when mA,T1 and mB,T2 are used as ground truth.
  • domain assumption Information asymmetry between role-playing LLMs induces false beliefs in an ecologically valid way.
    The ablation in Table 6 shows asymmetry increases false-belief frequency, but the human-like quality of these false beliefs is not established.
  • domain assumption SOTOPIA scenarios are representative of real-world social interactions.
    The benchmark's real-world relevance claim depends on the source of SOTOPIA's eight scenario categories.
  • domain assumption Four-option multiple-choice QA with random distractors is a reliable way to measure ToM.
    Common practice in ToM benchmarks; the paper reduces spurious correlations via random option sampling and z-statistics analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind." pith.science (2026). https://pith.science/paper/I55WMNR7

@misc{pith2026250108838,
  author       = {Pith},
  title        = {Pith review of: ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I55WMNR7}},
  note         = {Machine review of arXiv:2501.08838}
}
read the original abstract

Existing Theory of Mind (ToM) benchmarks diverge from real-world scenarios in three aspects: 1) they assess a limited range of mental states such as beliefs, 2) false beliefs are not comprehensively explored, and 3) the diverse personality traits of characters are overlooked. To address these challenges, we introduce ToMATO, a new ToM benchmark formulated as multiple-choice QA over conversations. ToMATO is generated via LLM-LLM conversations featuring information asymmetry. By employing a prompting method that requires role-playing LLMs to verbalize their thoughts before each utterance, we capture both first- and second-order mental states across five categories: belief, intention, desire, emotion, and knowledge. These verbalized thoughts serve as answers to questions designed to assess the mental states of characters within conversations. Furthermore, the information asymmetry introduced by hiding thoughts from others induces the generation of false beliefs about various mental states. Assigning distinct personality traits to LLMs further diversifies both utterances and thoughts. ToMATO consists of 5.4k questions, 753 conversations, and 15 personality trait patterns. Our analysis shows that this dataset construction approach frequently generates false beliefs due to the information asymmetry between role-playing LLMs, and effectively reflects diverse personalities. We evaluate nine LLMs on ToMATO and find that even GPT-4o mini lags behind human performance, especially in understanding false beliefs, and lacks robustness to various personality traits.

Figures

Figures reproduced from arXiv: 2501.08838 by the authors.

Figure 1
Figure 1. (a) Conversation between two role-playing LLMs with information asymmetry. Before speaking to the other, our Inner [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Statistical word-level correlation analysis (Gardner [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Statistical word-level correlation analysis (Gardner et al. 2021) on four benchmarks. Among the four, ToMATO (ours) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Conversation annotation [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: QA pair annotation [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: False belief annotation [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Human performance annotation [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Pairwise comparison [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Statistical word-level correlation analysis (Gardner et al. 2021) on the generated thoughts and utterances and the [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: z-statistics (Gardner et al. 2021) for the corre [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligence

    cs.CL 2025-05 conditional novelty 7.0 of 10

    A temporal-aware hierarchical reinforcement learning method improves a 7B LLM's social reasoning enough to rival DeepSeek-R1 and OpenAI-O3 on in-domain theory-of-mind benchmarks.

  2. MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A new benchmark shows MLLMs underperform humans on meeting Theory-of-Mind tasks, especially detecting pseudo-consensus and hidden dissent.

Reference graph

Works this paper leans on

75 extracted references · 67 canonical work pages · cited by 2 Pith papers

  1. [1]

    theory of mind

    Baron-Cohen, S.; Leslie, A. M.; and Frith, U. 1985. Does the autistic child have a “theory of mind” ? Cognition, 21(1): 37--46

    work page 1985

  2. [2]

    Beaudoin, C.; Leblanc, \'E .; Gagner, C.; and Beauchamp, M. H. 2020. Systematic review and inventory of theory of mind measures for young children. Frontiers in psychology, 10: 2905

    work page 2020

  3. [3]

    Bra \"u ner, T.; Blackburn, P.; and Polyanskaya, I. 2019. Being Deceived: Information Asymmetry in Second-Order False Belief Tasks. Topics in cognitive science

    work page 2019

  4. [4]

    T.; Li, Y.; Lundberg, S.; et al

    Bubeck, S.; Chandrasekaran, V.; Eldan, R.; Gehrke, J.; Horvitz, E.; Kamar, E.; Lee, P.; Lee, Y. T.; Li, Y.; Lundberg, S.; et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv:2303.12712

    arXiv 2023

  5. [5]

    Chen, Z.; Wu, J.; Zhou, J.; Wen, B.; Bi, G.; Jiang, G.; Cao, Y.; Hu, M.; Lai, Y.; Xiong, Z.; and Huang, M. 2024. T o MB ench: Benchmarking Theory of Mind in Large Language Models. In ACL, 15959--15983

    work page 2024

  6. [6]

    N.; Li, T.; Li, D.; Zhu, B.; Zhang, H.; Jordan, M.; Gonzalez, J

    Chiang, W.-L.; Zheng, L.; Sheng, Y.; Angelopoulos, A. N.; Li, T.; Li, D.; Zhu, B.; Zhang, H.; Jordan, M.; Gonzalez, J. E.; and Stoica, I. 2024. Chatbot Arena: An Open Platform for Evaluating LLM s by Human Preference. In ICML

    work page 2024

  7. [7]

    T.; and McCrae, R

    Costa, P. T.; and McCrae, R. R. 1980. Influence of extraversion and neuroticism on subjective well-being: happy and unhappy people. Journal of personality and social psychology, 38(4): 668

    work page 1980

  8. [8]

    L.; and Haber, N

    Cross, L.; Xiang, V.; Bhatia, A.; Yamins, D. L.; and Haber, N. 2024. Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models. arXiv:2407.07086

    arXiv 2024

Show all 75 references
  1. [9]

    De Raad, B. 2000. The big five personality factors: the psycholexical approach to personality. Hogrefe & Huber Publishers

    2000

  2. [10]

    Dettmers, T.; Pagnoni, A.; Holtzman, A.; and Zettlemoyer, L. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. In NeurIPS, 10088--10115

    2023

  3. [11]

    Dillion, D.; Tandon, N.; Gu, Y.; and Gray, K. 2023. Can AI language models replace human participants? Trends in Cognitive Sciences, 27(7): 597--600

    2023

  4. [12]

    Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783

    2024 arXiv

  5. [13]

    Fan, L.; Xu, M.; Cao, Z.; Zhu, Y.; and Zhu, S.-C. 2022. Artificial Social Intelligence: A Comparative and Holistic View. CAAI Artificial Intelligence Research, 1(2): 144--160

    2022

  6. [14]

    D.; Shwartz, V.; Sap, M.; and Choi, Y

    Forbes, M.; Hwang, J. D.; Shwartz, V.; Sap, M.; and Choi, Y. 2020. Social Chemistry 101: Learning to Reason about Social and Moral Norms. In EMNLP, 653--670

    2020

  7. [15]

    Gandhi, K.; Fr \"a nken, J.-P.; Gerstenberg, T.; and Goodman, N. 2023. Understanding Social Reasoning in Language Models with Language Models. In NeurIPS Datasets and Benchmarks Track

    2023

  8. [16]

    Gardner, M.; Merrill, W.; Dodge, J.; Peters, M.; Ross, A.; Singh, S.; and Smith, N. A. 2021. Competency Problems: On Finding and Removing Artifacts in Language Data. In EMNLP, 1801--1813

    2021

  9. [17]

    Gemma Team . 2024. Gemma

    2024

  10. [18]

    Gordon, J.; and Van Durme, B. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, 25–30

    2013

  11. [19]

    Gross, D.; and Harris, P. L. 1988. False Beliefs About Emotion: Children's Understanding of Misleading Emotional Displays. International Journal of Behavioral Development, 11(4): 475--488

    1988

  12. [20]

    Y.; Iwasawa, Y.; and Matsuo, Y

    Guo, J.; Yang, B.; Yoo, P.; Lin, B. Y.; Iwasawa, Y.; and Matsuo, Y. 2024. Suspicion Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT -4. In COLM

    2024

  13. [21]

    Guo, X.-Y.; Li, Y.-F.; and Haf, R. 2023. D e SIQ : Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding. In EMNLP, 3169--3180

    2023

  14. [22]

    Hayashi, H. 2007. YOUNG CHILDREN’S UNDERSTANDING OF SECOND-ORDER MENTAL STATES. Psychologia, 50: 15--25

    2007

  15. [23]

    He, H.; Balakrishnan, A.; Eric, M.; and Liang, P. 2017. Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings. In ACL, 1766--1776

    2017

  16. [24]

    He, H.; Chen, D.; Balakrishnan, A.; and Liang, P. 2018. Decoupling Strategy and Generation in Negotiation Dialogues. In EMNLP, 2333--2343

    2018

  17. [25]

    Hughes, C.; and Dunn, J. 1998. Understanding mind and emotion: longitudinal associations with mental-state talk between young friends. Developmental psychology, 34(5): 1026

    1998

  18. [26]

    E.; Libero, D

    Izard, C. E.; Libero, D. Z.; Putnam, P.; and Haynes, O. M. 1993. Stability of emotion experiences and their relations to traits of personality. Journal of personality and social psychology, 64(5): 847

    1993

  19. [27]

    Jiang, G.; Xu, M.; Zhu, S.-C.; Han, W.; Zhang, C.; and Zhu, Y. 2023. Evaluating and Inducing Personality in Pre-trained Language Models. In NeurIPS, 10622--10643

    2023

  20. [28]

    Jiang, H.; Zhang, X.; Cao, X.; Breazeal, C.; Roy, D.; and Kabbara, J. 2024. P ersona LLM : Investigating the Ability of Large Language Models to Express Personality Traits. In Findings of NAACL, 3605--3627

    2024

  21. [29]

    B.; and Rottenberg, J

    Kashdan, T. B.; and Rottenberg, J. 2010. Psychological flexibility as a fundamental aspect of health. Clinical psychology review, 30(7): 865--878

    2010

  22. [30]

    Kim, H.; Hessel, J.; Jiang, L.; West, P.; Lu, X.; Yu, Y.; Zhou, P.; Bras, R.; Alikhani, M.; Kim, G.; Sap, M.; and Choi, Y. 2023 a . SODA : Million-scale Dialogue Distillation with Social Commonsense Contextualization. In EMNLP, 12930--12949

    2023

  23. [31]

    Kim, H.; Sclar, M.; Zhou, X.; Bras, R.; Kim, G.; Choi, Y.; and Sap, M. 2023 b . FANT o M : A Benchmark for Stress-testing Machine Theory of Mind in Interactions. In EMNLP, 14397--14413

    2023

  24. [32]

    S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y

    Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022. Large Language Models are Zero-Shot Reasoners. In NeurIPS

    2022

  25. [33]

    Kosinski, M. 2024. Theory of mind may have spontaneously emerged in large language models. arXiv:2302.02083

    2024 arXiv

  26. [34]

    Le, M.; Boureau, Y.-L.; and Nickel, M. 2019. Revisiting the Evaluation of Theory of Mind through Question Answering. In EMNLP, 5872--5877

    2019

  27. [35]

    R.; and Prior, M

    Leekam, S. R.; and Prior, M. 1994. Can Autistic Children Distinguish Lies from Jokes? A Second Look at Second-order Belief Attribution. Journal of Child Psychology and Psychiatry, 35(5): 901--915

    1994

  28. [36]

    Lewis, M.; Yarats, D.; Dauphin, Y.; Parikh, D.; and Batra, D. 2017. Deal or No Deal? End-to-End Learning of Negotiation Dialogues. In EMNLP, 2443--2453

    2017

  29. [37]

    E.; and Diener, E

    Lucas, R. E.; and Diener, E. 2001. Understanding extraverts' enjoyment of social situations: the importance of pleasantness. Journal of personality and social psychology, 81(2): 343

    2001

  30. [38]

    Ma, Z.; Sansom, J.; Peng, R.; and Chai, J. 2023. Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models. In Findings of EMNLP, 1011--1031

    2023

  31. [39]

    Mangrulkar, S.; Gugger, S.; Debut, L.; Belkada, Y.; Paul, S.; and Bossan, B. 2022. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods

    2022

  32. [40]

    Mao, Y.; Lin, X.; Ni, Q.; and He, L. 2024. BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of Mind. In AAAI, 583--591

    2024

  33. [41]

    R.; Gosling, S

    Mehl, M. R.; Gosling, S. D.; and Pennebaker, J. W. 2006. Personality in its natural habitat: manifestations and implicit folk theories of personality in daily life. Journal of personality and social psychology, 90(5): 862

    2006

  34. [42]

    Nematzadeh, A.; Burns, K.; Grant, E.; Gopnik, A.; and Griffiths, T. 2018. Evaluating Theory of Mind in Question Answering. In EMNLP, 2392--2400

    2018

  35. [43]

    E.; and Wilson, T

    Nisbett, R. E.; and Wilson, T. D. 1977. Telling more than we can know: Verbal reports on mental processes. Psychological review, 84(3): 231

    1977

  36. [44]

    Norman, W. T. 1963. Toward an adequate taxonomy of personality attributes: Replicated factor structure in peer nomination personality ratings. The journal of abnormal and social psychology, 66(6): 574

    1963

  37. [45]

    OpenAI. 2024 a . Gpt-4 technical report. arXiv:2303.08774

    2024 arXiv

  38. [46]

    OpenAI. 2024 b . GPT-4o mini: advancing cost-efficient intelligence

    2024

  39. [47]

    John thinks that Mary thinks that…

    Perner, J.; and Wimmer, H. 1985. “John thinks that Mary thinks that…” attribution of second-order beliefs by 5- to 10-year-old children. Journal of Experimental Child Psychology, 39(3): 437--471

    1985

  40. [48]

    Premack, D.; and Woodruff, G. 1978. Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, 1(4): 515–526

    1978

  41. [49]

    This way!

    Rakoczy, H.; Warneken, F.; and Tomasello, M. 2007. “This way!”,“No! That way!”—3-year olds know that two people can have mutually incompatible desires. Cognitive Development, 22(1): 47--68

    2007

  42. [50]

    M.; and Gopnik, A

    Repacholi, B. M.; and Gopnik, A. 1997. Early reasoning about desires: evidence from 14-and 18-month-olds. Developmental psychology, 33(1): 12

    1997

  43. [51]

    R.; and Riggio, R

    Riggio, H. R.; and Riggio, R. E. 2002. Emotional expressiveness, extraversion, and neuroticism: A meta-analysis. Journal of Nonverbal Behavior, 26: 195--218

    2002

  44. [52]

    Sap, M.; Rashkin, H.; Chen, D.; Le Bras, R.; and Choi, Y. 2019. Social IQ a: Commonsense Reasoning about Social Interactions. In EMNLP, 4463--4473

    2019

  45. [53]

    Sclar, M.; Kumar, S.; West, P.; Suhr, A.; Choi, Y.; and Tsvetkov, Y. 2023. Minding Language Models ' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker. In ACL, 13960--13980

    2023

  46. [54]

    H.; Zhou, X.; Choi, Y.; Goldberg, Y.; Sap, M.; and Shwartz, V

    Shapira, N.; Levy, M.; Alavi, S. H.; Zhou, X.; Choi, Y.; Goldberg, Y.; Sap, M.; and Shwartz, V. 2024. Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models. In EACL, 2257--2273

    2024

  47. [55]

    Shapira, N.; Zwirn, G.; and Goldberg, Y. 2023. How Well Do Large Language Models Perform on Faux Pas Tests? In Findings of ACL, 10438--10451

    2023

  48. [56]

    M.; and Silber, S

    Shatz, M.; Wellman, H. M.; and Silber, S. 1983. The acquisition of mental verbs: A systematic investigation of the first reference to mental state. Cognition, 14(3): 301--321

    1983

  49. [57]

    Shinoda, K.; Sugawara, S.; and Aizawa, A. 2023. Which Shortcut Solution Do Question Answering Models Prefer to Learn? In AAAI, 13564--13572

    2023

  50. [58]

    M.; and Moore, C

    Shiverick, S. M.; and Moore, C. F. 2007. Second-order beliefs about intention and children’s attributions of sociomoral judgment. Journal of Experimental Child Psychology, 97(1): 44--60

    2007

  51. [59]

    S.; and Feigenson, L

    Smith-Flores, A. S.; and Feigenson, L. 2021. Preschoolers represent others’ false beliefs about emotions. Cognitive Development, 59: 101081

    2021

  52. [60]

    Sugawara, S.; and Tsugita, S. 2023. On Degrees of Freedom in Defining and Testing Natural Language Understanding. In Findings of ACL, 13625--13649

    2023

  53. [61]

    Ullman, T. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv:2302.08399

    2023 arXiv

  54. [62]

    Wang, X.; Shi, W.; Kim, R.; Oh, Y.; Yang, S.; Zhang, J.; and Yu, Z. 2019. Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good. In ACL, 5635--5649

    2019

  55. [63]

    Wang, Z.; and Shao, Y. 2024. Picture book reading improves children's learning understanding. British Journal of Developmental Psychology, 00: 1--24

    2024

  56. [64]

    Wimmer, H.; and Perner, J. 1983. Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children's understanding of deception. Cognition, 13(1): 103--128

    1983

  57. [65]

    Winner, E.; and Leekam, S. 1991. Distinguishing irony from deception: Understanding the speaker's second-order intention. British Journal of Developmental Psychology, 9(2): 257--270

    1991

  58. [66]

    Wu, Y.; He, Y.; Jia, Y.; Mihalcea, R.; Chen, Y.; and Deng, N. 2023. Hi- T o M : A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models. In Findings of EMNLP, 10691--10706

    2023

  59. [67]

    Xeophon. 2024. If you export your chat history from ChatGPT, you get the system prompt(s) for free, no jailbreaking or similar needed

    2024

  60. [68]

    Xu, H.; Zhao, R.; Zhu, L.; Du, J.; and He, Y. 2024. O pen T o M : A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models. In ACL, 8593--8623

    2024

  61. [69]

    Yu, W.; Jiang, Z.; Dong, Y.; and Feng, J. 2020. ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning. In ICLR

    2020

  62. [70]

    P.; Tong, E.; and Morency, L.-P

    Zadeh, A.; Chan, M.; Liang, P. P.; Tong, E.; and Morency, L.-P. 2019. Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence. In CVPR, 8799--8809

    2019

  63. [71]

    Zhou, X.; Su, Z.; Eisape, T.; Kim, H.; and Sap, M. 2024 a . Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLM s. In EMNLP, 21692--21714

    2024

  64. [72]

    Zhou, X.; Zhu, H.; Mathur, L.; Zhang, R.; Yu, H.; Qi, Z.; Morency, L.-P.; Bisk, Y.; Fried, D.; Neubig, G.; et al. 2024 b . Sotopia: Interactive evaluation for social intelligence in language agents. In ICLR

    2024

  65. [73]

    Ziems, C.; Dwivedi-Yu, J.; Wang, Y.-C.; Halevy, A.; and Yang, D. 2023. N orm B ank: A Knowledge Bank of Situational Social Norms. In ACL, 7756--7776

    2023

  66. [74]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  67. [75]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.