Pith. sign in

REVIEW 2 cited by

HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.16755 v1 pith:C7PPJU5K submitted 2023-10-25 cs.CL cs.AI

Yinghui He , Yufan Wu , Yilin Jia , Rada Mihalcea , Yulong Chen , Naihao Deng This is my paper · ORCID
classification cs.CLcs.AI
keywords higher-orderlanguagellmsmindtheorybenchmarkhi-tomlarge
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Theory of Mind (ToM) is the ability to reason about one's own and others' mental states. ToM plays a critical role in the development of intelligence, language understanding, and cognitive processes. While previous work has primarily focused on first and second-order ToM, we explore higher-order ToM, which involves recursive reasoning on others' beliefs. We introduce HI-TOM, a Higher Order Theory of Mind benchmark. Our experimental evaluation using various Large Language Models (LLMs) indicates a decline in performance on higher-order ToM tasks, demonstrating the limitations of current LLMs. We conduct a thorough analysis of different failure cases of LLMs, and share our thoughts on the implications of our findings on the future of NLP.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Decrypto Benchmark for Multi-Agent Reasoning and Theory of Mind

    cs.AI 2025-06 conditional novelty 6.0 of 10

    A new interactive language-game benchmark shows LLMs lag behind simple word-embedding baselines and that newer reasoning models regress on theory-of-mind tasks.

  2. THiNK: Can Large Language Models Think-aloud?

    cs.CL 2025-05 reject novelty 4.0 of 10

    THiNK uses a multi-agent, feedback-driven loop of problem revision and GPT-4O-based Bloom's Taxonomy scoring to measure and improve higher-order thinking in LLMs on math word problems.

Pith tools