Pith. sign in

REVIEW 4 major objections 8 minor 57 references

A world model that tracks only the physical scene will predict the wrong human action for the right-looking scene; mental state must be a core, coupled component of the model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 11:00 UTC pith:XLG4R2FO

load-bearing objection Clean formalization plus unusually careful ladder/oracle experiments; the necessity claim is real inside Menti-Bench but partly instrument-bound, and the “next stage of world modeling” rhetoric outruns the prompted-LLM setup. the 4 major comments →

arxiv 2607.27201 v1 pith:XLG4R2FO submitted 2026-07-29 cs.CL

Mental World Modeling

Hao Fei , Yiran Zhao This is my paper
classification cs.CL
keywords mental world modelingworld modelstheory of mindpartial observabilitycoupled physical-mental statesituated decision predictionLLM agentssocial intelligence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Physical world models answer what is where and how it will move. Human decisions, though, are driven by hidden mental variables—beliefs, goals, intentions, emotions, norms—so two agents in the same room can act differently because they know different things. This paper formulates Mental World Modeling (MWM): an external simulator that keeps a joint physical–mental world state, renders a first-person partial observation for a chosen target agent, and simulates how a candidate action updates both the material scene and the mental-social configuration. It implements the idea in MENTIS, a training-free, fully inspectable pipeline, and tests it on Menti-Bench, a process-annotated set of situated decisions in text, image, and sounding video. Across eight modern language-model world models, full MWM is consistently best; stripping the mental channel costs about twelve points on average, and the gains are largest precisely on interpersonal scenes where hidden minds decide the action. The remaining gap to human performance is concentrated in transition simulation—the stage that predicts how the coupled world changes.

Core claim

Explicit mental world modeling is necessary for predicting human decisions. On 448 process-annotated situated decision records, full MWM (MENTIS) reaches average final-action F1 of 87.9 across eight LLM-based world models; removing the mental channel drops performance by about 12 points, removing the physical channel by about 16, and decoupling the two transitions by about 6. The largest lift over direct answering is on interpersonal scenes, where hidden mental variables govern the choice. Oracle interventions show that most of the residual gap to human reference (98.5) lives in transition simulation, not in representing the current state.

What carries the argument

Mental World Modeling (MWM): a target-centric, action-conditioned process that maintains joint state st = (s_phy, s_ment), renders a first-person partial observation o_ϵ for a specified target, decomposes action into physical carrier and mental content, and applies a coupled transition that updates both components. MENTIS implements this as six inspectable stages—state parsing, observation generation, action decomposition, coupled physical and mental transition, branch-level value evaluation, and deterministic decision—without training.

Load-bearing premise

The hand-built test set is a fair external test of necessity, even though its gold states, observations, and per-option futures are written in the same physical–mental taxonomies and six-option branch structure that the method itself uses.

What would settle it

Re-run the necessity ladder and mental-channel ablation on a process-annotated decision set whose gold schema was written independently of MWM’s taxonomies; if full MWM no longer beats strong direct-answer baselines, or if removing the mental channel no longer costs double-digit F1—especially on interpersonal items—the necessity claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Human-facing agents (robots, tutors, care and advice systems) need joint physical–mental state, not physical dynamics alone, or they will choose the wrong action in the right-looking scene.
  • Interactive characters and social simulators should maintain explicit, inspectable belief, norm, and relationship transitions rather than prompt-level personality style.
  • Improving MWM systems should prioritize the coupled transition model first; oracle gains show that is the binding bottleneck.
  • Process faithfulness (correct state, leak-free observation, plausible successor) becomes a first-class evaluation target as outcome scores approach ceiling.
  • The same factored structure that improves prediction also makes mental-state assumptions auditable and contestable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Once transition models are learned rather than prompted, the same coupled state could support multi-step planning under false belief, not only single-step multiple choice.
  • The modality-closing effect of the typed state suggests MWM could serve as a shared interlingua between video world models and social-reasoning stacks.
  • Safety policy for assistants may be expressible as constraints on the mental transition (e.g., veto branches that update trust or consent unsafely) rather than only as output filters.
  • Counterfactual item pairs that flip one mental variable while holding the physical scene fixed would stress-test whether systems track minds or exploit schema surface form.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes Mental World Modeling (MWM), a framework that augments the POMDP-style world-model scaffold with latent mental-social state variables (beliefs, goals, intentions, emotions, norms, relations, atmosphere), so that a simulator maintains a coupled state s_t = (s^phy_t, s^ment_t), renders a target-specific partial observation, decomposes actions into physical carrier plus mental content, and simulates joint physical-mental successor states. The framework is instantiated as MENTIS, a training-free, modular prompted-LLM pipeline whose every stage emits machine-checkable JSON artifacts, and evaluated on Menti-Bench, a new 448-record process-annotated testbed (text/image/sounding-video) with gold states, observations, per-option successor states, and final actions. The central empirical claim is necessity: across 8 LLM world models from two families, full MWM (S6, avg F1 87.9) beats every rung of a structural ladder (options-only S0 through structured-state S5), removing the mental channel costs 12.1 points, removing the physical channel costs 16.5, decoupling the transitions costs 6.4, and gains are largest on interpersonal scenes (+26.4 S6−S1). Oracle interventions on one model localize most of the residual human gap to transition simulation (+3.5) and state parsing (+2.8), with sub-additive combined gains (+6.3 vs +8.7 summed) indicating propagating stage errors.

Significance. If the results hold, the paper makes two useful contributions. First, the necessity evidence is genuinely falsifiable in design: the ladder embeds standard baselines (CoT, self-consistency), every rung could have failed, the options-only floor and channel interventions rule out the cheapest shortcut explanations, and the finding replicates across eight models from two families with monotone orderings. Second, the process-complete annotation (2,688 gold successor states) plus the machine-checkable artifact requirement make this a diagnostic instrument rather than a leaderboard: the oracle cascade, per-module error taxonomy, and run-manifest discipline give the community a concrete way to localize failure in LLM-based social simulation. The conceptual novelty of the formalism itself is modest (Property 3.1 is an existence statement; Eqs. 16–17 are chain rule), but the operationalization and evaluation design are above the field's usual standard for prompt-pipeline papers. The dataset's small size and same-ontology construction bound how far the "necessity" conclusion can be exported beyond this instrument.

major comments (4)
  1. [§7.1, Table 4 (A1 ablation)] The 12.1-point cost of removing the mental channel is the strongest evidence for the necessity claim, but the ablation confounds channel removal with content removal. A1 strips mental state and mental observation outright; the comparison does not rule out that any equally extensive structured re-exposure of story evidence (additional physical detail, filler fields of matched length, or attention scaffolding from longer context) would recover part of the gap. The ladder partially mitigates this (S4 free-text state at 80.3 vs S5 structured at 82.6 isolates ~2.3 points for typing), but S4/S5 are different pipelines from A1 and do not isolate the *mental* content specifically. A content-matched control would settle it: e.g., A1 variants with (i) mental fields replaced by length-matched task-irrelevant content, (ii) mental fields populated with scrambled/cross-record values, or (iii) physical
  2. [§7.3, Table 6, Fig. 8 (scenario study)] The headline slice result — gains are largest (+26.4) "exactly where hidden mental variables govern the decision" — is partly by construction, because the interpersonal category is defined by the authors as scenes where belief/intention/emotion carry the decision, and the options were authored to differ in belief consistency, intention fit, norm compliance, and social consequence (§F.1). This makes the slice result partly a restatement of the category definitions rather than independent evidence. The domain-level flatness (19.1–22.8 across five domains) is a good check and argues against pure surface artifacts, but the category-level claim needs an independent anchor: e.g., blind human ratings of how strongly each record's decision depends on latent mental-state inference, collected without exposure to the MWM taxonomy, with the S6−S1 gain shown to correlate with that rating. Alternative
  3. [§7.1–7.4 (statistics reporting); Appendix G.6] Appendix G.6 defines the exact McNemar test and Wilson intervals, but no confidence intervals or p-values appear alongside any headline comparison (Tables 4–7, Figs. 6–9). For the full-data paired comparisons (S6 vs S5 = +5.3; S6 vs A1 = +12.1) significance is likely, but the paper makes per-model claims ("the ordering holds for each of the eight models") and slice-level claims on very small subsets (video n=28; workplace n=43; institutional n=52) where Wilson half-widths are ±7–14 points. The per-model A3 ordering (S6 > A3 for all eight) involves gaps as small as 4.2 (gpt-5.4), which on 448 paired records may or may not be significant. Please report paired McNemar results for the headline adjacent-rung and ablation comparisons per model, and add Wilson intervals to all slice tables. The machinery exists; it needs to be used where the claims are made.
  4. [§7.2, Table 5, Fig. 7 (oracle interventions)] The second contribution — "the binding bottleneck is transition simulation" — rests entirely on oracle interventions with a single model (gpt-5.6-sol). The +3.5 (O4) vs +2.8 (O1) separation is 3 records' worth of F1 on 448 items; without a paired test or a second model it is fragile. Since the claim "sets the agenda" for future work, it should either be replicated on at least one model from each family or explicitly scoped as a single-model diagnostic. Relatedly, O3 is not an oracle substitution of a predicted artifact in the same sense as O1/O2/O4: "skipping decomposition and using option text verbatim" tests whether decomposition adds error, not how much gold information helps, and the +0.7 should be interpreted accordingly in the text.
minor comments (8)
  1. [Abstract, §2.1, §3.2] Typos in the abstract/introduction: "MWM aintains", "notwhat each agent knows", "dinstinct" (§2.1), "how target will interpret the others' actions" (§3.2).
  2. [§6.3, Table 4] The human reference of 98.5 (Table 4 header) is load-bearing for the gap analysis in §7.2 but its provenance is thin: how many raters, inter-rater agreement, whether raters saw only the public scene or also annotations, and per-modality scores. Please add these details to §6.3 or Appendix F.
  3. [Appendix G.5] The RWW diagnostic (right answer / wrong world, Eq. 30) is defined and motivated as central to the inspectability argument, but no RWW values are reported for any system. Reporting RWW for S5 vs S6 (and per model) would directly support the claim that outcome gains track process faithfulness rather than shortcut use.
  4. [Appendix G.3] LLM-judge scores are stated to be "validated against human ratings on a stratified subsample" before entering headline claims, but the agreement statistics are not reported in the text provided. Please give the subsample size, correlation/agreement values, and judge model identity (Appendix G.3).
  5. [§6.3] Model identifiers need versioning detail for reproducibility: "gpt-5.6-sol" carries an unexplained suffix, and no snapshot dates or endpoint versions are given in §6.3 (the manifest contract in Appendix J suggests these exist — surface them in the main text).
  6. [§7.1 / Appendix I] The S0 floor (29.7–33.2) sits ~15 points above random; the paper attributes this to semantically weaker distractors but does not break it down. A per-category S0 analysis would quantify whether any category (e.g., object/resource) leaks more answer signal through the options alone.
  7. [Figure 5] Figure 5 (MENTIS pipeline) packs the full JSON flow, three branch lanes, and scoring into one figure; it is difficult to read at print size. Consider splitting the branch-simulation detail from the top-level flow.
  8. [Appendix M; §5.4] Appendix M is thoughtful but essayistic and duplicates §8; it could be halved without loss. Also, the sensitivity of results to K=6 fixed options (vs. free-proposal regime) is acknowledged in §5.4 but never probed; even a small K∈{4,6,8} check on the text subset would bound the forced-choice design's influence.

Circularity Check

2 steps flagged

No load-bearing circular derivation: Property 3.1 is a near-tautology, and Menti-Bench is ontology-aligned with MWM, but final-action gains over CoT/SC and ablations are not forced by definition.

specific steps
  1. self definitional [§3.2 Property 3.1, Eqs. (1)–(2)]
    "A physical-only representation, which retains only sphy_t, is insufficient whenever there exist mental components sment_t ≠ s̃ment_t such that p⋆(aϵ_t | sphy_t, sment_t) ≠ p⋆(aϵ_t | sphy_t, s̃ment_t). ... A model that observes only sphy_t cannot distinguish the two states ... so it must assign the same action distribution to both; Eq.(1) states that the true distributions differ, so no physical-only predictor can be sufficient in such cases."

    The 'necessity' of the joint state is assumed in the premise (action law depends on sment) and then restated as the conclusion. The proof sketch only unpacks that definitional dependence; it does not derive from independent dynamics that mental variables must enter the state.

  2. other [Appendix F.1 Conversion pipeline; cf. §6.1, Figs. 3–4, Appendix E]
    "the six options are constructed to be syntactically plausible while differing in physical feasibility, belief consistency, intention fit, norm compliance, or social consequence. ... The gold annotation covers the full MWM process (the joint current state s⋆_t, the target observation oϵ,⋆_t, one successor state per option, and the final action), follows the state taxonomies of Section 3.3"

    Not a fitted-input or equation identity, but the decision items are authored so distractors split on the same mental dimensions the framework treats as state variables, and process gold uses the MENTIS schemas. This makes 'mental variables are decision-relevant on Menti-Bench' partly true by item construction. It does not by itself force S6>S3 final-action F1, so it is mild instrument alignment rather than full circular proof of the headline claim.

full rationale

The paper is a framework-plus-benchmark empirical study, not a first-principles derivation that reduces a prediction to a fitted input or a self-cited uniqueness theorem. There is no parameter fitting renamed as prediction, no load-bearing self-citation chain, and no machine-unchecked uniqueness import. Property 3.1 only restates that if action distributions differ across mental components then a physical-only map is insufficient—true by the stated premise, not an independent necessity theorem. Menti-Bench options and gold process blocks are authored in the same physical/mental taxonomies MENTIS emits, which is instrument–method alignment and a validity concern, but the headline metric is final-action F1 on fixed labels: S1/S2/S3 can still reason about beliefs without emitting MWM JSON, the options-only floor is ~31%, and A1/A2/A3 can and do fail. Those gaps are empirical, not Eq.X=Eq.Y by construction. Score 2 for one minor definitional step plus mild ontology-matched test design; central empirical claims remain independently contestable.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

The work is a modeling framework plus LLM-pipeline empirics, not a theorem from minimal axioms. Load-bearing commitments are domain assumptions about mental variables as causal state, POMDP partial observability, the hand-built benchmark as a valid necessity test, and fixed evaluator weights/decision rules. No physical constants; free choices are engineering and annotation conventions that shape measured F1.

free parameters (4)
  • Branch value weights (mental consistency, physical plausibility, social appropriateness) and safety veto = Calibrated on frozen 30-record slice; medium effort + batched comparative-rank (Appendix H)
    Final selection is a weighted combination after veto (§5.5); weights and comparative-rank scoring discipline are chosen/calibrated, not derived.
  • Candidate set size K and forced multiple-choice action space = K=6
    Evaluation fixes K=6 natural-language options; free proposal is not measured, so necessity is conditional on this action space.
  • ToM inference depth L and mental-field schema cardinality = Schema per Figs. 3–4 / Tables 11–12; L left as modeling capacity
    Individual mental state fields (beliefs, goals, intentions, emotions, norms, etc.) and max ToM order are design choices that define what counts as correct state/observation.
  • Video frame-sampling budget and multimodal front-end settings = Configuration-exposed; not numerically fixed in main text
    Adaptive keyframe sampling and ASR transcription parameters affect parsed state for video items (§B.2, §7.4).
axioms (5)
  • domain assumption Beliefs, desires, intentions, affect, norms, and relations are causal latent state variables that can change the action distribution given the same physical scene (Property 3.1).
    Grounded in ToM/BDI citations (§3.1); assumed true for the necessity claim rather than independently measured outside the benchmark.
  • domain assumption Target agents act from partial observations oϵt rendered from a richer third-person joint state (POMDP scaffold).
    Standard POMDP commitment adopted in §3.3; defines observation generation and false-belief cases.
  • ad hoc to paper Actions factor into physical carrier and mental/semantic content with staged physical then mental transitions (Eqs. 16–17).
    Useful modeling factorization for inspectability; not uniquely forced by prior theory and shapes MENTIS modules.
  • domain assumption Menti-Bench gold actions are the uniquely defensible human next actions under the stated norms and scene cuts.
    Human reference 98.5 and all F1 scores presuppose gold quality after author adjudication (Appendix F).
  • ad hoc to paper Prompted LLM modules can stand in for Ω, Π, T, and value evaluation without task-specific training.
    Core of the training-free baseline; results measure scaffolded LLM behavior, not a learned world model per se.
invented entities (3)
  • Mental World Modeling (MWM) joint process no independent evidence
    purpose: Name and formalize coupled physical-mental state, target observation, and joint transition as one world-modeling law.
    Unifies existing ToM/POMDP/world-model pieces into a single target-centric simulator definition (§3.2–3.3).
  • MENTIS pipeline no independent evidence
    purpose: Training-free, inspectable reference implementation with logged artifacts for ablations and oracles.
    Operational scaffold; performance is not an independent natural phenomenon.
  • Menti-Bench process-complete dataset no independent evidence
    purpose: Provide gold st, oϵt, per-option st+1, and final action across text/image/video for necessity and bottleneck tests.
    Author-constructed diagnostic instrument aligned to MWM taxonomies; external validity unproven.

pith-pipeline@v1.2.0-grok45-kimik3 · 46140 in / 4133 out tokens · 86692 ms · 2026-07-30T11:00:06.816516+00:00 · methodology

0 comments
read the original abstract

World models enable a predictive substrate for planning and action, yet existing formulations merely answer a physical question: what/where it is, and how will it evolve. Human behavior, however, is driven by hidden mental state (what a person believes, wants, intends, feels, and considers socially permissible), so a model that tracks the physical scene but not what each agent knows and believes about it predicts the wrong action for the right-looking scene. We formulate Mental World Modeling (MWM), a generic theoretical framework that makes mental variables core components of a world model rather than posthoc rationales: MWM aintains a coupled physical-mental world state, renders a target-specific partial observation, and simulates how candidate actions jointly update both components. We instantiate the framework in MENTIS, a training-free and fully inspectable baseline that decomposes the process into state parsing, target-observation generation, action decomposition, coupled physical and mental transition, and branch-level value evaluation. On a manually constructed, quality-controlled dataset of situated decision scenarios spanning text, image, and sounding-video stories, experiments with 8 modern LLM-based world models demonstrate that explicitly modeling the mental state is essential for predicting human decisions. Deeper analyses further expose the bottlenecks of current mental world modeling. We expect MWM as a next stage of world modeling, from simulating physical scenes to simulating the minds that act in them.

Figures

Figures reproduced from arXiv: 2607.27201 by Hao Fei, Yiran Zhao.

Figure 1
Figure 1. Figure 1: Mental World Modeling (MWM) represents a scene as a coupled physical and mental world state, renders a target-specific observation, predicts candidate target actions, and simulates the next physical and mental states. Unlike physical world modeling alone, MWM explicitly tracks the unobserved beliefs, goals, intentions, emotions, relations, and norms that shape what the target agent will actually do. *Corre… view at source ↗
Figure 2
Figure 2. Figure 2: The world’s next state is also mental. Her mug is moved to the cabinet while she looks away. A world model that tracks only objects predicts the wrong action for the right-looking scene (top); one that also tracks her belief predicts what she will actually do (bottom). 1 Introduction “If the organism carries a ‘small-scale model’ of external reality and of its own possible actions within its head, it is ab… view at source ↗
Figure 3
Figure 3. Figure 3: Taxonomy of the physical world state s phy t . Entities (objects and characters) carry intrinsic and contextual attributes; relations record spatial and contact structure; the environment records time, place, region, and ambient visual, acoustic, air, and space conditions. Leaf boxes list the annotation fields (bold) with representative values (italics). Other agents need not be ignored. Their beliefs, goa… view at source ↗
Figure 4
Figure 4. Figure 4: Taxonomy of the mental world state s ment t . Individual mental entities carry identity, epistemic, motivational, affective, dispositional, normative, and constraint fields; group entities reuse the same structure at the collective level; mental relations record attitudes and role relations; a scene-level atmosphere summarizes collective mood. Leaf boxes list the annotation fields (bold) with representativ… view at source ↗
Figure 5
Figure 5. Figure 5: MENTIS pipeline. The system converts an input scene into a structured current state sˆt, renders the target pseudo-agent’s observation oˆ ϵ t , parses each option into an action branch, simulates physical and mental successor states in parallel, and evaluates the resulting futures to select the final action. The numbered regions correspond to the six stages described in Section 5.2. than relief, why a char… view at source ↗
Figure 6
Figure 6. Figure 6: Necessity ladder (final-action F1, 448 records). (a) F1 per ladder rung for the eight world models (thin lines) and their average (thick line). (b) Ablations of S6, averaged over the eight models; white dots mark the per-model values. 88 90 92 94 96 98 Final-action F1 (%), gpt-5.6-sol S6 full MWM (baseline) O3 gold action O2 gold observation O1 gold state O4 gold transition O1+O2 O2+O4 O1+O4 O1+O2+O4 all f… view at source ↗
Figure 7
Figure 7. Figure 7: Oracle interventions (gpt-5.6-sol). Bars show final-action F1 when gold information replaces predicted artifacts at one or more stages; bar labels give the gain over fully predictive S6. F1 increases at every rung, for every model. The average rises monotonically from 31.3 (S0) to 87.9 (S6), and the ordering holds for each of the models. The two largest increments come from reading the story (+32.0, S0→S1)… view at source ↗
Figure 8
Figure 8. Figure 8: Scenario study (gpt-5.6-sol). (a) F1 by scene category for four systems. (b) S6−S1 gain by scene category and by domain. 82.6) adds further. The largest modeling increment (+5.3) then comes from the final rung, observation rendering, branch simulation, and value evaluation. The conclusion is that both halves of MENTIS contribute: the state representation itself, and the simulation machinery built on top of… view at source ↗
Figure 9
Figure 9. Figure 9: Modality analysis (gpt-5.6-sol). (a) F1 by story modality per ladder system. (b) Effect of replacing images with captions. (c) F1 under video channel interventions [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Composition of Menti-Bench (448 records). (a) Scene categories and (b) domain categories, stacked by story modality; (c) gold answers are near-uniform over the six option letters (dashed line: exact uniformity), leaving no position prior to exploit [PITH_FULL_IMAGE:figures/full_fig_p032_10.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 15 linked inside Pith

  1. [1]

    Kenneth J. W. Craik. The Nature of Explanation. Cambridge University Press, Cambridge, 1943

    1943

  2. [2]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020

    2020

  3. [3]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022

    2022

  5. [5]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36: 0 38154--38180, 2023

    2023

  6. [6]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 2609--2634, 2023 a

    2023

  7. [7]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. In International Conference on Machine Learning, pages 53366--53397. PMLR, 2024

    2024

  8. [8]

    Voyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=ehfRiF0R3a

    2024

  9. [9]

    O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S

    Joon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), 2023

    2023

  10. [10]

    World models

    David Ha and J \"u rgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018. URL https://arxiv.org/abs/1803.10122

    Pith/arXiv arXiv 2018

  11. [11]

    A path towards autonomous machine intelligence

    Yann LeCun. A path towards autonomous machine intelligence. OpenReview, 2022. URL https://openreview.net/forum?id=BZ5a1r-kVsf

    2022

  12. [12]

    Mastering diverse control tasks through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640: 0 647--653, 2025 a

    2025

  13. [13]

    Video generation models as world simulators

    OpenAI . Video generation models as world simulators. https://openai.com/index/video-generation-models-as-world-simulators/, 2024. Accessed 2026-07-02

    2024

  14. [14]

    Genie: Generative interactive environments

    Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Si...

    Pith/arXiv arXiv 2024

  15. [15]

    Genie 3: A new frontier for world models

    Jack Parker-Holder and Shlomi Fruchter. Genie 3: A new frontier for world models. https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/, 2025. Accessed 2026-07-02

    2025

  16. [16]

    Marble: A multimodal world model

    World Labs . Marble: A multimodal world model. https://www.worldlabs.ai/blog/marble-world-model, 2025. Accessed 2026-07-02

    2025

  17. [17]

    Johnson-Laird

    Philip N. Johnson-Laird. Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Harvard University Press, Cambridge, MA, 1983

    1983

  18. [18]

    Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, 1 0 (4): 0 515--526, 1978

    David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, 1 0 (4): 0 515--526, 1978. doi:10.1017/S0140525X00076512

    doi:10.1017/s0140525x00076512 1978

  19. [19]

    Michael E. Bratman. Intention, Plans, and Practical Reason. Harvard University Press, Cambridge, MA, 1987

    1987

  20. [20]

    Varela, Evan Thompson, and Eleanor Rosch

    Francisco J. Varela, Evan Thompson, and Eleanor Rosch. The Embodied Mind: Cognitive Science and Human Experience. MIT Press, Cambridge, MA, 1991

    1991

  21. [21]

    Social IQ a: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQ a: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4463--4473, 2019. URL https://aclanthology.org/D19-1454/

    2019

  22. [22]

    ToMBench : Benchmarking theory of mind in large language models

    Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang. ToMBench : Benchmarking theory of mind in large language models. arXiv preprint arXiv:2402.15052, 2024. URL https://arxiv.org/abs/2402.15052

    Pith/arXiv arXiv 2024

  23. [23]

    Large language models fail on trivial alterations to theory-of-mind tasks

    Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399, 2023. URL https://arxiv.org/abs/2302.08399

    Pith/arXiv arXiv 2023

  24. [24]

    Littman, and Anthony R

    Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101 0 (1--2): 0 99--134, 1998. doi:10.1016/S0004-3702(98)00023-X

    doi:10.1016/s0004-3702(98)00023-x 1998

  25. [25]

    World models: A comprehensive survey of architectures, methodologies, reasoning paradigms, and applications

    Arif Hassan Zidan et al. World models: A comprehensive survey of architectures, methodologies, reasoning paradigms, and applications. arXiv preprint arXiv:2606.00133, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    V-JEPA 2: Self-supervised video models enable understanding, prediction and planning

    Meta AI . V-JEPA 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Training agents inside of scalable world models

    Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models. arXiv preprint arXiv:2509.24527, 2025 b

    Pith/arXiv arXiv 2025

  28. [28]

    Sora 2 system card

    OpenAI. Sora 2 system card. https://openai.com/index/sora-2-system-card/, 2025

    2025

  29. [29]

    Cosmos world foundation model platform for physical AI

    NVIDIA. Cosmos world foundation model platform for physical AI . arXiv preprint arXiv:2501.03575, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    GAIA-2 : A controllable multi-view generative world model for autonomous driving

    Wayve. GAIA-2 : A controllable multi-view generative world model for autonomous driving. arXiv preprint arXiv:2503.20523, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    NVIDIA OmniDreams : Real-time generative world model for closed-loop autonomous vehicle simulation

    NVIDIA. NVIDIA OmniDreams : Real-time generative world model for closed-loop autonomous vehicle simulation. arXiv preprint arXiv:2606.03159, 2026

    Pith/arXiv arXiv 2026

  32. [32]

    Matrix- Game 3.0: Real-time and streaming interactive world model with long-horizon memory

    Skywork AI . Matrix- Game 3.0: Real-time and streaming interactive world model with long-horizon memory. arXiv preprint arXiv:2604.08995, 2026

    Pith/arXiv arXiv 2026

  33. [33]

    HunyuanWorld 1.0: Generating immersive, explorable, and interactive 3D worlds from words or pixels

    Tencent Hunyuan . HunyuanWorld 1.0: Generating immersive, explorable, and interactive 3D worlds from words or pixels. arXiv preprint arXiv:2507.21809, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    HY-World 2.0: A multi-modal world model for reconstructing, generating, and simulating 3D worlds

    Tencent Hunyuan . HY-World 2.0: A multi-modal world model for reconstructing, generating, and simulating 3D worlds. arXiv preprint arXiv:2604.14268, 2026

    Pith/arXiv arXiv 2026

  35. [35]

    3D and 4D world modeling: A survey

    Lingdong Kong, Wesley Yang, Jianbiao Mei, Youquan Liu, Ao Liang, Dekai Zhu, et al. 3D and 4D world modeling: A survey. arXiv preprint arXiv:2509.07996, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children's understanding of deception

    Heinz Wimmer and Josef Perner. Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children's understanding of deception. Cognition, 13 0 (1): 0 103--128, 1983. doi:10.1016/0010-0277(83)90004-5

    doi:10.1016/0010-0277(83)90004-5 1983

  37. [37]

    Leslie, and Uta Frith

    Simon Baron-Cohen, Alan M. Leslie, and Uta Frith. Does the autistic child have a ``theory of mind''? Cognition, 21 0 (1): 0 37--46, 1985. doi:10.1016/0010-0277(85)90022-8

    doi:10.1016/0010-0277(85)90022-8 1985

  38. [38]

    Rosalind W. Picard. Affective Computing. MIT Press, 1997

    1997

  39. [39]

    Rao and Michael P

    Anand S. Rao and Michael P. Georgeff. BDI agents: From theory to practice. In Proceedings of the First International Conference on Multi-Agent Systems, pages 312--319, 1995

    1995

  40. [40]

    Rabinowitz, Frank Perbet, H

    Neil C. Rabinowitz, Frank Perbet, H. Francis Song, Chiyuan Zhang, S. M. Ali Eslami, and Matthew Botvinick. Machine theory of mind. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4218--4227. PMLR, 2018. URL https://proceedings.mlr.press/v80/rabinowitz18a.html

    2018

  41. [41]

    Revisiting the evaluation of theory of mind through question answering

    Matthew Le, Y-Lan Boureau, and Maximilian Nickel. Revisiting the evaluation of theory of mind through question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019

    2019

  42. [42]

    FANToM : A benchmark for stress-testing machine theory of mind in interactions

    Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. FANToM : A benchmark for stress-testing machine theory of mind in interactions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  43. [43]

    SOTOPIA : Interactive evaluation for social intelligence in language agents

    Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. SOTOPIA : Interactive evaluation for social intelligence in language agents. In International Conference on Learning Representations, 2024

    2024

  44. [44]

    Evaluating large language models in theory of mind tasks

    Michal Kosinski. Evaluating large language models in theory of mind tasks. Proceedings of the National Academy of Sciences, 121 0 (45): 0 e2405460121, 2024. doi:10.1073/pnas.2405460121

    doi:10.1073/pnas.2405460121 2024

  45. [45]

    Learning latent dynamics for planning from pixels

    Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2555--2565. PMLR, 2019. URL https://proceedings.mlr.press/v97/haf...

    2019

  46. [46]

    Gmytrasiewicz and Prashant Doshi

    Piotr J. Gmytrasiewicz and Prashant Doshi. A framework for sequential planning in multi-agent settings. Journal of Artificial Intelligence Research, 24: 0 49--79, 2005. doi:10.1613/jair.1579

    doi:10.1613/jair.1579 2005

  47. [47]

    Reasoning with language model is planning with world model

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  48. [48]

    Social world models

    Xuhui Zhou, Jiarui Liu, Akhila Yerukola, Hyunwoo Kim, and Maarten Sap. Social world models. arXiv preprint arXiv:2509.00559, 2025. URL https://arxiv.org/abs/2509.00559

    arXiv 2025

  49. [49]

    Lake, Tomer D

    Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40: 0 e253, 2017. doi:10.1017/S0140525X16001837

    doi:10.1017/s0140525x16001837 2017

  50. [50]

    The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 11 0 (2): 0 127--138, 2010

    Karl Friston. The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 11 0 (2): 0 127--138, 2010. doi:10.1038/nrn2787

    doi:10.1038/nrn2787 2010

  51. [51]

    James J. Gibson. The Ecological Approach to Visual Perception. Houghton Mifflin, Boston, 1979

    1979

  52. [52]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM -as-a-judge with MT-Bench and Chatbot Arena . In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2023

    2023

  53. [53]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022

    2022

  54. [54]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023 b

    2023

  55. [55]

    The Claude 3 model family: Opus , Sonnet , Haiku

    Anthropic. The Claude 3 model family: Opus , Sonnet , Haiku . Model card, Anthropic, 2024

    2024

  56. [56]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023

    2023

  57. [57]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, 2023

    2023