Pith. sign in

REVIEW 4 major objections 6 minor 9 cited by

Embodied AI Agents: Modeling the World

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that embodied AI agents should be built around predictive world models of the physical environment and of the user's mental state, making planning and human-agent collaboration the central design goal.

desk verdict A useful but thin Meta roadmap for embodied AI via world models; the framing is fine, the unsupported benchmarks and untested central premise keep it from being a strong paper. read the letter →

arxiv 2506.22355 v3 pith:P3QXPPJP submitted 2025-06-27 cs.AI

Pascale Fung , Yoram Bachrach , Asli Celikyilmaz , Kamalika Chaudhuri , Delong Chen , Willy Chung , Emmanuel Dupoux , Hongyu Gong
show 13 more authors
Hervé Jégou Alessandro Lazaric Arjun Majumdar Andrea Madotto Franziska Meier Florian Metze Louis-Philippe Morency Théo Moutakanni Juan Pino Basile Terver Joseph Tighe Paden Tomasello Jitendra Malik
This is my paper
classification cs.AI
keywords embodiedAIagentsworldmodelsmentalplanningandreasoningmultimodalperceptiontheoryofmindwearableautonomous
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Embodied AI agents—virtual avatars, wearables such as smart glasses, and robots—need more than large language models that predict the next token or pixel. The paper argues that their core should be a world model: an internal predictive representation of the physical environment that lets an agent imagine how the world will evolve under candidate actions, score those futures, and plan the lowest-cost sequence. In addition, the paper proposes a mental world model of the user, capturing beliefs, goals, intentions, and emotions, so that agents can anticipate needs and collaborate rather than merely react. If this framework is right, the road to capable embodied AI runs through learning predictive world models from unlabelled video and through benchmarking physical and social reasoning, where current models still fall far short of human performance.

What carries the argument

The load-bearing mechanism is the predictive world model, defined as a learned module that forecasts future world states under candidate actions without regenerating raw sensory detail. Two instantiations carry the argument: a joint-embedding predictive world model that imagines futures in a compressed latent space and supports gradient-based or sampling-based planning via a cost function, and a vision-language world model (VLWM) that predicts interleaved language descriptions of actions and resulting states from unlabelled video, giving an interpretable planning substrate. The same predictive logic is extended to a mental world model of the user—beliefs, goals, intentions, emotions—so the agent can anticipate, not just respond. Memory supplies the third pillar: the paper distinguishes fixed weights, working memory such as the KV cache, and external retrieval, and argues for a scalable episodic memory so the world model can grow with interaction.

What would settle it

Train a joint-embedding world model and a generative video model on identical unlabelled egocentric video, then compare zero-shot accuracy on the WorldPrediction-WM task; if the joint-embedding model does not beat both the generative baseline and a prompted vision-language model, the paper's central claim that abstract predictive world models are the right substrate for planning is not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that a world model—a learned representation of how the environment and the people in it behave—should be the organizing component of embodied AI. Physically, the model captures objects, their spatial relations, dynamics, and the causal effects of actions; for planning, the agent rolls the model forward under alternative actions, applies a cost or distance-to-goal function to the imagined futures, and executes only the first step of the best plan before re-planning. Socially, a mental world model represents a user's attention, goals, emotions, beliefs, and beliefs about others, enabling theory-of-mind reasoning for assistance, tutoring, and multi-agent collaboration. The paper contrasts this predictive approach with generative models that recreate every pixel, arguing that joint-embedding world models which predict in abstract space are more efficient and stable, and with language-planning LLMs that hallucinate; it reports benchmarks where state-of-the-art models reach 40.2% on physical video reasoning versus 92.9% for humans, and 57% and 38% on procedural planning versus near-perfect human performance, framing the gap as the motivating problem.

Load-bearing premise

The framework stands on the premise that predictive world models trained on unlabelled video capture enough causal structure to plan in novel environments, a premise the paper itself flags as open in Section 3.2.

Editorial extensions

If this is right

  • Embodied agents should be designed around predictive world models rather than prompted LLM or VLM planners, since the paper reports that language-planned actions hallucinate and that generative models are inefficient for long-horizon planning.
  • Wearable assistants can infer user goals from egocentric context: the goal-inference benchmark shows the best VLM reaches 84% multiple-choice accuracy against 93% for humans, making proactive assistance a concrete target.
  • A mental world model of the user enables theory-of-mind behaviours such as detecting belief discrepancies, anticipating goals, and predicting emotional responses, which the paper argues is required for tutoring and collaboration.
  • Robotic agents should combine learned policies, task-agnostic world models, and cost or scoring models, enabling re-planning when a vision-language-action policy's predicted action would fail.
  • World-model progress can be measured on physical and procedural benchmarks such as Minimal Video Pairs, IntPhys2, CausalVQA, and WorldPrediction, where current models remain far below human performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's bet implies that large-scale egocentric video, not text or annotated robotics data, may be the highest-value training resource for embodied intelligence; this is an extrapolation, since the paper describes the approach but does not demonstrate it at scale.
  • The mental world model could be the bridge to multi-agent negotiation: if each agent models the others' beliefs and goals, conflicts could be resolved through prediction rather than hand-coded protocols; the paper lists multi-agent collaboration as future work rather than a solved claim.
  • A testable extension would measure whether goal-inference accuracy improves when a joint-embedding world model is used to filter or score candidate goals against predicted futures, isolating the causal-abstraction advantage from raw VLM pattern matching.
  • Persistent episodic memory plus a mental world model raises a privacy tension the paper acknowledges but does not resolve: the same representations that enable personalization can encode intimate user data, so privacy-preserving training needs to be co-designed with world-model learning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This manuscript is a position and overview paper from a large industrial research group. It argues that embodied AI agents—virtual avatars, wearables, and robots—should be built around world models, which integrate multimodal perception, planning, and memory, and additionally around a 'mental world model' of the user for human-agent collaboration. It surveys three agent types, proposes architectures (e.g., VLWM, dyadic motion models, world-model-based robot planning), introduces several benchmarks (MVP, IntPhys2, CausalVQA, WorldPrediction, Goal Inference), and discusses future directions (embodied learning, multi-agent interaction, ethics). The central claim is that predictive world models are essential for reasoning and planning in embodied agents.

Significance. If the central thesis is correct, the paper provides a useful organizing framework for embodied-AI research and highlights evaluation gaps; the benchmark designs, especially MVP and WorldPrediction, are valuable in exposing shortcut exploitation and the large headroom between current VLM/LLM performance and human performance. The paper is also candid about limitations, explicitly noting in Section 3.2 that joint-embedding world models are only useful if their abstraction matches the causal structure of the task. However, the manuscript offers no experimental validation of the proposed framework: the few quantitative claims are presented without protocols or artifacts, and the benchmark results largely quantify failures of current models rather than successes of the proposed architecture. As a position statement it is coherent, but as a technical claim it is not yet supported.

major comments (4)
  1. [5.2] The VLWM evaluation is not reproducible as reported: the section claims +20% SR, +10% mAcc, and +4% mIoU on 'VPA benchmarks' and preference in PlannerArena, but gives no definition of VPA, no baseline configuration, no dataset reference, no error bars, and no artifact link. Because these numbers are the manuscript's only direct quantitative evidence that the proposed predictive world-model architecture improves planning, the claim is load-bearing and must either be backed by a published protocol or explicitly reframed as preliminary and non-evaluative.
  2. [3.2 / 3.6] The central premise—that joint-embedding predictive world models capture causal structure for planning in novel environments—is stated in Sections 1 and 3 but conceded in Section 3.2 to depend on 'how well that abstraction captures the causal structure of the task.' The benchmarks in Section 3.6 (MVP 40.2% vs 92.9% human; WorldPrediction 57%/38% vs near-perfect humans; IntPhys2 at chance) quantify the gap between current models and humans but provide no evidence that the proposed latent or linguistic world-model abstraction closes it. The paper should formulate a concrete, falsifiable research milestone (e.g., a specified target on WorldPrediction or a controlled zero-shot planning experiment comparing VLWM/JEPA against behavior cloning) so that the framework's central claim is testable.
  3. [3.3] The mental world model is a core component of the proposed framework, but the section provides no operational definition, no evaluation protocol, and no experiment; the cited external benchmarks (ToMi, Hi-ToM, ExploreToM) demonstrate limitations of current models rather than benefits of the proposed mental world model. The claim that mental world models 'proactively and strategically plan' user guidance is therefore unsupported as stated and should be presented with a concrete evaluation proposal or removed from the load-bearing argument.
  4. [3.6 / 5.3] The manuscript's evidence relies heavily on the authors' own unpublished or self-published benchmarks (MVP, IntPhys2, CausalVQA, WorldPrediction, Goal Inference, VPA/PlannerArena). This is not circular reasoning, but it raises a verification burden: the reader cannot assess whether these benchmarks meet the stated design goals (e.g., shortcut avoidance) without the full protocols, data releases, and evaluation code. The revision should either include an appendix with benchmark statistics, artifact links, and access conditions, or explicitly state that these are internal evaluations that have not yet been independently validated.
minor comments (6)
  1. [2.1 / 4.3] There are several spacing and typographical errors, including 'Seamless Ineraction dataset' and 'over4, 000 hours'; these should be corrected.
  2. [5.2] The phrases 'withoutrequiring annotatedsupervision' and 'interleaved natural language sequences' contain missing spaces and should be cleaned up.
  3. [5.3.2] The text 'pixel-level continuit' should read 'pixel-level continuity'.
  4. [6.2] The term 'kyno-dynamic planning' should be spelled 'kino-dynamic planning'; also, 'extract extractions' in Section 6.2.1 should be reworded.
  5. [7] The subheadings 'SystemAHelpingSystemB' and 'SystemBHelpingSystemA' need spaces for readability, and the table cell 'Not done to our knowledge' should be clarified as an open research gap.
  6. [9] The text 'Tomitigateethicalissues' at the start of a paragraph should be split into 'To mitigate ethical issues'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the central claim is a position proposal, and its supporting benchmarks, while partly self-cited, are externally checkable rather than fitted or definitionally forced.

full rationale

The paper is a position/overview, not a derivation chain. The load-bearing claim—that world models should be central to embodied-agent reasoning and planning—is argued qualitatively (Section 3, Sections 3.2–3.3) and motivated by empirical gaps, not derived from a fitted parameter or from an earlier self-citation that encodes the conclusion. The cited benchmarks (MVP, IntPhys2, CausalVQA, WorldPrediction) are external, human-validated evaluations; even where WorldPrediction (Chen et al., 2025) and IntPhys2 (Bordes et al., 2025) involve current authors, their results are falsifiable measurements with human baselines, so they do not reduce to the paper's assertion by construction. Section 3.2 explicitly concedes the key assumption: 'the usefulness of joint-embedding world models depends on how well that abstraction captures the causal structure of the task,' and Section 5.2 reports VLWM gains on 'VPA benchmarks' and PlannerArena without providing a protocol or artifact. That is a missing-support problem for the paper's evidence, but no equation or definition forces the VLWM numbers from the training objective; the report is an unverified empirical claim, not a circular one. Weighing the manuscript's own caveats and missing evaluations, there is no circular step that meets the quote-and-reduction standard, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 2 invented entities

The paper introduces no fitted parameters; its numbers are borrowed from self-authored benchmarks. The axioms are the domain assumptions on which the proposal rests. The mental world model and episodic memory are new conceptual constructs without external falsifiable handles.

assumptions (6)
  • domain assumption Embodiment is beneficial for AI agents because it enables physical interaction and fosters user trust.
    Introduction, Section 1, second paragraph: 'Embodiment serves two primary purposes...' This is a foundational assumption for the whole paper, but no comparative evidence is given in this paper.
  • domain assumption World models are necessary for reasoning and planning in embodied agents.
    Section 3, opening: 'World modeling is essential for embodied AI agents to understand and interact with their environment effectively.' The paper asserts this centrality.
  • domain assumption Generative models trained to predict the next token or pixel are inefficient and prone to hallucination for planning.
    Section 1, paragraph 5: 'one fundamental flaw of generative models is the inefficiency in their model size.' No quantitative comparison is provided.
  • domain assumption A mental world model can be represented as textual descriptions of the user's attention, memory, intentions, emotions, and beliefs.
    Section 3.3: 'the mental state Xm,s of a subject s in X consists of textual descriptions...' This is a strong simplification of Theory of Mind.
  • domain assumption High-level world models can be learned from unlabelled egocentric video.
    Section 5.2: 'VLWM is trained to condition on visual context and generate plausible procedural futures in language, without requiring annotated supervision.' This is the feasibility premise.
  • domain assumption The benchmarks cited (MVP, IntPhys2, CausalVQA, WorldPrediction) are valid measures of world modeling ability.
    Sections 3.6.1 through 3.6.4 use these benchmarks as evidence, but their validity is assumed.
invented entities (2)
  • Mental world model
    purpose: AI agent's representation of a human user's beliefs, goals, intentions, emotions, and attention
    Proposed as a core component for human-agent collaboration, but no falsifiable measurement or external validation is provided; the paper only lists dimensions that 'can be flexibly adapted' (Section 3.3).
  • Episodic memory
    purpose: A scalable memory that grows sublinearly with interaction time for world models
    Proposed in Section 3.5.2 as a 'new form of memory' but no implementation or external validation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Embodied AI Agents: Modeling the World." pith.science (2026). https://pith.science/paper/P3QXPPJP

@misc{pith2026250622355,
  author       = {Pith},
  title        = {Pith review of: Embodied AI Agents: Modeling the World},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P3QXPPJP}},
  note         = {Machine review of arXiv:2506.22355}
}
read the original abstract

This paper describes our research on AI agents embodied in visual, virtual or physical forms, enabling them to interact with both users and their environments. These agents, which include virtual avatars, wearable devices, and robots, are designed to perceive, learn and act within their surroundings, which makes them more similar to how humans learn and interact with the environments as compared to disembodied agents. We propose that the development of world models is central to reasoning and planning of embodied AI agents, allowing these agents to understand and predict their environment, to understand user intentions and social contexts, thereby enhancing their ability to perform complex tasks autonomously. World modeling encompasses the integration of multimodal perception, planning through reasoning for action and control, and memory to create a comprehensive understanding of the physical world. Beyond the physical world, we also propose to learn the mental world model of users to enable better human-agent collaboration.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ViSAGE: Constructing Self-Correcting Memories for Long-Form Video Understanding

    cs.AI 2026-07 conditional novelty 6.0 of 10

    ViSAGE builds entity-centered, self-correcting memories for long-form video understanding and reports state-of-the-art accuracy on M3-Bench and Video-MME-long.

  2. SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios

    cs.CV 2025-11 conditional novelty 6.0 of 10

    SWITCH introduces a 193-video benchmark of tangible control-interface interactions and shows that frontier LMMMs struggle with fine-grained grounding and outcome verification.

  3. A Comprehensive Survey on World Models for Embodied AI

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A unified three-axis taxonomy — functionality, temporal modeling, spatial representation — organizes the world-model literature for embodied AI.

  4. Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details

    cs.AI 2026-08 conditional novelty 5.0 of 10

    For Other-Play in Yokai, agents trained with different implementation details coordinate across implementations about as well as across seeds, supporting inter-seed cross-play as a proxy for cross-implementation evaluation.

  5. 3D and 4D World Modeling: A Survey

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A survey that defines 3D/4D world modeling, organizes methods into VideoGen, OccGen, and LiDARGen categories, and compiles datasets, metrics, and benchmark numbers.

  6. How to Build Digital Humans? From Priors to Photorealistic Avatars

    cs.GR 2026-07 accept novelty 4.0 of 10

    A taxonomy-driven state-of-the-art report that structures controllable 3D human avatar creation around prior learning and personalization, reviewing full-body, head, and layered (hair/hands/garments) methods.

  7. OpenWorldLib: A Unified Codebase and Definition of Advanced World Models

    cs.CV 2026-04 unverdicted novelty 4.0 of 10

    OpenWorldLib offers a standardized codebase and definition for world models that combine perception, interaction, and memory to understand and predict the world.

  8. Embodied AI: Emerging Risks and Opportunities for Policy Action

    cs.CY 2025-08 conditional novelty 4.0 of 10

    A policy analysis arguing that embodied AI risks are real, under-covered by current US/EU/UK frameworks, and best handled through certification, benchmarks, clarified liability, and economic adaptation.

  9. Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systems

    cs.AI 2025-07 conditional novelty 4.0 of 10

    A position paper arguing that Bayesian inference could become a key design principle for embodied AI in open physical worlds, using Sutton's search-and-learning lens to explain its current absence.

Reference graph

Works this paper leans on

50 extracted references · 18 canonical work pages · cited by 9 Pith papers

  1. [1]

    Seamless interaction: Dyadic audiovisual motion modeling and large-scale dataset

    Vasu Agrawal, Akinniyi Akinyemi, Kathryn Alvero, Morteza Behrooz, Julia Buffalini, Fabio Maria Carlucci, Joy Chen, Junming Chen, Zhang Chen, Shiyang Cheng, Praveen Chowdary, Joe Chuang, Antony D’Avirro, Jon Daly, Ning Dong, Mark Duppenthaler, Cynthia Gao, Jeff Girard, Martin Gleize, Sahir Gomez, Hongyu Gong, Srivathsan Govindarajan, Brandon Han, Sen He, D...

    arXiv 2025

  2. [5]

    Intphys 2: Benchmarking intuitive physics understanding in complex synthetic environments.arXiv preprint arXiv:2506.09849,

    Florian Bordes, Quentin Garrido, Justine T Kao, Adina Williams, Michael Rabbat, and Emmanuel Dupoux. Intphys 2: Benchmarking intuitive physics understanding in complex synthetic environments.arXiv preprint arXiv:2506.09849,

    arXiv

  3. [9]

    The mit humanoid robot: Design, motion planning, and control for acrobatic behaviors

    Matthew Chignoli, Donghyun Kim, Elijah Stanger-Jones, and Sangbae Kim. The mit humanoid robot: Design, motion planning, and control for acrobatic behaviors. In2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids), pages 1–8. doi: 10.1109/HUMANOIDS47582.2021.9555782. Enrique Coronado, Takuya Kiyokawa, Gustavo A Garcia Ricardez, Ixchel ...

    arXiv

  4. [10]

    BERT: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings 37 of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V...

    work page 2019

  5. [13]

    Calibrating noise to sensitivity in private data analysis

    Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7,

    2006

  6. [16]

    Robbie: Robust bias evaluation of large generative language models

    David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi-Yu, Eleonora Presani, Adina Williams, and Eric Michael Smith. Robbie: Robust bias evaluation of large generative language models. In The 2023 Conference on Empirical Methods in Natural Language Processing,

    work page 2023

  7. [18]

    Towards empathetic human-robot interactions

    Pascale Fung, Dario Bertero, Yan Wan, Anik Dey, Ricky Ho Yin Chan, Farhad Bin Siddique, Yang Yang, Chien-Sheng Wu, and Ruixi Lin. Towards empathetic human-robot interactions. InComputational Linguistics and Intelligent Text Processing: 17th International Conference, CICLing 2016, Konya, Turkey, April 3–9, 2016, Revised Selected Papers, Part II 17, pages 1...

    work page 2016

  8. [19]

    Sparsh: Self-supervised touch representations for vision-based tactile sensing

    Carolina Higuera, Akash Sharma, Chaithanya Krishna Bodduluri, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishnan, Michael Kaess, Byron Boots, Mike Lambeta, Tingfan Wu, and Mustafa Mukadam. Sparsh: Self-supervised touch representations for vision-based tactile sensing. 2024.https://openreview.net/forum?id=xYJn2e1uu8. Felix Hill, Sona Mokra, Nathaniel Wong,...

    arXiv 2024

Show all 50 references
  1. [20]

    Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al

    doi: 10.1109/IROS.2016.7758092. Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. pi0.5: a vision-language-action model with open-world generalization. arXiv preprint a...

    2016

  2. [22]

    Dino-foresight: Looking into the future with dino.CoRR, abs/2412.11673, 2024.https://doi.org/10.48550/arXiv.2412.11673

    Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. Dino-foresight: Looking into the future with dino.CoRR, abs/2412.11673, 2024.https://doi.org/10.48550/arXiv.2412.11673. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishn...

    2024 doi

  3. [23]

    A shortcut-aware video-qa benchmark for physical understanding via minimal video pairs.arXiv preprint arXiv:2506.09987,

    Benno Krojer, Mojtaba Komeili, Candace Ross, Quentin Garrido, Koustuv Sinha, Nicolas Ballas, and Mahmoud Assran. A shortcut-aware video-qa benchmark for physical understanding via minimal video pairs.arXiv preprint arXiv:2506.09987,

    arXiv

  4. [24]

    Matthew Le, Y-Lan Boureau, and Maximilian Nickel

    ISBN 0521862051. Matthew Le, Y-Lan Boureau, and Maximilian Nickel. Revisiting the evaluation of theory of mind through question answering. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat...

    2019

  5. [25]

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra

    https: //openreview.net/forum?id=LZh48DTg71. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971,

    arXiv

  6. [26]

    Yuxin Liu and Keng L Siau

    https://proceedings.neurips.cc/paper_files/paper/2023/file/ 8c3c666820ea055a77726d66fc7d447f-Paper-Datasets_and_Benchmarks.pdf. Yuxin Liu and Keng L Siau. Human-ai interaction and ai avatars. InInternational Conference on Human-Computer Interaction, pages 120–130. Springer,

    2023

  7. [30]

    Ellma-t: an embodied llm-agent for supporting english language learning in social vr.arXiv preprint arXiv:2410.02406,

    Mengxu Pan, Alexandra Kitson, Hongyu Wan, and Mirjana Prpa. Ellma-t: an embodied llm-agent for supporting english language learning in social vr.arXiv preprint arXiv:2410.02406,

    arXiv

  8. [31]

    Position: Episodic memory is the missing piece for long-term LLM agents.arXiv preprint arXiv:2502.06975,

    Mathis Pink, Qinyuan Wu, Vy Ai Vo, Javier Turek, Jianing Mu, Alexander Huth, and Mariya Toneva. Position: Episodic memory is the missing piece for long-term LLM agents.arXiv preprint arXiv:2502.06975,

    arXiv

  9. [32]

    Perturbation augmentation for fairer nlp

    Rebecca Qian, Candace Ross, Jude Fernandes, Eric Michael Smith, Douwe Kiela, and Adina Williams. Perturbation augmentation for fairer nlp. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9496–9521,

    2022

  10. [33]

    Intphys 2019: A benchmark for visual intuitive physics understanding.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5016–5025,

    41 Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys 2019: A benchmark for visual intuitive physics understanding.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5016–5025,

    2019

  11. [34]

    Yara Rizk, Mariette Awad, and Edward W Tunstel

    doi: 10.1109/TPAMI.2021.3083839. Yara Rizk, Mariette Awad, and Edward W Tunstel. Cooperative heterogeneous multi-robot systems: A survey.ACM Computing Surveys (CSUR), 52(2):1–31,

    2021

  12. [37]

    Smolvla: A vision-language-action model for affordable and efficient robotics.arXiv preprint arXiv:2506.01844,

    Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine, Michel Aractingi, Caroline Pascal, Martino Russi, Andres Marafioti, et al. Smolvla: A vision-language-action model for affordable and efficient robotics.arXiv preprint arXiv:2506.01844,

    arXiv

  13. [38]

    Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage

    Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, et al. Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188,

    arXiv

  14. [39]

    Sneha Silwal, Karmesh Yadav, Tingfan Wu, Jay Vakil, Arjun Majumdar, Sergio Arnaud, Claire Chen, Vincent-Pierre Berges, Dhruv Batra, Aravind Rajeswaran, Mrinal Kalakrishnan, Franziska Meier, and Oleksandr Maksymets. What do we learn from a large-scale study of pre-trained visua...

    2024

  15. [40]

    Linda Smith and Michael Gasser

    doi: 10.1109/ICRA57147.2024.10610218. Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies.Artificial life, 11 (1-2):13–29,

    2024

  16. [41]

    http://incompleteideas.net/book/the-book-2nd.html. Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimír Vondruš, Sameer Dharur, Franziska Meier...

    2021

  17. [42]

    Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503.20020,

    Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world.arXiv preprint arXiv:2503.20020,

    arXiv

  18. [43]

    Exploring virtual agents for augmented reality

    Isaac Wang, Jesse Smith, and Jaime Ruiz. Exploring virtual agents for augmented reality. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–12,

    2019

  19. [44]

    Hi-tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models

    Yufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. Hi-tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 10691–10706,

    2023

  20. [45]

    Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese

    Fei Xia, Amir R. Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson Env: real-world perception for embodied agents. InComputer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on. IEEE,

    2018

  21. [46]

    Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun

    https://arxiv.org/abs/2409.11494. Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR...

    arXiv

  22. [47]

    Homerobot: Open-vocabulary mobile manipulation

    Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung- Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, et al. Homerobot: Open-vocabulary mobile manipulation. arXiv preprint arXiv:2306.11565,

    arXiv

  23. [48]

    Robust llm safeguarding via refusal feature adversarial training

    Lei Yu, Virginie Do, Karen Hambardzumyan, and Nicola Cancedda. Robust llm safeguarding via refusal feature adversarial training. arXiv preprint arXiv:2409.20089, 2024a. Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming-Hsuan ...

    arXiv

  24. [49]

    Vip: a differentially private foundation model for computer vision

    Yaodong Yu, Maziar Sanjabi, Yi Ma, Kamalika Chaudhuri, and Chuan Guo. Vip: a differentially private foundation model for computer vision. In Proceedings of the 41st International Conference on Machine Learning, pages 57639–57658, 2024b. Arman Zharmagambetov, Chuan Guo, Ivan Ev...

  25. [50]

    Dino-wm: World models on pre-trained visual features enable zero-shot planning, 2024.https://arxiv.org/abs/2411.04983

    Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning, 2024.https://arxiv.org/abs/2411.04983. 43 Zhiyuan Zhou, Pranav Atreya, You Liang Tan, Karl Pertsch, and Sergey Levine. Autoeval: Autonomous e...

    2024 arXiv

  26. [1945]

    Plumbley

    Annamaria Mesaros, Romain Serizel, Toni Heittola, Tuomas Virtanen, and Mark D. Plumbley. A decade of dcase: Achievements, practices, evaluations and future challenges, 2024.https://arxiv.org/abs/2410.04951. Ruaridh Mon-Williams, Gen Li, Ran Long, Wenqian Du, and Christopher G ...

    2024 arXiv

  27. [1991]

    Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning, 2024.https: //arxiv.org/abs/2412.12175

    Melanie Sclar, Jane Yu, Maryam Fazel-Zarandi, Yulia Tsvetkov, Yonatan Bisk, Yejin Choi, and Asli Celikyilmaz. Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning, 2024.https: //arxiv.org/abs/2412.12175. Ben Shneiderman and Catherine ...

    2024 arXiv

  28. [2005]

    doi: 10.1145/1067860.1067867

    ISSN 1073-0516. doi: 10.1145/1067860.1067867. https://doi.org/10.1145/1067860.1067867. 36 Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. Gr00t n1: An open foundation model for gener...

  29. [2006]

    Moshi: a speech-text foundation model for real-time dialogue, 2024.https://arxiv.org/abs/2410.00037

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue, 2024.https://arxiv.org/abs/2410.00037. Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenne...

    2024 arXiv

  30. [2007]

    Plan-and-act: Improving planning of agents for long-horizon tasks.arXiv preprint arXiv:2503.09572,

    Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. Plan-and-act: Improving planning of agents for long-horizon tasks.arXiv preprint arXiv:2503.09572,

    arXiv

  31. [2010]

    The art of llm refinement: Ask, refine, trust

    Kumar Shridhar, Koustuv Sinha, Andrew Cohen, Tianlu Wang, Ping Yu, Ramakanth Pasunuru, Mrinmaya Sachan, Jason E Weston, and Asli Celikyilmaz. The art of llm refinement: Ask, refine, trust. InICLR 2024 Workshop on Large Language Model (LLM) Agents,

    2024

  32. [2011]

    Worldprediction: A benchmark for high-level world modeling and long-horizon procedural planning, 2025.https://arxiv.org/abs/2506.04363

    Delong Chen, Willy Chung, Yejin Bang, Ziwei Ji, and Pascale Fung. Worldprediction: A benchmark for high-level world modeling and long-horizon procedural planning, 2025.https://arxiv.org/abs/2506.04363. Neeraj Cherakara, Finny Varghese, Sheena Shabana, Nivan Nelson, Abhiram Kar...

    2025 arXiv

  33. [2015]

    Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748,

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748,

    arXiv

  34. [2016]

    Causalvqa: A physically grounded causal reasoning benchmark for video models.arXiv preprint arXiv:2506.09943,

    Aaron Foss, Chloe Evans, Sasha Mitts, Koustuv Sinha, Ammar Rizvi, and Justine T Kao. Causalvqa: A physically grounded causal reasoning benchmark for video models.arXiv preprint arXiv:2506.09943,

    arXiv

  35. [2017]

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al

    ISBN 1107652871. Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:23...

    arXiv

  36. [2018]

    Ai robots and humanoid ai: Review, perspectives and directions.Perspectives and Directions (March 19, 2024),

    Longbing Cao. Ai robots and humanoid ai: Review, perspectives and directions.Perspectives and Directions (March 19, 2024),

    2024

  37. [2019]

    doi: 10.18653/v1/N19-1423.https://aclanthology.org/N19-1423/

    Association for Computational Linguistics. doi: 10.18653/v1/N19-1423.https://aclanthology.org/N19-1423/. Xin Luna Dong. Next-generation intelligent assistants for wearable devices. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24,...

    doi

  38. [2020]

    Calibrating verbal uncertainty as a linear feature to reduce hallucinations.arXiv preprint arXiv:2503.14477,

    Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, and Nicola Cancedda. Calibrating verbal uncertainty as a linear feature to reduce hallucinations.arXiv preprint arXiv:2503.14477,

    arXiv

  39. [2022]

    Quan Nguyen, Ayush Agrawal, Xingye Da, William Martin, Hartmut Geyer, Jessy Grizzle, and Koushil Sreenath

    doi: 10.1109/CDC51059.2022.9993259. Quan Nguyen, Ayush Agrawal, Xingye Da, William Martin, Hartmut Geyer, Jessy Grizzle, and Koushil Sreenath. Dynamic walking on randomly-varying discrete terrain with one-step preview. InProceedings of Robotics: Science and Systems (RSS ’17), July

    2022

  40. [2023]

    Hallulens: Llm hallucination benchmark.arXiv preprint arXiv:2504.17550,

    Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. Hallulens: Llm hallucination benchmark.arXiv preprint arXiv:2504.17550,

    arXiv

  41. [2024]

    ISBN 9798400704901

    Association for Computing Machinery. ISBN 9798400704901. doi: 10.1145/3637528.3672500.https://doi.org/10.1145/3637528. 3672500. Emmanuel Dupoux. Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner. Cognition, 1...

  42. [2025]

    Pranav Atreya, Karl Pertsch, Tony Lee, Moo Jin Kim, Arhan Jain, Artur Kuramshin, Clemens Eppner, Cyrus Neary, Edward Hu, Fabio Ramos, et al

    https://ai.meta.com/blog/v-jepa-2-world-model-benchmarks/ . Pranav Atreya, Karl Pertsch, Tony Lee, Moo Jin Kim, Arhan Jain, Artur Kuramshin, Clemens Eppner, Cyrus Neary, Edward Hu, Fabio Ramos, et al. Roboarena: Distributed real-world evaluation of generalist robot policies.ar...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.