Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Ella: Embodied Social Agents with Lifelong Memory

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Ella shows that an embodied agent can learn social behavior over days by storing visual and conversational experiences in a dual memory and retrieving them with foundation models, and demonstrates gains over re-implemented baselines in…

desk verdict A promising memory-system architecture for embodied social agents, but the evidence is too thin to support the lifelong-learning and memory-causality claims. read the letter →

arxiv 2506.24019 v1 pith:ZE2WX7ZC submitted 2025-06-30 cs.CV cs.CL

Hongxin Zhang , Zheyuan Zhang , Zeyuan Wang , Zunzhe Zhang , Lixing Fang , Qinhong Zhou , Chuang Gan This is my paper
classification cs.CVcs.CL
keywords embodiedagentlifelongmemorysemanticepisodic3Dopenworldsocialreasoningscenegraphfoundationmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Ella is an embodied agent designed to keep learning while living in a simulated town. The paper claims that a structured long-term memory, split into semantic memory for facts and episodic memory for experiences, lets the agent accumulate knowledge from visual observation and conversation and then use that knowledge to plan days, persuade others, and lead group tasks. If the claim holds, it shows that foundation-model agents can develop social competence in an open world without retraining, by storing and retrieving experience rather than by updating weights. The paper tests this in three three-dimensional communities with fifteen agents, measuring how often invited agents attend parties and how many group task items get completed.

What carries the argument

The load-bearing object is the dual long-term memory with its retrieval scoring. Episodic events are ranked by a weighted average of spatial proximity to the query location, a joint text-image cosine similarity for content relevance, and an exponential-decay recency score; semantic memory is a name-centric graph whose spatial backbone is a hierarchical scene graph (volume grid, object layer, region layer) built incrementally from posed RGB-depth frames. This memory feeds a planning-reaction loop: retrieve context, generate a structured daily schedule with explicit commute times, then update memory and react to new observations or conversations. The retrieval mechanism is what lets a foundation-model backbone act on accumulated experience instead of only on the current frame.

What would settle it

Extend the simulation to thirty simulated days without resetting any agent's memory and rerun the Influence Battle and Leadership Quest evaluations; if show-up rates or completion rates decay toward the no-memory baseline, or if retrieval latency grows so fast that decisions degrade, then the observed 1.5-day advantage was not lifelong learning. A second check is to run the same scenario multiple times and report variance, since single runs per community cannot distinguish memory-driven behavior from luck in which agents happen to meet.

Watch

Extended reading notes

Core claim

Ella's central claim is that lifelong learning in an open social world can be achieved by pairing a dual-form long-term memory with foundation models. Semantic memory is organized around names and includes a hierarchical scene graph built from RGB-depth observations, so the agent knows places, objects, and people; episodic memory stores events with time, location, text, and image content. At decision time the agent retrieves relevant memories by spatial proximity, content similarity, and recency, generates a daily schedule, and reacts to new observations by revising plans, interacting, or conversing. Across three communities, Ella reports higher party show-up rates and higher task completion rates than the two re-implemented baselines, and the authors interpret this as evidence that the agent learns through observation and social interaction rather than through explicit training.

Load-bearing premise

The entire lifelong-learning claim rests on the assumption that nine hours of first-day simulation plus two evaluations over one and a half simulated days, with fifteen agents and no repeated runs, is enough to demonstrate that knowledge accumulates, is retained, and is reused over time.

Editorial extensions

If this is right

  • If the claim holds, an embodied agent can acquire social knowledge from perception and dialogue without gradient updates, by writing experiences to memory and reading them back at decision time.
  • Structured, name-centric organization should keep retrieval efficient as memory grows, so agents can operate over days rather than minutes.
  • The reported gains in persuasion and leadership suggest that capability-oriented evaluations in open 3D worlds can expose whether an agent actually retains and reuses what it learned.
  • Stronger perception improves social outcomes, so progress in open-set detection and segmentation should transfer directly to social-agent competence.
  • Closer performance with an open-source backbone indicates the memory design, not the specific proprietary model, is what carries the result.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension is knowledge transfer between agents: if semantic memory were shared or queried across agents, one agent's learned facts could bootstrap another's planning, which would amplify the community-level effect.
  • The paper's own limitation list implies that adding graph-based multi-hop retrieval to semantic memory could turn social recall into reasoning, for example inferring a target's availability from a chain of observed events rather than from a direct match.
  • A direct stress test of the memory itself would be to measure retrieval latency and decision quality as memory grows past the 1.5-day horizon; the current evaluations do not isolate memory size from agent behavior.
  • The synchronous-thinking assumption suggests a natural next experiment: give agents a per-step thinking budget and see whether planning quality degrades or agents learn to prioritize, mirroring bounded human cognition.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Ella, an embodied social agent in the 3D open-world Virtual Community simulator, equipped with a dual long-term memory system: a name-centric semantic memory (hierarchical scene graph plus knowledge graph) and a spatiotemporal episodic memory. The agent uses foundation models to plan daily schedules, react to observations, communicate, and update memory. Ella is evaluated in three communities with 15 agents over 1.5 simulated days in two tasks: Influence Battle and Leadership Quest. The reported results show higher show-up and completion rates than re-implemented CoELA and Generative Agents baselines. The abstract and conclusion claim that Ella can influence, lead, and cooperate with other agents, demonstrating effective learning through observation and social interaction.

Significance. If the central attribution to the memory system were established, this work would be a valuable step toward embodied social agents with persistent, structured memory in a visually realistic simulation. The paper is strong in system description: it details the scene graph construction, episodic memory scoring, planning and reaction prompts, and it includes full prompt templates in the appendix, compute details, and an explicit limitations section. The use of multiple scenes and a preliminary open-source backbone comparison (Table 2) are also useful. However, the current evaluation does not isolate the memory system from the rest of the pipeline, and the 'lifelong' claim is tested only over 1.5 simulated days. As a result, the evidence is not yet sufficient to support the paper's headline claims.

major comments (4)
  1. [§5.1, Table 1] No ablation isolates the memory system. The baselines differ from Ella not only in memory but also in planning prompts, reaction and communication modules, perception integration, and, for Generative Agents, the backbone model (gpt-35-turbo instead of gpt-4o). Consequently, the reported gains (e.g., 53.4% vs 24.5% show-up rate, 32.5% vs 3.8% completion rate) could be due to any of these differences. The statement in §5.2 that CoELA fails because it 'lacks long-term memory' is an interpretation, not a measurement. A memory ablation—e.g., Ella with episodic/semantic retrieval disabled while keeping planner, reactor, and communicator fixed—is required to support the central claim that the memory system enables learning and social reasoning.
  2. [§5.1, Table 1] All quantitative results come from a single simulation run per scene and community, with no repeated seeds, error bars, or confidence intervals. With only 15 agents per scene, a show-up rate of 6.7% (Detroit, CoELA) corresponds to a single agent, and a 53.4% average is about eight agents. Given the stochasticity of LLM-based planning, navigation, and conversation, the differences between methods may be within run-to-run noise. Repeated runs, at least over multiple random seeds, are necessary to establish the robustness of the reported margins.
  3. [§6] The 'lifelong' claim is not supported by the experimental timescale. The paper itself acknowledges in Section 6: 'our experiments span only 1.5 simulated days—seemingly short for a "lifelong" setting.' While the paper adopts the interpretation of lifelong learning as accumulation, retention, and reuse of knowledge, the evaluation only tests one day of experience followed by a second-day controlled task. Phenomena central to lifelong learning—such as forgetting, memory consolidation, interference, and knowledge transfer across many days—are not measured. To justify 'lifelong', the evaluation should either span substantially longer simulated time or include tasks explicitly requiring retention and reuse across multiple days.
  4. [§4.3.3, §5.2] There is no direct evidence that the knowledge extracted from conversations is correct, retained, or subsequently used in decision-making. The communication module summarizes conversations and extracts knowledge items, but the experiments do not probe the accuracy or the downstream utility of these items. Figure 4b only counts memory nodes, which does not demonstrate functional use. A knowledge-probe evaluation (e.g., asking agents about other characters or places after day one) and a decision-trace analysis showing that retrieved memories actually influence reactions and schedules would be needed to support the claim of 'learning effectively through observation and social interaction.'
minor comments (4)
  1. [§4.2] The temporal recency formula is given as 'Recency(e) = exp(te − tq)' without defining te and tq, and the text says the decay is based on 'time elapsed since the memory was last accessed.' If this is intended as an exponential decay, please clarify the time variables and the decay constant.
  2. [§4.1.1] In the Region Layer description, 'partitioning the graph into p |B| regions' appears to be a formatting error; likely it should read the square root of |B| (i.e., √|B|). Please clarify.
  3. [§5.2] There is a typo: 'the performance further boots' should be 'the performance further boosts'.
  4. [§5.1] The footnote explaining why Generative Agents uses gpt-35-turbo (original prompts break with gpt-4o and cost) is useful, but it introduces a backbone confound. At minimum, the paper should acknowledge this confound more prominently, or provide a smaller-scale comparison with the same backbone.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reduction: task metrics are independent of memory parameters; self-developed simulator and reimplemented baselines are attribution risks, not tautologies.

full rationale

Ella is an empirical systems paper rather than a derivation, so the circularity check reduces to whether any reported result is defined by the method's own parameters or supported only by a self-citation chain. It is not. The evaluation metrics (show-up rate, completion rate, conversation counts, Section 5.1) are task-level quantities defined independently of Ella's memory internals; no coefficient in the retrieval equations of Section 4.2 (spatial proximity, cosine relevance, exponential recency) is fitted to those metrics, and the memory system has no trainable parameters tuned on the benchmark. The comparisons against CoELA and Generative Agents do differ in planner, reaction, and communication modules and in backbone (Section 5.1), so the causal attribution that memory enables the gains is not isolated by an ablation; that is a confounding/attribution limitation, not a circular reduction. The paper's own Section 6 admits the 1.5-day horizon is 'seemingly short for a lifelong setting,' which limits external validity but does not make the claim tautological. The only notable self-citation is Virtual Community [105], which supplies the simulation platform rather than the result; it is not load-bearing in the sense of importing the paper's conclusion. No equation or metric reduces to the memory design by construction, and no fitted parameter is relabeled as a prediction. Hence no significant circularity.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard AI tooling assumptions (foundation model competence), a simulation proxy (Virtual Community), and a cognitive taxonomy mapping. No new physical entities are introduced; the free parameters are hand-set design choices, not fitted to the evaluation. The most consequential assumption is that the 1.5-day horizon is long enough to show lifelong memory.

free parameters (6)
  • Message range threshold theta_msg = 10 m
    Hand-set in Section 5.1; controls which conversations agents can hear, directly shaping social interaction outcomes.
  • Reaction trigger interval theta_react = not specified
    Mentioned in Section 4.3.2 but value not reported; determines how often agents reevaluate their situation.
  • Retrieval score weights = equal weights on proximity, relevance, recency
    Section 4.2: scores are normalized and averaged with equal weight; no justification or tuning is reported.
  • Retrieval top-k = unspecified
    Section 4.2 says the top k events are retrieved, but the value of k is not given; this directly controls how much memory the planner sees.
  • Navigation cost weights = unknown=5, obstacle=infinity, proximity=100/d
    Appendix C.1; hand-set A* costs that influence whether agents arrive at events on time.
  • Occupancy grid parameters = 0.5m blocks, 0.1m cells, 0.5m height difference
    Section 4.1.1; geometric thresholds for obstacle detection and navigation.
assumptions (6)
  • domain assumption Foundation models (GPT-4o, CLIP, text-embedding-3-small, open-source LLMs) provide sufficient perception, language, and planning competence.
    Every decision, from perception summaries to schedules to utterances, is produced by these models (Sections 4.1-4.3); if a backbone is incompetent, Ella's memory cannot compensate.
  • domain assumption Virtual Community is a valid proxy for an open 3D world with realistic physics, rendering, and social dynamics.
    All experiments run inside this simulator (Section 5.1 and [105]); there is no validation against real-world interaction or an external embodied benchmark.
  • domain assumption The psychological distinction between episodic and semantic memory transfers to the implemented memory structures.
    The design is justified by Tulving's taxonomy (Section 1 and Section 4); the mapping from cognitive categories to data structures is assumed rather than validated.
  • domain assumption All agents think synchronously and with effectively unlimited computational resources.
    Stated explicitly in Section 6: every agent's deliberation costs exactly one simulated second; this abstracts away timing constraints that could change social dynamics.
  • standard math A* on a discretized occupancy grid returns feasible navigation paths.
    The navigation module (Appendix C.1) treats A* as correct on the constructed grid; standard, but an assumption about the environment discretization.
  • domain assumption Lifelong learning can be defined as continuous accumulation, retention, and reuse of knowledge.
    Section 6 adopts the definition from [14] to characterize 1.5 days of simulation as lifelong; this is a definitional choice, not an empirical result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ella: Embodied Social Agents with Lifelong Memory." pith.science (2026). https://pith.science/paper/ZE2WX7ZC

@misc{pith2026250624019,
  author       = {Pith},
  title        = {Pith review of: Ella: Embodied Social Agents with Lifelong Memory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZE2WX7ZC}},
  note         = {Machine review of arXiv:2506.24019}
}
read the original abstract

We introduce Ella, an embodied social agent capable of lifelong learning within a community in a 3D open world, where agents accumulate experiences and acquire knowledge through everyday visual observations and social interactions. At the core of Ella's capabilities is a structured, long-term multimodal memory system that stores, updates, and retrieves information effectively. It consists of a name-centric semantic memory for organizing acquired knowledge and a spatiotemporal episodic memory for capturing multimodal experiences. By integrating this lifelong memory system with foundation models, Ella retrieves relevant information for decision-making, plans daily activities, builds social relationships, and evolves autonomously while coexisting with other intelligent beings in the open world. We conduct capability-oriented evaluations in a dynamic 3D open world where 15 agents engage in social activities for days and are assessed with a suite of unseen controlled evaluations. Experimental results show that Ella can influence, lead, and cooperate with other agents well to achieve goals, showcasing its ability to learn effectively through observation and social interaction. Our findings highlight the transformative potential of combining structured memory systems with foundation models for advancing embodied intelligence. More videos can be found at https://umass-embodied-agi.github.io/Ella/.

Figures

Figures reproduced from arXiv: 2506.24019 by the authors.

Figure 1
Figure 1. (a) Embodied agents require lifelong learning to accumulate experiences and acquire knowledge [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An example community of 15 agents and 4 social groups in New York. The character and observation of agent Elizabeth Mensah are shown on the right. 2 Related Work 2.1 Embodied Social Intelligence Social intelligence has been widely studied in embodied multi-agent environments [105, 58, 11, 2, 7, 39, 70, 89, 72, 95, 87, 100, 48], while one branch focuses on simplified symbolic or game￾like environments [78, 84, 38, 6,… view at source ↗
Figure 3
Figure 3. Method Overview. We build long-term memory in two forms: (a) name-centric semantic memory organizes the knowledge in a name-centric graph including a hierarchical scene graph serving as the spatial memory; (b) spatiotemporal episodic memory stores the experience as a series of events consisting of time, location, and multimodal contents. (c) Ella first generates a daily schedule according to the knowledge and experi… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: (a) Social interaction pattern in Influence Battle. The thickness of a line reflects the frequency of interaction. Members from Creative Minds Collective successfully persuaded Elizabeth Mensah to join their group’s party. (b) Comparison of memory growth over time. The…
Figure 5
Figure 5. Figure 5: An illustration of a community in New York City with places of different functionalities annotated. There are 6 types of functional places: accommodation, entertainment, food, office, stores, and transit, each labeled with different colors on the figure. Social group i…
Figure 6
Figure 6. Figure 6: Close-up views of different scenes in Virtual Community. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: A visualization of the final spatial coverage on the Detroit community. Explored regions are shown in red, buildings are shown in white, and unexplored regions are shown in black. The buildings in the agent’s schedule are denoted with green circles. C Additional Implem…
Figure 8
Figure 8. Figure 8: Prompt template for generating the daily schedule. $Character$ is replaced with the agent’s character description, $Context$ is replaced with the retrieved memory. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Prompt template for generating the reaction. $Character$ is replaced with the agent’s character description, $Schedule$ is replaced with today’s remaining schedules, $Experience$ is replaced with the retrieved memory, $Context$ is replaced with the latest memory. 21 […
Figure 10
Figure 10. Figure 10: Prompt template for generating the utterance. $Character$ is replaced with the agent’s character description, $Target_knowledge$, $Target_experience$, $Context$ are replaced with the retrieved memory, $Conversation_history$ is replaced with the last 4 messages. Prompt…
Figure 11
Figure 11. Figure 11 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Prompt template for extracting knowledge from a conversation. $Conversation_history$ is replaced with the full conversation, $Knowledge_items$ is replaced with sampled knowledge items from semantic memory. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Observable Social Life Spaces: Exploring User Interpretations of agent-side life context in human-agent interaction

    cs.HC 2026-03 conditional novelty 6.0 of 10

    Seeing an AI agent's autonomous virtual life increased users' perceived equality with it in a small study, but the effect needs replication.

Reference graph

Works this paper leans on

106 extracted references · 53 canonical work pages · cited by 1 Pith paper

  1. [1]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022

    arXiv 2022

  2. [2]

    Amato, G

    C. Amato, G. Konidaris, L. P. Kaelbling, and J. P. How. Modeling and planning with macro- actions in decentralized pomdps. Journal of Artificial Intelligence Research , 64:817–859, 2019

    2019

  3. [3]

    Amershi, D

    S. Amershi, D. Weld, M. V orvoreanu, A. Fourney, B. Nushi, P. Collisson, J. Suh, S. Iqbal, P. N. Bennett, K. Inkpen, et al. Guidelines for human-ai interaction. In Proceedings of the 2019 chi conference on human factors in computing systems, pages 1–13, 2019

    2019

  4. [4]

    Asgharivaskasi and N

    A. Asgharivaskasi and N. Atanasov. Semantic octree mapping and shannon mutual information computation for robot exploration. IEEE Transactions on Robotics, 39(3):1910–1928, 2023

    1910

  5. [5]

    G. Authors. Genesis: A universal and generative physics engine for robotics and beyond, December 2024

    2024

  6. [6]

    Baker, I

    B. Baker, I. Kanitscheider, T. Markov, Y . Wu, G. Powell, B. McGrew, and I. Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference on Learning Representations, 2020

    2020

  7. [7]

    N. Bard, J. N. Foerster, S. Chandar, N. Burch, M. Lanctot, H. F. Song, E. Parisotto, V . Dumoulin, S. Moitra, E. Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280:103216, 2020

    2020

  8. [8]

    Blukis, C

    V . Blukis, C. Paxton, D. Fox, A. Garg, and Y . Artzi. A persistent spatial semantic representation for high-level natural language instruction execution. In Conference on Robot Learning, pages 706–717. PMLR, 2022

    2022

Show all 106 references
  1. [9]

    A. Bobu, A. Peng, P. Agrawal, J. Shah, and A. D. Dragan. Aligning robot and human representations. arXiv preprint arXiv:2302.01928, 2023

    2023 arXiv

  2. [10]

    Borgeaud, A

    S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driess- che, J.-B. Lespiau, B. Damoc, A. Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR, 2022

    2022

  3. [11]

    Carroll, R

    M. Carroll, R. Shah, M. K. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019

    2019

  4. [12]

    D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems , 33:4247–4258, 2020

    2020

  5. [13]

    H. Chen, H. Chen, M. Yan, W. Xu, G. Xing, W. Shen, X. Quan, C. Li, J. Zhang, and F. Huang. Socialbench: Sociality evaluation of role-playing conversational agents. In Findings of the Association for Computational Linguistics ACL 2024, pages 2108–2126, 2024

    2024

  6. [14]

    Chen and B

    Z. Chen and B. Liu. Lifelong machine learning. Morgan & Claypool Publishers, 2018

    2018

  7. [15]

    Crosby, B

    M. Crosby, B. Beyret, and M. Halina. The animal-ai olympics. Nature Machine Intelligence, 1(5):257–257, 2019

    2019

  8. [16]

    G. Dai, W. Zhang, J. Li, S. Yang, S. Rao, A. Caetano, M. Sra, et al. Artificial leviathan: Exploring social evolution of llm agents through the lens of hobbesian social contract theory. arXiv preprint arXiv:2406.14373, 2024

    2024 arXiv

  9. [17]

    Dautenhahn

    K. Dautenhahn. Socially intelligent robots: dimensions of human–robot interaction. Philo- sophical transactions of the royal society B: Biological sciences, 362(1480):679–704, 2007. 10

    2007

  10. [18]

    Y . Du, M. Yang, P. Florence, F. Xia, A. Wahid, B. Ichter, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenenbaum, et al. Video language planning. arXiv preprint arXiv:2310.10625, 2023

    2023 arXiv

  11. [19]

    J. S. B. Evans. In two minds: dual-process accounts of reasoning. Trends in cognitive sciences, 7(10):454–459, 2003

    2003

  12. [20]

    S. Y . Gadre, K. Ehsani, S. Song, and R. Mottaghi. Continuous scene representations for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14849–14859, 2022

    2022

  13. [21]

    S. Y . Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23171–23181, 2023

    2023

  14. [22]

    C. Gan, J. Schwartz, S. Alter, D. Mrowca, M. Schrimpf, J. Traer, J. D. Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, M. Sano, K. Kim, E. Wang, M. Lingelbach, A. Curtis, K. T. Feigelis, D. Bear, D. Gutfreund, D. D. Cox, A. Torralba, J. J. DiCarlo, J. B. Tenenbaum, J. Mcdermot...

    2021

  15. [23]

    M. C. Gombolay, R. A. Gutierrez, S. G. Clarke, G. F. Sturla, and J. A. Shah. Decision- making authority, team efficiency and human worker satisfaction in mixed human–robot teams. Autonomous Robots, 39:293–312, 2015

    2015

  16. [24]

    M. A. Goodrich, A. C. Schultz, et al. Human–robot interaction: a survey. Foundations and Trends® in Human–Computer Interaction, 1(3):203–275, 2008

    2008

  17. [25]

    Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pa...

    2024

  18. [26]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  19. [27]

    I. Gur, H. Furuta, A. Huang, M. Safdari, Y . Matsuo, D. Eck, and A. Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856, 2023

    2023 arXiv

  20. [28]

    B. J. Gutiérrez, Y . Shu, Y . Gu, M. Yasunaga, and Y . Su. Hipporag: Neurobiologically inspired long-term memory for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  21. [29]

    B. J. Gutiérrez, Y . Shu, W. Qi, S. Zhou, and Y . Su. From rag to memory: Non-parametric continual learning for large language models, 2025

    2025

  22. [30]

    H. Han, Y . Wang, H. Shomer, K. Guo, J. Ding, Y . Lei, M. Halappanavar, R. A. Rossi, S. Mukherjee, X. Tang, et al. Retrieval-augmented generation with graphs (graphrag). arXiv preprint arXiv:2501.00309, 2024

    2024 arXiv

  23. [31]

    W. Hong, W. Wang, Q. Lv, J. Xu, W. Yu, J. Ji, Y . Wang, Z. Wang, Y . Dong, M. Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024

    2024

  24. [32]

    Hornung, K

    A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard. Octomap: An efficient probabilistic 3d mapping framework based on octrees. Autonomous robots, 34:189–206, 2013

    2013

  25. [33]

    Huang, O

    C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 10608–10615. IEEE, 2023. 11

    2023

  26. [34]

    Huang, C

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. In Conference on Robot Learning, pages 540–562. PMLR, 2023

    2023

  27. [35]

    Huang, Y .-J

    X. Huang, Y .-J. Huang, Y . Zhang, W. Tian, R. Feng, Y . Zhang, Y . Xie, Y . Li, and L. Zhang. Open-set image tagging with multi-grained text supervision. arXiv e-prints, pages arXiv–2310, 2023

    2023

  28. [36]

    Hughes, Y

    N. Hughes, Y . Chang, and L. Carlone. Hydra: A real-time spatial perception system for 3d scene graph construction and optimization. arXiv preprint arXiv:2201.13360, 2022

    2022 arXiv

  29. [37]

    Ilharco, M

    G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V . Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. If you use this software, please cite it as below

    2021

  30. [38]

    Jaderberg, W

    M. Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castaneda, C. Beattie, N. C. Rabinowitz, A. S. Morcos, A. Ruderman, et al. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science, 364(6443):859– 865, 2019

    2019

  31. [39]

    U. Jain, L. Weihs, E. Kolve, A. Farhadi, S. Lazebnik, A. Kembhavi, and A. Schwing. A cordial sync: Going beyond marginal policies for multi-agent embodied tasks. In Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, p...

    2020

  32. [40]

    Jiang, F

    X. Jiang, F. Li, H. Zhao, J. Wang, J. Shao, S. Xu, S. Zhang, W. Chen, X. Tang, Y . Chen, et al. Long term memory: The foundation of ai self-evolution. arXiv preprint arXiv:2410.15665, 2024

    2024 arXiv

  33. [41]

    Kurenkov, M

    A. Kurenkov, M. Lingelbach, T. Agarwal, E. Jin, C. Li, R. Zhang, L. Fei-Fei, J. Wu, S. Savarese, and R. Martın-Martın. Modeling dynamic environments with scene graph memory. In International Conference on Machine Learning, pages 17976–17993. PMLR, 2023

    2023

  34. [42]

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  35. [43]

    J. E. Laird. Introduction to soar. arXiv preprint arXiv:2205.03854, 2022

    2022 arXiv

  36. [44]

    P. A. Lasota, T. Fong, J. A. Shah, et al. A survey of methods for safe human-robot interaction. Foundations and Trends® in Robotics, 5(4):261–349, 2017

    2017

  37. [45]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020

    2020

  38. [46]

    C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, M. Lingelbach, J. Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80–93. PMLR, 2023

    2023

  39. [47]

    G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991–52008, 2023

    2023

  40. [48]

    S. Li, Y . Wu, X. Cui, H. Dong, F. Fang, and S. Russell. Robust multi-agent reinforcement learn- ing via minimax deep deterministic policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4213–4220, 2019

    2019

  41. [49]

    X. Li, D. Guo, H. Liu, and F. Sun. Embodied semantic scene graph generation. In Conference on robot learning, pages 1585–1594. PMLR, 2022

    2022

  42. [50]

    Z. Li, Y . Xie, R. Shao, G. Chen, D. Jiang, and L. Nie. Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 12

    2024

  43. [51]

    Lieder and T

    F. Lieder and T. L. Griffiths. Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and brain sciences, 43:e1, 2020

    2020

  44. [52]

    Lindes and J

    P. Lindes and J. E. Laird. Toward integrating cognitive linguistics and cognitive language processing. In Proceedings of the 14th International Conference on Cognitive Modeling (ICCM), 2016

    2016

  45. [53]

    R. Liu, R. Yang, C. Jia, G. Zhang, D. Yang, and S. V osoughi. Training socially aligned language models on simulated social interactions. In The Twelfth International Conference on Learning Representations, 2024

    2024

  46. [54]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023

    2023 arXiv

  47. [55]

    X. Liu, J. Zhang, S. Guo, H. Shang, C. Yang, and Q. Zhu. Exploring prosocial irrationality for llm agents: A social cognition view. arXiv preprint arXiv:2405.14744, 2024

    2024 arXiv

  48. [56]

    Z. Liu, A. Anand, P. Zhou, J.-t. Huang, and J. Zhao. Interintent: Investigating social intel- ligence of llms via intention understanding in an interactive game context. arXiv preprint arXiv:2406.12203, 2024

    2024 arXiv

  49. [57]

    D. P. Losey, H. J. Jeon, M. Li, K. Srinivasan, A. Mandlekar, A. Garg, J. Bohg, and D. Sadigh. Learning latent actions to control assistive robots. Autonomous robots, 46(1):115–147, 2022

    2022

  50. [58]

    R. Lowe, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30, 2017

    2017

  51. [59]

    Maggio, Y

    D. Maggio, Y . Chang, N. Hughes, M. Trang, D. Griffith, C. Dougherty, E. Cristofalo, L. Schmid, and L. Carlone. Clio: Real-time task-driven open-set 3d scene graphs. arXiv preprint arXiv:2404.13696, 2024

    2024 arXiv

  52. [60]

    Mastrogiuseppe, N

    M. Mastrogiuseppe, N. Bertelsen, M. F. Bedeschi, and S. A. Lee. The spatiotemporal orga- nization of episodic memory and its disruption in a neurodevelopmental disorder. Scientific reports, 9(1):18447, 2019

    2019

  53. [61]

    S. Y . Min, D. S. Chaplot, P. K. Ravikumar, Y . Bisk, and R. Salakhutdinov. Film: Following instructions in language with modular methods. In International Conference on Learning Representations, 2022

    2022

  54. [62]

    Natarajan and M

    M. Natarajan and M. Gombolay. Effects of anthropomorphism and accountability on trust in human robot interaction. In Proceedings of the 2020 ACM/IEEE international conference on human-robot interaction, pages 33–42, 2020

    2020

  55. [63]

    Nikolaidis, R

    S. Nikolaidis, R. Ramakrishnan, K. Gu, and J. Shah. Efficient model learning from joint- action demonstrations for human-robot collaborative tasks. In Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction, pages 189–196, 2015

    2015

  56. [64]

    Y . Niu, R. R. Paleja, and M. C. Gombolay. Multi-agent graph-attention communication and teaming. In AAMAS, volume 21, page 20th, 2021

    2021

  57. [65]

    A. M. Nuxoll and J. E. Laird. Extending cognitive architecture with episodic memory. In AAAI, pages 1560–1564, 2007

    2007

  58. [66]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  59. [67]

    Packer, V

    C. Packer, V . Fang, S. Patil, K. Lin, S. Wooders, and J. Gonzalez. Memgpt: Towards llms as operating systems. arXiv preprint, 2023

    2023

  60. [68]

    J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023. 13

    2023 arXiv

  61. [69]

    Pavlakos, V

    G. Pavlakos, V . Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3D hands, face, and body from a single image. InProceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10975–10985, 2019

    2019

  62. [70]

    X. Puig, T. Shu, S. Li, Z. Wang, Y .-H. Liao, J. B. Tenenbaum, S. Fidler, and A. Torralba. Watch- and-help: A challenge for social perception and human-ai collaboration. In International Conference on Learning Representations, 2021

    2021

  63. [71]

    X. Puig, E. Undersander, A. Szot, M. D. Cote, T.-Y . Yang, R. Partsey, R. Desai, A. Clegg, M. Hlavac, S. Y . Min, et al. Habitat 3.0: A co-habitat for humans, avatars, and robots. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  64. [72]

    X. Puig, E. Undersander, A. Szot, M. D. Cote, T.-Y . Yang, R. Partsey, R. Desai, A. W. Clegg, M. Hlavac, S. Y . Min, et al. Habitat 3.0: A co-habitat for humans, avatars and robots.arXiv preprint arXiv:2310.13724, 2023

    2023 arXiv

  65. [73]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PMLR, 2021

    2021

  66. [74]

    S. K. Ramakrishnan, D. S. Chaplot, Z. Al-Halah, J. Malik, and K. Grauman. Poni: Potential functions for objectgoal navigation with interaction-free learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18890–18900, 2022

    2022

  67. [75]

    K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. D. Reid, and N. Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. CoRR, 2023

    2023

  68. [76]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V . Alwala, N. Carion, C.-Y . Wu, R. Girshick, P. Dollár, and C. Feichtenhofer. Sam 2: Segment anything in images and videos, 2024

    2024

  69. [77]

    L. Rozo, S. Calinon, D. G. Caldwell, P. Jimenez, and C. Torras. Learning physical collaborative robot behaviors from human demonstrations. IEEE Transactions on Robotics, 32(3):513–527, 2016

    2016

  70. [78]

    Samvelyan, T

    M. Samvelyan, T. Rashid, C. Schroeder de Witt, G. Farquhar, N. Nardelli, T. G. Rudner, C.-M. Hung, P. H. Torr, J. Foerster, and S. Whiteson. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pa...

    2019

  71. [79]

    S. H. Schwartz. An overview of the schwartz theory of basic values. Online readings in Psychology and Culture, 2(1):11, 2012

    2012

  72. [80]

    N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam. Clip-fields: Weakly supervised semantic fields for robotic memory. arXiv preprint arXiv:2210.05663, 2022

    2022 arXiv

  73. [81]

    Sharon, R

    G. Sharon, R. Stern, A. Felner, and N. R. Sturtevant. Conflict-based search for optimal multi-agent pathfinding. Artificial intelligence, 219:40–66, 2015

    2015

  74. [82]

    W. Shi, S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettlemoyer, and W.-t. Yih. Replug: Retrieval-augmented black-box language models. In Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language...

    2024

  75. [83]

    Shinn, F

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  76. [84]

    Suarez, Y

    J. Suarez, Y . Du, P. Isola, and I. Mordatch. Neural mmo: A massively multiagent game environment for training and evaluating intelligent agents. arXiv preprint arXiv:1903.00784, 2019. 14

    1903 arXiv

  77. [85]

    Sumers, S

    T. Sumers, S. Yao, K. Narasimhan, and T. L. Griffiths. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427, 2023

    2023 arXiv

  78. [86]

    J. Sun, C. Xu, L. Tang, S. Wang, C. Lin, Y . Gong, H.-Y . Shum, and J. Guo. Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph.arXiv preprint arXiv:2307.07697, 2023

    2023 arXiv

  79. [87]

    A. Szot, U. Jain, D. Batra, Z. Kira, R. Desai, and A. Rai. Adaptive coordination in social embodied rearrangement. In International Conference on Machine Learning, pages 33365– 33380. PMLR, 2023

    2023

  80. [88]

    J. B. Tenenbaum, C. Kemp, T. L. Griffiths, and N. D. Goodman. How to grow a mind: Statistics, structure, and abstraction. science, 331(6022):1279–1285, 2011

    2011

  81. [89]

    N. Tsoi, M. Hussein, J. Espinoza, X. Ruiz, and M. Vázquez. Sean: Social environment for autonomous navigation. In Proceedings of the 8th international conference on human-agent interaction, pages 281–283, 2020

    2020

  82. [90]

    E. Tulving. Episodic and semantic memory. Organization of memory/Academic Press, 1972

    1972

  83. [91]

    E. Tulving. Elements of episodic memory, 1983

    1983

  84. [92]

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar. V oyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  85. [93]

    Z. Wang, S. Cai, G. Chen, A. Liu, X. Ma, and Y . Liang. Describe, explain, plan and se- lect: interactive planning with llms enables open-world multi-task agents. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  86. [94]

    Z. Wang, B. Yu, J. Zhao, W. Sun, S. Hou, S. Liang, X. Hu, Y . Han, and Y . Gan. Karma: Augmenting embodied ai agents with long-and-short term memory systems. arXiv preprint arXiv:2409.14908, 2024

    2024 arXiv

  87. [95]

    M. Wen, J. Kuba, R. Lin, W. Zhang, Y . Wen, J. Wang, and Y . Yang. Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, 35:16509–16521, 2022

    2022

  88. [96]

    Weston, S

    J. Weston, S. Chopra, and A. Bordes. Memory networks. arXiv preprint arXiv:1410.3916, 2014

    2014 arXiv

  89. [97]

    Y . Yang, H. Yang, J. Zhou, P. Chen, H. Zhang, Y . Du, and C. Gan. Snapmem: Snapshot-based 3d scene memory for embodied exploration and reasoning. arXiv preprint arXiv:2411.17735, 2024

    2024 arXiv

  90. [98]

    Yasunaga, A

    M. Yasunaga, A. Aghajanyan, W. Shi, R. James, J. Leskovec, P. Liang, M. Lewis, L. Zettle- moyer, and W.-T. Yih. Retrieval-augmented multimodal language modeling. In International Conference on Machine Learning, pages 39755–39769. PMLR, 2023

    2023

  91. [99]

    X. Yu, J. Fu, R. Deng, and W. Han. Mineland: Simulating large-scale multi-agent interactions with limited multimodal senses and physical needs. arXiv preprint arXiv:2403.19267, 2024

    2024 arXiv

  92. [100]

    Zhang, W

    H. Zhang, W. Du, J. Shan, Q. Zhou, Y . Du, J. B. Tenenbaum, T. Shu, and C. Gan. Building cooperative embodied agents modularly with large language models, 2023

    2023

  93. [101]

    Zhang, L

    L. Zhang, L. Wei, P. Shen, W. Wei, G. Zhu, and J. Song. Semantic slam based on object detection and improved octomap. IEEE Access, 6:75545–75559, 2018

    2018

  94. [102]

    Zhang, S

    Q. Zhang, S. Chen, Y . Bei, Z. Yuan, H. Zhou, Z. Hong, J. Dong, H. Chen, Y . Chang, and X. Huang. A survey of graph retrieval-augmented generation for customized large language models, 2025

    2025

  95. [103]

    Zheng, A

    K. Zheng, A. Paul, and S. Tellex. Asystem for generalized 3d multi-object search. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1638–1644. IEEE, 2023. 15

    2023

  96. [104]

    F. Zhou, H. Liu, H. Zhao, and L. Liang. Long-term object search using incremental scene graph updating. Robotica, 41(3):962–975, 2023

    2023

  97. [105]

    Q. Zhou, H. Zhang, X. Lin, Z. Zhang, Y . Chen, W. Liu, Z. Zhang, S. Chen, L. Fang, Q. Lyu, X. Sun, J. Yang, Z. Wang, B. C. Dang, Z. Chen, D. Ladia, J. Liu, and C. Gan. Virtual community: An open world for humans, robots, and society. 2025

    2025

  98. [106]

    type": "activity type, should be one of the following: 'commute', 'meal', 'sleep', 'main'

    X. Zhou, H. Zhu, L. Mathur, R. Zhang, H. Yu, Z. Qi, L.-P. Morency, Y . Bisk, D. Fried, G. Neubig, et al. Sotopia: Interactive evaluation for social intelligence in language agents. In The Twelfth International Conference on Learning Representations, 2024. 16 A Broader Impact A...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.