Pith. sign in

REVIEW 4 major objections 7 minor 90 references

STMA: A Spatio-Temporal Memory Agent for Long-Horizon Embodied Task Planning

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a spatio-temporal memory module, pairing a compressed temporal belief with a dynamic knowledge graph and a planner-critic loop, improves long-horizon embodied task planning, reporting a 31.25% success-rate gain over…

desk verdict Plausible memory-agent architecture, clearly written, but the headline numbers rest on 8 games per cell with no significance testing and an internal KG-update inconsistency; worth a serious referee but only after more seeds, metric honesty, and code. read the letter →

arxiv 2502.10177 v2 pith:LRVRRG6V submitted 2025-02-14 cs.AI

Mingcong Lei , Yiming Zhao , Ge Wang , Zhixin Mai , Shuguang Cui , Yatong Han , Jinke Ren This is my paper · ORCID
classification cs.AI
keywords spatio-temporalmemorylong-horizontaskplanningembodiedagentknowledgegraphplanner-criticTextWorldLLMagentspartialobservability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that long-horizon embodied planning fails less from weak reasoning than from missing memory: an agent that cannot recall its path or track where objects are cannot complete multi-step tasks in a partially observable environment. To test this, the authors build STMA, which compresses raw interaction history into a temporal belief, maintains a dynamic knowledge graph of spatial relations, and runs a planner-critic loop that checks each planned action before execution. On 32 TextWorld cooking tasks across four difficulty levels, STMA reports higher success rates and average scores than ReAct, Reflexion, and AdaPlanner, with the largest gains when powered by the open-source Qwen2.5-72b model. The central claim is that structured spatio-temporal memory plus closed-loop planning, not raw model scale, is what drives the improvement.

What carries the argument

The load-bearing object is the factored spatio-temporal belief, $b_i = (b^t_i, b^s_i)$: the agent compresses its history into a temporal belief and represents the environment's layout as a dynamic knowledge graph, then feeds both beliefs to a planner-critic loop. The dynamic KG is the spatial half: semantic triples extracted from summarized history replace outdated edges in real time, so the graph tracks rooms, objects, and containers as they move. The temporal half is a first-in-first-out history buffer plus an LLM summarizer that turns raw interaction tuples into a concise temporal belief. The planner-critic loop is what turns memory into behavior: the planner emits multi-step plans in one pass, the critic checks each action before execution, and a rejected action returns feedback that restarts the planning cycle.

What would settle it

Rerun the same TextWorld tasks with many more seeds, say 100 per difficulty level, and compute confidence intervals for success rate and average score; the central claim collapses if STMA's advantage over the best baseline falls within the noise band of the baselines. A cheaper check is to keep the seed set fixed but disable the critic while retaining memory and observe whether the success-rate drop matches Table 2.

Watch

Extended reading notes

Core claim

The paper's central claim is that a language-model agent can plan long-horizon embodied tasks much better when it separates what it has done from where things are. STMA's temporal memory stores raw action-observation tuples in a history buffer and summarizes them into a temporal belief; its spatial memory extracts semantic triples from that belief into a dynamic knowledge graph, retrieves task-relevant subgraphs, and aggregates them into a spatial belief. A planner proposes multi-step action sequences from these two beliefs, and a critic validates each planned action against temporal consistency, spatial feasibility, the current observation, and safety, sending feedback that triggers replanning when an action is rejected. In TextWorld cooking tasks, the paper reports a 31.25% improvement in success rate over the best baseline with Qwen2.5-72b and a 24.7% increase in average score, with ablation results showing that removing the full memory module drops success to zero and removing the critic or spatial memory sharply degrades harder tasks. The authors read these results as evidence that memory architecture and closed-loop verification matter more than model scale.

Load-bearing premise

The reported success-rate advantages rest on eight randomly generated games per difficulty level (32 in total), with no confidence intervals or significance tests; if those seeds are unrepresentative, the headline gains could shrink or disappear.

Editorial extensions

If this is right

  • A well-designed memory system can substitute for part of the raw capability gap between open-source and proprietary models: Qwen2.5-72b powered STMA reaches higher success rates than GPT-4o powered baselines on most difficulty levels.
  • Memory becomes a precondition rather than a luxury: without the spatio-temporal memory module the agent completes no tasks, so any embodied agent facing similar partial observability needs persistent structured memory.
  • The critic's value scales with task difficulty: removing it costs little on level 1 but sharply reduces success on levels 2 through 4, meaning closed-loop verification matters most when plans are long and hallucination-prone.
  • Spatial beliefs must be accurate to help: the ablation suggests an incorrect spatial belief can mislead the agent more than no spatial belief at all, so memory quality and freshness are as important as memory presence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension is whether the same memory stack transfers to visual embodied benchmarks or real robots, where perception noise and continuous state make dynamic knowledge-graph updates harder; the paper only evaluates TextWorld.
  • The result that inaccurate spatial belief is worse than none suggests a design principle—agents should mark spatial triples with confidence or timestamps and downweight stale relations—that the paper does not explore.
  • Because the history buffer keeps only the most recent 25 turns and tasks cap at 50 turns, the claimed benefits are demonstrated on a short-horizon slice; scaling to hundreds of steps would test whether the summarizer compresses without losing critical details.
  • Treating the critic as a classifier rather than a generator suggests a cheap improvement: generate several candidate plans and have the critic rank them instead of refining a single plan sequentially; the paper does not report such a comparison.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes STMA, an LLM-based agent framework for long-horizon embodied task planning in the TextWorld cooking environment. STMA combines a temporal memory (history buffer plus summarizer), a spatial memory built as a dynamically updated knowledge graph with retrieval and aggregation, and a planner-critic loop that iteratively proposes and validates action plans. The authors evaluate STMA against ReAct, Reflexion, and AdaPlanner using GPT-4o and Qwen2.5-72b-instruct on 32 generated games (8 per difficulty level), reporting higher success rates and average scores, with the largest gains on the open-source model. Ablation studies examine the contribution of each module. The central empirical claim is that STMA outperforms the best baseline by 31.25 percentage points in success rate and 24.7 percentage points in average score on Qwen2.5-72b.

Significance. If the reported improvements are reliable, the paper makes a useful modular contribution: it demonstrates that a memory-augmented planner-critic architecture can improve the performance of a moderately sized open-source LLM on multi-step embodied tasks, and it provides fairly detailed prompts and pseudocode that would help others reproduce or adapt the approach. The framework itself is clearly presented and the ablation design is sensible. However, the evaluation rests on only 8 games per condition, no confidence intervals or significance tests, an overstatement of absolute percentage-point differences as percentage improvements, and no released code or seeds. These issues are directly load-bearing for the paper's headline claim, so the current evidence is not sufficient to establish the stated advantage.

major comments (4)
  1. [§5.2, Table 1] The headline comparison is not statistically supported. Each difficulty level uses only 8 randomly generated games, so a single game changes the level's success rate by 12.5 percentage points, yet no confidence interval, standard error, or significance test is reported for any success-rate comparison. For example, the Level 3 Qwen2.5-72b success-rate gap (STMA 62.5% vs. Reflexion 25.0%, a 3-of-8 vs. 2-of-8 split in actual counts) would not approach conventional significance. The reported average-score standard deviations describe variation across the 8 games and do not account for LLM sampling variability or multiple comparisons. The authors should report confidence intervals, run significance tests (e.g., Fisher's exact test per level, or a paired bootstrap over the 32 games), increase the number of games, and separately quantify run-to-run stochasticity of the LLMs.
  2. [Abstract and §5.2] The claimed "31.25% improvement in success rate" and "24.7% increase in average score" are absolute percentage-point differences, not relative improvements, and the wording overstates the result. From Table 1, the Qwen2.5-72b success rate averaged over the four levels is 71.875% for STMA versus 40.625% for the best baseline (Reflexion); the 31.25 figure is the difference in percentage points, while the relative improvement is about 77%. The same applies to the 24.7% average-score figure. The abstract, introduction, and Section 5.2 should be rewritten to distinguish percentage-point gaps from relative improvements, or the analysis should report both consistently.
  3. [§5.1 and Appendix A] The baseline comparison is difficult to verify because all three baselines (ReAct, Reflexion, AdaPlanner) were implemented by the authors, and the paper does not provide their prompts, hyperparameters, tuning procedure, or the complete list of game seeds and generation commands. Only three example commands are shown in Appendix A, and no code is released. Since the paper claims superiority over these baselines using small samples, the absence of reproducible baseline configurations and evaluation seeds is a major gap. The authors should release code and seeds, or at least provide full baseline prompts, hyperparameter settings, and a clear statement of whether any hyperparameters were tuned on the same 32-game set used for the reported numbers.
  4. [§D.5 and §5.3] The critic prompt explicitly instructs the model to "Encourage Exploration, Give less 'False' prediction," which biases the critic toward accepting proposed actions. This weakens the interpretation of the planner-critic mechanism as an independent validity check and complicates the ablation "STMA w/o Critic," since the critic is not a neutral classifier. If the critic is deliberately lenient, the performance drop without it may partly reflect the loss of a biased (but sometimes useful) filter rather than the value of principled verification. The authors should either remove this instruction and rerun the experiments, or quantify the critic's false-acceptance and false-rejection rates to show that the bias does not drive the reported gains.
minor comments (7)
  1. [Table 1 caption] The caption states that AS is reported as mean ± standard deviation, but it does not clarify that SR has no associated uncertainty; adding a standard error or a confidence interval for SR would make the table more informative.
  2. [Abstract/Introduction/§5.2] The percentage-point versus relative-improvement terminology appears in three places; the correction identified in the major comments should be applied consistently across all of them.
  3. [Appendix A] The paper shows only three example game-generation commands with seeds 1001, 20002, and 303; please provide the full list of seeds and commands for all 32 games, or release the scripts, so that the exact evaluation set is reproducible.
  4. [Table 2] The ablation "STMA w/o Spatio-Temporal Memory" is a sanity check rather than an informative ablation, because without any memory the agent cannot recall the recipe and therefore cannot complete any task by construction; this should be stated explicitly in the text.
  5. [Equation (4) and Algorithm 2] The summarizer operator is rendered as the unusual symbol "S⊓⇕," which appears to be a placeholder or font artifact; it should be typeset as a proper mathematical symbol or replaced with a named function such as Summarize.
  6. [Figure 4] The caption of Figure 4 says "Average score vs. steps" but does not state which difficulty level or which aggregate over levels is shown; please clarify the plot's construction.
  7. [Related Work] SwiftSage is discussed in the related work but is not included as a baseline; if it was excluded for practical reasons, state this, since the claim of outperforming "state-of-the-art" baselines would be stronger with a broader baseline set.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: STMA is an empirically evaluated architecture; its reported gains are direct measurements against external baselines, not derivations from fitted inputs or self-citations.

full rationale

This paper makes no derivation-based circularity claims. STMA is a proposed framework whose components (temporal memory, dynamic knowledge graph, planner-critic) are design choices, not quantities derived from equations that already contain the target result. The central evidence, Table 1, reports success rate and average score as directly measured outcomes in TextWorld, with SR and AS defined in Section 5.1 from task completion and intermediate scores; STMA's parameters (retrieval top-n, K-hop, history buffer length) are fixed hyperparameters, not fitted to maximize the reported SR or AS. The comparison against ReAct, Reflexion, and AdaPlanner is an external empirical benchmark, and the paper contains no author self-citations that carry the argument. Ablations remove components and measure performance drops; the zero score when spatio-temporal memory is completely removed follows from the environment's requirement to recall recipe steps and is an expected consequence, not a circular equivalence. Concerns about the 32-game evaluation, lack of significance testing, and the absolute-versus-relative framing of the 31.25% improvement are statistical reliability and reporting issues, not circularity. Accordingly, the derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The framework's constructs (temporal belief, spatial belief) are conceptual and do not carry independent falsifiable handles. The free parameters are hand-chosen hyperparameters, not fitted to optimize the target metric.

free parameters (3)
  • top_n = 8
    Number of entities selected in the semantic filtering step of the retrieval algorithm; chosen by the authors based on environment size (Appendix A.2).
  • K (hop count) = 3
    K-hop neighborhood expansion depth; chosen because the environment has at most 12 rooms (Appendix A.2).
  • history_buffer_length = 25
    FIFO buffer stores the most recent 25 interactions; chosen because each task is limited to 50 turns (Appendix A.2).
assumptions (5)
  • domain assumption TextWorld cooking tasks are a valid proxy for long-horizon embodied task planning.
    Used to ground all experimental claims; Section 5.1 defines the environment and difficulty levels.
  • domain assumption LLMs with the provided prompts can reliably summarize history, extract spatial triples, and aggregate them, without any verification of extraction accuracy.
    No manual or automated evaluation of summarization or relation extraction quality is reported; Sections 4.1, 4.2, and Appendix D.
  • domain assumption The 32 generated games (8 per difficulty level) with the chosen seeds adequately represent task difficulty.
    No seed sensitivity analysis is reported; Table 1 and Section 5.2 use these games as the full evidence.
  • ad hoc to paper A critic prompted to 'Encourage Exploration, Give less False prediction' still provides a meaningful validity check rather than rubber-stamping actions.
    Appendix D.5 explicitly instructs the critic to be permissive; the paper does not evaluate how this bias affects the planner-critic loop.
  • standard math The POMDP formalism in Section 3 accurately models the agent-environment loop, even though the method does not use it in the algorithm or experiments.
    Section 3 introduces the POMDP tuple and belief factorization but the rest of the paper does not invoke these equations in the design or analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of STMA: A Spatio-Temporal Memory Agent for Long-Horizon Embodied Task Planning." pith.science (2026). https://pith.science/paper/LRVRRG6V

@misc{pith2026250210177,
  author       = {Pith},
  title        = {Pith review of: STMA: A Spatio-Temporal Memory Agent for Long-Horizon Embodied Task Planning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LRVRRG6V}},
  note         = {Machine review of arXiv:2502.10177}
}
read the original abstract

A key objective of embodied intelligence is enabling agents to perform long-horizon tasks in dynamic environments while maintaining robust decision-making and adaptability. To achieve this goal, we propose the Spatio-Temporal Memory Agent (STMA), a novel framework designed to enhance task planning and execution by integrating spatio-temporal memory. STMA is built upon three critical components: (1) a spatio-temporal memory module that captures historical and environmental changes in real time, (2) a dynamic knowledge graph that facilitates adaptive spatial reasoning, and (3) a planner-critic mechanism that iteratively refines task strategies. We evaluate STMA in the TextWorld environment on 32 tasks, involving multi-step planning and exploration under varying levels of complexity. Experimental results demonstrate that STMA achieves a 31.25% improvement in success rate and a 24.7% increase in average score compared to the state-of-the-art model. The results highlight the effectiveness of spatio-temporal memory in advancing the memory capabilities of embodied agents.

Figures

Figures reproduced from arXiv: 2502.10177 by the authors.

Figure 1
Figure 1. Comparative overview of ReAct and STMA. (a) ReAct uses a simple history buffer to store action-feedback pairs and reasoning information, generating actions one step at a time. This approach lacks structured spatio-temporal reasoning, limiting its adaptability in complex, long￾horizon tasks. (b) STMA utilizes dedicated spatial memory and temporal memory, summarized into refined spatial belief and temporal belief usin… view at source ↗
Figure 2
Figure 2. Overview of STMA. STMA consists of two components: a spatio-temporal memory module and a planner-critic module. The spatio-temporal memory module is divided into a temporal memory submodule and a spatial memory submodule, which provide temporal and spatial beliefs, respectively. These beliefs serve as the spatio-temporal context for the planner-critic module. The planner-critic module consists of a planner and a cri… view at source ↗
Figure 3
Figure 3. Interaction with the Textworld Environment. The interaction pattern between Textworld and our framework involves the environment providing the agent with the current observation, inventory, and a list of possible actions. Based on the agent’s executed actions, the environment returns feedback. These pieces of information are recorded in STMA’s spatio-temporal memory, serving as the necessary context for the planner-… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Average score vs. steps of different frameworks (powered by GPT-4o) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: STMA versus Reflexion in Case 1. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: STMA versus Reflexion in Case 2. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

90 extracted references · 51 canonical work pages

  1. [1]

    Larp: Language-agent role play for open-world games

    Ming Yan, Ruihao Li, Hao Zhang, Hao Wang, Zhilan Yang, and Ji Yan. Larp: Language-agent role play for open-world games. arXiv preprint arXiv:2312.17653, 2023

    arXiv 2023

  2. [2]

    A survey on recent advances in llm-based multi-turn dialogue systems

    Zihao Yi, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. A survey on recent advances in llm-based multi-turn dialogue systems. arXiv preprint arXiv:2402.18013, 2024

    arXiv 2024

  3. [3]

    Llm-rec: Personalized recommendation via prompting large language models

    Hanjia Lyu, Song Jiang, Hanqing Zeng, Yinglong Xia, Qifan Wang, Si Zhang, Ren Chen, Christopher Leung, Jiajie Tang, and Jiebo Luo. Llm-rec: Personalized recommendation via prompting large language models. arXiv preprint arXiv:2307.15780, 2023

    arXiv 2023

  4. [4]

    A survey of embodied ai: From simulators to research tasks

    Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2):230–244, 2022

    2022

  5. [5]

    Embodied artificial intelligence: Trends and challenges

    Rolf Pfeifer and Fumiya Iida. Embodied artificial intelligence: Trends and challenges. Lecture notes in computer science, pages 1–26, 2004

    2004

  6. [6]

    Memory matters: The need to improve long-term memory in llm-agents

    Kostas Hatalis, Despina Christou, Joshua Myers, Steven Jones, Keith Lambert, Adam Amos- Binks, Zohreh Dannenhauer, and Dustin Dannenhauer. Memory matters: The need to improve long-term memory in llm-agents. In Proceedings of the AAAI Symposium Series, volume 2, pages 277–280, 2023

    2023

  7. [7]

    Scene memory transformer for embodied agents in long-horizon tasks

    Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 538–547, 2019

    2019

  8. [8]

    Neural map: Structured memory for deep reinforce- ment learning

    Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforce- ment learning. arXiv preprint arXiv:1702.08360, 2017

    arXiv 2017

Show all 90 references
  1. [9]

    Prediction of temporal atmospheric boundary layer height using long short-term memory network

    Nishant Kumar, Kirti Soni, and Ravinder Agarwal. Prediction of temporal atmospheric boundary layer height using long short-term memory network. Tellus A: Dynamic Meteorology and Oceanography, 73(1):1–14, 2021

    2021

  2. [10]

    Graph neural networks: A review of methods and applications

    Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI open, 1:57–81, 2020

    2020

  3. [11]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  4. [12]

    Textworld: A learning environment for text-based games

    Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Conjunction w...

    2018

  5. [13]

    Rule-driven coordination agents: A self-configurable agent architecture for distributed control

    M Brian Blake. Rule-driven coordination agents: A self-configurable agent architecture for distributed control. In Proceedings 5th International Symposium on Autonomous Decentralized Systems, pages 271–277. IEEE, 2001

    2001

  6. [14]

    Q-decomposition for reinforcement learning agents

    Stuart J Russell and Andrew Zimdars. Q-decomposition for reinforcement learning agents. In Proceedings of the 20th international conference on machine learning (ICML-03) , pages 656–663, 2003

    2003

  7. [15]

    Expel: Llm agents are experiential learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642, 2024

    2024

  8. [16]

    A comparative study of a-star algorithms for search and rescue in perfect maze

    Xiang Liu and Daoxiong Gong. A comparative study of a-star algorithms for search and rescue in perfect maze. In 2011 international conference on electric information and control engineering, pages 24–27. IEEE, 2011. 11

    2011

  9. [17]

    A survey of robotic motion planning in dynamic environments

    MG Mohanan and Ambuja Salgoankar. A survey of robotic motion planning in dynamic environments. Robotics and Autonomous Systems, 100:171–185, 2018

    2018

  10. [18]

    Automated Planning: theory and practice

    Malik Ghallab, Dana Nau, and Paolo Traverso. Automated Planning: theory and practice . Elsevier, 2004

    2004

  11. [19]

    Constraint-based scheduling and planning

    Philippe Baptiste, Philippe Laborie, Claude Le Pape, and Wim Nuijten. Constraint-based scheduling and planning. In Foundations of artificial intelligence, volume 2, pages 761–799. Elsevier, 2006

    2006

  12. [20]

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(64...

    2018

  13. [21]

    Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards

    Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:170...

    2017 arXiv

  14. [22]

    Distributed prioritized experience replay

    Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933, 2018

    2018 arXiv

  15. [23]

    Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications

    Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE transactions on cybernetics, 50(9):3826–3839, 2020

    2020

  16. [24]

    Progressive neural networks

    Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016

    2016 arXiv

  17. [25]

    Model- based reinforcement learning for atari

    Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model- based reinforcement learning for atari. arXiv preprint arXiv:1903.00374, 2019

    1903 arXiv

  18. [26]

    Model-based reinforcement learning: A survey

    Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. Model-based reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 16(1):1–118, 2023

    2023

  19. [27]

    Human-level control through deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015

    2015

  20. [28]

    Deep reinforcement learning (drl): Another perspective for unsupervised wireless localization

    You Li, Xin Hu, Yuan Zhuang, Zhouzheng Gao, Peng Zhang, and Naser El-Sheimy. Deep reinforcement learning (drl): Another perspective for unsupervised wireless localization. ieee internet of things journal, 7(7):6279–6287, 2019

    2019

  21. [29]

    An overview of the action space for deep reinforcement learning

    Jie Zhu, Fengge Wu, and Junsuo Zhao. An overview of the action space for deep reinforcement learning. In Proceedings of the 2021 4th International Conference on Algorithms, Computing and Artificial Intelligence, pages 1–10, 2021

    2021

  22. [30]

    Hierarchical rein- forcement learning: A comprehensive survey

    Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical rein- forcement learning: A comprehensive survey. ACM Computing Surveys (CSUR), 54(5):1–35, 2021

    2021

  23. [31]

    A survey of meta-reinforcement learning

    Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028, 2023

    2023 arXiv

  24. [32]

    Model-agnostic meta-learning for fast adap- tation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017. 12

    2017

  25. [33]

    Chateval: Towards better llm-based evaluators through multi-agent debate

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201, 2023

    2023 arXiv

  26. [34]

    Smart-llm: Smart multi-agent robot task planning using large language models

    Shyam Sundar Kannan, Vishnunandan LN Venkatesh, and Byung-Cheol Min. Smart-llm: Smart multi-agent robot task planning using large language models. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 12140–12147. IEEE, 2024

    2024

  27. [35]

    Llm-based multi-agent reinforcement learning: Current and future directions

    Chuanneng Sun, Songjun Huang, and Dario Pompili. Llm-based multi-agent reinforcement learning: Current and future directions. arXiv preprint arXiv:2405.11106, 2024

    2024 arXiv

  28. [36]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  29. [37]

    A generalist agent

    Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022

    2022 arXiv

  30. [38]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  31. [39]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    2024

  32. [40]

    Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks

    Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. Advances in Neural Information Processing Syst...

    2024

  33. [41]

    Dual-process and dual-system theories of reasoning

    Keith Frankish. Dual-process and dual-system theories of reasoning. Philosophy Compass, 5(10):914–926, 2010

    2010

  34. [42]

    Thinking, fast and slow/farrar

    Daniel Kahneman. Thinking, fast and slow/farrar. Straus and Giroux, 2011

    2011

  35. [43]

    Adaplanner: Adaptive planning from feedback with language models

    Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  36. [44]

    Hipporag: Neurobiologically inspired long-term memory for large language models

    Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobiologically inspired long-term memory for large language models. arXiv preprint arXiv:2405.14831, 2024

    2024 arXiv

  37. [45]

    The human hippocampus and spatial and episodic memory

    Neil Burgess, Eleanor A Maguire, and John O’Keefe. The human hippocampus and spatial and episodic memory. Neuron, 35(4):625–641, 2002

    2002

  38. [46]

    A review: Knowledge reasoning over knowledge graph

    Xiaojun Chen, Shengbin Jia, and Yang Xiang. A review: Knowledge reasoning over knowledge graph. Expert systems with applications, 141:112948, 2020

    2020

  39. [47]

    Planning and acting in partially observable stochastic domains

    Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134, 1998

    1998

  40. [48]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  41. [49]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024

    2024

  42. [50]

    Morris, Brandon Duderstadt, and Andriy Mulyar

    Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic embed: Training a reproducible long context text embedder, 2024

    2024

  43. [51]

    prepare meal

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314, 2024. 13 Task Difficulty Setti...

    2024 arXiv

  44. [52]

    ** Read the T r a j e c t o r y :** Ca ref ul ly review each ** O b s e r v a t i o n ** and the c o r r e s p o n d i n g ** Action ** taken by the robot in the provided t r a j e c t o r y

  45. [53]

    Consider how each action responds to the pr ec ed in g o b s e r v a t i o n

    ** Analyze I n t e r a c t i o n s :** For each pair of ** O b s e r v a t i o n ** and ** Action ** , u n d e r s t a n d the context and purpose of the robot ’ s behavior . Consider how each action responds to the pr ec ed in g o b s e r v a t i o n

  46. [54]

    ** Exclude Latest Feedback :** Do not include the ** FeedBack ** section in your summary , as it r e p r e s e n t s the current state awaiting the robot ’ s next action

  47. [55]

    Your summary should : - Describe the sequence of actions the robot has p er for me d

    ** S umm ar iz e the History :** Compile a c o m p r e h e n s i v e and detailed summary of the robot ’ s actions and i n t e r a c t i o n s based on the analyzed t r a j e c t o r y . Your summary should : - Describe the sequence of actions the robot has p er for me d . - H...

  48. [56]

    ** Format the Summary :** Present the summary in clear , coherent English , ensuring that it a c c u r a t e l y reflects the robot ’ s h i s t o r i c a l behavior as derived from the t r a j e c t o r y

  49. [57]

    go xxx

    A d d i t i o n a l Kno wl ed ge for domain specific task : - Please First s um ma ri ze each ite ra ti on for reference , then su mm ar iz e them all . Plases Notice the objects and entrance ’ s di re ct io n of rooms .! - The cookbook in the kitchen is e xt rem el y im por t...

  50. [58]

    This means that the a va il ab le actions and their meanings will shift after each step

    ** Dynamic E n v i r o n m e n t **: Each time you take an action , the e n v i r o n m e n t will change . This means that the a va il ab le actions and their meanings will shift after each step . It ’ s crucial that you c are fu ll y consider how each current action will aff...

  51. [59]

    This means your plan might quickly become outdated as you proceed

    ** Plan Validity **: The plan you create is based on the current environment , but the e n v i r o n m e n t is dynamic , and the set of a vai la bl e actions may change s i g n i f i c a n t l y during ex ec ut io n . This means your plan might quickly become outdated as you ...

  52. [60]

    ** E x p l o r a t i o n and E x p e r i m e n t a t i o n **: We en co ur age you to embrace e x p l o r a t i o n and learning through trial and error . Don ’ t hesitate to test di ff er en t a p p r o a c h e s and learn from outcomes , as this will help refine your s t r a...

  53. [61]

    ** Only generate action that is proper **: only valid action allowed , ** No if st at em en t action allowed !** ** Your task is :**

  54. [62]

    ** Think through each step in a chain of thought ** to de ter mi ne the next useful subgoal that will help achieve the primary task

  55. [63]

    U n d e r s t a n d the overall e n v i r o n m e n t ( your belief ) .**

    ** Based on what you have done and the spatial memory which describe the r e l a t i o n s h i p of the object . U n d e r s t a n d the overall e n v i r o n m e n t ( your belief ) .**

  56. [64]

    ** Propose a subgoal ** based on the current e n v i r o n m e n t and information , ensuring it c o n t r i b u t e s to the main goal

  57. [65]

    ** Plan a sequence of actions ** that will complete the proposed subtask , drawing from the list of av ail ab le actions

  58. [66]

    ** Ensure ** that the sequence of actions e f f e c t i v e l y fulfills the subtask and p r o g r e s s e s toward the overall goal

  59. [67]

    ** Fully Consider what spatial memory shows ( This provide you with overall insight of the whole e n v i r o n m e n t ) .**

  60. [68]

    use correct action chop / cut / slice the correct i n g r e d i e n t follow the cookbook

    ** Get as much score as you can **: You can get 1 point if you : - 1. use correct action chop / cut / slice the correct i n g r e d i e n t follow the cookbook . - 2. use correct method fry / grill / roast to cook the i n g r e d i e n t follow the cookbook . - 3. collect corr...

  61. [69]

    Base on the Spatio Memory , What s itu at io n you are facing ?

  62. [70]

    Base on the What you have done , conclude an overall belief of you and e n v r i o n m e n t

  63. [71]

    What is your next subgoal that can achieve the task ? Why ? - Is this subgoal r ed un den t ? - What is the c o n s e q u e n c e s when this subgoal com pl et ed ? - Whether this subgoal fit ** Every command ** in your K now le dg e

  64. [72]

    You can modify it

    Re - think whether it is a good subgoal . You can modify it

  65. [73]

    What is your plan to achieve the subgoal ?

  66. [74]

    Please go through ** each step ** in your plan ( each step is a s e l e c t e d _ a c t i o n ) , consider : - what may be the c o n s e q u e n c e s of the action ? - Is this a proper action ? - Is this a useless or re du nd ent action ? - Whether this action fit ** Every co...

  67. [75]

    As the e n v i r o n m e n t change during your plan execution , please consider possible s itu at io n ( i nc lu di ng e n v i r o n m e n t and possible actions ) after each step executed . ... ( more thought ) ** Replan **: after go thought the thought step above , please t...

  68. [76]

    < s e l e c t e d _ a c t i o n 1 >

    Plan : ‘‘‘ YAML Subgoal : "..." # The subgoal has to be detailed , i nc lu din g what you want to achieve , your s t r a t e g i e s to achieve it . ( and your plan action sequence in natural language ) Action Plan : - " < s e l e c t e d _ a c t i o n 1 >" - " < s e l e c t e...

  69. [77]

    Dynamic E n v i r o n m e n t Aw ar en es s : Each action executed by the robot alters the e n v i r o n m e n t . This means that the o b s e r v a t i o n s available , the s i g n i f i c a n c e of current ca nd id at e actions , and the 25 c o n s e q u e n c e s of ex ec...

  70. [78]

    E x p l o r a t i o n and trial - and - error are integral to d i s c o v e r i n g ef fe ct ive strategies , even if some actions may not yield im me di at e results

    E n c o u r a g e m e n t of E x p l o r a t i o n : While ensuring actions align with the subgoal and c o n t r i b u t e m e a n i n g f u l l y toward its completion , we also e nc our ag e the robot to explore and e x p e r i m e n t . E x p l o r a t i o n and trial - and...

  71. [79]

    Robot ’ s Subgoal : A clear d e s c r i p t i o n of the specific subgoal the robot is c ur re ntl y working toward

  72. [80]

    E xe cut io n History : A s u m m a r i z e d report de ta il ing the robot ’ s actions up to this point

  73. [81]

    Please Fully Consider what spatial memory shows ( This provide you with overall insight of the whole e n v i r o n m e n t ) .*

    ** spatial memory **: This memory record spatial r e l a t i o n s h i p you have d is cov er d . Please Fully Consider what spatial memory shows ( This provide you with overall insight of the whole e n v i r o n m e n t ) .*

  74. [82]

    The K now le dg e part will tell you how to achieve the goal !

    K no wle dg e : This is the most i mp or ta nt part in your input . The K now le dg e part will tell you how to achieve the goal !

  75. [83]

    Current E n v i r o n m e n t : The robot ’ s current environment , showing details of its surroundings , relevant objects , and a va il ab le paths

  76. [84]

    Please judge the proposed action based on possible action choice

    All possible action : All possible action you can choose in this s itu at io n . Please judge the proposed action based on possible action choice

  77. [85]

    Your r e s p o n s i b i l i t i e s :

    Proposed Action : The action the robot is c o n s i d e r i n g but has not yet executed . Your r e s p o n s i b i l i t i e s :

  78. [86]

    - Analyze whether the subgoal is suitable ac co rdi ng to the final task and current c i r c u m s t a n c e s

    R ea son in g : - Base on the Spatio Memory , What s it uat io n you are facing ? - Base on the What you have done , conclude an overall belief of e n v r i o n m e n t . - Analyze whether the subgoal is suitable ac co rdi ng to the final task and current c i r c u m s t a n c...

  79. [87]

    Action S u i t a b i l i t y : E x p l i c i t l y indicate if the proposed action is suitable by s el ec ti ng either ‘ True ‘ ( action is relevant and b e n e f i c i a l ) or ‘ False ‘ ( action is u n s u i t a b l e or p o t e n t i a l l y c o u n t e r p r o d u c t i v e )

  80. [88]

    26 - The key problem with the action in relation to the current e n v i r o n m e n t and subgoal

    C o n s t r u c t i v e Feedback : Provide a detailed e x p l a n a t i o n if the action is unsuitable , covering : - Why it was not suitable . 26 - The key problem with the action in relation to the current e n v i r o n m e n t and subgoal . - How to improve : Provide speci...

  81. [89]

    Internal Re aso ni ng : Before pro vi di ng the output , use a Chain of Thought approach to reason through your decision based on the subgoal , current state , and action s u i t a b i l i t y

  82. [90]

    S t r u c t u r e d Output : Follow this format : ‘‘‘ YAML Action S u i t a b i l i t y : True / False Feedback : " Detailed e x p l a n a t i o n covering the reasons for s u i t a b i l i t y or unsuitability , key issues , a c t i o n a b l e improvements , and a d d i t i ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.