Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

CRAFT: Coaching Reinforcement Learning Autonomously using Foundation Models for Multi-Robot Coordination Tasks

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that a foundation-model coaching loop of LLM-generated curricula, LLM-generated rewards, and VLM-guided reward refinement can teach teams of robots long-horizon coordination tasks that standard multi-agent reinforcement…

desk verdict A credible closed-loop LLM/VLM coaching framework for multi-robot coordination that deserves peer review, but the VLM success gate is unvalidated and the headline numbers are maxima over five seeds. read the letter →

arxiv 2509.14380 v3 pith:G3PBFQPD submitted 2025-09-17 cs.RO

Seoyeon Choi , Kanghyun Ryu , Jonghoon Ock , Negar Mehr This is my paper · ORCID
classification cs.RO
keywords multi-agentreinforcementlearningcurriculumlargelanguagemodelsvision-languagerewarddesignrobotcoordinationrefinementquadrupednavigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that foundation models can act as an automated coach for multi-robot coordination, replacing the human-designed curricula and hand-calibrated rewards that multi-agent reinforcement learning usually needs. CRAFT generates a curriculum of subtasks with a language model, writes a reward function for each subtask in executable code, and then refines those rewards through a visual-language-model evaluation and advice loop until the learned policy succeeds. The claim is supported by experiments in three long-horizon coordination benchmarks where CRAFT reaches up to 90% success in Quadruped Gate, 60% in Quadruped Seesaw, and 100% in Two Arm Lift, while the environment rewards alone often fail completely. A zero-shot transfer of one learned policy to physical quadruped robots achieved 65% success. If the claim holds, it offers a path to teach teams of robots new coordination skills without expert reward engineering.

What carries the argument

The load-bearing mechanism is the closed coaching loop made of five modules: a curriculum LLM that decomposes the task and refines several candidate curricula into one; a reward-generation LLM that turns each subtask description into executable Python reward code with named reward components; a policy-evaluation VLM that judges rollouts from snapshot images and state metrics; an advice VLM that reads failure reasons and plots of per-component reward learning curves and proposes concrete changes; and a refine LLM that rewrites the reward accordingly. Sequential subtasks are trained by initializing from the previous policy, resetting the exploration noise and the value network to avoid loss of plasticity. The working parts are the curriculum, the executable dense rewards, and the VLM-guided refinement loop acting together.

What would settle it

Count, on a sample of 100 saved rollouts, how often the evaluation VLM's success or failure verdict disagrees with a ground-truth check of whether the task (gate crossed, or pot lifted above 0.1 m with tilt under 30 degrees) was actually completed; if disagreement exceeds about 10%, the loop's premise is violated.

Watch

Extended reading notes

Core claim

The central discovery is that an LLM-VLM "coaching" loop can convert a single natural-language task description into a trainable curriculum plus rewards, and that this loop, rather than any single component, is what makes complex coordination learnable. In the paper's own framing, curriculum learning changes the problem from optimizing one long-horizon reward to optimizing a sequence of rewards, and CRAFT is a method for automatically producing that sequence. The authors report that without the curriculum (even with CRAFT's final rewards) and without the VLM-based refinement, success rates fall substantially, and that the environment-provided rewards almost never succeed.

Load-bearing premise

The pipeline's reliability rests on the VLM consistently recognizing failure from snapshot images and state metrics and on the advice VLM proposing reward changes that actually improve learning; the paper itself notes that foundation models lack native 3D understanding and that trial outcomes vary due to stochasticity.

Editorial extensions

If this is right

  • If CRAFT works as reported, multi-robot coordination tasks that currently require expert curriculum design can be tackled from a language prompt alone, reducing human effort to environment setup and state-variable definitions.
  • The same coaching loop should apply to other long-horizon multi-agent problems beyond the three benchmarks, since the curriculum and reward modules only need the environment description and a set of interpretable state variables.
  • Real-world transfer at 65% success without fine-tuning suggests that policies trained in simulation by CRAFT can serve as initializations for on-robot learning with modest additional data.
  • The failure analysis suggests that standard dense environment rewards can lock policies into local optima (e.g., grasping without lifting); CRAFT's iterative reward refinement is a way to escape such optima automatically.
  • Because CRAFT logs per-component reward curves, the refinement loop doubles as a diagnostic tool: it identifies which reward term is too weak or too dominant, providing human-readable explanations of training failures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors do not pursue is to use the learned per-component reward curves to detect when a task has multiple equally good coordination strategies; the VLM advice could then be prompted to preserve diversity rather than converge to a single behavior.
  • CRAFT's reliance on snapshot images for evaluation could be stress-tested by swapping the VLM judge for a scripted success detector; if success stays high, the visual pipeline is redundant, and if it collapses, visual feedback is the key ingredient.
  • The same loop could be applied to asymmetric roles (where agents have different capabilities) by letting the curriculum LLM name role-specific subtasks instead of a shared team reward, which the paper's team-reward framing does not address.
  • The reported 65% real-world transfer suggests that the gap to robust deployment is mostly perceptual and control calibration; a promising next step is to close that gap by fine-tuning the transferred policy with a few real-world rollouts using the same VLM evaluation loop.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents CRAFT, a framework in which LLMs generate a curriculum and reward functions for multi-robot MARL tasks, and a VLM evaluates trained policies and provides advice that is used to iteratively refine rewards. The authors evaluate CRAFT on two multi-quadruped navigation tasks and one bimanual manipulation task, report success-rate advantages over several baselines, and demonstrate transfer of a learned quadruped policy to physical hardware. The central claim is that foundation-model coaching enables learning of coordination behaviors that are infeasible with standard MARL training and hand-specified rewards.

Significance. If the reported effects are real, CRAFT is a valuable step toward reducing human effort in curriculum design and reward engineering for multi-robot coordination, and the real-world transfer result is a useful existence proof. The paper's strengths include a clearly described pipeline, full prompts and example outputs in the appendix, and the demonstration that LLM/VLM-generated rewards can support successful MARL training in three nontrivial tasks. However, the evaluation as reported does not yet establish that the coaching loop is the cause of the success: the key internal gate is unvalidated, and the headline numbers are selected maxima.

major comments (3)
  1. [Section V-B, Fig. 6 and Table I] The headline success rates (90%, 60%, 100%) are the maximum values over five experiments, and Fig. 6 shows only the top-three of five runs; no per-seed averages, error bars, or distributions are provided. Because the comparison against baselines uses this best-case selection, the claim in Q1 that CRAFT 'achieves the highest success rate on every environment' is not supported by the reported evidence. Please report mean and variance over all five seeds, or otherwise show that the advantage is not driven by one lucky run.
  2. [Section IV-C and Section IV-D] The VLM-based success gate that advances the curriculum is never validated against the environment's ground-truth success condition. The paper reports no false-positive rate, false-negative rate, or agreement statistics between the VLM's judgment and the simulator's success flag, and it does not ablate the gate by substituting ground-truth success. Since a false-positive VLM judgment can move training to the next subtask before the current behavior is actually learned, the internal decision-making of CRAFT is a load-bearing but unmeasured component; please add a calibration study or an oracle-gate ablation.
  3. [Section V-B, Table I] The 'effective curricula ratio' is defined as the fraction of curricula that achieve any nonzero success rate; under this definition, a curriculum with a single successful episode out of 100 is counted as effective. This metric is too weak to support the Q3 conclusion that VLM-guided reward refinement 'improves curricula reliability,' because reliability should be reflected in the full success-rate distribution rather than in a binary nonzero threshold. Please report per-curriculum summary statistics or use a threshold that reflects meaningful success.
minor comments (5)
  1. [Appendix A.1, Prompt 1] There are several typos in the prompt text, e.g., 'chage' for 'change' and 'Varaibles' for 'Variables'; please proofread the appendix.
  2. [Section IV-D, Fig. 3] The example in Fig. 3 shows a single refinement; the paper states that J=3 iterations were sufficient, but it does not report the number of refinement iterations actually needed per subtask across runs. Please include this information or clarify.
  3. [Section V-A2] The description of no_refinement should specify how many reward resamples are allowed and whether the resampling uses any information from the failed reward beyond stochasticity.
  4. [Section V-C] The hardware section mentions 'action component-wise scaling' but does not describe how the scaling was determined; please provide details or a reference.
  5. [Section V-B and Fig. 6] The sentence 'its unreliable curriculum generation strongly degrades its average performance shown in Fig. 6' refers to an average, but Fig. 6 plots top-three curves; please reconcile the terminology.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: CRAFT's headline success rates are measured against ground-truth environment conditions, not against the VLM's own judgments.

full rationale

The paper's central claim is that CRAFT learns coordination policies that are infeasible without human-designed curricula or hand-calibrated rewards. This claim is supported by simulator-based success rates defined in Section V-A: 'We define a task as success when both agents have passed the gate' (Quadruped Gate), 'success when one of the agents reaches the target platform' (Quadruped Seesaw), and 'success when the pot is lifted above 0.1m and no more tilted than 30 degrees' (Two Arm Lift). These are environment ground-truth conditions, not the evaluation VLM's verdicts, so the measured success is not defined in terms of the VLM's own outputs. The VLM gates curriculum advancement and reward refinement, but the final reported numbers are computed independently by the simulator over 100 random initial conditions. There is no equation in the paper that makes the target success identical to a fitted parameter or to a VLM judgment. The reward refinement loop is an empirical heuristic search; its output is tested by training and evaluation, not by construction. The no_curriculum and no_refinement baselines are reasonable ablations, and the paper does not rename a known result as a new one. A few self-citations exist, notably [20] (CurricuLLM) and [6], but they are used for related-work context and for describing a naive resampling baseline, not as load-bearing justification for CRAFT's core mechanism. The unvalidated VLM success gate is a legitimate correctness and robustness concern, and the paper itself concedes VLMs 'lack a native understanding of 3D motion' in Section IV-C, but this is a reliability limitation, not a circular derivation. The central result remains externally measured and independently checkable against the simulator, so no significant circularity is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the capability of foundation models to act as a coach, which is a domain assumption rather than standard math. The reward weights are chosen by the LLM/VLM loop and vary across trials. No new physical entities are introduced.

free parameters (2)
  • Reward component weights chosen by LLM/VLM = e.g., lift_reward multiplier from 80 to 200, balance_reward weight from 2 to 1 (Fig. 3)
    The reward refinement loop adjusts these scalar weights and shaping terms in response to VLM advice, so the final rewards are effectively tuned by hand (by the foundation models) to make each subtask learnable. They are not learned from data and vary across trials.
  • Refinement loop budget and exploration reset parameters = J=3 refinement iterations; policy standard deviation reset; value network reinitialization
    These design choices affect whether the loop converges to a successful reward and are introduced to make sequential training work. They are chosen by hand and not justified by a theoretical argument.
assumptions (4)
  • standard math The Dec-POMDP formulation with a shared CTDE policy is an adequate model for the coordination tasks.
    Section III models the problem as a Dec-POMDP and uses MAPPO with a shared policy, which is standard in MARL.
  • ad hoc to paper Foundation models can reliably generate coherent curricula, executable reward code, and accurate policy evaluations and advice.
    The entire CRAFT pipeline in Sections IV-A through IV-D depends on this. The paper itself notes variability and limited 3D understanding in the conclusion, so this is an unproven, load-bearing assumption.
  • domain assumption MAPPO is a competent enough MARL algorithm for these tasks to expose differences in reward and curriculum quality.
    Section V-A uses MAPPO for all experiments; if MAPPO were too weak, the comparisons would be less informative, and if it were too strong the baselines might underperform for other reasons.
  • domain assumption Simulation-to-real transfer by component-wise action scaling is sufficient without fine-tuning.
    Section V-C applies action scaling and zero-shot deployment; the hardware success rate of 65% suggests this holds, but it is only shown for one task.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CRAFT: Coaching Reinforcement Learning Autonomously using Foundation Models for Multi-Robot Coordination Tasks." pith.science (2026). https://pith.science/paper/G3PBFQPD

@misc{pith2026250914380,
  author       = {Pith},
  title        = {Pith review of: CRAFT: Coaching Reinforcement Learning Autonomously using Foundation Models for Multi-Robot Coordination Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G3PBFQPD}},
  note         = {Machine review of arXiv:2509.14380}
}
read the original abstract

Multi-Agent Reinforcement Learning (MARL) provides a powerful framework for learning coordination in multi-agent systems. However, applying MARL to robotics remains challenging due to their high-dimensional continuous joint action spaces, complex reward design, and non-stationarity from concurrently learning agents. On the other hand, humans often learn complex coordination with the help of coaches, who guide learning through carefully designed curricula and detailed feedback. Building on the reasoning capabilities of foundation models, we argue that these models can similarly coach robots to learn coordination. Motivated by this, we propose CRAFT: Coaching Reinforcement learning Autonomously using Foundation models for learning coordination Tasks, a framework that leverages foundation models to act as a "coach" for multi-robot coordination. CRAFT automatically decomposes long-horizon coordination tasks into sequences of subtasks using the planning capability of Large Language Models (LLMs). Then, CRAFT trains each subtask using LLM-generated reward functions, and refines them through a Vision Language Model (VLM)-guided reward-refinement loop. We evaluate CRAFT on multi-quadruped navigation and bimanual manipulation tasks, and demonstrate its capability to learn complex coordination behaviors. In addition, in a multi-quadruped navigation setting, we show that our learned policies transfer to the real world. Project website is https://iconlab.negarmehr.com/CRAFT/

Figures

Figures reproduced from arXiv: 2509.14380 by the authors.

Figure 1
Figure 1. (left) Overview of CRAFT. Given a natural language description of the target task, CRAFT generates a curriculum C = [M1, . . . ,MK] of subtasks that progressively build coordination towards the target task. For each subtask Mk, a reward LLM generates a reward function R j k and a decentralized policy π j k , shared among agents, is trained via MARL. A VLM evaluates this policy using state variable trajectories and s… view at source ↗
Figure 2
Figure 2. Example of curriculum refinement for task [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Example of reward refinement of subtask Coordinate Preliminary Lift. Through the first reward-refinement loop, R 1 k=3 was produced and the evaluation VLM marked the policy as a failure since the pot never reached the required elevation of 0.05 m. The reward component learning curves were then passed to the advice VLM, which identified that lift_reward was too weak compared to balance_reward. It recommended removing… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Illustrative snapshot of policies trained with env_reward [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Success Rate of the top-three curricula from each environment. Each environment is evaluated by 100 random initial conditions. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Trajectory of quadrupeds in our hardware experiments [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MARS-RA: Rank Aggregation for Credit Assignment via Multimodal Comparisons in Embodied Multi-Agent Cooperation

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Credit assignment via LMM pairwise comparisons plus Bradley–Terry rank aggregation and potential-based shaping improves cooperative MARL under sparse rewards and dynamic agent counts.

Reference graph

Works this paper leans on

55 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    Grandmaster level in starcraft ii using multi-agent reinforcement learning,

    O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev,et al., “Grandmaster level in starcraft ii using multi-agent reinforcement learning,”nature, vol. 575, no. 7782, pp. 350–354, 2019

    2019

  2. [2]

    Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning,

    B. Ellis, J. Cook, S. Moalla, M. Samvelyan, M. Sun, A. Mahajan, J. Foerster, and S. Whiteson, “Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning,”Advances in Neural Information Processing Systems, vol. 36, pp. 37567–37593, 2023

    work page 2023

  3. [3]

    Google research football: A novel reinforcement learning environ- ment,

    K. Kurach, A. Raichuk, P. Sta ´nczyk, M. Zaj ˛ ac, O. Bachem, L. Es- peholt, C. Riquelme, D. Vincent, M. Michalski, O. Bousquet,et al., “Google research football: A novel reinforcement learning environ- ment,” inProceedings of the AAAI conference on artificial intelligence, vol. 34, pp. 4501–4510, 2020

    work page 2020

  4. [4]

    Toward real-world cooperative and competitive soccer with quadrupedal robot teams,

    Z. Su, Y . Gao, E. Lukas, Y . Li, J. Cai, F. Tulbah, F. Gao, C. Yu, et al., “Toward real-world cooperative and competitive soccer with quadrupedal robot teams,”arXiv preprint arXiv:2505.13834, 2025

    arXiv 2025

  5. [5]

    Marladona-towards cooperative team play using multi-agent reinforcement learning,

    Z. Li, F. Bjelonic, V . Klemm, and M. Hutter, “Marladona-towards cooperative team play using multi-agent reinforcement learning,” in 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 15014–15020, IEEE, 2025

    work page 2025

  6. [6]

    Leveraging large language models for effective and explainable multi-agent credit assignment,

    K. Nagpal, D. Dong, and N. Mehr, “Leveraging large language models for effective and explainable multi-agent credit assignment,” inProceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, pp. 1501–1510, 2025

    work page 2025

  7. [7]

    Variational automatic curriculum learning for sparse-reward cooperative multi-agent problems,

    J. Chen, Y . Zhang, Y . Xu, H. Ma, H. Yang, J. Song, Y . Wang, and Y . Wu, “Variational automatic curriculum learning for sparse-reward cooperative multi-agent problems,”Advances in Neural Information Processing Systems, vol. 34, pp. 9681–9693, 2021

    work page 2021

  8. [8]

    Au- tomatic curriculum learning for deep rl: a short survey,

    R. Portelas, C. Colas, L. Weng, K. Hofmann, and P.-Y . Oudeyer, “Au- tomatic curriculum learning for deep rl: a short survey,” inProceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 4819–4825, 2021

    work page 2021

Show all 55 references
  1. [9]

    Cooperative multi- agent control using deep reinforcement learning,

    J. K. Gupta, M. Egorov, and M. Kochenderfer, “Cooperative multi- agent control using deep reinforcement learning,” inInternational conference on autonomous agents and multiagent systems, pp. 66–83, Springer, 2017

    2017

  2. [10]

    A survey on curriculum learning,

    X. Wang, Y . Chen, and W. Zhu, “A survey on curriculum learning,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 9, pp. 4555–4576, 2021

    2021

  3. [11]

    V oyager: An open-ended embodied agent with large language models,

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar, “V oyager: An open-ended embodied agent with large language models,”Transactions on Machine Learning Research, 2024

    2024

  4. [12]

    Progprompt: Generating situated robot task plans using large language models,

    I. Singh, V . Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: Generating situated robot task plans using large language models,” in2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), pp. 11523– 11530, IEEE, 2023

    2023

  5. [13]

    Eureka: Human-level reward design via coding large language models,

    Y . J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y . Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-level reward design via coding large language models,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  6. [14]

    Vision-language models are zero-shot reward models for reinforce- ment learning,

    J. Rocamonde, V . Montesinos, E. Nava, E. Perez, and D. Lindner, “Vision-language models are zero-shot reward models for reinforce- ment learning,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  7. [15]

    Autoeval: Autonomous evaluation of generalist robot manipulation policies in the real world,

    Z. Zhou, P. Atreya, Y . L. Tan, K. Pertsch, and S. Levine, “Autoeval: Autonomous evaluation of generalist robot manipulation policies in the real world,”arXiv preprint arXiv:2503.24278, 2025

    2025 arXiv

  8. [16]

    Aha: A vision- language-model for detecting and reasoning over failures in robotic manipulation,

    J. Duan, W. Pumacay, N. Kumar, Y . R. Wang, S. Tian, W. Yuan, R. Krishna, D. Fox, A. Mandlekar, and Y . Guo, “Aha: A vision- language-model for detecting and reasoning over failures in robotic manipulation,”arXiv preprint arXiv:2410.00371, 2024

    arXiv

  9. [17]

    Reverse forward curriculum learning for extreme sample and demo efficiency,

    S. Tao, A. Shukla, T. kai Chan, and H. Su, “Reverse forward curriculum learning for extreme sample and demo efficiency,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  10. [18]

    Unsupervised curricula for visual meta-reinforcement learning,

    A. Jabri, K. Hsu, A. Gupta, B. Eysenbach, S. Levine, and C. Finn, “Unsupervised curricula for visual meta-reinforcement learning,”Ad- vances in Neural Information Processing Systems, vol. 32, 2019

    2019

  11. [19]

    Tizero: Mastering multi-agent football with curriculum learning and self-play,

    F. Lin, S. Huang, T. Pearce, W. Chen, and W.-W. Tu, “Tizero: Mastering multi-agent football with curriculum learning and self-play,” inProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pp. 67–76, 2023

    2023

  12. [20]

    Curricullm: Automatic task curricula design for learning complex robot skills using large language models,

    K. Ryu, Q. Liao, Z. Li, P. Delgosha, K. Sreenath, and N. Mehr, “Curricullm: Automatic task curricula design for learning complex robot skills using large language models,” in2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 4470–4477, IEEE, 2025

    2025

  13. [21]

    Environment curriculum generation via large language models,

    W. Liang, S. Wang, H.-J. Wang, O. Bastani, D. Jayaraman, and Y . J. Ma, “Environment curriculum generation via large language models,” in8th Annual Conference on Robot Learning, 2024

    2024

  14. [22]

    Aura: Agentic upskilling via reinforced abstractions,

    A. Zhu, Y . Tanaka, and D. Hong, “Aura: Agentic upskilling via reinforced abstractions,”arXiv preprint arXiv:2506.02507, 2025

    2025

  15. [23]

    Self-refined large language model as automated reward function designer for deep reinforcement learning in robotics,

    J. Song, Z. Zhou, J. Liu, C. Fang, Z. Shu, and L. Ma, “Self-refined large language model as automated reward function designer for deep reinforcement learning in robotics,”arXiv preprint arXiv:2309.06687, 2023

    2023 arXiv

  16. [24]

    Learning a high-quality robotic wiping policy using systematic reward analysis and visual-language model based curriculum,

    Y . Liu, D. Kang, and S. Ha, “Learning a high-quality robotic wiping policy using systematic reward analysis and visual-language model based curriculum,”arXiv preprint arXiv:2502.12599, 2025

    2025 arXiv

  17. [25]

    Learning multi-agent loco-manipulation for long-horizon quadrupedal pushing,

    Y . Feng, C. Hong, Y . Niu, S. Liu, Y . Yang, and D. Zhao, “Learning multi-agent loco-manipulation for long-horizon quadrupedal pushing,” in2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 14441–14448, IEEE, 2025

    2025

  18. [26]

    De- centralized navigation of a cable-towed load using quadrupedal robot team via marl,

    W.-T. Chen, M. Nguyen, Z. Li, G. N. Sue, and K. Sreenath, “De- centralized navigation of a cable-towed load using quadrupedal robot team via marl,”arXiv preprint arXiv:2503.18221, 2025

    2025 arXiv

  19. [27]

    Resolving conflicting constraints in multi-agent rein- forcement learning with layered safety,

    J. J. Choi, J. J. Aloor, J. Li, M. G. Mendoza, H. Balakrishnan, and C. J. Tomlin, “Resolving conflicting constraints in multi-agent rein- forcement learning with layered safety,” inProceedings of Robotics: Science and Systems, (Los Angeles, USA), June 2025

    2025

  20. [28]

    Learning differentiable and safe multi-robot control for generalization to novel environments using control barrier functions,

    V . Sharma, N. Mehr, and N. Hovakimyan, “Learning differentiable and safe multi-robot control for generalization to novel environments using control barrier functions,” in2024 IEEE 63rd Conference on Decision and Control (CDC), pp. 8423–8428, IEEE, 2024

    2024

  21. [29]

    The surprising effectiveness of ppo in cooperative multi-agent games,

    C. Yu, A. Velu, E. Vinitsky, J. Gao, Y . Wang, A. Bayen, and Y . Wu, “The surprising effectiveness of ppo in cooperative multi-agent games,”Advances in neural information processing systems, vol. 35, pp. 24611–24624, 2022

    2022

  22. [30]

    Monotonic value function factorisation for deep multi- agent reinforcement learning,

    T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson, “Monotonic value function factorisation for deep multi- agent reinforcement learning,”Journal of Machine Learning Research, vol. 21, no. 178, pp. 1–51, 2020

    2020

  23. [31]

    Lever- aging pre-trained large language models to construct and utilize world models for model-based task planning,

    L. Guan, K. Valmeekam, S. Sreedharan, and S. Kambhampati, “Lever- aging pre-trained large language models to construct and utilize world models for model-based task planning,”Advances in Neural Information Processing Systems, vol. 36, pp. 79081–79094, 2023

    2023

  24. [32]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,

    B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia, “Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455–14465, 2024

    2024

  25. [33]

    3d-vla: A 3d vision-language-action generative world model,

    H. Zhen, X. Qiu, P. Chen, J. Yang, X. Yan, Y . Du, Y . Hong, and C. Gan, “3d-vla: A 3d vision-language-action generative world model,” inInternational Conference on Machine Learning, pp. 61229–61245, PMLR, 2024

    2024

  26. [34]

    Synthesizing interpretable control poli- cies through large language model guided search,

    C. Bosio and M. W. Mueller, “Synthesizing interpretable control poli- cies through large language model guided search,” in2025 American Control Conference (ACC), pp. 583–590, IEEE, 2025

    2025

  27. [35]

    Large language model based multi-agents: a survey of progress and challenges,

    T. Guo, X. Chen, Y . Wang, R. Chang, S. Pei, N. V . Chawla, O. Wiest, and X. Zhang, “Large language model based multi-agents: a survey of progress and challenges,” inProceedings of the Thirty-Third Inter- national Joint Conference on Artificial Intelligence, pp. 8048–8057, 2024

    2024

  28. [36]

    Multi-agent collaboration: Harnessing the power of intelligent llm agents,

    Y . Talebirad and A. Nadiri, “Multi-agent collaboration: Harnessing the power of intelligent llm agents,”arXiv preprint arXiv:2306.03314, 2023

    2023 arXiv

  29. [37]

    Loss of plasticity in continual deep reinforcement learning,

    Z. Abbas, R. Zhao, J. Modayil, A. White, and M. C. Machado, “Loss of plasticity in continual deep reinforcement learning,” inConference on lifelong learning agents, pp. 620–636, PMLR, 2023

    2023

  30. [38]

    Loss of plasticity in deep continual learning,

    S. Dohare, J. F. Hernandez-Garcia, Q. Lan, P. Rahman, A. R. Mah- mood, and R. S. Sutton, “Loss of plasticity in deep continual learning,” Nature, vol. 632, no. 8026, pp. 768–774, 2024

    2024

  31. [39]

    Mqe: Unleashing the power of interaction with multi-agent quadruped envi- ronment,

    Z. Xiong, B. Chen, S. Huang, W.-W. Tu, Z. He, and Y . Gao, “Mqe: Unleashing the power of interaction with multi-agent quadruped envi- ronment,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5918–5924, IEEE, 2024

    2024

  32. [40]

    robosuite: A modular simulation framework and benchmark for robot learning,

    Y . Zhu, J. Wong, A. Mandlekar, R. Martín-Martín, A. Joshi, S. Nasiriany, Y . Zhu, and K. Lin, “robosuite: A modular simulation framework and benchmark for robot learning,” inarXiv preprint arXiv:2009.12293, 2020

    2009 arXiv

  33. [41]

    Openrl: A unified reinforcement learning framework,

    S. Huang, W. Chen, Y . Sun, F. Bie, and W.-W. Tu, “Openrl: A unified reinforcement learning framework,”arXiv preprint arXiv:2312.16189, 2023

    2023 arXiv

  34. [42]

    skrl: Modular and flexible library for reinforcement learning,

    A. Serrano-Muñoz, D. Chrysostomou, S. Bøgh, and N. Arana- Arexolaleiba, “skrl: Modular and flexible library for reinforcement learning,”Journal of Machine Learning Research, vol. 24, no. 254, pp. 1–9, 2023. APPENDIX A. Prompts In this section, we provide prompts used in CRAFT....

    2023

  35. [43]

    One for generating candidate curricula, and another for refining the final curriculum from the candidates

    Curriculum Generation Module (Section IV-A):We use two prompts which are given to the curriculum LLM. One for generating candidate curricula, and another for refining the final curriculum from the candidates. Prompt 1: LLM prompt for generating curriculum candidates. You are a...

  36. [44]

    Prompt 3: LLM prompt for generating base reward function

    Reward Function Generation Module (Section IV-B):Below is the prompt given to the reward generation LLM for generating a base reward function. Prompt 3: LLM prompt for generating base reward function. You should write reward function for given task using useful variables from ...

  37. [45]

    Prompt 4: VLM prompt for policy evaluation

    Policy Evaluation Module (Section IV-C):Below is the prompt given to the evaluation VLM for evaluating a trained policy. Prompt 4: VLM prompt for policy evaluation. You are a multi-agent reinforcement learning engineer asked to evaluate a rollout from a task in a curriculum. Y...

  38. [46]

    Whether the agents achieved the current goal

  39. [47]

    very close

    Whether they have forgotten earlier subtasks. Your output should be Decision: [Success or Failure] Reason: - [reason 1] - [reason 2] ... Fill out the [] brackets in the given format. (Don’t keep the [] brackets) Current task is: <<Current task>> Former tasks that RL succeeded ...

  40. [48]

    One for generating advice on how to refine the reward (VLM), and one for refining the reward function given the advice (LLM)

    Reward Refinement Module (Section IV-D):We use two prompts. One for generating advice on how to refine the reward (VLM), and one for refining the reward function given the advice (LLM). Prompt 5: VLM prompt for generating reward refinement advice. You are a multi agent reinfor...

  41. [49]

    reach_reward

    Task-Specific Substitutions for Two Arm Lift:Below are task-specific substitutions (e.g., environment details and task descriptions) that are inserted into the general prompts above. Prompt 7: «Environment_Description» for Two Arm Lift. A large pot with two handles is placed o...

  42. [50]

    Because the agents rarely reach that threshold early on, they get almost zero signal to lift beyond ~0.016 m

    Sparse elevation feedback: Controlled elevation and stability rewards only kick in once elevation >= 0.1 m and tilt within threshold. Because the agents rarely reach that threshold early on, they get almost zero signal to lift beyond ~0.016 m

  43. [51]

    Binary tilt gating: direction_coef zeroes out any elevation reward if tilt is even slightly over 30 degree, so there is no gradient to correct small tilt errors during the lift

  44. [52]

    Once the handles are touched or grasped, there is little extra push to actually raise the pot

    Incentives decoupled from lift: Proximity, alignment, grasp and cooperation rewards are largely satisfied independently of successful coordinated lifting. Once the handles are touched or grasped, there is little extra push to actually raise the pot. Three targeted improvements

  45. [53]

    This gives gradient toward any increase in height, not just surpassing 0.1 m

    Continuous elevation shaping - Replace the hard step for elevation with a smooth, monotonically increasing reward - for example r_elev = min(elevation / threshold, 1.0) or a quadratic/ sigmoid on elevation. This gives gradient toward any increase in height, not just surpassing 0.1 m

  46. [54]

    This still rewards low tilt but provides a gradient that gently pushes the pot back toward upright whenever it begins to tilt

    Tilt-aware bonus rather than gate - Instead of direction_coef={0,1}, use a continuous tilt bonus such as r_tilt = (cos_z - cos_max) / (1 - cos_max), clipped to [0,1], and multiply it with the elevation reward. This still rewards low tilt but provides a gradient that gently pus...

  47. [55]

    touch and release

    Tighten coupling of grasp, proximity, and lift - Multiply (or weight more heavily) your proximity/alignment rewards by the current elevation factor (e.g. r_prox x r_elev). That way, the agents only obtain full proximity/alignment credit if they hold on while actually lifting. ...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.