Pith. sign in

REVIEW 4 major objections 6 minor 115 references

GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks

T0 review · 4 major / 6 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A multi-agent coding harness can assemble and self-refine modular robot computation graphs that outperform model-free policies on variational automation tasks.

desk verdict Solid systems paper: multi-agent graph harness + sim self-learning + MORSL beats VLAs/CaP/TAMP on new VA benchmarks, but the big gaps partly reflect privileged geometry and skills rather than the harness alone. read the letter →

arxiv 2607.05369 v1 pith:6TFVCMZ3 submitted 2026-07-06 cs.RO cs.AIcs.CLcs.LG

Kaiyuan Chen , Shuangyu Xie , Letian Fu , Justin Yu , William Pacini , Sandeep Bajamahal , Hudson Kim , Jaimyn Drake
show 16 more authors
Daehwa Kim Haoru Xue Jonathan Francis Christian Juette Peter Schaldenbrand Muhammet Yunus Seker Ruwan Wickramarachchi Uksang Yoo Guanzhi Wang Adithyavairavan Murali Balakumar Sundaralingam S. Shankar Sastry Spencer Huang Yuke Zhu Linxi "Jim" Fan Ken Goldberg
This is my paper · ORCID
classification cs.ROcs.AIcs.CLcs.LG
keywords VariationalAutomationGraph-as-Policyagenticcodingself-learningrobotcomputationgraphsskilllibrariesTaskandMotionPlanningmulti-agentsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Commercial robots need policies that run for long periods on tasks with real variation in object geometry and pose—not fixed factory motions and not open-ended household generalism. The paper calls this class Variational Automation and argues that end-to-end model-free policies often fail under that variation while classical programming is too expensive to retune for every instance distribution. Graph-as-Policy answers by having multiple coding agents build a directed graph of perception, planning, and control nodes from an open skill library, then rehearse many sampled instances in simulation and rewrite graph structure and parameters until success rate and throughput stop improving. On eight new open benchmarks (four simulated, four physical)—grocery order fulfillment and packing, making popcorn, USB-C cable insertion, and dual-arm crate washing—the refined graphs reach high success rates that substantially exceed single-agent code generation, modular planners, and finetuned vision-language-action models. A sympathetic reader cares because the approach keeps policies interpretable and composable while still using agents and learned skills where they help, aiming to cut the human setup cost of reliable automation.

What carries the argument

Graph-as-Policy (GaP): a policy is a directed computation graph whose nodes are atomic skills from the Modular Open Robot Skill Library (MORSL). An orchestration multi-agent system decomposes a language task into subgraphs, wires them, then runs self-learning—sample belief-space instances, parallel sim rollouts, analyze contacts and state differences, update graph topology and parameters—until a weighted success-and-throughput objective plateaus; the resulting graph is interpreted on the edge without further agents.

What would settle it

On grocery fulfillment and packing with mixed large pose variation, if GaP success collapsed to the level of the vision-language-action baselines (near 0.2 over the same hundred-instance cells) while those baselines remained high, or if make-popcorn self-learning gains in simulation failed to transfer so real success stayed near the initial one-third rather than near 90 percent, the central reliability claim would be falsified.

Watch

Extended reading notes

Core claim

Representing a robot policy as a directed computation graph of modular skills, authored by a multi-agent coding harness and iteratively refined by parallel simulation rehearsal with failure analysis, yields persistent policies for variational automation whose success rates significantly outperform free-form coding agents, modular task-and-motion planners, and vision-language-action baselines under large pose and object variation.

Load-bearing premise

The workcell, robot, sensors, object models, and the distribution of instance poses are already known and fixed, so the system only has to handle variation inside that closed operating envelope.

Editorial extensions

If this is right

  • Structured multi-agent graph authoring with static validation can avoid the total collapse to zero success that free-form single-agent script generation shows on the same tasks.
  • Graph-based approach and camera-centering steps can more than double vision-language-action success under pose variation by bringing wrist-camera observations into distribution before handing off control.
  • Self-learning from simulation contact and state feedback can raise a long-horizon kitchen graph from roughly one-third to over 90 percent success under pan pose variation, with transfer to physical trials.
  • The same harness can emit ROS-compatible force-aware insertion workflows and dual-arm coordination graphs whose reliability approaches a hand-engineered industrial baseline on crate washing.
  • Open variational benchmarks with controlled larger pose variation expose reliability gaps that near-fixed-pose evaluations hide.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As skill libraries grow, hierarchical multi-agent graph harnesses may become the practical bridge between classical automation engineering and learned skills, rather than pure end-to-end generalists for factory-like settings.
  • Sim-to-real fidelity of contact and state feedback is the hinge of self-learning; sites without usable digital twins may need physical rehearsal or the loop will optimize the wrong failures.
  • Cycle time remains far below industrial unit-per-hour targets; cutting vision-model and motion-planning latency at execution may matter as much as raising success rate for adoption.
  • Adding force-rich, deformable, and moving-target nodes would test whether the same authoring-and-rehearsal loop covers a larger share of industrial work beyond quasi-static pick-and-place.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Graph-as-Policy (GaP), a multi-agent LLM coding harness that synthesizes directed computation graphs of perception, planning, and control nodes drawn from a Modular Open Robot Skill Library (MORSL), then refines graph structure and parameters via parallel Isaac-Lab rehearsal (Algorithm 1). It defines a Variational Automation (VA) task class with known workcell, robot/sensors, object models, and pose belief (tuple T in §3), and evaluates on eight new open benchmarks (grocery fulfill/pack, make popcorn, USB-C insertion, dual-arm crate wash; 4 sim / 4 real). Reported results claim large gains over CaP-X, π0.5, MolmoAct2, and TipTop (e.g., Table 1: 0.93–0.99 vs ~0.20 under large pose variation; real make-popcorn 18/20; cable insertion 121/130; crate wash 0.95 matching a hand-engineered graph), with ablations that graphless or single-agent generation collapses to zero.

Significance. If the results hold under fairer controls, GaP is a practical bridge between classical modular stacks (ROS/TAMP-style graphs) and agentic coding for commercial-style tasks with bounded geometric variation. Concrete strengths include: large sim trial counts (100 instances per cell, >5000 trials for grocery tables), real-robot tables, explicit ablations of graph structure and multi-agent authoring, a documented self-learning case study on make-popcorn (33%→94% sim), hybrid GaP+VLA rows showing scaffolding value, an open MORSL catalog (51 skills), and promised code/benchmarks. The VA framing and open benchmarks are useful even if the absolute reliability/throughput remain below industrial targets (as the Limitations section acknowledges).

major comments (4)
  1. The headline claim that GaP “significantly outperform[s]” CaP-X, π0.5, MolmoAct2, and TipTop (Abstract; §5.2–5.3; Table 1) is not cleanly isolated to the multi-agent graph harness. §5.2 itself states CaP-X is “not a fair comparison” (single agent, image+language only, no self-learning). VLAs are LIBERO-finetuned without GaP’s known E/R/O/B, MORSL skills, or interactive-perception centering; TipTop failures are attributed largely to M2T2/cuRobo/cuTAMP planning limits on cubic/tall objects that GaP’s skill stack handles. Ablations in §5.4 show graphless/single-agent collapse to zero inside GaP’s stack, but that does not show the harness—not privileged models, skills, and sim edits—drives the 0.93–0.99 vs ~0.20 gap. A load-bearing revision is either (i) baselines given the same O/B/MORSL/sim loop, or (ii) a carefully scoped claim that GaP is an effective integration/self-refinement system u
  2. Self-learning (Algorithm 1; §4.2; Fig. 2) is central to the paper’s contribution narrative, yet evidence is concentrated on Make Popcorn (initial ~33% → 94% sim after ~10 iterations; 18/20 real). §5.2 states that for Benchmarks I–II the first generated graph already achieves high performance, so self-learning is not exercised. Without multi-task statistics on when GraphUpdate changes topology vs parameters, how often sim-improved G* transfers, and failure modes of AnalyzeFailure under contact/state feedback, the claim that parallel rehearsal “iteratively refine[s] the graph… to improve success rates and throughput” remains under-supported outside one long-horizon example.
  3. §3 Assumptions and the VA tuple T=⟨L,E,R,O,X,B,J⟩ grant known fixed workcell, robot/sensor config, object models (including articulated joints), and a known belief B over instance poses. The weakest assumption for deployment is that Isaac-Lab state/contact feedback is faithful enough that graph edits raising sim success transfer to real hardware. Real N is modest relative to sim (Table 2: 25/25, 28/30, 18/20; Table 3: 130 insertions; Table 4: 150 crate trials). The manuscript should quantify sim-to-real discrepancy for self-refined graphs (e.g., make-popcorn IK/kinematic failures) and state more clearly which results use self-learning vs first-shot generation, so readers can judge generalization under misspecified B or imperfect sim.
  4. Reward J = ws·I(success) + wt·Φ (§3) and free parameters (ws, wt; M, N; cable force/depth thresholds in §5.6; LLM choice/temperature 0.1) are not systematically ablated. Throughput claims (Table 4: ~18 successes/hr; Limitations: far below ~500 units/hr industrial targets) depend on cycle time dominated by VLM/IK. Without sensitivity of G* to J weights and rehearsal budget, it is hard to assess robustness of the optimization objective that defines G* = arg max_G E[J].
minor comments (6)
  1. Figure 1 and several captions use corrupted glyphs (e.g., “V A”, “Pck”, “tributinterpreted”); clean for camera-ready.
  2. Table 1 header/column labels mix LIBERO, LIBERO-Pro, and new VA variants; a short legend defining “mixed all” and packing success (items/6 after 6 attempts) in the table caption would help.
  3. Related Work repeats CaP/self-improving agent paragraphs almost verbatim between §2 and Appendix A; consolidate.
  4. MORSL is said to have “51 initial skills” (§1, §4.1) while Appendix D lists many primitives; a single inventory table mapping composite skills to gRPC methods would improve reproducibility.
  5. Notation: belief B and instance sampling xi∼p(x|X) (§3) vs later {ŝi}∼B in Algorithm 1; keep symbols consistent.
  6. Project URL and “code and data will be posted” should be checked for availability at review time; if not yet public, provide an anonymized artifact or clear release plan.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: GaP is an empirical systems paper whose success rates are measured against external task criteria and independent baselines, not forced by construction from fitted inputs or self-citation.

full rationale

The load-bearing claim is empirical outperformance of GaP-generated graphs on eight VA benchmarks (Tables 1–4), not a first-principles derivation. Success is defined by external task outcomes (item in basket, pan covering burner, USB-C insertion depth/force, dual-arm crate placement) via J = w_s·I(success) + w_t·Φ and by comparison to CaP-X, π0.5, MolmoAct2, TipTop, and a hand-engineered crate graph. Self-learning (Alg. 1) samples instances from belief B, rolls out in Isaac-Lab, and edits graph topology/parameters from contact/state feedback; that is iterative optimization against physical sim outcomes, not redefinition of success as the graph itself. MORSL skills (GraspGen, cuRobo, DINO/SAM, ROS nodes) are library components, including some from overlapping NVIDIA coauthors; using prior tools as nodes is normal engineering and does not make reported success rates tautological. Ablations (graphless/single-agent → zero) show structure helps inside the stack but do not close a circular loop. No equation equates a claimed prediction to a fitted input by construction; no uniqueness theorem is imported to forbid alternatives. Score 0 is the honest finding.

Assumptions & free parameters 4 free parameters · 5 assumptions · 4 invented entities

The central claim rests on domain assumptions that define VA (known cell and object set), on the adequacy of modular skill libraries and LLM coding agents, and on simulation as a valid rehearsal oracle. Free parameters are mostly engineering thresholds and reward weights, not a single fitted constant that forces the headline success rates. Invented entities are methodological (graph-as-policy, harness, MORSL, VA class), not new physical particles.

free parameters (4)
  • Reward weights ws, wt in J = ws·I(success) + wt·Φ
    Multi-objective tradeoff between success and throughput used to drive graph optimization; values not uniquely derived.
  • Self-learning iterations M and parallel rollouts N
    Algorithm 1 hyperparameters that control how long/how widely graphs are refined before plateau.
  • Cable-insertion force/depth thresholds (e.g. >3 mm at <10 N; success depth >6 mm at 30 N; extract lateral <20 N)
    Hand-set contact criteria that define success for Benchmark IV policy nodes.
  • LLM temperature 0.1 and model choice (Gemini-3.1-Flash-Lite)
    Generation stochasticity and capability are fixed by experiment setup; results depend on this agent stack.
assumptions (5)
  • domain assumption VA tasks have known fixed workcell E, robot/sensors R, object set O with models, and known belief B over instance poses.
    Section 3 Assumptions; without this, graph generation and sim rehearsal as described do not apply to open generalist settings.
  • domain assumption A robot policy can be represented as a directed computation graph of typed skill nodes with data/control edges (ROS/TAMP-inspired).
    Problem Formulation and Section 4; load-bearing design choice that enables multi-agent decomposition and validation.
  • domain assumption Isaac-Lab (or equivalent) physics, contacts, and rendered observations provide failure signals sufficient to improve real-world graphs.
    Self-learning Algorithm 1 and make-popcorn transfer; sim-to-real is assumed workable for the reported tasks.
  • domain assumption MORSL skills expose correct typed I/O contracts and implement the named perception/planning/control behaviors.
    Graph correctness depends on skill library fidelity; ablations show unstructured code fails structural checks.
  • standard math Standard probability and SE(3) geometry for sampling xi ~ B and evaluating success indicators.
    Used in formal task tuple and expectation G* = arg max_G E[J].
invented entities (4)
  • Variational Automation (VA) task class
    purpose: Position the problem between fixed automation and generalist robotics with bounded variation.
    New framing with formal tuple T; independent evidence is the eight benchmarks, not external prior definition.
  • Graph-as-Policy (GaP) multi-agent harness
    purpose: Generate, validate, and self-refine robot computation graphs from language + MORSL.
    Core system contribution; evidence is empirical performance, not a prior named system.
  • Modular Open Robot Skill Library (MORSL)
    purpose: Provide 51 typed atomic/composite skills for agents to wire into graphs.
    Library is introduced and catalogued in the paper/appendix; skills wrap prior models (SAM, GraspGen, cuRobo) but the library contract is new.
  • Eight open VA benchmarks (grocery fulfill/pack, make popcorn, cable insert, wash crates)
    purpose: Evaluate reliability under pose/geometry variation in sim and real.
    New evaluation suite; some assets from LIBERO but variations and tasks are paper-defined.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks." pith.science (2026). https://pith.science/paper/6TFVCMZ3

@misc{pith2026260705369,
  author       = {Pith},
  title        = {Pith review of: GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6TFVCMZ3}},
  note         = {Machine review of arXiv:2607.05369}
}
read the original abstract

For robots to work reliably in commercial and industrial applications, can recent advances in agentic coding systems combine interpretable robot programming with the open-world adaptability of model-free policies? We focus on "Variational Automation" (VA), a class of tasks that have larger variations in object geometry and pose than fixed automation. Model-free policies often struggle to close the reliability gap for VA tasks, which must be executed persistently and reliably in commercial and industrial applications. Motivated by prior work on Task and Motion Planning (TAMP) and the Robot Operating System (ROS), we introduce Graph-as-Policy (GaP), a multi-agent coding harness that generates directed computation graphs with perception, planning, and control nodes from a Modular Open Robot Skill Library (MORSL). GaP then generates an internal simulation environment to rehearse task instances with different graphs in parallel to iteratively refine the graph structure and parameters to improve success rates and throughput. Evaluation with 8 new open VA task benchmarks, 4 in-simulation and 4 in real-world, suggests that GaP can achieve success rates that significantly outperform baselines. Details, code, and data can be found online: https://graph-robots.github.io/gap

Figures

Figures reproduced from arXiv: 2607.05369 by the authors.

Figure 1
Figure 1. GaP system architecture. Given a “Variational Automation (VA)” task specification, GaP uses a multi-agent harness for coding tools such as Claude and Gemini to automatically generate computation graphs that include “skill” nodes from a Modular Open Robot Skill Library (MORSL), which includes model-based procedures (e.g. ROS [1]) and model-free policies (e.g. GraspGen [2]). GaP then orchestrates self-learning using s… view at source ↗
Figure 2
Figure 2. Self-Learning for Making Popcorn Benchmark. Pan pose variations (left) drives an 10-iteration sequence graph update ; (blue, left axis) iteration phases shaded by class of edit. The Make Popcorn task requires the robot to grasps the stove knob and rotate it to turn on the burner, then the robot must find and pick up the handle of the JiffyPop popcorn pan, place it on the stove burner, wait, and then turn off the sto… view at source ↗
Figure 3
Figure 3. Seven-ports cable insertion setup [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Left: ROS computation graph that is hand-engineered using traditional ROS nodes and topics. Middle: [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

115 extracted references · 1 canonical work pages

  1. [1]

    Macenski, T

    S. Macenski, T. Foote, B. Gerkey, C. Lalancette, and W. Woodall. Robot operating system 2: Design, architecture, and uses in the wild.Science robotics, 7(66):eabm6074, 2022

    2022

  2. [2]

    Murali, B

    A. Murali, B. Sundaralingam, Y .-W. Chao, J. Yamada, W. Yuan, M. Carlson, F. Ramos, S. Birch- field, D. Fox, and C. Eppner. Graspgen: A diffusion-based framework for 6-dof grasping with on-generator training. InProceedings of the IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026

    2026

  3. [3]

    Isaac Sim, 2025

    NVIDIA. Isaac Sim, 2025. URL https://github.com/isaac-sim/IsaacSim. Version 5.1.0

    2025

  4. [4]

    M. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. Openvla: An open-source vision-language-action model.arXiv preprint arXiv:2406.09246, 2024

    arXiv 2024

  5. [5]

    P. I. Team.π0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    arXiv 2024

  6. [6]

    O. X.-E. Co-Authors. Open X-Embodiment: Robotic learning datasets and RT-X models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903. IEEE, 2024

    2024

  7. [7]

    H. Fang, J. Duan, D. Clay, S. Wang, S. Liu, W. Huang, X. Fan, W.-C. Tsai, S. Chen, Y . R. Wang, et al. Molmoact2: Action reasoning models for real-world deployment.arXiv preprint arXiv:2605.02881, 2026

    arXiv 2026

  8. [8]

    Black, N

    K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fu- sai, M. Y . Galliker, et al.π0.5: a vision-language-action model with open-world generalization. In9th Annual Conference on Robot Learning, 2025

    2025

Show all 115 references
  1. [9]

    Zawalski, W

    M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine. Robotic control via embodied chain-of-thought reasoning. InConference on Robot Learning, pages 3157–3181. PMLR, 2025

    2025

  2. [10]

    J. Gao, S. Belkhale, S. Dasari, A. Balakrishna, D. Shah, and D. Sadigh. A taxonomy for evaluating generalist robot manipulation policies.IEEE Robotics and Automation Letters (RA-L), 2026

    2026

  3. [11]

    Goldberg

    K. Goldberg. Should robot generalists get off their high horse?, 2026. Available online. 11

    2026

  4. [12]

    Solowjow, I

    E. Solowjow, I. Ugalde, Y . Shahapurkar, J. Aparicio, J. Mahler, V . Satish, K. Goldberg, and H. Claussen. Industrial robot grasping with deep learning using a programmable logic controller (plc). In2020 IEEE 16th International Conference on Automation Science and Engineering ...

    2020 doi

  5. [13]

    Adebola, T

    S. Adebola, T. Sadjadpour, K. El-Refai, W. Panitch, Z. Ma, R. Lin, T. Qiu, S. Ganti, C. Le, J. Drake, and K. Goldberg. Automating deformable gasket assembly. In2024 IEEE 20th International Conference on Automation Science and Engineering (CASE), pages 4146–4153. IEEE, 2024

    2024

  6. [14]

    S. Xie, K. Goldberg, and D. Song. Energy efficient planning for repetitive heterogeneous tasks in precision agriculture. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 7139–7145, 2025. doi:10.1109/ICRA55743.2025.11128083

    2025 doi

  7. [15]

    Adebola, R

    S. Adebola, R. Parikh, M. Presten, S. Sharma, S. Aeron, A. Rao, S. Mukherjee, T. Qu, C. Wistrom, E. Solowjow, and K. Goldberg. Can machines garden? systematically comparing the alphagarden vs. professional horticulturalists. In2023 IEEE International Conference on Robotics and...

    2023 doi

  8. [16]

    Q. Team. Qwen3. 5-omni technical report.arXiv preprint arXiv:2604.15804, 2026

    2026 arXiv

  9. [17]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  10. [18]

    D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  11. [19]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In M. Meila and T. Zhang, editors,Proceedings of the 38th Inter- n...

    2021

  12. [20]

    Bordes, R

    F. Bordes, R. Y . Pang, A. Ajay, A. C. Li, A. Bardes, S. Petryk, O. Ma ˜nas, Z. Lin, A. Mah- moud, B. Jayaraman, et al. An introduction to vision-language modeling.arXiv preprint arXiv:2405.17247, 2024

    2024 arXiv

  13. [21]

    G. R. Team, S. Abeyruwan, et al. Gemini robotics: Bringing ai into the physical world, 2025. URLhttps://arxiv.org/abs/2503.20020

    2025 arXiv

  14. [22]

    Huang, C

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. InConference on Robot Learning, pages 540–562. PMLR, 2023

    2023

  15. [23]

    C. Ning, K. Fang, and W.-C. Ma. Prompting with the future: Open-world model predictive control with interactive digital twins. InProceedings of Robotics: Science and Systems (RSS), 2025

    2025

  16. [24]

    Claude 3.5 Sonnet

    Anthropic. Claude 3.5 Sonnet. https://www.anthropic.com/product/claude-code, June 2024

    2024

  17. [25]

    K. Chen, S. Xie, Z. Ma, P. R. Sanketi, and K. Goldberg. Robo2vlm: Improving visual question answering using large-scale robot manipulation data. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025

    2025

  18. [26]

    Y . Ji, H. Tan, J. Shi, X. Hao, Y . Zhang, H. Zhang, P. Wang, M. Zhao, Y . Mu, P. An, et al. Robobrain: A unified brain model for robotic manipulation from abstract to concrete.CVPR, 2025. 12

    2025

  19. [27]

    Harness design for long-running application development

    Anthropic. Harness design for long-running application development. https://www. anthropic.com/engineering/harness-design-long-running-apps, Mar. 2026

    2026

  20. [28]

    Liang, W

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In2023 IEEE International conference on robotics and automation (ICRA), pages 9493–9500. IEEE, 2023

    2023

  21. [31]

    M. Fu, J. Yu, K. El-Refai, E. Kou, H. Xue, H. Huang, W. Xiao, G. Wang, F.-F. Li, G. Shi, et al. Cap-x: A framework for benchmarking and improving coding agents for robot manipulation. arXiv preprint arXiv:2603.22435, 2026

    2026 arXiv

  22. [32]

    L. P. Kaelbling and T. Lozano-P´erez. Hierarchical task and motion planning in the now. In2011 IEEE international conference on robotics and automation, pages 1470–1477. IEEE, 2011

    2011

  23. [33]

    C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-P´erez. Integrated task and motion planning.Annual review of control, robotics, and autonomous systems, 4(1):265–293, 2021

    2021

  24. [34]

    W. Shen, N. Kumar, S. Chintalapudi, J. Wang, C. Watson, E. S. Hu, J. Cao, D. Jayaraman, L. P. Kaelbling, and T. Lozano-P ´erez. TiPToP: A modular open-vocabulary planning system for robotic manipulation.arXiv preprint arXiv:2603.09971, 2026

    2026

  25. [35]

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning.Advances in Neural Information Processing Systems, 36: 44776–44791, 2023

    2023

  26. [36]

    Levine, C

    S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1–40, 2016

    2016

  27. [37]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 44(10-11):1684–1704, 2025

    2025

  28. [38]

    N. J. Nilsson.Artificial intelligence: a new synthesis. Elsevier, 1998

    1998

  29. [39]

    N. J. Nilsson.Principles of artificial intelligence. Morgan Kaufmann, 2014

    2014

  30. [40]

    Bucker, P

    A. Bucker, P. Ortega-Kral, J. Francis, and J. Oh. Grappa: Generalizing and adapting robot policies via online agentic guidance.IEEE Robotics and Automation Letters, 2026

    2026

  31. [41]

    J. Shi, R. Yang, K. Chao, B. S. Wan, Y . S. Shao, J. Lei, J. Qian, L. Le, P. Chaudhari, K. Daniilidis, et al. Maestro: Orchestrating robotics modules with vision-language models for zero-shot generalist robots. InNeurIPS 2025 Workshop on Space in Vision, Language, and Embodied...

    2025

  32. [42]

    S. Wang, M. Han, Z. Jiao, Z. Zhang, Y . N. Wu, S.-C. Zhu, and H. Liu. LLM3: Large Language Model-based Task and Motion Planning with Motion Failure Reasoning. In2024 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 12086–12092. IEEE, 2024

    2024

  33. [43]

    Curtis, N

    A. Curtis, N. Kumar, J. Cao, T. Lozano-P´erez, and L. P. Kaelbling. Trust the proc3s: Solving long-horizon robotics problems with llms and constraint satisfaction. InConference on Robot Learning, pages 1362–1383. PMLR, 2025. 13

    2025

  34. [44]

    Z. Yang, C. Garrett, D. Fox, T. Lozano-P´erez, and L. P. Kaelbling. Guiding long-horizon task and motion planning with vision language models. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 16847–16853. IEEE, 2025

    2025

  35. [45]

    Kumar, W

    N. Kumar, W. Shen, F. Ramos, D. Fox, T. Lozano-P´erez, L. P. Kaelbling, and C. R. Garrett. Open-world task and motion planning via vision-language model generated constraints.IEEE Robotics and Automation Letters, 2026

    2026

  36. [46]

    N. Cote, J. Drake, and S. Chitta. Agentic language-grounded adaptive robotic assembly. In The first CVPR workshop on 3D Vision Language Models (VLMs) for Robotics Manipulation: Opportunities and Challenges, 2024

    2024

  37. [47]

    G. Yin, Y . Li, Y . Wang, D. Mcconachie, P. Shah, K. Hashimoto, H. Zhang, K. Liu, and Y . Li. CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity. InProceedings of Robotics: Science and Systems, LosAngeles, CA, USA, June

  38. [48]

    doi:10.15607/RSS.2025.XXI.072

    2025 doi

  39. [49]

    B. Wu, A. Jones, A. Renault, H. Tay, J. Noble, N. Picard, S. Jiang, et al. Introducing advanced tool use on the claude developer platform, 2024

    2024

  40. [50]

    X. Hou, Y . Zhao, S. Wang, and H. Wang. Model context protocol (mcp): Landscape, security threats, and future research directions.ACM Transactions on Software Engineering and Methodology, 2025

    2025

  41. [51]

    GPT-4o System Card

    OpenAI. GPT-4o System Card. https://openai.com/index/gpt-4o-system-card/ , Aug. 2024

    2024

  42. [52]

    Claude 3.5 Sonnet

    Anthropic. Claude 3.5 Sonnet. https://www.anthropic.com/news/claude-3-5-sonnet , June 2024

    2024

  43. [53]

    Kavukcuoglu

    K. Kavukcuoglu. Gemini 2.5: Our most intelligent AI model. https://blog.google/ technology/google-deepmind/gemini-model-thinking-updates-march-2025/ , Mar. 2025

    2025

  44. [54]

    Zhang, J

    J. Zhang, J. Xiang, Z. Yu, F. Teng, X. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, et al. Aflow: Automating agentic workflow generation. InInternational Conference on Learning Representations, volume 2025, pages 34040–34077, 2025

    2025

  45. [55]

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar. V oyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024

    2024

  46. [56]

    Zehle, T

    T. Zehle, T. Heiß, M. Schlager, M. Aßenmacher, and M. Feurer. promptolution: A unified, mod- ular framework for prompt optimization. InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations),...

    2026

  47. [57]

    L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning. InFirst Workshop on Foundations of Reasoning in Language Models, 2025

    2025

  48. [58]

    Y . Lee, J. Boen, and C. Finn. Feedback descent: Open-ended text optimization via pairwise comparison.arXiv preprint arXiv:2511.07919, 2025

    2025

  49. [59]

    Goldberg, K

    A. Goldberg, K. Kondap, T. Qiu, Z. Ma, L. Fu, J. Kerr, H. Huang, K. Chen, K. Fang, and K. Goldberg. Blox-net: Generative design-for-robot-assembly using vlm supervision, physics, simulation, and a robot with reset. In2025 International Conference on Robotics and Automation (IC...

    2025

  50. [60]

    Equipping agents for the real world with Agent Skills

    Anthropic. Equipping agents for the real world with Agent Skills. https://www.anthropic. com/engineering/equipping-agents-for-the-real-world-with-agent-skills , Oct. 2025

    2025

  51. [61]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. InInternational Conference on Learning Representations, volume 2025, pages 28085–28128, 2025

    2025

  52. [62]

    Carion, L

    N. Carion, L. Gustafson, Y .-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V . Alwala, H. Khedr, A. Huang, J. Lei, T. Ma, B. Guo, A. Kalla, M. Marks, J. Greer, M. Wang, P. Sun, R. R¨adle, T. Afouras, E. Mavroudi, K. Xu, T.-H. Wu, Y . Zhou, L. Momeni, R. Hazra, S. Ding, S. V...

    2025 arXiv

  53. [63]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean Conference on Computer Vision, pages 38–55. Springer, 2024

    2024

  54. [64]

    Minderer, A

    M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Ma- hendran, A. Arnab, M. Dehghani, Z. Shen, et al. Simple open-vocabulary object detection. In European conference on computer vision, pages 728–755. Springer, 2022

    2022

  55. [65]

    Deitke, C

    M. Deitke, C. Clark, S. Lee, R. Tripathi, Y . Yang, J. S. Park, M. Salehi, N. Muennighoff, K. Lo, L. Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

    2025

  56. [66]

    Kamath, J

    A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ram´e, M. Rivi`ere, L. Rouillard, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 4, 2025

    2025 arXiv

  57. [67]

    Sundermeyer, A

    M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. In2021 IEEE international conference on robotics and automation (ICRA), pages 13438–13444. IEEE, 2021

    2021

  58. [68]

    W. Yuan, A. Murali, A. Mousavian, and D. Fox. M2t2: Multi-task masked transformer for object-centric pick and place. In7th Annual Conference on Robot Learning, 2023

    2023

  59. [69]

    Sundaralingam, S

    B. Sundaralingam, S. K. S. Hari, A. Fishman, C. Garrett, K. Van Wyk, V . Blukis, A. Millane, H. Oleynikova, A. Handa, F. Ramos, et al. Curobo: Parallelized collision-free robot motion generation. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 811...

    2023

  60. [70]

    Sundaralingam, A

    B. Sundaralingam, A. Murali, and S. Birchfield. curobov2: Dynamics-aware motion generation with depth-fused distance fields for high-dof robots.arXiv preprint arXiv:2603.05493, 2026

    2026 arXiv

  61. [71]

    X. Zhou, Y . Xu, G. Tie, Y . Chen, G. Zhang, D. Chu, P. Zhou, and L. Sun. Libero-pro: Towards robust and fair evaluation of vision-language-action models beyond memorization.[arXiv preprint arXiv:2510.03827], 2025

    2025 arXiv

  62. [72]

    Minkowski.Geometrie der Zahlen

    H. Minkowski.Geometrie der Zahlen. B.G. Teubner, Leipzig, 1896

  63. [73]

    W. Shen, C. Garrett, N. Kumar, A. Goyal, T. Hermans, L. P. Kaelbling, T. Lozano-P´erez, and F. Ramos. Differentiable gpu-parallelized task and motion planning. InRobotics science and systems. Robotics; Science and Systems, 2025

    2025

  64. [74]

    A. Pun, K. Deng, R. Liu, D. Ramanan, C. Liu, and J.-Y . Zhu. Generating physically stable and buildable brick structures from text. InICCV, 2025. 15 Appendix Contents A More Related Work 17 B Integrating ROS with GaP in Cable Insertion Benchmark 17 C Sample Generated Graphs 18...

    2025

  65. [75]

    orchestrator

    and multi-agent prompt optimization frameworks [55, 56]. To improve code generation quality and performance, CaP-X [ 30] used a VLM to provide feedback before and after execution (i.e. Visual Differencing), but the VLM can suffer from hallucinations and cannot handle geometric...

  66. [76]

    The block MUST end with a module-level variable named ‘‘spec‘‘ bound to a ‘‘WorkflowSpec‘‘ instance

  67. [77]

    Imports: ‘‘from vos.builder import WorkflowSpec, START‘‘ and ‘‘from vos.runtime.workflow import ServiceCall‘‘ are pre-provided in the sandbox

  68. [78]

    No I/O, no other imports. 33

  69. [79]

    OrientedBoundingBox

    Subgraph ‘inputs‘ / ‘outputs‘ MUST be dicts of ‘{name: proto_type_str}‘. Each value is a **bare proto type string** (e.g. ‘‘"OrientedBoundingBox"‘‘, ‘‘"Mask"‘‘, ‘‘"PointCloud"‘‘) -- never a ‘‘Ref‘‘ and never a nested dict. Cross-subgraph data flow is established implicitly by ...

  70. [80]

    done"]‘ and ‘on_error=

    Every ‘declare_subgraph(...)‘ call MUST pass BOTH ‘exit_success_values=‘ AND ‘on_error=‘ as keyword arguments -- they are required and have no defaults. Omitting either raises ‘declare_subgraph() missing 2 required keyword-only arguments‘ and the whole spec rejects. Convention...

  71. [81]

    Every end node must be reachable from ‘START‘

    **Edges and conditional_edges.** For each subgraph node, every one of its ‘exit_success_values‘ PLUS its ‘on_error‘ symbol must appear as a key in the corresponding ‘add_conditional_edges‘ mapping, and every mapping target must be a node declared at the top level. Every end no...

  72. [82]

    **The entry node must be a subgraph node** (not an end node)

  73. [83]

    The universal subgraph_agent fills those in per subgraph

    **No internal nodes / edges / on_error** for any subgraph in your output. The universal subgraph_agent fills those in per subgraph

  74. [84]

    Declared ‘outputs‘ must be a subset of the skill’s ‘produces_outputs‘ (omit outputs nothing downstream consumes)

    **Inputs and outputs.** A subgraph’s declared ‘inputs‘ must equal its skill’s ‘required_inputs‘ after ‘<name>‘ substitution, and every input must have an upstream subgraph on some path to it that produces a matching output name with a matching proto type. Declared ‘outputs‘ mu...

  75. [85]

    version": 3,

    **Pick the right specialized variant.** When multiple variants of a role appear in Available Skills (e.g. ‘perception_multi‘ vs ‘perception_single‘, ‘grasp_curobo_obb‘ vs ‘grasp_direct_ik‘), read each skill’s *When to use* guidance and pick the best fit -- default to the more ...

  76. [86]

    The shared workflow spec (top-level shape, node types, edge semantics, ‘$ref‘ syntax, validation rules) -- see ‘_workflow_spec.md‘

  77. [87]

    The chosen skill’s SKILL.md body (this is the per-skill guidance -- recommended node sequence, hard rules, exit-value semantics, and contract -- including whether the skill is ‘streaming: true‘)

  78. [88]

    The filtered tool catalog: only the tools listed in the skill’s ‘allowed-tools‘ frontmatter, intersected with what is deployed

  79. [89]

    The chosen skill’s ‘canonical_scripts‘ list (for composite skills) -- file references the subgraph may use as ‘type="script"‘ nodes

  80. [90]

    ## Your job Compose the minimal sequence of nodes and edges that:

    The coordinator-supplied subgraph spec (name, inputs, outputs, exit values, context). ## Your job Compose the minimal sequence of nodes and edges that:

  81. [91]

    in.{name}

    Consumes declared inputs (referenced as ‘Ref(f"in.{name}")‘)

  82. [92]

    Produces values bound to the declared outputs via ‘sg.set_outputs(...)‘

  83. [93]

    Names every success-path exit via ‘sg.add_exit(name)‘ (creates the terminal ‘noop‘ marker), and names the single failure-path exit via ‘sg.set_on_error(value)‘

  84. [94]

    Reaches each success-exit marker on its own path with an explicit edge to ‘END‘

  85. [95]

    found")‘ + ‘sg.add_edge(

    Calls only tools in the filtered catalog and scripts in the skill’s ‘canonical_scripts‘ list. If you need a primitive that’s missing, call ‘report_missing_capability(name, why)‘. Postcondition checkpoints (‘sg.add_checkpoint(...)‘) are authored by a separate ‘checkpoint_agent‘...

  86. [96]

    Anything else (including stray top-level ‘‘print‘‘ calls or ‘‘Workflow‘‘ instances) is rejected

    The block MUST end with a module-level variable named ‘‘sg‘‘ bound to a ‘‘Subgraph‘‘ instance. Anything else (including stray top-level ‘‘print‘‘ calls or ‘‘Workflow‘‘ instances) is rejected

  87. [97]

    Imports: ‘‘from vos.builder import Subgraph, Ref, START, END‘‘ is provided in the sandbox -- you may re-import it (idempotent) but no other imports are needed

  88. [98]

    Imports outside the allow list are rejected

    No I/O: do not open files, call ‘‘requests‘‘, spawn threads, or import packages beyond ‘‘vos.builder‘‘. Imports outside the allow list are rejected

  89. [99]

    Canonical scripts

    No mutation of nodes/edges after they’re added (the builder has no ‘remove‘/‘rename‘/‘replace‘ -- re-author from scratch instead). ### Inline-script blocks (unchanged) Optional ‘ ‘‘‘python:scripts/<sg>/<file>.py‘ blocks for inline scripts -- **only for paths whose stem is NOT ...

  90. [100]

    Any node referenced by an edge or conditional-edge mapping is not declared in ‘nodes‘ (and is not ‘START‘/‘END‘)

  91. [101]

    Any non-‘END‘ node is unreachable from ‘START‘

  92. [102]

    Any non-streaming, non-end node has no outgoing edge or conditional-edges entry

  93. [103]

    A streaming node has any outgoing edge or conditional-edges entry

  94. [104]

    A node with ‘streaming: true‘ invokes a skill whose contract has ‘streaming: false‘ (or vice versa) -- when the skill registry is available

  95. [105]

    ‘conditional_edges‘ from a non-router source omits ‘router_field‘

  96. [106]

    ‘conditional_edges‘ from a router source sets ‘router_field‘ to non-null (router scripts return the target directly)

  97. [107]

    ‘outputs‘ binding references an unknown node or an end node

  98. [108]

    ‘exit.success_values‘ is empty (S7)

  99. [109]

    ‘on_error‘ collides with a declared node (S9) or appears as a ‘conditional_edges‘ mapping target (S10)

  100. [110]

    With ‘router_field: null‘, any name in ‘exit.success_values‘ is not declared as a ‘noop‘ node; OR with ‘router_field‘ set, any name in ‘exit.success_values‘ collides with a node name (S11)

  101. [111]

    $ref": "in.<name>

    Any ‘{"$ref": "in.<name>"}‘ references an input name not declared in your ‘inputs‘ schema (the executor-injected ‘observation_stream‘ is exempt). 47

  102. [112]

    START", node]‘ edge | | EndState marker (‘{

    ‘inputs.<name>‘ declared on a reachable subgraph has no upstream producer subgraph that declares an output of the same name. Errors are fed back; fix every error and re-emit the full subgraph JSON. ## v2 -> v3 mapping (for migration) | v2 concept | v3 replacement | |---|---| |...

  103. [113]

    observation.GetObservation

    **‘observe‘** -- ‘type: tool‘, ‘tool: "observation.GetObservation"‘, ‘inputs: {}‘. Auto-registered gRPC method; flat name only

  104. [114]

    observe.cameras

    **‘perceive‘** -- ‘type: script‘, file ‘scripts/<sg>/perceive_dino_vlm.py‘ from this bundle. Inputs: ‘cameras=Ref("observe.cameras")‘, ‘object_name="basket"‘ (replace with the actual target noun phrase from this subgraph’s description), plus any optional fields (‘object_descri...

  105. [115]

    geometry_svc.FilterAndComputeOBB

    **‘filter_obb‘** -- ‘type: tool‘, ‘tool: "geometry_svc.FilterAndComputeOBB"‘, ‘inputs={"point_cloud": Ref("perceive.cloud")}‘. Returns a bare ‘OrientedBoundingBox‘. ### Wiring the exit (HARD) Use the linear edge ‘filter_obb -> found -> END‘. The ‘perceive‘ script already raise...

  106. [116]

    The cloud is the fused world-frame point cloud needed by learned-grasp skills (e.g

    Subgraph-level outputs MUST emit ALL THREE: ‘<name>_obb‘, ‘<name>_mask‘, AND ‘<name>_cloud‘. The cloud is the fused world-frame point cloud needed by learned-grasp skills (e.g. ‘grasp_moe‘); emit it unconditionally so the downstream agent can wire it without round-tripping. Se...

  107. [117]

    filter_obb

    ‘geometry_svc.FilterAndComputeOBB‘ returns a bare ‘OrientedBoundingBox‘; bind via ‘Ref("filter_obb")‘ (no trailing field). See ‘references/geometry_calling_conventions.md‘. ## Required end states | End state | Meaning | |---|---| | ‘found‘ | OBB + mask bound; route to next sub...

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.