Pith. sign in

REVIEW 4 major objections 6 minor 29 references

This paper claims that a frozen robot policy, wrapped in an inference-time agent loop that plans, verifies, and recovers, recovers most task-level reasoning that direct prompting misses — with no retraining.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:52 UTC pith:BMKHQWZE

load-bearing objection A useful orchestration-gap framing and a real closed-loop recipe, but the 'frozen VLA reasons' claim is partly a prompt-engineering artifact; worth engaging, needs ablation and code. the 4 major comments →

arxiv 2607.21725 v1 pith:BMKHQWZE submitted 2026-07-23 cs.RO

Addressing the Orchestration Gap in Generalist Robots via Physical Agency

Liane Galanti , Dhruv Shah , Tri Dao This is my paper
classification cs.RO
keywords orchestration gapphysical agencyclosed-loop robot controlvision-language-action modelsinference-time planningsuccess verificationfailure recoveryfrozen policies
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that much of what looks like a robot's missing motor capability is actually missing orchestration: a frozen, already-trained control policy fails on reasoning-heavy instructions when prompted directly, but succeeds when an inference-time agent decides what to do, checks whether it worked, and repairs. The proposed system, Pigey, is that agent loop. Its headline numbers — from 12.8% to 53.3% on one simulation perturbation benchmark, and from near zero to over 90% on real-robot reasoning probes — are meant to show that the gains come from closing the loop, not from new data or policy training. The claim matters because, if true, robot learning should measure the orchestration gap before spending scarce robot data on teaching policies to reason. The causal reading rests on the resolution strategies written into the system prompt being genuinely task-agnostic rather than encoding the evaluation tasks.

Core claim

Pigey (Physical Agency) is a closed-loop orchestrator that runs a frontier vision-language model over two frozen motor backends — one geometric grasp-and-place planner and one learned visuomotor policy. At each step it perceives the scene, picks a short subgoal, selects a backend, verifies the outcome from gripper sensors and wrist images, and retries or escalates on failure. The central discovery is the orchestration gap: holding the robot, cameras, scenes, and policy weights fixed, the same policy succeeds far more often when driven by this loop than when given the full instruction directly. Reported performance goes from 12.8% to 53.3% on a six-suite simulation benchmark, and from roughly

What carries the argument

The central object is Pigey, an inference-time agent loop — observe, reason, act, verify, recover — that never emits motor commands itself. Its tools frame each action as a call whose outcome must be verified before the plan advances: perceive the scene, pick and place through a geometric planner, execute a short subgoal through a frozen vision-language-action policy, and declare done only after a fresh verification. The carrying mechanism is the split of an open-loop pick-and-place into separately verified steps: a grasp counts only if the gripper sensor reports an object and the wrist image confirms it, a place is issued only after a verified grasp, and any unverified step triggers re-perc

Load-bearing premise

The claim that orchestration, not the hand-written resolution strategies in the system prompt, produces the gains depends on those strategies being task-agnostic; several of them closely track the evaluation tasks — occlusion search, superlative selection, absolute-coordinate restoration, and sequential imitation — so if they encode the tasks, the result is substantially a prompt-engineering artifact.

What would settle it

Run the identical robot, scenes, and frozen motor skills with the 'general resolution strategies' block removed or randomized from the system prompt, on the same task suite. If success rates fall toward the direct-prompting baseline, the loop alone is not carrying the reasoning. A second check: in the simulation benchmark, route every manipulation tool through the learned policy instead of analytic controllers; if the gains vanish, the benchmark is measuring the tools, not the orchestrated policy.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Before training a policy to reason, a practitioner should measure how much of the apparent deficit an inference-time agent already recovers around the frozen policy; the paper argues this often removes the need for new robot data.
  • Closed-loop verification around an open-loop planner is itself a source of robustness: tasks that fail by construction — target moved, target hidden, grasp slipped — become recoverable without any change to the policy.
  • The gain is a property of the loop, not of a single reasoner: the paper reports qualitatively consistent improvements across a sweep of different vision-language models orchestrating the same frozen motor skills.
  • Structured execution traces — subgoals, backend choices, verifications, recoveries — could later be distilled into a smaller orchestrator or task-level adapter, turning test-time orchestration into a path for policy improvement.
  • Reasoning-limited failures such as negation, world knowledge, conditionals, long-horizon memory, and recovery are the main bottleneck in generalist robot policies; the paper's decomposition attributes failures to the missing component rather than to insufficient data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the hand-written resolution strategies in the system prompt — occlusion search, superlative-constrained selection, absolute-coordinate restoration, sequential imitation — are later shown to encode the evaluation tasks rather than being task-agnostic, the headline numbers would stand as an engineering result, but the interpretation that the loop itself supplies the reasoning
  • Editorial extension: the same design pattern — fast frozen control, slow semantic verification, typed failure signals, bidirectional escalation — transfers beyond tabletop manipulation to mobile navigation, assembly, and other closed-loop settings where the motor skill exists but the task logic does not.
  • Editorial extension: the paper's own limitation notes imply that the approach is bounded by the skills being orchestrated and by verifier reliability; a natural next test is whether orchestrating a stronger frozen policy closes the remaining reasoning cases without additional training.
  • Editorial extension: a direct test of the causal attribution would be to run the same system with the resolution-strategy block removed or randomized; high success would confirm emergent orchestration, while a collapse would indicate that the prompt is doing much of the work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Pigey, an inference-time orchestrator that wraps a frozen VLA policy (π0.5) and a frozen TAMP planner (TiPToP) behind a frontier VLM agent loop. The VLM perceives, decomposes the instruction into subgoals, selects a backend tool, verifies execution via gripper sensors and wrist images, and retries/escalates on failure. The claimed contribution is that a large part of the 'task-level stack' — planning, decomposition, verification, recovery, memory — can be supplied at inference time with no additional robot data or policy training, and the paper reports a 12.8%→53.3% gain on LIBERO-PRO and 16.7%→97.3% on 30 real-robot tasks (Tables 2 and 3). The paper includes detailed appendices with per-task results, tool schemas, the system prompt, and a failure-mode analysis.

Significance. If the central claim holds, the paper is significant: it would show that a frozen VLA plus a closed-loop VLM orchestrator can recover much of the reasoning stack that current approaches try to train into the policy, with clear practical implications for when to spend robot data versus inference compute. The paper's design has real strengths: the real-robot data are reported per-task (Appendix F.1), the frozen-weight protocol is clearly stated (Appendix H.1), and the paper is unusually transparent about its system prompt (Appendix J) and its failure modes (Table 14). The LIBERO-PRO reasoner sweep (Table 15) is a useful check that the effect is not tied to one VLM. However, the paper's own appendices contain passages that undercut the strongest interpretation of the claim, and the evaluation as presented does not cleanly separate 'orchestration' from task-aligned prompt recipes and analytic controllers.

major comments (4)
  1. [Appendix J, 'General resolution strategies'] The system prompt contains five hand-written strategies that closely mirror the evaluation tasks: occlusion search (OB1), superlative-constrained selection (CL1/SR3), absolute-coordinate restoration (LM1), sequential imitation (LM2), and existential/universal reading (MS1, WK-style tasks). The paper says these are 'task-agnostic' but does not provide an ablation that removes them. Under the direct-prompting baseline, the VLA receives only the instruction; under Pigey it receives the instruction plus these explicit recipes. Since the headline claim is that 'orchestration—not extra machinery—causes the gains,' the evaluation changes two variables at once. The claim is not yet supported; the near-zero-to-90% gains on LM1/LM2 and the OB/SR suites may be substantially a prompt-engineering artifact. An ablation that runs Pigey with a generic tool-use prompt (or with the strategies block remove
  2. [Appendix H.1] In the LIBERO-PRO experiments, only VLARollout invokes the frozen learned policy; the other manipulation tools (Grasp, Place, VerifyCandidate, GoHome, Release) are analytic controllers. Table 2 therefore does not compare 'Pigey over a frozen VLA' against baselines in the way the abstract implies: the 12.8%→53.3% gain is obtained by adding analytic grasp/place tools, a VLM verifier, and the agent loop, not by 'orchestrating the frozen policy.' The claim that Pigey 'advances the state of the art by over 4x with no task-specific fine-tuning' is technically true but conflates the orchestration contribution with the analytic-controller contribution. The paper should either re-run the sim experiment with the VLA as the only manipulation backend, or clearly label the LIBERO-PRO result as evaluating the full Pigey system rather than the orchestration-gap claim.
  3. [Table 3 and Appendix F.1] The real-robot results are reported as success rates over 5 trials per task, with no error bars or confidence intervals, and the scoring protocol (Appendix K) says success is determined by 'a human evaluator' without stating that scoring was blind to condition. For tasks where π0.5 scores 0/5 and Pigey scores 5/5, the difference is qualitatively large, but for the overall 97.3% figure the absence of uncertainty quantification matters, and for categories with intermediate counts (e.g., OB3 3/5, ER2 4/5) the single-trial-count uncertainty is large. The authors should report binomial confidence intervals or at least per-task counts with a clear statement of whether the evaluator was blind to the method.
  4. [Appendix J, LookAway/LookBack tools] The memory tasks LM1 and LM2 rely on dedicated tools (LookAway, LookBack) that are purpose-built for the evaluation protocol. In LM1, the agent is 'physically prevented from observing the change' by a tool that rotates the wrist away; in LM2, the demonstration is broken into discrete moves that the agent logs. These tools are not generic orchestration; they are task-specific affordances inserted for the benchmark. The paper does not discuss how much of the LM1/LM2 success depends on these bespoke tools versus on the VLM's memory and reasoning. This should be acknowledged and ideally ablated.
minor comments (6)
  1. [Abstract / Table 3] The phrase 'lifts the frozen policy from near-zero to over 90%' should be qualified: the 97.3% is the overall mean, while on the simple pick-and-place control subset the frozen policy already attains 95%. The gains are concentrated on reasoning-limited tasks, as the paper states, but the abstract's wording could mislead.
  2. [Section 4, trials] The paper says 5 trials per task (real robot) and 10 per sim task per reasoner, but the headline '150 trials, 97.3%' in the reader's summary is not explicitly broken out in the main text; Appendix F.1 gives per-task counts. It would help to state total trial counts and the scoring rubric in the main text.
  3. [Table 1 and Section 2] Table 1's entries are based on the authors' reading of other systems and use qualitative symbols ('–', 'Partial', 'Explicit'). The table is labeled 'Operational profile' but the criteria (e.g., 'Context', 'Retry', 'Verify') are not formally defined; consider adding precise definitions or moving the table to the appendix.
  4. [Appendix J] The 'General resolution strategies' block is described in prose but not reproduced verbatim. Since the paper's claim depends on what the VLM is told, the exact prompt text (or a representative excerpt) should be included in a supplementary or appendix.
  5. [Section 5.2 and Table 3] The comparison of Pigey to TiPToP is described as isolating 'closed-loop verification around TAMP,' but Pigey also adds a VLM that can re-plan and choose VLARollout as an alternative backend. The phrase 'the cost of staying open-loop' is therefore not the only factor; the comparison includes more than closing the loop.
  6. [Appendix F.3] Table 14's failure-mode distribution is useful, but 'Verifier false-success' is reported as 2 failures for Pigey; this is a rare but important failure class that the paper does not analyze further. A short discussion of when the verifier is fooled would strengthen the limitations section.

Circularity Check

0 steps flagged

No significant circularity: the orchestration results are empirical comparisons, not derivations that reduce to their inputs.

full rationale

The paper's central claim is an empirical one: a frozen policy plus an inference-time orchestrator succeeds where direct prompting fails. I walked the claimed derivation chain and found no step in which a 'prediction' is equivalent by construction to an input. There are no fitted parameters; the only quantity defined from the data is the 'orchestration gap,' which is simply the measured success difference between the direct and agentic conditions, not an output derived from that difference. The main adjacent concern is evaluation co-design: Appendix J's 'General resolution strategies' (occlusion search, superlative-constrained selection, absolute-coordinate restoration, sequential imitation) closely mirror the Appendix B probes (OB1, CL1/SR3, LM1, LM2), so the benchmark partly tests instruction-following rather than autonomous reasoning, and the paper does not ablate these prompt blocks. That is a confound about attribution of the gains, not circularity in the sense of a derivation reducing to its inputs; the system's performance is still measured against external tasks and the strategies are disclosed components of the system under test. Citations to prior work, including any author-involved reference, are background context and are not load-bearing for the central comparison. Accordingly, no circular step is identified and the score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical entities, forces, or conserved quantities. Its free parameters are engineering hyperparameters. The main implicit postulates are about the competence of the frozen frontier VLM and the reliability of perception/verification signals.

free parameters (4)
  • T_max (max tool calls per episode)
    Chosen by hand; not fitted, but no ablation shows sensitivity of the headline gains to this budget.
  • K=300 control steps per VLARollout = 300
    Manual rollout horizon; not fitted, but determines how often verification can be triggered.
  • Retry-once-then-escalate rule
    Hand-specified recovery policy in Appendix E; no ablation comparing other retry counts.
  • H=8 action-chunk stride = 8
    Inherited from π0.5 inference loop; chosen by hand, not part of a demonstrated optimization.
axioms (4)
  • domain assumption A frozen frontier VLM can verify physical outcomes from wrist images and sensor flags well enough to drive a closed loop.
    Central to verification (§3.4). The paper tests this empirically, but it is not proven and has known false-success cases (Appendix F.3, 2 failures).
  • domain assumption The frozen backends (TiPToP/TAMP and π0.5) cover the motor skills required by the task suite.
    Explicitly acknowledged in Limitations: 'Pigey is limited by the skills it orchestrates... there is a ceiling set by the support of the low-level tools.'
  • domain assumption The open-vocabulary detector returns accurate and complete object labels on every Perceive call.
    The orchestrator's vocabulary is these labels (Appendix D); if detection misses or mislabels objects, the agent cannot reason about them.
  • domain assumption Human evaluator scoring of real-robot trials is unbiased and consistent with the task predicates.
    Appendix K states success is determined by a human evaluator; it does not state whether the evaluator is blind to the method, which matters for 5-trial binary judgments.

pith-pipeline@v1.3.0-alltime-deepseek · 18382 in / 11767 out tokens · 114244 ms · 2026-08-01T06:52:11.737296+00:00 · methodology

0 comments
read the original abstract

General-purpose robots need to reason about their actions, combining perception, world knowledge, planning, success detection, recovery, and low-level control. Today's state-of-the-art models attempt to combine all these capabilities into the learned policy via large-scale pre-training. Instead, we show that these capabilities can be decomposed into a general language-conditioned policy/control agent and a high-level agent manager/orchestrator. Rather than training policies to reason via pre-training, we build a closed-loop physical agent orchestrator that can do high-level planning, decompose the goal into achievable subgoals, command low-level motor commands, track and verify the outcome from low-level observations, and recover from failures. Our Physical Agency orchestrator (Pigey) can control existing vision-language-action (VLA) policies as well as parametrized skills to solve complex reasoning tasks in the real world, without any additional data collection or post-training. We evaluate Pigey extensively across simulation benchmarks and challenging real-world robotic manipulation tasks, and demonstrate significant performance improvements over existing generalist policies. On LIBERO-PRO, Pigey advances the state-of-the-art by over 4x (12.8% -> 53.3%) with no task-specific fine-tuning. On a real robot, Pigey lifts the frozen policy from near-zero to over 90% on reasoning-limited tasks. We call the difference between what frozen motor skills achieve alone and inside the agentic loop the orchestration gap.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 10 linked inside Pith

  1. [1]

    InRobotics: Science and Systems (RSS), 2025

    Kevin Black et al.𝜋0: A vision-language-action flow model for general robot control. InRobotics: Science and Systems (RSS), 2025

    2025

  2. [2]

    𝜋0.5: A vision-language-action model with open-world generalization

    Physical Intelligence. 𝜋0.5: A vision-language-action model with open-world generalization. arXiv:2504.16054, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Kim et al

    M.J. Kim et al. OpenVLA: An open-source vision-language-action model.arXiv:2406.09246, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Khazatsky et al

    A. Khazatsky et al. DROID: A large-scale in-the-wild robot manipulation dataset. InRobotics: Science and Systems (RSS), 2024

    2024

  5. [5]

    GR00T N1: An open foundation model for generalist humanoid robots

    Johan Bjorck et al. GR00T N1: An open foundation model for generalist humanoid robots. arXiv:2503.14734, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Liang et al

    J. Liang et al. Code as policies: Language model programs for embodied control. InICRA, 2023

    2023

  7. [7]

    Singh et al

    I. Singh et al. ProgPrompt: Generating situated robot task plans using large language models. InICRA, 2023

    2023

  8. [8]

    Ahn et al

    M. Ahn et al. Do as i can, not as i say: Grounding language in robotic affordances. InCoRL, 2022

    2022

  9. [9]

    Huang et al

    W. Huang et al. VoxPoser: Composable 3d value maps for robotic manipulation with language models. InCoRL, 2023

    2023

  10. [10]

    Shi et al

    L. Shi et al. Hi robot: Teaching robots to listen and think harder.arXiv:2502.19417, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Zawalski et al

    M. Zawalski et al. Robotic control via embodied chain-of-thought reasoning. InCoRL, 2024

    2024

  12. [12]

    Shen et al

    W. Shen et al. TiPToP: A modular open-vocabulary planning system for robotic manipulation. arXiv:2603.09971, 2026

    Pith/arXiv arXiv 2026

  13. [13]

    Zhou et al

    X. Zhou et al. LIBERO-PRO: Towards comprehensive evaluation of robotic manipulation under perturba- tions.arXiv:2510.03827, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Huang et al

    W. Huang et al. Inner monologue: Embodied reasoning through planning with language models. In CoRL, 2022

    2022

  15. [15]

    Fu et al

    M. Fu et al. CaP-X: A framework for benchmarking and improving coding agents for robot manipulation. arXiv:2603.22435, 2026

    Pith/arXiv arXiv 2026

  16. [16]

    Gemini robotics: Bringing ai to the physical world

    Google DeepMind. Gemini robotics: Bringing ai to the physical world. Technical report, 2025

    2025

  17. [17]

    Chen et al

    W. Chen et al. Steerable vision-language-action policies for embodied reasoning and hierarchical control. arXiv:2602.13193, 2026

    Pith/arXiv arXiv 2026

  18. [18]

    Pan et al

    J. Pan et al. MemER: Scaling up memory for robot control via episodic-keyframe selection.https: //jen-pan.github.io/memer/, 2026

    2026

  19. [19]

    Kang et al

    J. Kang et al. RoboMonkey: Scaling test-time sampling and verification for vision-language-action models. arXiv:2506.17811, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Robometer: Scaling general-purpose robotic reward models via trajectory comparisons.arXiv preprint arXiv:2603.02115, 2026

    Anthony Liang, Yigit Korkmaz, Jiahui Zhang, Minyoung Hwang, Abrar Anwar, Sidhant Kaushik, Aditya Shah, Alex S Huang, Luke Zettlemoyer, Dieter Fox, et al. Robometer: Scaling general-purpose robotic reward models via trajectory comparisons.arXiv preprint arXiv:2603.02115, 2026

    Pith/arXiv arXiv 2026

  21. [21]

    Vision language models are in-context value learners

    Yecheng Jason Ma, Joey Hejna, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, et al. Vision language models are in-context value learners. InInternational Conference on Learning Representations, volume 2025, pages 33984–34009, 2025

    2025

  22. [22]

    Pick up the doll and put it in the bowl

    B. Liu et al. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. InNeurIPS Datasets and Benchmarks, 2023. 11 Appendix A Full Problem Setup We study general-purpose tabletop manipulation from natural-language instructions. The agent observes the scene through cameras, receives a language instruction, and must execute physical actions unti...

    2023

  23. [23]

    childproof the table

    Perceive before acting on abstract tasks.If the instruction names a specific target object, the orchestrator may callPickdirectly. If the instruction is abstract or multi-object (e.g. “childproof the table”), it first callsPerceiveto obtain the detected object labels, then maps the instruction’s semantics onto those concrete labels. EveryPick/DropAbovearg...

  24. [24]

    Rigid grasp via TAMP, then verify.For a rigid target the orchestrator callsPickand verifies the grasp from the wrist view and the gripper-width sensor (is_grasped)

  25. [25]

    A second failure escalates toVLARollout

    Retry once, then escalate.An unverifiedPickis retried once; the retry re-perceives and re-plans at the object’s current location. A second failure escalates toVLARollout

  26. [26]

    Deformables go straight to the VLA.Cable-, cloth-, or rope-like targets bypass TAMP entirely, since the grasp predictor is not trained for deformables

  27. [27]

    Placement.A held object is placed withDropAboveon the cached target location from a prior Perceive/Pick

  28. [28]

    Bidirectional fallback.If aVLARolloutmakes no progress, the orchestrator may fall back to a TAMPPickon a freshly perceived scene, and vice versa

  29. [29]

    There’s the tape

    Stop condition.The orchestrator callsDoneonly after verifying the task predicate; for clear- ing/sorting tasks it stops once all in-scope objects have been moved, rather than reaching for borderline items. The detected object labels come from an open-vocabulary detector and are regenerated on each Perceive; the orchestrator therefore re-reads the label se...