REVIEW 4 major objections 5 minor 16 references
Beyond Flat Policies: Hierarchical Post-Training for Embodied Agents in Robotic Manipulation
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read HiRoC: separating a frozen subgoal planner from a reinforcement-learned executor outperforms flat VLA post-training by 10.06% on average.
desk verdict Sensible hierarchical VLA post-training framework, but its distinguishing local-GRPO branch rests on an undefined subgoal score and a weighting that makes the branch nearly negligible; the results are promising but the paper overclaims and is not yet reproducible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the hierarchical GRPO advantage used during reinforcement learning. Each rollout i receives a task-level advantage from the group-standardized terminal return and a subgoal-level advantage from the group-standardized aggregated subtask score S_i, and the final advantage used in the clipped PPO-style actor loss is a weighted sum of the two, with weights that depend on K, the average number of subgoals per trajectory, and L, the trajectory length. This converts flat outcome-level RL into subgoal-structured RL. The other key object is the alignment dataset, built by cutting each demonstration into subgoal-conditioned chunks of observation, subgoal, and action chunk; its supervised fine-tuning loss gives the executor an initialization that mitigates the distribution misalignment between global instructions and fine-grained subgoals before online interaction.
What would settle it
A concrete falsifier is to re-run the w/oL ablation on all four LIBERO suites with S_i defined by a dense per-subgoal reward or by text-match against an annotated subgoal sequence and then measure the rank correlation between the subgoal-level and task-level advantages within each GRPO group; if the correlation is near 1, or if the reported w/oL gain is reproduced when S_i is replaced by terminal success, the local GRPO term is not doing independent work.
Extended reading notes
Core claim
HiRoC's central claim is that the prevailing flat-policy post-training protocol for VLA models is the wrong optimization target for long-horizon manipulation, and that a two-level decomposition is the right one. A high-level planner, initialized from a vision-language model and fine-tuned on reorganized subgoal annotations, is frozen and repeatedly emits intermediate subgoals such as approach, lift, move, and place from the current observation. The low-level executor, initialized from OpenVLA, is first fine-tuned on subgoal-conditioned trajectory chunks—this alignment stage lets it understand planner-generated subgoals—and then updated by online reinforcement learning. The RL objective combines a standardized global advantage from terminal success with a standardized local advantage from per-subgoal scores, weighted by the average number of subgoals relative to trajectory length. The paper reports that this combination outperforms flat RL baselines, world-model baselines, and step-advantage baselines, with the largest margins on the Spatial and Long suites, and that removing the planner, the alignment stage, or the local advantage degrades performance.
Load-bearing premise
The subgoal-level advantage in Eq. (6) depends on an aggregated subtask score S_i that the paper never defines; if S_i is derived from the terminal success reward or from the planner's own outputs, the local term carries no independent information and the benefit attributed to subgoal-level RL is uninterpretable.
Editorial extensions
If this is right
- If correct, VLA post-training should stop conditioning on the full task instruction throughout an episode; stage-level subgoal conditions are learnable and improve long-horizon success.
- The same recipe—frozen planner, alignment SFT, subgoal-level GRPO—should transfer to other embodied benchmarks where tasks can be segmented into coarse stages, and the zero-shot LIBERO-Plus results suggest it also generalizes to perturbed observations.
- The gains concentrate on the Spatial and Long suites, so task decomposition matters most where stage transitions are semantically distinct and error accumulation is otherwise fatal.
- Removing the planner or the distribution-misalignment SFT is reported to hurt both final performance and learning stability, so hierarchy and alignment are essential rather than optional components.
Reading between the lines
- Because the aggregated subtask score S_i is never defined in the paper, the local-advantage term is the least secured part of the recipe; the reader should ask whether the reported gains survive when S_i is computed from a dense, independently defined subgoal reward rather than from terminal success or the planner's own outputs.
- The frozen-planner design implies that the ceiling of the whole system is the planner's decomposition quality; an end-to-end variant that lets the executor influence replanning could close the gap when the planner missegments a task, but that extension is beyond the paper.
- The same hierarchical post-training logic could apply beyond manipulation, such as navigation or multi-room household tasks, wherever a language-conditioned low-level policy can be aligned to subgoals and optimized with subgoal-level advantage estimates; the paper demonstrates it only on manipulation.
- Reorganizing existing trajectory data into subgoal-conditioned chunks may be a data-efficient way to reuse long-horizon demonstrations, since it avoids collecting new annotations while still teaching the executor to respond to intermediate objectives.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HiRoC, a hierarchical post-training framework for vision-language-action (VLA) manipulation policies. A frozen high-level planner, trained by supervised fine-tuning, decomposes a task instruction into subgoals; a low-level executor is first aligned to those subgoals with SFT and then optimized online with a hierarchical GRPO objective that combines task-level and subgoal-level advantages. Experiments are reported on the four LIBERO suites, on LIBERO-Plus perturbations, and in a real-world sim-to-real deployment; the headline numbers are a 93.5% average success rate across LIBERO and an average 10.06% improvement over ten baselines. The central methodological novelty is the local-GRPO term in Eq. (8), which depends on an aggregated subtask score S_i that the paper never operationally defines.
Significance. If the method and results were fully specified, the paper would make a useful contribution: it addresses a real gap in VLA post-training (flat policies lack explicit stage structure), and the three-stage recipe (planner SFT, subgoal-conditioned executor SFT, then hierarchical GRPO) is intuitive and potentially practical. The evaluation scope is creditable, covering a standard benchmark, a perturbation benchmark, ablations, and a real-world deployment. However, the local-GRPO contribution is not currently reproducible because S_i is never defined, the executor SFT loss in Eq. (2) is not a well-formed likelihood as printed, and the headline claim of consistent superiority over all baselines is contradicted by the paper's own Table 1 on the Object and Goal suites. The significance of the empirical gains is also reduced by the absence of variance reporting. The core idea is salvageable, but the manuscript in its current form does not yet substantiate the claimed benefits.
major comments (4)
- [Methodology, 'RL Tuning', Eq. (6)] The aggregated subtask score S_i is never operationally defined. The sentence 'Let S_i denote the aggregated subtask score of trajectory i, computed from the intermediate subgoal segments contained in that trajectory' does not say how each segment is scored or how the segment scores are aggregated. This matters because Eq. (6) enters the final advantage Eq. (8) and the actor loss Eq. (12), and because the w/oL row of Table 3 is the only ablation that validates the local-GRPO branch. If S_i is derived only from terminal success, then A_sub^i is a re-scaled version of A_task^i and Eq. (8) carries no independent information; if it is derived from the planner's own subgoal transitions, the local advantage may reward planner statements rather than physical progress. The authors should define S_i, state its source (terminal reward, planner transitions, dense per-subgoal reward, or a learned critic), and specify the aggregation rule.
- [Methodology, Eq. (2)] The executor SFT loss in Eq. (2) is not a well-formed log-likelihood as written: the expectation is over (s_i, l_i, a_i), but the logarithm inside is log Pπ(·|s_i, l_i), which conditions only on the observation and the subgoal and contains no target action tokens; the sums over t and d therefore have no argument that depends on the sampled action chunk. This should presumably be a token-level cross-entropy of the form Σ_{t,d} log π(a_{t,d} | s_i, l_i, a_{i,<t,d}) (or the equivalent over flattened action tokens). As printed, the equation cannot be implemented, and this is load-bearing because the subgoal-conditioned SFT step is the mechanism the paper claims mitigates planner–executor distribution misalignment.
- [Experiments, Table 1] The text states that 'HiRoC achieves state-of-the-art performance' and 'consistently outperforms strong baselines,' but Table 1 itself shows the opposite on two of the four suites: on Object, VAL-OS-A-S achieves 96.5% versus HiRoC's 96.0%; on Goal, VAL-OS-A-S achieves 92.7% and OpenVLA*-Full achieves 90.6% versus HiRoC's 84.4%. HiRoC is best only on Spatial (95.6%) and Long (98.0%), and its average rank of 10.3 reflects that it is not rank 1 in every suite. The claims and the abstract's 'consistently outperforms' should be narrowed to 'achieves the best average success rate and the best results on Spatial and Long,' and the Goal deficit should be discussed rather than attributed to visual grounding without supporting analysis.
- [Experiments, Table 1 and Table 3] No variance, confidence interval, or significance test is reported for any success rate, although the evaluation protocol is only 50 episodes per task. This is especially important for the local-GRPO ablation: the difference between w/oL (95.20%) and HiRoC (96.00%) in Table 3 is 0.8 percentage points, which is likely within sampling noise at that sample size, and the w/oG row collapses to 4%, suggesting that the local term contributes little on its own. Please report standard errors or confidence intervals across the eight environment seeds mentioned in the appendix, and state the number of trials behind each number in Table 3.
minor comments (5)
- [Appendix, Experimental Details] The real-world paragraph introduces JoyRA-0.1 and Flow-SDE without any description, and it is unclear whether the real-world executor is the same OpenVLA-OFT plus GRPO pipeline used in simulation; please clarify the relationship between the simulated and real-world training setups.
- [Table 2] The column headers of Table 2 are garbled ('L. Method Subgoal Episode Avg. Short'), and the metrics 'text similarity' and 'overall similarity' are never defined; RoboBrain2's near-zero values are implausible for a similarity measure, so the reader cannot tell what is being reported.
- [Experiments, zero-shot performance] The text refers to 'OpenVAL*-One' while Table 1 uses 'OpenVLA*-Full'; please standardize baseline names and define the asterisk convention consistently.
- [Methodology, Eq. (9)] The definitions of K and L are too loose: K is 'the average number of recorded subgoals per trajectory' and L is 'the length of collected trajectories,' but it is not specified over which trajectories these are computed (the group of N, the replay buffer, or the whole rollout), nor whether L is measured in environment steps, action chunks, or tokens; this makes the adaptive weights non-reproducible.
- [Introduction and notation] There is a typo 'brige' for 'bridge' in the third paragraph of the introduction, and 'Robobrain2' and 'RoboBrain2' are used inconsistently across the text and Table 2.
Circularity Check
No significant circularity: HiRoC's headline gains are benchmarked against external LIBERO baselines; the undefined subgoal score S_i is a reproducibility gap, not a circular derivation.
full rationale
Walking the derivation chain, the planner is obtained by SFT on reannotated VLA-OS data and the executor by subgoal-conditioned SFT followed by RL. The hierarchical advantage in Eq. (8) combines the task-level GRPO advantage of Eq. (4) with the subgoal-level advantage of Eq. (6). The paper states that S_i is 'computed from the intermediate subgoal segments contained in that trajectory,' which the text does not equate with the terminal reward used in Eq. (4); although the absence of an operational definition for S_i makes the local-GRPO branch and the w/oL ablation hard to reproduce, this is an under-specification rather than a circular reduction. The headline results are comparisons against external baselines on LIBERO, LIBERO-Plus, and a real-robot deployment, so the main empirical claim is not derived from the paper's own equations. The only self-citation of note is TGRPO (Chen et al. 2025), which shares overlapping authors with this paper; it appears as a related-work reference and as one baseline in Table 1, but the central claims do not depend on TGRPO's correctness or on any uniqueness theorem imported from it. The planner's cross-validation measures agreement with its own reannotated label space, which is self-referential, but the downstream success-rate results provide independent external evidence. No step reduces a stated prediction to its own input by construction, so the finding is no significant circularity, with a minor score adjustment only for the non-load-bearing self-citation.
Assumptions & free parameters
free parameters (3)
- Subgoal-level score S_i =
not defined
- Advantage weights w_t and w_s =
Eq. (9), not numerically reported
- Replanning interval =
20 policy calls
assumptions (4)
- domain assumption Average success rate on LIBERO is the right proxy for hierarchical manipulation quality
- domain assumption Planner subgoals trained on cleaned VLA-OS data transfer to LIBERO states
- domain assumption Subgoal-conditioned SFT removes the planner-executor distribution mismatch
- standard math GRPO group standardization gives unbiased advantage estimates
Cite this review
Pith. "Pith review of Beyond Flat Policies: Hierarchical Post-Training for Embodied Agents in Robotic Manipulation." pith.science (2026). https://pith.science/paper/ENGNYTA3
@misc{pith2026260805999,
author = {Pith},
title = {Pith review of: Beyond Flat Policies: Hierarchical Post-Training for Embodied Agents in Robotic Manipulation},
year = {2026},
howpublished = {\url{https://pith.science/paper/ENGNYTA3}},
note = {Machine review of arXiv:2608.05999}
}
read the original abstract
Vision-language-action (VLA) models have demonstrated remarkable capabilities in robotic manipulation by leveraging pretrained vision-language models. However, existing post-training methods predominantly optimize VLA models as flat policies, making it difficult to explicitly model task progression and perform robust long-horizon manipulation. Although hierarchical approaches introduce task decomposition, they mainly rely on supervised learning from offline demonstrations and cannot effectively improve execution through online interaction. To address this limitation, we propose Hierarchical Robotic Control (HiRoC), a hierarchical post-training framework that decouples high-level task planning from low-level action execution. The planner decomposes complex tasks into executable subgoals to provide explicit semantic guidance, while the executor continuously improves subgoal-conditioned action generation through reinforcement learning. To enable effective collaboration between the two modules, we further align the executor with planner-generated subgoals before reinforcement learning, mitigating the distribution misalignment between planning and execution. Extensive experiments across diverse robotic manipulation benchmarks demonstrate that HiRoC consistently outperforms strong baselines. Comprehensive analyses further validate the effectiveness of hierarchical post-training and the contribution of each key component.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Rt-h: Action hierarchies using language.arXiv preprint arXiv:2403.01823
Belkhale, S.; Ding, T.; Xiao, T.; Sermanet, P.; Vuong, Q.; Tompson,J.;Chebotar,Y.;Dwibedi,D.;andSadigh,D.2024. Rt-h: Action hierarchies using language.arXiv preprint arXiv:2403.01823. Cao,L.;Chen,R.;Li,L.;Wang,P.;Peng,M.;andLi,Y.2026. Z-1:EfficientReinforcementLearningforVision-Language- Action Models.arXiv preprint arXiv:2606.31846. Chen, Z.; Niu, R.; Ko...
arXiv 2024
-
[3]
Guo, D.; Yang, D.; Zhang, H.; Song, J.; Wang, P.; Zhu, Q.; Xu, R.; Zhang, R.; Ma, S.; Bi, X.; et al
Vla- os: Structuring and dissecting planning representations and paradigmsinvision-language-actionmodels.arXiv preprint arXiv:2506.17561. Guo, D.; Yang, D.; Zhang, H.; Song, J.; Wang, P.; Zhu, Q.; Xu, R.; Zhang, R.; Ma, S.; Bi, X.; et al. 2025a. DeepSeek- R1 incentivizes reasoning in LLMs through reinforcement learning.Nature, 645(8081): 633–638. Guo,Y.;Z...
-
[7]
Molmoact: Action reasoning models that can reason in space.arXiv preprint arXiv:2508.07917. Li, H.; Zuo, Y.; Yu, J.; Zhang, Y.; Yang, Z.; Zhang, K.; Zhu, X.;Zhang,Y.;Chen,T.;Cui,G.;etal.2025.SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning.arXiv preprint arXiv:2509.09674. Liu, B.; Zhu, Y.; Gao, C.; Feng, Y.; Liu, Q.; Zhu, Y.; and Stone, P
arXiv 2025
-
[8]
InProceedings of the 37th Conference on Neural Information Processing Systems, 44776–44791
LIBERO: Benchmarking Knowledge Trans- fer for Lifelong Robot Learning. InProceedings of the 37th Conference on Neural Information Processing Systems, 44776–44791. Long, Q.; Wang, Y.; Song, J.; Zhang, J.; Li, P.; Wang, W.; Wang,Y.;Li,H.;Xie,S.;Yao,G.;etal.2026. Scalingworld modelforhierarchicalmanipulationpolicies.arXiv preprint arXiv:2602.10983. Lu, G.; G...
arXiv 2026
-
[9]
Ma, Y.; Song, Z.; Zhuang, Y.; Hao, J.; and King, I
Vla-rl: Towards masterful and general robotic manipulation with scalable reinforce- ment learning.arXiv preprint arXiv:2505.18719. Ma, Y.; Song, Z.; Zhuang, Y.; Hao, J.; and King, I
-
[10]
arXiv preprint arXiv:2405.14093
Asurveyonvision-language-actionmodelsforembodiedai. arXiv preprint arXiv:2405.14093. Ma, Y.; Song, Z.; Zhuang, Y.; Hao, J.; and King, I
-
[11]
A Survey on Vision–Language–Action Models for Embodied AI.IEEE Transactions on Neural Networks and Learning Systems, 37(7): 3031–3051. Shukor, M.; Aubakirova, D.; Capuano, F.; Kooijmans, P.; Palma,S.;Zouitine,A.;Aractingi,M.;Pascal,C.;Russi,M.; Marafioti,A.;etal.2025. Smolvla:Avision-language-action model for affordable and efficient robotics.arXiv prepri...
arXiv 2025
-
[12]
Wang, Z.; Chen, Y.; Liu, Y.; Ye, J.; Chen, P.; Lu, C.; Liu, S.; and Jia, J
RoboBrain 2.0 Technical Report.arXiv preprint arXiv:2507.02029. Wang, Z.; Chen, Y.; Liu, Y.; Ye, J.; Chen, P.; Lu, C.; Liu, S.; and Jia, J
Show all 16 references
-
[13]
Xiao, J.; Yang, Y.; Chang, X.; Chen, R.; Xiong, F.; Xu, M.; Zheng, W.-S.; and Zhang, Q
VP-VLA: Visual Prompting as an In- terface for Vision-Language-Action Models.arXiv preprint arXiv:2603.22003. Xiao, J.; Yang, Y.; Chang, X.; Chen, R.; Xiong, F.; Xu, M.; Zheng, W.-S.; and Zhang, Q
-
[14]
arXiv preprint arXiv:2509.24948
World-env: Leveraging world model as a virtual environment for vla post-training. arXiv preprint arXiv:2509.24948. Yi, T.; Yang, Q.; and Chen, E
-
[15]
Yu,C.;Wang,Y.;Guo,Z.;Lin,H.;Xu,S.;Zang,H.;Zhang, Q.;Wu,Y.;Zhu,C.;Hu,J.;etal.2025
ActionX: pre- training action experts with reinforcement learning for vision-language action models.Frontiers in Neurorobotics, 20: 1806605. Yu,C.;Wang,Y.;Guo,Z.;Lin,H.;Xu,S.;Zang,H.;Zhang, Q.;Wu,Y.;Zhu,C.;Hu,J.;etal.2025. Rlinf:Flexibleandef- ficientlarge-scalereinforcementle...
2025
-
[16]
Zhao,Q.;Lu,Y.;Kim,M.J.;Fu,Z.;Zhang,Z.;Wu,Y.;Li,Z.; Ma, Q.; Han, S.; Finn, C.; Handa, A.; Liu, M.-Y.; Xiang, D.; Wetzstein,G.;andLin,T.-Y.2025
Grape: Generalizing robot policy via preference alignment.arXiv preprint arXiv:2411.19309. Zhao,Q.;Lu,Y.;Kim,M.J.;Fu,Z.;Zhang,Z.;Wu,Y.;Li,Z.; Ma, Q.; Han, S.; Finn, C.; Handa, A.; Liu, M.-Y.; Xiang, D.; Wetzstein,G.;andLin,T.-Y.2025. CoT-VLA:VisualChain- of-Thought Reasoning f...
2025 arXiv
-
[2023]
Huang, C.-P.; Wu, Y.-H.; Chen, M.-H.; Wang, F.; and Yang, F.-E
Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis.arXiv preprint: arXiv:2312.08782. Huang, C.-P.; Wu, Y.-H.; Chen, M.-H.; Wang, F.; and Yang, F.-E
-
[2024]
Lee, J.; Duan, J.; Fang, H.; Deng, Y.; Liu, S.; Li, B.; Fang, B.; Zhang, J.; Wang, Y
Openvla: An open-source vision-language- action model.arXiv preprint arXiv:2406.09246. Lee, J.; Duan, J.; Fang, H.; Deng, Y.; Liu, S.; Li, B.; Fang, B.; Zhang, J.; Wang, Y. R.; Lee, S.; et al
-
[2025]
Fei,S.;Wang,S.;Shi,J.;Dai,Z.;Cai,J.;Qian,P.;Ji,L.;He, X.;Zhang,S.;Fei,Z.;etal.2025
Tgrpo: Fine-tuning vision-language-action model viatrajectory-wisegrouprelativepolicyoptimization.arXiv preprint arXiv:2506.08440. Fei,S.;Wang,S.;Shi,J.;Dai,Z.;Cai,J.;Qian,P.;Ji,L.;He, X.;Zhang,S.;Fei,Z.;etal.2025. Libero-plus:In-depthRo- bustnessAnalysisofVision-Language-Acti...
2025
-
[2026]
Thinkact: Vision-language-action reasoning via reinforced visual latent planning.Advances in Neural Infor- mation Processing Systems, 38: 82782–82802. Jiang, Z.; Xie, Y.; Lin, K.; Xu, Z.; Wan, W.; Mandlekar, A.; Fan,L.J.;andZhu,Y.2025.DexMimicGen:AutomatedData GenerationforBim...
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.