Pith. sign in

REVIEW 4 major objections 5 minor 145 references

RL$^2$-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Pith's one-line read This paper argues that test-time action diversity from reinforcement-learned steering helps only when the base robot policy is failing, and that gating steering on a failure detector yields gains up to +17.3% on out-of-domain tasks.

desk verdict A practical, well-built adaptive steering method whose 'scaling law' foundation is thinner than it looks — worth refereeing, but the central asymmetry claim needs direct validation. read the letter →

arxiv 2607.26991 v2 pith:GADTS5BF submitted 2026-07-29 cs.RO

classification cs.RO
keywords vision-language-actionmodelsofflinereinforcementlearningflowmatchingtest-timescalingadaptiveinferencefailuredetectionconformalpredictionrobotmanipulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks when and how to steer a frozen vision-language-action (VLA) model at inference time. It argues that sampling more actions from the same policy is not enough, because those samples inherit the base policy's failure modes. Its RL² framework trains a lightweight offline RL policy on the VLA's internal feature vectors, composes the RL policy's flow velocity with the frozen VLA's flow velocity to generate diverse action candidates, and enables that composition only when a learned failure detector says failure is likely. The supporting discovery is a state-dependent scaling law: under an oracle verifier, RL compositional steering reduces action error on failure tuples but degrades already-accurate actions on success tuples. In the paper's experiments, adaptive steering beats both the base VLA and always-on steering, with success-rate gains up to +17.3% in out-of-domain simulation and +17.5% on real hardware.

What carries the argument

Compositional steering of flow velocities. In a flow-matching VLA, actions are generated by following a velocity field that carries noise onto action trajectories. The paper combines the frozen VLA's velocity field with that of a lightweight offline RL steering policy—both conditioned on latents (internal feature vectors) from the VLA action expert—as a weighted average, with the weight drawn from a Gaussian. This composes the imitation prior with RL-induced diversity. The RL policy is trained offline with an adjoint-matching Q-learning objective. A separate lightweight failure detector, calibrated by conformal prediction, decides when to activate the composition; an external verifier ranks

What would settle it

On an out-of-domain task with success/failure defined by task completion, compare always-on composition, adaptive composition, and the base policy using the same verifier. If a perfect failure oracle does not outperform always-on steering, or if compositional steering reduces action error on success states when failure labels come from task outcome rather than action NRMSE, the central asymmetry claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is a pair of test-time scaling laws. With an oracle verifier, generating more action candidates from a flow-matching VLA reduces normalized action error as a power law in the number of samples. The law is not uniform, however: RL-based compositional steering is the strongest reducer of action error on failure tuples and among the worst on success tuples, where it unnecessarily perturbs accurate actions. The paper reads this asymmetry as evidence that inference-time steering should be adaptive—switched on only when the base policy is predicted to fail—and reports that this adaptive rule outperforms both the base policy and always-on steering across two simulation benchmarks,

Load-bearing premise

The load-bearing assumption is that the states where the base policy is about to fail can be recognized at run time: the scaling law is derived with an oracle that knows the ground-truth action, while deployment relies on a learned failure detector making the same distinction.

Editorial extensions

If this is right

  • Adaptive steering beats always-on steering in the paper's experiments, implying that fixed test-time intervention wastes its benefit on states where the base policy already succeeds.
  • Latent-conditioned RL steering outperforms behavior-cloned steering and raw-observation-conditioned steering, implying that internal VLA features carry the state information needed for recovery.
  • Scaling the number of action samples and the number of language rephrases both improve success rates, and compositional steering adds to gains from either form of scaling.
  • The gains transfer from simulation to a real manipulator (+17.5% over the strongest baseline), suggesting the gating logic is not an artifact of the simulator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's scaling law labels failure states using ground-truth actions that exist only in the offline dataset; re-deriving the law with failure defined by actual task outcomes would test whether the adaptive trigger gives the same benefit in deployment.
  • The conditional-steering recipe is generic: any large policy with internal latents and a flow or diffusion head could gate resampling or self-correction on predicted failure instead of applying it unconditionally.
  • The failure detector is trained on collected rollouts, so failures identified by the external verifier during deployment could become new labels for the detector, making the steering increasingly selective over time.
  • The conformal-threshold sweep could be replaced by learning to schedule the threshold online from reward feedback, removing the per-task calibration step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes RL2, an inference-time steering framework for vision-language-action (VLA) models. A lightweight offline RL flow-matching policy is trained on latents extracted from a frozen VLA action expert, and its velocity field is composed with the VLA's flow velocity during action generation. A failure detector (SAFE) with conformal prediction is used to trigger this compositional steering only when the base VLA is predicted to fail; otherwise, the base VLA samples are used. The paper reports a scaling-law analysis on BridgeV2 claiming that compositional steering reduces action error under high-NRMSE 'failure' tuples and degrades under low-NRMSE 'success' tuples. Based on this, it argues for adaptive steering. Experiments on SIMPLER, PolaRiS, and a real PiperX manipulator report success-rate gains over repeated and rephrased sampling baselines, with ablations on RL training, latent conditioning, and the failure-detection trigger.

Significance. If the central scaling-law asymmetry held for deployment-time failure states, the adaptive gating idea would be a meaningful contribution to test-time scaling for VLAs, since existing methods apply a uniform intervention at every timestep. The paper has clear strengths: it evaluates across three VLAs, two simulation benchmarks, and real hardware; reports timing overhead; and provides detailed hyperparameters. The real-robot gains (+17.5% over Rephrase, +14.2% over non-adaptive) are encouraging. However, the 'when to steer' insight is currently derived from an oracle NRMSE-based partition of BridgeV2 tuples, not from the task-outcome failure states that SAFE sees at deployment, and the scaling-law measurements lack error bars and quantitative fits. These gaps are load-bearing because the adaptivity is the paper's main novelty. The claimed results may be salvageable with additional analyses, but the evidence presented does not yet establish the central claim.

major comments (4)
  1. [Sec. IV and Sec. V-C] The scaling law is established on tuples partitioned by NRMSE between pi0 actions and ground-truth expert actions (top/bottom 1,024 BridgeV2 tuples), not on actual task success or failure. A high-NRMSE state need not be a state where the base policy will fail the task: demonstrations are multimodal, and a low-NRMSE action can still be followed by downstream failure. At deployment, RL2 uses SAFE, a learned binary failure predictor trained on online rollouts with task outcomes. No analysis is provided showing that SAFE-triggered states coincide with the NRMSE-defined failure set, or that the scaling-law asymmetry persists when states are labeled by true success/failure. This is load-bearing because the adaptive-vs-always comparison is the paper's headline. Concretely, I would like to see the scaling-law analysis repeated on rollout states labeled by actual task outcomes, and/or an analysis
  2. [Sec. IV, Eq. (2)] The paper claims the action-error curves follow an 'exponential power law,' but no functional form, fitted parameters, residuals, or confidence intervals are provided. It is also unclear whether the NRMSE at sample count N is the oracle-selection error (minimum over N candidates) or an average over the N candidates; this distinction determines whether a decreasing curve is meaningful at all. Please state the aggregation rule explicitly and report the fitted law with error bars. Without this, the scaling-law claim is a qualitative description rather than a quantitative result.
  3. [Sec. VI-C2, Table II] The ablation supporting the use of VLA latents compares a QAM policy conditioned on latents (39.3% for OpenVLA) with a V-GPS/CQL policy conditioned on raw observations (0.5%). These two conditions differ not only in the conditioning signal but also in the RL algorithm, network architecture, training procedure, and base VLA being steered. This is not a controlled ablation of the latent representation. A clean test would hold the RL algorithm fixed and vary only the conditioning (latents vs. raw observations), or at least add a V-GPS/CQL-on-latents condition.
  4. [Sec. VI-C1, Table III] The comparison of SAFE with CoVer as a trigger may be unfair because CoVer is a contrastive action verifier, not a failure-detection model; the paper itself shows that CoVer success/failure scores are highly correlated. The more important issue is whether the SAFE trigger realizes the Sec. IV scaling-law asymmetry. The current results show only that SAFE-gated steering improves average success over always steering. This could happen even if the scaling-law asymmetry does not hold, e.g., if SAFE selects states where the verifier happens to rank the RL-policy samples highly. I recommend reporting success-rate improvements of RL2-Adaptive over Repeated/Rephrase separately on SAFE-positive and SAFE-negative timesteps, or gating with true outcome labels, to directly test the claimed mechanism.
minor comments (5)
  1. [Eq. (2)] The notation in the NRMSE formula is ambiguous: N is said to be the number of flattened action dimensions, while T is also referenced in the denominator as 'max_T a*_i' and in the surrounding text as the set of tuples. Please define all symbols precisely.
  2. [Fig. 20] The caption states that the PolaRiS Pan Cleaning task is run with pi0 [4], but Sec. VI and Table I indicate PolaRiS experiments use pi0.5 [5]. This inconsistency should be corrected.
  3. [Sec. IV] The phrase 'exponential power law' is used repeatedly, but an exponential law and a power law are different functional forms. The authors should state the intended model explicitly.
  4. [Sec. VII] The self-stated limitations are substantial: the failure detector requires online rollout collection and per-task calibration, and the alpha-selection heuristic needs test-time evaluation. In particular, the real-world experiments had to collect new rollouts to retrain SAFE for the real environment. These points should be reflected in the abstract and conclusion, which currently describe RL2 as practically deployable without qualification.
  5. [Reproducibility] The paper says code will be released with the final paper, but no code or checkpoints are currently provided. Given the number of modules (QAM training, SAFE, CP calibration, verifier integration), releasing code and evaluation scripts is important for reproducibility of the reported gains.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline scaling-law 'discovery' is defined on the paper's own NRMSE-based success/failure split, so the asymmetry that motivates adaptive steering is partly constructed rather than independently established.

  1. self definitional [Sec. IV (Inference-Time Scaling Law), Eq. (2); Abstract]
    "Using π0 [4] as our base VLA, we first generate the policy actions for each tuple(s, a∗, I)within the validation set of the large-scale BridgeV2 dataset [36]. We then compute the normalized root-mean-squared error (NRMSE) between policy and ground truth actions, and extract the top and bottom 1,024 tuples based on NRMSE to define the failure and success tuple sets respectively. ... For failure tuples, RL2 compositional steering exhibits the strongest scaling behavior with significantly lower action error ... In contrast, under success tuples, most methods exhibit the opposite trend."

    The 'failure' and 'success' states are literally the top and bottom 1,024 validation tuples ranked by the NRMSE of the base VLA's actions against ground-truth actions, and the scaling law is then reported as average NRMSE on those same tuples under an oracle verifier. The label used to partition the data is the same outcome metric used to measure improvement, so the claimed asymmetry ('steering improves failure states, degrades success states') is partly guaranteed by selection on the outcome variable: high-error tuples have mechanical room for error reduction, while low-error tuples are near the floor and perturbation tends to worsen min-error. The paper then invokes this asymmetry to justify SAFE-gated adaptive steering, making the central 'when to steer' rationale depend on a self-defin

full rationale

The main circularity is confined to the framing of the scaling-law insight. The paper defines 'failure' and 'success' tuples by base-VLA NRMSE against ground truth, then 'discovers' that steering reduces NRMSE on the failure set and degrades it on the success set; that reduction is an artifact of selecting extreme values of the very metric being averaged. The subsequent claim that action diversity is most useful when the base VLA is likely to fail inherits this construction, so the adaptive-gating motivation is partially circular. The rest of the paper is substantially independent and externally grounded: the RL steering policy is trained on BridgeV2/DROID with QAM, evaluated against RoboMonkey and CoVer verifiers on SIMPLER and PolaRiS, and validated on real hardware, so those benchmark gains do not reduce to the paper's own definitions. The reliance on SAFE is not scored as load-bearing self-citation because the paper directly trains SAFE, reports ROC-AUC, and ablates against an alternative trigger; however, the real-world section acknowledges that SAFE 'could not generalize similarly' after simulation-only training, which is a robustness gap rather than a circularity. Overall, the central empirical results are real, but the headline scaling-law discovery reduces by construction, yielding a partial-circularity score of 6.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entity or mathematical object is introduced. The framework consists of existing modules: QAM, SAFE, CoVer, and velocity composition. The invented conceptual entity is the 'adaptive scaling-law' partitioning, but it is an empirical analysis, not an entity. The main free parameters are the composition weight, CP alpha, and RL temperature, all tuned per task or per model.

free parameters (4)
  • Composition weight distribution w ~ N(0.5, 0.25) = mu=0.5, sigma=0.25
    The velocity composition weight is chosen by hand and drawn from a Gaussian with mean 0.5, variance 0.25; the paper says [19] used exhaustive sweeps but here it is set without per-task tuning (Sec. V-B).
  • Conformal prediction significance level alpha = selected per task via balanced-accuracy heuristic (top-3 among 0.05..0.50 by 0.05)
    Alpha directly controls how often steering is triggered. The paper's own heuristic selects alpha on validation rollouts to maximize balanced accuracy, so this is a fitted per-task threshold rather than a fixed a priori risk level (Sec. V-C).
  • Offline RL inverse temperature tau = 0.1 for pi0, 0.02 for pi0.5
    The temperature in QAM controls how strongly the policy is pushed toward high-Q actions. It is set per base VLA and affects how diverse the steering actions are (Appendix Table V).
  • Positive reward steps H (3) = 3
    The RL reward design uses only the first 3 steps as positive reward; this is a hand-chosen reward-shaping choice that the central steering behavior depends on (Appendix Tables V-VI).
assumptions (4)
  • domain assumption VLA internal feature embeddings e_t are informative enough to train an effective RL steering policy and failure detector.
    The entire method conditions on latent features extracted from the frozen VLA action expert. If these latents lack state information required for recovering from failure, compositional steering fails. Motivated by SAFE [20], but not independently verified here.
  • domain assumption An oracle or proxy verifier can reliably rank the generated action candidates.
    The method always selects actions via a verifier (RoboMonkey or CoVer). The paper explicitly lists this as a limitation and assumes a robust verifier (Sec. VII).
  • domain assumption The flow-matching VLA velocity field can be guided by the RL policy's velocity with no representational conflict.
    Compositional steering assumes a weighted average of two velocity fields stays on valid action trajectories. This is assumed via [19], not demonstrated with ODE properties in this paper (Sec. V-B).
  • domain assumption BridgeV2/DROID ground-truth action NRMSE is a valid proxy for policy success/failure in deployment.
    Used to partition success/failure tuples for the scaling law. This is a pseudo-labeling assumption; high deviation from ground truth can include linguistically valid alternative behaviors that are actually successful (Sec. IV).

how reviews work

0 comments
Cite this review

Pith. "Pith review of RL$^2$-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models." pith.science (2026). https://pith.science/paper/GADTS5BF

@misc{pith2026260726991,
  author       = {Pith},
  title        = {Pith review of: RL$^2$-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GADTS5BF}},
  note         = {Machine review of arXiv:2607.26991}
}
abstract

Despite the impressive visuomotor capabilities enabled by Vision-Language-Action (VLA) models, their performance often degrades on challenging and out-of-domain tasks. Recent test-time steering and scaling methods improve performance without extensive data collection and retraining, but action samples often remain concentrated around similar behaviors and therefore inherit correlated failure modes. Moreover, existing methods apply the same intervention strategy at every timestep, regardless of whether the base policy is already likely to succeed. To address these limitations, we introduce $RL^2$, an adaptive inference-time steering framework that leverages Reinforcement Learning on VLA Latents. First, we train a lightweight offline RL policy conditioned on expressive latents extracted from the VLA action expert and compose its flow velocity with that of the frozen VLA during inference. This compositional steering strategy combines the behavioral priors of large-scale imitation learning with the action diversity induced by offline RL beyond dominant demonstration modes. We further discover that inference-time steering follows fundamentally different scaling laws under success and failure states, revealing that action diversity is most beneficial when the base VLA is likely to fail, but can unnecessarily perturb already-accurate actions when success is likely. Building on this insight, $RL^2$ activates compositional steering only when failure is predicted. Across the SIMPLER and PolaRiS benchmarks, $RL^2$ improves success rates by up to +17.3% in out-of-domain settings, while ablations and scaling studies demonstrate the importance of latent representations and RL training. Finally, real-world experiments demonstrate that these gains transfer beyond simulation, establishing $RL^2$ as a practical and modular steering framework for VLA deployment.

Figures

Figures reproduced from arXiv: 2607.26991 by the authors.

Figure 1
Figure 1. Overview: RL2 improves VLA test-time scaling by adaptively applying RL compositional steering when the base VLA is likely to fail, particularly in out-of-domain settings, without modifying the pretrained VLA. (Left) Our scaling laws indicate that RL compositional steering (green) reduces action error most effectively under failure states, but can unnecessarily perturb already-accurate success states, motivating adap… view at source ↗
Figure 2
Figure 2. Types of In-domain and OOD Tasks: While VLAs often perform well for in-domain tasks seen during training, their success rates tend to deteriorate when encountering unseen language instructions (left, 38.2% → 14.2%) [1] or unseen task environments (right, 70.2% → 36.0%) [2]. Recent methods focus on inference-time steering for VLAs to achieve better generalization to these OOD tasks without extensive data collection a… view at source ↗
Figure 3
Figure 3. Real-Robot Experiment for Tape in Toolbox Task (OOD Environment): This task is challenging for the base VLA (π0 [4]) because the tape was never seen during training. (Left) Rephrase correctly grasps the tape but often collides with the side of the toolbox. (Middle) Non-adaptive RL2 inaccurately approaches the tape, as it unnecessarily steers already-accurate VLA action samples. (Right) Adaptive RL2 relies on accurat… view at source ↗
Figures from the paper (26 more)
Figure 4
Figure 4. Figure 4: Test-time Scaling Laws (Steering Improves Failure States but Degrades Success States): We observe that the action error between ground truth and predicted VLA actions (π0 [4]) consistently decreases as we scale the number of action samples across different offline RL a…
Figure 5
Figure 5. Figure 5: RL2 Framework (Flow-Matching VLAs): (i) For flow-matching-based VLAs like π0 [4], given language instructions and observations, RL2 first extracts action expert latents (et) from its frozen VLA action expert. (ii) The latents et is fed into the RL flow-matching steerin…
Figure 7
Figure 7. Figure 7: SIMPLER OOD Prompt Evaluation: Using π0 [4], adaptive RL2 improves robustness under OOD language instructions [1], achieving up to +14.7% in task-wise success rate (average +10.1%) over Rephrase baseline. state vectors from the VLA action expert; we ablate different ag…
Figure 8
Figure 8. Figure 8: SIMPLER OOD Environment Evaluation: Using π0 [4], adaptive RL2 improves robustness in OOD environments [1], by up to +14.6% in task-wise success rate (average +8.5%) over the strongest Rephrase baseline. SIMPLER (OOD): We test π0 using the SIMPLER bench￾mark with the s…
Figure 10
Figure 10. Figure 10: CP Alpha Selection Heuristic: Our heuristic selects one of the top alphas within top-3 selection across all baselines. TABLE IV SCALING TIME ANALYSIS: RL STEERING AND FAILURE DETECTION REMAIN EFFICIENT WITH INCREASING BATCH SIZES (IN MS, ON NVIDIA RTX5090 GPU). Batch …
Figure 12
Figure 12. Figure 12: PCA Heatmap Analysis: We represent VLA actions along its two principal axes via Principal Component Analysis on tuples from BridgeV2 dataset [36]. During failure, RL2 will direct action distribution (green) to be closer to the ground truth action. 1) RL2 scales well w…
Figure 13
Figure 13. Figure 13: Results of our Real-Robot Experiments: We curate two in-domain and two challenging OOD task environments to compare RL2 against baseline approaches on-hardware. Adaptive RL2 achieves an average of +17.5% over Rephrase [13] baseline, highlighting its robustness to fail…
Figure 14
Figure 14. Figure 14: Real-Robot Experiment for Screwdriver in Toolbox Task (OOD Environment): This task is challenging for the base VLA (π0 [4]) because the screwdriver was never seen during training and the scene includes a tape distractor. (Left) Rephrase correctly grasps the screwdrive…
Figure 15
Figure 15. Figure 15: Simulation Tasks: Task execution for eight SIMPLER tasks [21] (WidowX manipulator) and three PolaRiS tasks [22] (Franka manipulator) [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 15
Figure 15. Figure 15: Simulation Tasks: Task execution for eight SIMPLER tasks [21] (WidowX manipulator) and three PolaRiS tasks [22] (Franka manipulator) [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: Real-World Tasks: Task execution for two in-domain and two OOD real-world task environments on the PiperX manipulator [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]
Figure 16
Figure 16. Figure 16: Real-World Tasks: Task execution for two in-domain and two OOD real-world task environments on the PiperX manipulator [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]
Figure 17
Figure 17. Figure 17: CP Alpha Selection Heuristics: Examples of our Conformal Prediction α selection heuristic comparing balanced accuracy against significance level α plots for real-world tasks, with sample task-wise CP plots (left) and the combined CP plot for OOD tasks (right). From th…
Figure 17
Figure 17. Figure 17: CP Alpha Selection Heuristics: Examples of our Conformal Prediction α selection heuristic comparing balanced accuracy against significance level α plots for real-world tasks, with sample task-wise CP plots (left) and the combined CP plot for OOD tasks (right). From th…
Figure 18
Figure 18. Figure 18: Comparison of SAFE Failure Probabilities and CoVer Scores: For both π0 and π0.5 as the base VLA, we visualize SAFE [20] failure probabilities (left) and CoVer [13] VLM normalized score (right) against normalized episode progress. The solid lines and their correspondin…
Figure 18
Figure 18. Figure 18: Comparison of SAFE Failure Probabilities and CoVer Scores: For both π0 and π0.5 as the base VLA, we visualize SAFE [20] failure probabilities (left) and CoVer [13] VLM normalized score (right) against normalized episode progress. The solid lines and their correspondin…
Figure 19
Figure 19. Figure 19: SIMPLER Experiment for Spoon on Towel Task (OOD Prompt): This task is challenging for the base VLA (π0 [4]) because the base language prompt was never seen during training. (Left) Rephrase repeatedly fails to grasp the spoon during replanning. (Right) Adaptive RL2 is …
Figure 19
Figure 19. Figure 19: SIMPLER Experiment for Spoon on Towel Task (OOD Prompt): This task is challenging for the base VLA (π0 [4]) because the base language prompt was never seen during training. (Left) Rephrase repeatedly fails to grasp the spoon during replanning. (Right) Adaptive RL2 is …
Figure 20
Figure 20. Figure 20: PolaRiS Experiment for Pan Cleaning Task (OOD Prompt): This task is challenging for the base VLA (π0 [4]) because the base language prompt was never seen during training. (Left) Rephrase repeatedly fails to grasp the sponge during replanning and eventually gets distra…
Figure 20
Figure 20. Figure 20: PolaRiS Experiment for Pan Cleaning Task (OOD Prompt): This task is challenging for the base VLA (π0 [4]) because the base language prompt was never seen during training. (Left) Rephrase repeatedly fails to grasp the sponge during replanning and eventually gets distra…
Figure 21
Figure 21. Figure 21: SIMPLER Experiment for Toy Dinosaur on Towel Task (OOD Environment): This task is challenging for the base VLA (π0 [4]) because the toy dinosaur and elephant were never seen during training. (Left) Rephrase initially approaches the dinosaur but is eventually distracte…
Figure 21
Figure 21. Figure 21: SIMPLER Experiment for Toy Dinosaur on Towel Task (OOD Environment): This task is challenging for the base VLA (π0 [4]) because the toy dinosaur and elephant were never seen during training. (Left) Rephrase initially approaches the dinosaur but is eventually distracte…
Figure 22
Figure 22. Figure 22: Real-Robot Experiment for Carrot on Plate Task (OOD Prompt): This task is challenging for the base VLA (π0 [4]) because the base language prompt was never seen during training. (Left) Rephrase repeatedly fails to grasp the carrot during replanning. (Right) Adaptive RL…
Figure 22
Figure 22. Figure 22: Real-Robot Experiment for Carrot on Plate Task (OOD Prompt): This task is challenging for the base VLA (π0 [4]) because the base language prompt was never seen during training. (Left) Rephrase repeatedly fails to grasp the carrot during replanning. (Right) Adaptive RL…
Figure 23
Figure 23. Figure 23: Real-Robot Experiment for Cube in Toolbox Task (OOD Prompt): This task is challenging for the base VLA (π0 [4]) because the base language prompt was never seen during training. (Left) Rephrase repeatedly fails to grasp the cube during replanning. (Right) Adaptive RL2 …
Figure 23
Figure 23. Figure 23: Real-Robot Experiment for Cube in Toolbox Task (OOD Prompt): This task is challenging for the base VLA (π0 [4]) because the base language prompt was never seen during training. (Left) Rephrase repeatedly fails to grasp the cube during replanning. (Right) Adaptive RL2 …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

145 extracted references · 11 linked inside Pith

  1. [1]

    From intention to execution: Probing the generalization boundaries of vision-language-action mod- els,

    I. Fang, J. Zhang, S. Tong, and C. Feng, “From intention to execution: Probing the generalization boundaries of vision-language-action mod- els,”arXiv preprint arXiv2506.09930, 2025

  2. [2]

    Interleave-VLA: Enhancing robot ma- nipulation with image-text interleaved instructions,

    C. Fan, X. Jia, Y . Sun, Y . Wang, J. Wei, Z. Gong, X. Zhao, M. Tomizuka, X. Yang, J. Yan, and M. Ding, “Interleave-VLA: Enhancing robot ma- nipulation with image-text interleaved instructions,” inThe Fourteenth International Conference on Learning Representations, 2026

  3. [3]

    Openvla: An open-source vision-language-action model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketiet al., “Openvla: An open-source vision-language-action model,” inProceedings of The 8th Conference on Robot Learning, 2025

  4. [4]

    π 0: A vision-language-action flow model for general robot control,

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausmanet al., “π 0: A vision-language-action flow model for general robot control,”arXiv preprint arXiv:2410.24164, 2026

  5. [5]

    π 0.5: a vision- language-action model with open-world generalization,

    K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y . Gallikeret al., “π 0.5: a vision- language-action model with open-world generalization,” inProceedings of The 9th Conference on Robot Learning, 2025

  6. [6]

    Molmoact: Action reasoning models that can reason in space,

    J. Lee, J. Duan, H. Fang, Y . Deng, S. Liu, B. Li, B. Fang, J. Zhang, Y . R. Wang, S. Leeet al., “Molmoact: Action reasoning models that can reason in space,”arXiv preprint arXiv:2508.07917, 2025

  7. [7]

    Open x-embodiment: Robotic learning datasets and rt-x models

    A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jainet al., “Open x-embodiment: Robotic learning datasets and rt-x models.” in2024 IEEE International Conference on Robotics and Automation, 2024

  8. [8]

    GRAPE: Generalizing robot policy via preference alignment,

    Z. Zhang, K. Zheng, Z. Chen, J. Jang, Y . Li, S. Han, C. Wang, M. Ding, D. Fox, and H. Yao, “GRAPE: Generalizing robot policy via preference alignment,” inarXiv preprint arXiv:2411.19309, 2025

Show all 145 references
  1. [9]

    Robotic control via embodied chain-of-thought reasoning,

    M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine, “Robotic control via embodied chain-of-thought reasoning,” inProceed- ings of The 8th Conference on Robot Learning, 2024

  2. [10]

    Fast ecot: Efficient embodied chain-of-thought via thoughts reuse,

    Z. Duan, Y . Zhang, S. Geng, G. Liu, J. Boedecker, and C. X. Lu, “Fast ecot: Efficient embodied chain-of-thought via thoughts reuse,”IEEE International Conference on Robotics and Automation, 2025

  3. [11]

    Steering your generalists: Improving robotic foundation models via value guidance,

    M. Nakamoto, O. Mees, A. Kumar, and S. Levine, “Steering your generalists: Improving robotic foundation models via value guidance,” Proceedings of The 8th Conference on Robot Learning, 2024

  4. [12]

    Robomonkey: Scaling test-time sampling and verifi- cation for vision-language-action models,

    J. Kwok, C. Agia, R. Sinha, M. Foutter, S. Li, I. Stoica, A. Mirhoseini, and M. Pavone, “Robomonkey: Scaling test-time sampling and verifi- cation for vision-language-action models,” inProceedings of The 9th Conference on Robot Learning, 2025

  5. [13]

    Scaling verification can be more effective than scaling policy learning for vision-language-action alignment,

    J. Kwok, X. Zhang, M. Xu, Y . Liu, A. Mirhoseini, C. Finn, and M. Pavone, “Scaling verification can be more effective than scaling policy learning for vision-language-action alignment,” inEuropean Conference on Computer Vision, 2026

  6. [14]

    From foresight to fore- thought: Vlm-in-the-loop policy steering via latent alignment,

    Y . Wu, R. Tian, G. Swamy, and A. Bajcsy, “From foresight to fore- thought: Vlm-in-the-loop policy steering via latent alignment,”Robotics: Science and Systems (RSS), 2025

  7. [15]

    Do what you say: Steering vision-language-action models via runtime reasoning-action alignment verification,

    Y . Wu, A. Li, T. Hermans, F. Ramos, A. Bajcsy, and C. P´erez-D’Arpino, “Do what you say: Steering vision-language-action models via runtime reasoning-action alignment verification,”2026 IEEE International Con- ference on Robotics & Automation, 2026

  8. [16]

    Dynaguide: Steering diffusion policies with active dynamic guidance,

    M. Du and S. Song, “Dynaguide: Steering diffusion policies with active dynamic guidance,” inThe Thirty-ninth Conference on Neural Information Processing Systems, 2026

  9. [17]

    VLS: Steering pretrained robot policies via vision-language models,

    S. Liu, I. S. Singh, Y . Xu, J. Duan, and R. Krishna, “VLS: Steering pretrained robot policies via vision-language models,”arXiv preprint arXiv:2602.03973, 2026

  10. [18]

    Towards deploying vla without fine-tuning: Plug-and-play inference- time vla policy steering via embodied evolutionary diffusion,

    Z. Li, J. Liu, Z. Dong, T. Teng, Q. Rouxel, D. Caldwell, and F. Chen, “Towards deploying vla without fine-tuning: Plug-and-play inference- time vla policy steering via embodied evolutionary diffusion,”arXiv preprint arXiv:2511.14178, 2026

  11. [19]

    Compose your policies! improving diffusion-based or flow-based robot policies via test-time distribution- level composition,

    J. Cao, Y . Huang, H. Guo, Q. Zhang, R. Zhang, W. Mai, M. Nan, J. Wang, H. Cheng, J. Sunet al., “Compose your policies! improving diffusion-based or flow-based robot policies via test-time distribution- level composition,” inThe Fourteenth International Conference on Learning ...

  12. [20]

    Safe: Multitask failure detection for vision-language-action models,

    Q. Gu, Y . Ju, S. Sun, I. Gilitschenski, H. Nishimura, M. Itkina, and F. Shkurti, “Safe: Multitask failure detection for vision-language-action models,”Advances in Neural Information Processing Systems, 2026

  13. [21]

    Evaluating real-world robot manipulation policies in simulation,

    X. Li, K. Hsu, J. Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Lunawat, I. Siehet al., “Evaluating real-world robot manipulation policies in simulation,”arXiv preprint arXiv:2405.05941, 2024

  14. [22]

    Polaris: Scalable real-to-sim evaluations for generalist robot policies,

    A. Jain, M. Zhang, K. Arora, W. Chen, M. Torne, M. Z. Irshad, S. Zakharov, Y . Wanget al., “Polaris: Scalable real-to-sim evaluations for generalist robot policies,”arXiv preprint arXiv:2512.16881, 2025

  15. [23]

    π ∗ 0.6: a vla that learns from experience,

    P. Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Con- ley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, D. Driess et al., “π ∗ 0.6: a vla that learns from experience,”arXiv preprint arXiv:2511.14759, 2026

  16. [24]

    SimpleVLA-RL: Scaling VLA training via reinforcement learning,

    H. Li, Y . Zuo, J. Yu, Y . Zhang, Y . Zhaohui, K. Zhang, X. Zhu, Y . Zhang, T. Chen, G. Cuiet al., “SimpleVLA-RL: Scaling VLA training via reinforcement learning,” inThe Fourteenth International Conference on Learning Representations, 2026

  17. [25]

    Conrft: A reinforced fine-tuning method for vla models via consistency policy,

    Y . Chen, S. Tian, S. Liu, Y . Zhou, H. Li, and D. Zhao, “Conrft: A reinforced fine-tuning method for vla models via consistency policy,” in Proceedings of Robotics: Science and Systems, RSS, 2025

  18. [26]

    Policy decorator: Model-agnostic online refinement for large policy model,

    X. Yuan, T. Mu, S. Tao, Y . Fang, M. Zhang, and H. Su, “Policy decorator: Model-agnostic online refinement for large policy model,” in Thirteenth International Conference on Learning Representations, 2025

  19. [27]

    Self-improving vision-language-action models with data generation via residual RL,

    W. Xiao, H. Lin, A. Peng, H. Xue, T. He, Z. Luo, Y . Xie, F. Hu, L. Fan, G. Shi, and Y . Zhu, “Self-improving vision-language-action models with data generation via residual RL,” inThe Fourteenth International Conference on Learning Representations, 2026

  20. [28]

    Steering your diffusion policy with latent space reinforcement learning,

    A. Wagenmaker, M. Nakamoto, Y . Zhang, S. Park, W. Yagoub, A. Naga- bandi, A. Gupta, and S. Levine, “Steering your diffusion policy with latent space reinforcement learning,”Proceedings of The 9th Conference on Robot Learning, 2025

  21. [29]

    Rl token: Bootstrapping online rl with vision-language-action models,

    C. Xu, J. T. Springenberg, M. Equi, A. Amin, A. Esmail, S. Levine, and L. Ke, “Rl token: Bootstrapping online rl with vision-language-action models,”arXiv preprint arXiv:2604.23073, 2026

  22. [30]

    OnetwoVLA: A unified vision-language-action model with adaptive reasoning,

    F. Lin, R. Nai, Y . Hu, J. You, J. Zhao, and Y . Gao, “OnetwoVLA: A unified vision-language-action model with adaptive reasoning,” inThe Fourteenth International Conference on Learning Representations, 2026

  23. [31]

    Recurrent-depth vla: Implicit test-time compute scaling of vision-language-action models via latent iterative reasoning,

    Y . Tur, J. Naghiyev, H. Fang, W.-C. Tsai, J. Duan, D. Fox, and R. Krishna, “Recurrent-depth vla: Implicit test-time compute scaling of vision-language-action models via latent iterative reasoning,”arXiv preprint arXiv:2602.07845, 2026

  24. [32]

    Vla- attc: Adaptive test-time compute for vla models with relative action critic model,

    W. Li, X. Su, Y . Cao, H. Xu, X. Xia, S. You, Y . Chen, and C. Xu, “Vla- attc: Adaptive test-time compute for vla models with relative action critic model,”arXiv preprint arXiv2605.01194, 2026

  25. [33]

    Scale: Self-uncertainty conditioned adaptive looking and execution for vision- language-action models,

    H. Choi, D. Ahn, Y . Lee, T. Kang, S. Cho, and J. Choi, “Scale: Self-uncertainty conditioned adaptive looking and execution for vision- language-action models,” inThe Fourteenth International Conference on Learning Representations, 2026

  26. [34]

    Diffusion models beat GANs on image synthesis,

    P. Dhariwal and A. Q. Nichol, “Diffusion models beat GANs on image synthesis,” inThe Thirty-fifth Conference on Neural Information Processing Systems, 2021

  27. [35]

    Tree-guided diffusion planner,

    H. Jeon, C. Min, and J. Park, “Tree-guided diffusion planner,”Advances in Neural Information Processing Systems, 2026

  28. [36]

    Bridgedata v2: A dataset for robot learning at scale,

    H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, P. Hansen-Estruch, Q. Vuong, Heet al., “Bridgedata v2: A dataset for robot learning at scale,” inThe 7th Conference on Robot Learning, 2023

  29. [37]

    Q-learning with adjoint matching,

    Q. Li and S. Levine, “Q-learning with adjoint matching,” inThe Fourteenth International Conference on Learning Representations, 2026

  30. [38]

    Droid: A large-scale in-the-wild robot manipulation dataset,

    A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karam- cheti, S. Nasiriany, M. K. Srirama, and L. Y . C. et al., “Droid: A large-scale in-the-wild robot manipulation dataset,”Robotics: Science and Systems (RSS), 2024

  31. [39]

    Flow q-learning,

    S. Park, Q. Li, and S. Levine, “Flow q-learning,” inForty-second International Conference on Machine Learning, 2025

  32. [40]

    Conservative q-learning for offline reinforcement learning,

    A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative q-learning for offline reinforcement learning,” inAdvances in Neural Information Processing Systems, 2020

  33. [41]

    Can we detect failures without failure data? uncertainty-aware runtime failure detection for imitation learning policies,

    C. Xu, T. K. Nguyen, E. Dixon, C. Rodriguez, P. Miller, R. Lee, P. Shah, R. Ambrus, H. Nishimura, and M. Itkina, “Can we detect failures without failure data? uncertainty-aware runtime failure detection for imitation learning policies,”arXiv preprint arXiv2503.08558, 2024

  34. [42]

    Embodied red teaming for auditing robotic foundation models,

    S. Karnik, Z.-W. Hong, N. Abhangi, Y .-C. Lin, T.-H. Wang, and P. Agrawal, “Embodied red teaming for auditing robotic foundation models,”arXiv preprint arXiv2411.18676, 2024

  35. [43]

    OGBench: Bench- marking offline goal-conditioned RL,

    S. Park, K. Frans, B. Eysenbach, and S. Levine, “OGBench: Bench- marking offline goal-conditioned RL,” inThe Thirteenth International Conference on Learning Representations, 2025. 13 APPENDIX A. Additional Implementation Details In this section, we provide additional training ...

  36. [51]

    Continued on next page 19 TABLE IX LANGUAGEINSTRUCTIONSREPHRASES(CONTINUED FROM PREVIOUS PAGE)

    Place the vegetable on the ceramic plate. Continued on next page 19 TABLE IX LANGUAGEINSTRUCTIONSREPHRASES(CONTINUED FROM PREVIOUS PAGE). Task Name Original Instruction Red-Team Instruction Generated Rephrases Spoon on Towel (SIMPLER) Put the spoon on the towel. Set the spoon ...

  37. [52]

    Place the spoon in the center of the towel

  38. [58]

    Lay the green-handled spoon in the middle of the fabric

  39. [59]

    Stack Cubes (SIMPLER) Stack the green block on the yellow block

    Set the spoon in the center of the towel. Stack Cubes (SIMPLER) Stack the green block on the yellow block. Arrange the lush green element atop the yellowish-orange element

  40. [60]

    Place the green cube on the yellow block

  41. [61]

    Put the grassy block on top of the yellow piece

  42. [62]

    Set the green block over the yellow cube

  43. [63]

    Stack the green piece onto the yellowish block

  44. [64]

    Position the lush cube above the yellow element

  45. [65]

    Place the green cube on the yellow cube

  46. [66]

    Set the green block on top of the yellow block

  47. [67]

    Eggplant in Basket (SIMPLER) Put eggplant into yellow basket

    Put the green element over the yellow cube. Eggplant in Basket (SIMPLER) Put eggplant into yellow basket. Arrange the eggplant neatly in the yellow bin

  48. [68]

    Place the vegetable in the yellow rack

  49. [69]

    Put the eggplant in the yellow holder

  50. [70]

    Set the purple vegetable into the yellow bin

  51. [71]

    Organize the eggplant in the yellow rack

  52. [72]

    Position the vegetable in the yellow holder

  53. [73]

    Place the purple vegetable in the yellow container

  54. [74]

    Set the eggplant into the dish rack

  55. [75]

    Orange Juice on Plate (SIMPLER) Put orange juice on plate

    Move the purple produce to the yellow bin. Orange Juice on Plate (SIMPLER) Put orange juice on plate. – 1. Set the carton of juice onto the dish

  56. [76]

    Move the orange drink to the saucer

  57. [77]

    Lay the juice carton on the platter

  58. [78]

    Position the orange juice on the dish

  59. [79]

    Put the drink carton on the yellow plate

  60. [80]

    Place the orange drink container onto the platter

  61. [81]

    Set the juice on the yellow saucer

  62. [82]

    Spoon on Towel (Google) (SIMPLER) Put the spoon on the towel google

    Move the orange box to the plate. Spoon on Towel (Google) (SIMPLER) Put the spoon on the towel google. – 1. Place the spoon in the center of the towel

  63. [83]

    Put the utensil on the middle of the cloth

  64. [84]

    Position the scoop at the center of the fabric

  65. [85]

    Lay the green-handled spoon in the middle of the blue towel

  66. [86]

    Set the utensil in the central part of the towel

  67. [87]

    Position the spoon on the center of the blue towel

  68. [88]

    Set the spoon exactly in the middle of the towel

  69. [89]

    Toy Dinosaur on Towel (SIMPLER) Put the toy dinosaur on the towel

    Set the spoon in the center of the towel. Toy Dinosaur on Towel (SIMPLER) Put the toy dinosaur on the towel. – 1. Place the toy on the cloth

  70. [90]

    Move the figure to the blue fabric

  71. [91]

    Transfer the creature to the piece of cloth

  72. [92]

    Lay the toy on the rag

  73. [93]

    Position the dinosaur on the towel

  74. [94]

    Put the object on the blue fabric

  75. [95]

    Set the figure on the table cloth

  76. [96]

    Tape Measure in Basket (SIMPLER) Put tape measure into yellow basket

    Move the dinosaur over to the rag. Tape Measure in Basket (SIMPLER) Put tape measure into yellow basket. – 1. Move the measuring tape to the yellow basket

  77. [97]

    Set the tape measure inside the yellow basket

  78. [98]

    Transfer the measuring tape to the yellow container

  79. [99]

    Put the tape measure in the yellow dish rack

  80. [100]

    Place the measuring tape into the yellow holder

  81. [101]

    Place the tape measure in the yellow rack

  82. [102]

    Set the measuring tape in the yellow rack

  83. [103]

    Continued on next page 20 TABLE IX LANGUAGEINSTRUCTIONSREPHRASES(CONTINUED FROM PREVIOUS PAGE)

    Position the tape measure in the yellow basket. Continued on next page 20 TABLE IX LANGUAGEINSTRUCTIONSREPHRASES(CONTINUED FROM PREVIOUS PAGE). Task Name Original Instruction Red-Team Instruction Generated Rephrases Pan Cleaning (PolaRiS) Use the yellow sponge to scrub the blu...

  84. [104]

    Scrub the blue-handled frying pan with the sponge

  85. [105]

    Use the cleaning sponge to clean the pan with the blue handle

  86. [106]

    Rub the sponge against the frying pan to scrub it

  87. [107]

    Clean the skillet with the blue handle using the sponge

  88. [108]

    Touch the cleaning sponge to the blue-handled pan

  89. [109]

    Wipe the frying pan with the sponge

  90. [110]

    Apply the sponge to the pan with the blue handle for scrubbing

  91. [111]

    Tape into Container (PolaRiS) Put the tape into the container

    Wash the blue-handled pan using the cleaning sponge. Tape into Container (PolaRiS) Put the tape into the container. Carefully drop the adhesive roll inside the bin

  92. [112]

    Place the tape roll in the box

  93. [113]

    Drop the masking tape into the tray

  94. [114]

    Move the tape to the brown container

  95. [115]

    Set the roll of tape inside the bin

  96. [116]

    Lift the tape and place it in the tray

  97. [117]

    Insert the adhesive tape into the box

  98. [118]

    Deposit the tape roll inside the receptacle

  99. [119]

    Move Latte Cup (PolaRiS) Put the latte art cup on top of the cutting board

    Grab the tape and put it into the container. Move Latte Cup (PolaRiS) Put the latte art cup on top of the cutting board. Relocate the pitcher with the decoration onto the wooden board

  100. [120]

    Place the metal pitcher on the cutting board

  101. [121]

    Move the latte cup onto the wooden board

  102. [122]

    Set the silver cup on the cutting board

  103. [123]

    Lift the milk jug and put it on the board

  104. [124]

    Position the frothing pitcher on the wooden board

  105. [125]

    Take the metal cup and place it on the cutting board

  106. [126]

    Move the latte pitcher to the board

  107. [127]

    Carrot on Plate (Real-World) Put carrot on plate

    Put the latte cup on top of the cutting board. Carrot on Plate (Real-World) Put carrot on plate. Balance the carrot on the ceramic platter

  108. [128]

    Place the carrot on the green dish

  109. [129]

    Set the orange vegetable on the small plate

  110. [130]

    Position the carrot upright on the platter

  111. [131]

    Stand the carrot on the green plate

  112. [132]

    Set the vegetable on the round dish

  113. [133]

    Position the carrot on the small platter

  114. [134]

    Balance the orange carrot on the green dish

  115. [135]

    Cube in Toolbox (Real-World) Place the cube inside the toolbox

    Place the vegetable on the ceramic plate. Cube in Toolbox (Real-World) Place the cube inside the toolbox. Move the cube towards the container and put it inside

  116. [136]

    Put the cube into the toolbox

  117. [137]

    Insert the cube into the case

  118. [138]

    Move the cube into the container

  119. [139]

    Set the cube inside the red box

  120. [140]

    Place the cube inside the red box

  121. [141]

    Place the cube into the box

  122. [142]

    Insert the cube in the open toolbox

  123. [143]

    Tape in Toolbox (Real-World) Place the green tape inside the toolbox

    Put the cube inside the case. Tape in Toolbox (Real-World) Place the green tape inside the toolbox. – 1. Put the green tape in the toolbox

  124. [144]

    Set the tape inside the red box

  125. [145]

    Move the green roll into the storage box

  126. [146]

    Insert the green adhesive into the toolbox

  127. [147]

    Place the colored tape in the red container

  128. [148]

    Drop the green roll inside the toolbox

  129. [149]

    Put the adhesive inside the storage box

  130. [150]

    Screwdriver in Toolbox (Real-World) Place the screwdriver tool inside the toolbox

    Set the tape in the container. Screwdriver in Toolbox (Real-World) Place the screwdriver tool inside the toolbox. – 1. Put the screwdriver into the toolbox

  131. [151]

    Insert the tool into the case

  132. [152]

    Move the screwdriver into the container

  133. [153]

    Set the tool inside the red box

  134. [154]

    Place the tool inside the red box

  135. [155]

    Place the green and black tool into the box

  136. [156]

    Insert the screwdriver in the open toolbox

  137. [157]

    Put the green-handled tool inside the case

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.