Pith. sign in

REVIEW 4 major objections 6 minor 46 references

SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibration

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SAFECAST claims that augmenting hidden-state risk probe training and conformal calibration with visual and language contrast-set rollouts makes failure detection for vision-language-action policies more reliable under deployment-time…

desk verdict Core idea is plausible and the simulation evidence is strong, but the real-world evaluation may be confounded and the abstract oversells significance. read the letter →

arxiv 2608.04246 v1 pith:QGGVUFRJ submitted 2026-08-04 cs.RO cs.CV

Harshitha Rajaprakash , Aditeya Prajapati , Rong Xue , Abrar Anwar , Jesse Thomason This is my paper · ORCID
classification cs.ROcs.CV
keywords failuredetectionvision-language-actionpoliciescontrastsetsconformalpredictiondistributionshiftrobotmanipulationhidden-stateprobessim-to-realtransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language-action robot policies can fail when deployment conditions differ from training, and hidden-state risk probes only stay reliable if their calibration data resembles deployment. This paper argues that deliberately collecting rollouts under contrast-set perturbations—visual changes, paraphrased instructions, and their combination—and adding them to both probe training and conformal calibration makes failure detection more reliable under distribution shift. In real-world DROID experiments and LIBERO simulation, the resulting SAFECAST detector improves rollout-level F1 and ROC-AUC over the vanilla SAFE baseline across π0 and OpenVLA backbones. The paper also reports that combining visual and language perturbations helps more than either alone, and that a probe trained entirely in simulation and calibrated on a small set of real contrast-set rollouts can beat a probe trained on real rollouts only.

What carries the argument

The mechanism is a hidden-state risk probe paired with functional conformal prediction: at each timestep a lightweight MLP maps the pre-final-layer hidden state of the frozen policy to a scalar failure score, and functional conformal prediction converts the score sequence into a time-varying threshold $\delta_t = \mu_t + q_\alpha$ calibrated on successful rollouts. SAFECAST intervenes on the data feeding both components, replacing the source-only pools with $\mathcal{D}_{\text{aug}} = \mathcal{D}_{\text{src}} \cup \mathcal{D}_{\text{CS}}$, where $\mathcal{D}_{\text{CS}}$ contains re-executed contrast-set rollouts under visual, language, and joint perturbations, filtered by a DTW-based active rejection procedure to reduce near-duplicates. This changes both the learned failure-score function and the threshold location, aligning the risk trajectories seen in calibration with those seen under deployment shift.

What would settle it

Collect a deployment evaluation set whose shifts are deliberately outside the contrast-set family used for training and calibration—for example, contrast sets built from distractor objects and paraphrases, but deployment featuring camera repositioning and novel spatial layouts—and compare SAFECAST's ROC-AUC or F1 against SAFE; if the gain disappears or reverses, the representativeness assumption is violated.

Watch

Extended reading notes

Core claim

The central claim is that calibration mismatch, not probe capacity, is the main bottleneck for hidden-state failure detection under deployment shift, and that mismatch can be reduced by exposing the probe and the conformal thresholds to perturbed rollouts that approximate the shifts expected at deployment. SAFECAST re-executes the frozen VLA policy under contrast-set perturbations to obtain new hidden-state risk trajectories, pools them with source rollouts, and uses the augmented pool for both probe training and functional conformal calibration. Across π0 and OpenVLA, on real DROID/Franka rollouts and LIBERO simulation, these augmented configurations improve failure detection F1 and ROC-AUC relative to the SAFE baseline, with joint visual-language contrast sets giving the strongest robustness. In the sim-to-real variant, a probe trained in LIBERO and calibrated on real contrast-set rollouts outperforms a probe trained and calibrated only on the smaller real-world dataset.

Load-bearing premise

The load-bearing premise is that the contrast-set perturbations made from source rollouts resemble the shifts the robot will actually meet at deployment; if deployment shifts fall outside that perturbation family, the calibration advantage shrinks and detection performance can degrade.

Editorial extensions

If this is right

  • Failure detectors for VLA policies should be trained and calibrated on perturbed rollouts rather than only source rollouts whenever deployment-time shift is expected.
  • Joint visual-language contrast sets yield stronger robustness than visual-only or language-only augmentation, so shift coverage should span multiple modalities.
  • Simulation-trained probes with real contrast-set calibration can substitute for larger real-world data collection when real rollouts are scarce.
  • The improvement persists across a sweep of conformal significance levels $\alpha$, so the gain is not confined to a single risk tolerance or operating point.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: using the detector's own uncertainty to propose new perturbations online could let contrast sets adapt to failures as they appear; this is testable by comparing fixed contrast pools with adaptive ones in the same evaluation suite.
  • Beyond the paper: the anti-goal stress tests suggest a cheap overfitting check for VLA policies—checking whether a negated instruction changes behavior—which the paper flags as future benchmark work.
  • Beyond the paper: the sim-to-real result implies the bottleneck is calibration alignment rather than probe capacity; one could test this by measuring how the gap to a real-only probe closes as the number of real contrast-set rollouts grows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. SAFECAST proposes to improve hidden-state failure detection for vision-language-action (VLA) policies under deployment-time distribution shift by augmenting both probe training and functional-conformal calibration with contrast-set rollouts (visual perturbations, language paraphrases, and their combination). The authors compare SAFE against three SAFECAST variants (train-augmentation only, calibration-augmentation only, and both) on LIBERO simulation with two base policies (π0 and OpenVLA) and on a real Franka/DROID setup with two checkpoints (π0 and π0-FAST), reporting F1 and ROC-AUC averaged over conformal significance levels α and 30 seeds. They report statistically significant improvements in most settings, a benefit of joint visual-language contrast sets, and a sim-to-real transfer result in which a simulation-trained probe calibrated on real contrast-set rollouts outperforms a real-only probe.

Significance. If the claims hold, SAFECAST is a practical, low-overhead intervention that strengthens runtime failure detection under distribution shift, which matters for safe deployment of VLA policies. The experimental methodology is a clear strength: 30 seeds, seed-paired t-tests, full alpha sweeps in the appendices, and controlled source/calibration/evaluation splits in simulation. The appendices also provide unusually thorough perturbation-family ablations and precision–recall diagnostics. However, the paper's headline significance claim is broader than the statistical evidence supports, and the real-world evaluation may be confounded by overlap between the contrast-set pool and the evaluation distribution. The significance of the work is therefore conditional on addressing these points.

major comments (4)
  1. [Abstract; §5.2; Table 6] The abstract states that SAFECAST 'statistically significantly improves' failure detection in both real-world DROID and LIBERO simulation experiments, but Table 6 shows that this is not true for all headline comparisons after the authors' own Bonferroni correction (m = 12, threshold 0.00417). In particular, Sim-OpenVLA SAFECAST vs. SAFE has p = 0.0256, Sim-OpenVLA SAFECAST_TRAINAUG has p = 0.4093, and Real-π0 SAFECAST_TRAINAUG has p = 0.8753. Please qualify the significance claim to the specific settings where it holds, or present a corrected multiple-testing analysis that justifies the abstract's wording.
  2. [§5.1; Appendix A.2; Table 3] The real-world DROID evaluation may be confounded because the evaluation distribution D_eval overlaps with the contrast-set pool D_CS. Section 5.1 says evaluation rollouts contain 'stronger shifts including novel objects, new tasks, clutter, lighting variation, and altered robot initial states,' and Table 3 lists real-robot contrast-set perturbations including 'Visual distractors/clutter,' 'Language paraphrases/negations,' and 'Novel objects/tasks.' The paper never states that different task instances, object sets, or paraphrase templates were used in D_eval versus D_CS, nor that the perturbation operators were applied disjointly. If D_eval and D_CS are drawn from the same perturbation families, the reported ROC-AUC gains over SAFE may reflect in-distribution calibration on eval-like rollouts rather than robustness to truly held-out deployment shift. Please either document explicit disjointness of D_eval from D_CS (e.g., separate objects, tasks, paraphrase templates, or perturbation operators), or re-analyze the real-world results on perturbation families that were never used in contrast-set construction.
  3. [§5.3; Table 11] The claim that joint visual-language contrast sets produce 'the strongest OOD failure detection robustness' is not supported by the ROC-AUC metric for π0. Table 11 reports α-marginal cumulative ROC-AUC for π0 as 0.469 (visual only), 0.441 (language only), and 0.282 (visual + language), so the joint setting is substantially worse than either single modality on this metric. The main text reports only the F1-based Figure 4 and does not mention this discrepancy. Please report both metrics in the main text and either reconcile the difference (e.g., via the class-imbalance discussion in Appendix F.11) or soften the claim to be metric- and policy-specific.
  4. [§5.4; Table 12] The sim-to-real transfer result is a central finding but is reported without significance testing or variance estimates. Table 12 gives α-marginal F1 and ROC-AUC for 'Real-only probe and calibration' (0.478/0.228) versus 'Sim-trained probe + real CS calibration' (0.523/0.410), and Figure 5 shows curves without confidence bands. Given the small real-world rollout counts in Table 2 (e.g., π0 train: 17 successes/27 failures; calibration: 14/28), please provide seed-paired tests or confidence intervals for this comparison, and state whether the improvement is statistically significant.
minor comments (6)
  1. [§3 heading] The heading 'Task Definiton' contains a typo; it should be 'Task Definition.'
  2. [Appendix F.11] The sentence 'Anti-goal Perturbations Do Not Improve Failure Detection but they are really, really funny' is informal and out of place in a scientific paper; please remove or rewrite it in a neutral register.
  3. [Figure 20 caption] The caption reads 'Pecision–recall curves'; 'Pecision' should be 'Precision.'
  4. [§5.1; Tables 8 and 10] The term 'Cumulative ROC-AUC' is used in the appendix tables and figures but is not defined in the main text. Please clarify how cumulative ROC-AUC is computed from the TPR–FPR curves and how it differs from the standard ROC-AUC defined in §5.1.
  5. [Appendix B.3] The DTW rejection threshold τ_DTW = 0.02 is fixed with no sensitivity analysis. Since this threshold controls the diversity of the contrast-set training pool, please provide an ablation over τ_DTW or justify the chosen value.
  6. [Appendix A.2] The language contrast-set construction uses three paraphrases per task as a budget choice, but no ablation is reported on the number of paraphrases. Please add a sentence justifying this budget or a small ablation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SAFECAST's claims are empirical comparisons against external baselines, and no fitted quantity or self-citation chain forces the reported gains.

full rationale

The paper's central claim is an empirical result: contrast-set-augmented probe training and functional-conformal calibration improve failure-detection ROC-AUC relative to the SAFE baseline on DROID and LIBERO-Plus evaluations. The derivation chain is data-driven rather than algebraic or definitional. The probe is a lightweight MLP f_phi trained on hidden states with success/failure labels, and the conformal threshold delta_t = mu_t + q_alpha is computed from calibration nonconformity scores; no equation in Sections 3-4 equates the reported ROC-AUC with the training or calibration inputs by construction. The only potentially load-bearing self-citation is [11] (Anwar, Gupta, Thomason) for the contrast-set idea, but the paper also draws on the external contrast-set literature [47], builds on the external SAFE baseline [3], and evaluates on the external LIBERO-Plus benchmark [20]; the citation is not used to forbid alternatives or to supply a uniqueness theorem. The Limitations section explicitly concedes that the method's success depends on D_CS being representative of deployment shifts and that conformal exchangeability guarantees are not restored, which is an honest statement of an assumption rather than a circular reduction. The closest concern to circularity is that the real-world DROID evaluation perturbations (novel objects, clutter, paraphrases) overlap in kind with the D_CS perturbation families, which could make the real-world evaluation partially in-distribution; however, the paper does not define D_eval as D_CS, the LIBERO evaluation uses the separate pre-existing LIBERO-Plus suite, and this overlap is a validity threat rather than an equation-level reduction. Therefore no circular step is established.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The claim rests on domain assumptions about hidden-state informativeness and the representativeness of contrast sets, plus the standard conformal exchangeability assumption that the paper explicitly relaxes. The only hand-chosen numeric hyperparameter that materially shapes the training data is the DTW threshold; the paraphrase budget is a cost-saving design choice. No new entities are postulated.

free parameters (2)
  • DTW active-rejection threshold tau_DTW = 0.02 normalized DTW distance
    Hand-chosen threshold in Algorithm 1 that filters near-duplicate contrast-set rollouts from probe training; no sensitivity analysis is reported, yet it shapes the training pool.
  • Language contrast-set budget = 3 paraphrases per task, 3 episodes per paraphrase
    Chosen to reduce rollout collection cost (Appendix A.2); the number of paraphrase variants and episodes per variant is not swept, so it is a fixed design choice that could affect the diversity of language contrast sets.
assumptions (4)
  • domain assumption Hidden states of the frozen VLA policy carry information predictive of eventual rollout failure.
    SAFECAST inherits this from SAFE [3]; if false, contrast-set training and calibration have no signal to improve. Invoked in Section 3.
  • ad hoc to paper Contrast-set perturbations constructed from the source distribution are representative of deployment shifts in D_eval.
    Section 4 states contrast sets are used to build distributions whose risk trajectories better approximate deployment shifts; Limitations admit degradation if the deployment shift is outside the contrast-set family.
  • standard math Conformal calibration assumes exchangeability between calibration and deployment trajectories.
    Section 3 describes the functional conformal prediction step; Section 4 and Limitations state SAFECAST does not restore formal exchangeability guarantees, so the method is evaluated empirically rather than by conformal coverage.
  • domain assumption LLM-generated paraphrases preserve the target object, action, and goal, so success/failure labels remain valid.
    Appendix A.2 prompt instructs preservation; if paraphrases changed semantics, rollout labels could be misaligned with instructions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibration." pith.science (2026). https://pith.science/paper/QGGVUFRJ

@misc{pith2026260804246,
  author       = {Pith},
  title        = {Pith review of: SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QGGVUFRJ}},
  note         = {Machine review of arXiv:2608.04246}
}
read the original abstract

Vision-language-action policies often fail under deployment-time distribution shifts such as clutter, distractor objects, lighting changes, novel objects, altered initial states, and reworded instructions. Hidden-state-based risk probes combined with functional conformal prediction can detect rollout failures, but their reliability depends on calibration data matching deployment conditions. We introduce SAFECAST, which leverages contrast set perturbations to improve hidden-state probe training and calibration for deployment time shift. SAFECAST statistically significantly improves failure detection ROC-AUC scores over a state of the art baseline in both real-world DROID and LIBERO simulation experiments across multiple VLM backbones. We further find that SAFECAST benefits most when both visual and language contrast set perturbations are used to augment data, and that with contrast set perturbations, sim-to-real calibration leads to better probes than using real rollout data only.

Figures

Figures reproduced from arXiv: 2608.04246 by the authors.

Figure 1
Figure 1. SAFECAST. Left: VLA policies trained on source-distribution rollouts can encounter visual, language, and multimodal deployment shifts during execution. Middle: Hidden-state failure detectors calibrated only on source-distribution trajectories can become misaligned under deploy￾ment mismatch, leading to delayed or missed failure warnings. Right: SAFECAST intervenes at probe training and conformal calibration time wit… view at source ↗
Figure 2
Figure 2. Real-world failure detection across varying [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Simulation failure detection across varying [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Effect of contrast-set modality on simulation failure detection. Left: F1 across conformal [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Sim-to-real failure detection robustness [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Simulation failure detection in the low- [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Simulation failure detection in the high- [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Real-world failure detection in the low- [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Real-world failure detection in the high- [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Examples of goal-preserving visual contrast variations used in the extended LIBERO [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Combined contrast-family ablations for SAFECAST on LIBERO-Plus. The top row [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Lighting variation ablations for SAFECAST on LIBERO-Plus. The curves compare [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Texture variation ablations for SAFECAST on LIBERO-Plus. The curves compare [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: Goal-preserving language variation ablations for SAFECAST on LIBERO-Plus. The [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: Anti-goal ablation for SAFECAST on LIBERO-Plus. Anti-goal is evaluated separately [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 16
Figure 16. Figure 16: Precision–recall curves for combined SAFECAST ablations on LIBERO-Plus. Each [PITH_FULL_IMAGE:figures/full_fig_p030_16.png]
Figure 17
Figure 17. Figure 17: precision–recall curves for lighting variation ablations on LIBERO-Plus. Curves closer to [PITH_FULL_IMAGE:figures/full_fig_p030_17.png]
Figure 18
Figure 18. Figure 18: Precision–recall curves for texture variation ablations on LIBERO-Plus. The curves com [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]
Figure 19
Figure 19. Figure 19: Precision–recall curves for goal-preserving language variation ablations on LIBERO [PITH_FULL_IMAGE:figures/full_fig_p031_19.png]
Figure 20
Figure 20. Figure 20: Pecision–recall curves for anti-goal contrasts on LIBERO-Plus. Anti-goal reverses [PITH_FULL_IMAGE:figures/full_fig_p032_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 29 canonical work pages

  1. [1]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. OpenVLA: An open-source vision-language-action model. InConference on Robot Learning (CoRL), 2024

    2024

  2. [3]

    Q. Gu, Y . Ju, S. Sun, I. Gilitschenski, H. Nishimura, M. Itkina, and F. Shkurti. Safe: Multitask failure detection for vision-language-action models. InNeural Information Processing Systems (NeurIPS), 2025

    work page 2025

  3. [4]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. Leal, K.-H. Lee, S. Levine, Y . Lu, U. Malla, D. Manju- nath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsc...

    2023

  4. [5]

    Brohan, Y

    A. Brohan, Y . Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances.Confer- ence on Robot Learning (CoRL), 2023

    work page 2023

  5. [6]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Haus- man, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky. pi0: A vision-language-action flow model for general robot control.CoRR, abs/2410.24164, 2024

    arXiv

  6. [7]

    Black, N

    Physical Intelligence, K. Black, N. Brown, D. Driess, C. Finn, S. Levine, et al. pi0.5: A vision-language-action model with open-world generalization.arXiv preprint arXiv:2504.16054, 2025

    arXiv 2025

  7. [8]

    O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy.arXiv preprint arXiv:2405.12213, 2024

    arXiv 2024

  8. [9]

    Pumacay, I

    W. Pumacay, I. Singh, J. Duan, R. Krishna, J. Thomason, and D. Fox. THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation.Robotics: Science and Systems (RSS), 2024

    work page 2024

Show all 46 references
  1. [10]

    X. Li, K. Hsu, J. Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kir- mani, S. Levine, J. Wu, C. Finn, H. Su, Q. Vuong, and T. Xiao. Evaluating real-world robot manipulation policies in simulation.arXiv preprint arXiv:2405.05941, 2024

    2024 arXiv

  2. [11]

    Anwar, R

    A. Anwar, R. Gupta, and J. Thomason. Contrast sets for evaluating language-guided robot policies.Conference on Robot Learning (CoRL), 2024. 9

    2024

  3. [12]

    Anwar, R

    A. Anwar, R. Gupta, Z. Merchant, S. Ghosh, W. Neiswanger, and J. Thomason. Efficient evaluation of multi-task robot policies with active experiment selection. InConference on Robot Learning (CoRL), 2025

    2025

  4. [13]

    Kress-Gazit, K

    H. Kress-Gazit, K. Hashimoto, N. Kuppuswamy, P. Shah, P. Horgan, G. Richardson, S. Feng, and B. Burchfiel. Robot learning as an empirical science: Best practices for policy evaluation. arXiv, 2024

    2024

  5. [14]

    Parekh, N

    A. Parekh, N. Vitsakis, A. Suglia, and I. Konstas. Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks.arXiv preprint arXiv:2407.03967, 2024

    2024 arXiv

  6. [15]

    A. Xie, L. Lee, T. Xiao, and C. Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation.International Conference on Robotics and Automation (ICRA), 2024

    2024

  7. [16]

    J. Gao, S. Belkhale, S. Dasari, A. Balakrishna, D. Shah, and D. Sadigh. A taxonomy for evaluating generalist robot policies.RA-L, 2025

    2025

  8. [17]

    Mitra, Y

    C. Mitra, Y . Luo, R. Saravanan, D. Niu, A. Pai, J. Thomason, T. Darrell, A. Anwar, D. Ra- manan, and R. Herzig. Mechanistic finetuning of vision-language-action models via few-shot demonstrations.arXiv preprint, 2025

    2025

  9. [18]

    Nasiriany, A

    S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y . Zhu. RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots.arXiv preprint arXiv:2406.02523, 2024

    2024 arXiv

  10. [19]

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone. Libero: Benchmarking knowl- edge transfer for lifelong robot learning.NeurIPS, 2023

    2023

  11. [20]

    S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, J. Fu, J. Gong, and X. Qiu. Libero-plus: In-depth robustness analysis of vision-language-action models.arXiv preprint arXiv:2510.13626, 2025

    2025 arXiv

  12. [21]

    T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. InConference on Robot Learning (CoRL), 2019

    2019

  13. [22]

    Sinha, A

    R. Sinha, A. Sharma, S. Banerjee, T. Lew, R. Luo, S. M. Richards, Y . Sun, E. Schmerling, and M. Pavone. A system-level view on out-of-distribution data in robotics.arXiv preprint arXiv:2212.14020, 2022

    2022 arXiv

  14. [23]

    Natarajan, S

    R. Natarajan, S. Reddy, S. C. Bose, H. L. Gururaj, F. Flammini, and S. Velmurugan. Fault detection and state estimation in robotic automatic control using machine learning.Array, 19: 100298, 2023

    2023

  15. [24]

    Q. M. Rahman, P. Corke, and F. Dayoub. Run-time monitoring of machine learning for robotic perception: A survey of emerging trends.IEEE Access, 9:20067–20075, 2021

    2021

  16. [25]

    C. Xu, T. K. Nguyen, E. Dixon, C. Rodriguez, P. Miller, R. Lee, P. Shah, R. Ambrus, H. Nishimura, and M. Itkina. Can we detect failures without failure data? uncertainty-aware runtime failure detection for imitation learning policies.arXiv preprint arXiv:2503.08558, 2025

    2025 arXiv

  17. [26]

    H. Liu, Y . Zhang, V . Betala, E. Zhang, J. Liu, C. Ding, and Y . Zhu. Multi-task interactive robot fleet learning with visual world models. InConference on Robot Learning, 2024

    2024

  18. [27]

    Sinha, A

    R. Sinha, A. Elhafsi, C. Agia, M. Fouftter, E. Schmerling, and M. Pavone. Real-time anomaly detection and reactive planning with large language models. InRobotics: Science and Systems (RSS), 2024. 10

    2024

  19. [28]

    J. Wong, A. Tung, A. Kurenkov, A. Mandlekar, L. Fei-Fei, S. Savarese, and R. Mart´ın-Mart´ın. Error-aware imitation learning from teleoperation data for mobile manipulation. InConference on Robot Learning, 2022

    2022

  20. [29]

    Majumdar, M

    A. Majumdar, M. Sharma, D. Kalashnikov, S. Singh, P. Sermanet, and V . Sindhwani. Predictive red teaming: Breaking policies without breaking robots.arXiv preprint arXiv:2502.06575, 2025

    2025 arXiv

  21. [30]

    C. Agia, R. Sinha, J. Yang, Z. Cao, R. Antonova, M. Pavone, and J. Bohg. Unpacking failure modes of generative policies: Runtime monitoring of consistency and progress. InConference on Robot Learning, pages 689–723. PMLR, 2025

    2025

  22. [31]

    H. Liu, S. Dass, R. Mart ´ın-Mart´ın, and Y . Zhu. Model-based runtime monitoring with interac- tive imitation learning. InIEEE International Conference on Robotics and Automation, pages 4154–4161. IEEE, 2024

    2024

  23. [32]

    Gokmen, D

    C. Gokmen, D. Ho, and M. Khansari. Asking for help: Failure prediction in behavioral cloning through value approximation. InInternational Conference on Robotics and Automa- tion (ICRA), 2023

    2023

  24. [33]

    A. Xie, F. Tajwar, A. Sharma, and C. Finn. When to ask for help: Proactive interventions in autonomous reinforcement learning. InNeural Information Processing Systems (NeurIPS), 2022

    2022

  25. [34]

    Sinha, E

    R. Sinha, E. Schmerling, and M. Pavone. Closing the loop on runtime monitors with fallback- safe mpc. InConference on Decision and Control, pages 6533–6540. IEEE, 2023

    2023

  26. [35]

    Farid, D

    A. Farid, D. Snyder, A. Z. Ren, and A. Majumdar. Failure prediction with statistical guarantees for vision-based robot control. InRobotics: Science and Systems, 2022

    2022

  27. [36]

    Ablett, F

    T. Ablett, F. Maric, and J. Kelly. Fighting failures with fire: Failure identification to reduce expert burden in intervention-based learning.arXiv preprint arXiv:2007.00245, 2020

    2007 arXiv

  28. [37]

    Huang, J

    Y . Huang, J. Song, Z. Wang, S. Zhao, H. Chen, F. Juefei-Xu, and L. Ma. Look before you leap: An exploratory study of uncertainty measurement for large language models.arXiv preprint arXiv:2307.10236, 2023

    2023 arXiv

  29. [38]

    L. Kuhn, Y . Gal, and S. Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. InInternational Conference on Learning Represen- tations, 2023

    2023

  30. [39]

    Shorinwa, Z

    O. Shorinwa, Z. Mei, J. Lidard, A. Z. Ren, and A. Majumdar. A survey on uncertainty quantifi- cation of large language models: Taxonomy, open research challenges, and future directions. arXiv preprint arXiv:2412.05563, 2024

    2024 arXiv

  31. [40]

    J. Duan, W. Pumacay, N. Kumar, Y . R. Wang, S. Tian, W. Yuan, R. Krishna, D. Fox, A. Man- dlekar, and Y . Guo. Aha: A vision-language-model for detecting and reasoning over failures in robotic manipulation.arXiv preprint arXiv:2410.00371, 2024

    arXiv

  32. [41]

    Y . Du, K. Konyushkova, M. Denil, A. Raju, J. Landon, F. Hill, N. de Freitas, and S. Cabi. Vision-language models as success detectors. InConference on Lifelong Learning Agents, pages 120–136. PMLR, 2023

    2023

  33. [42]

    Liang, Y

    A. Liang, Y . Korkmaz, J. Zhang, M. Hwang, A. Anwar, S. Kaushik, A. Shah, A. S. Huang, L. Zettlemoyer, D. Fox, et al. Robometer: Scaling general-purpose robotic reward models via trajectory comparisons.Robotics: Science and Systems (RSS), 2026

    2026

  34. [43]

    V ovk, A

    V . V ovk, A. Gammerman, and G. Shafer.Algorithmic Learning in a Random World. Springer, 2005. 11

    2005

  35. [44]

    A. N. Angelopoulos and S. Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification.arXiv preprint arXiv:2107.07511, 2021

    2021 arXiv

  36. [45]

    Diquigiovanni, M

    J. Diquigiovanni, M. Fontana, and S. Vantini. The importance of being a band: Finite-sample exact distribution-free prediction sets for functional data.Statistica Sinica, 34:1–41, 2024

    2024

  37. [46]

    Devic, T

    S. Devic, T. Srinivasan, J. Thomason, W. Neiswanger, and V . Sharan. From calibration to collaboration: Llm uncertainty quantification should be more human-centered.arXiv preprint arXiv:2506.07461, 2025

    2025 arXiv

  38. [47]

    [ORIGINAL IN- STRUCTION]

    M. Gardner, Y . Artzi, V . Basmov, J. Berant, B. Bogin, S. Chen, P. Dasigi, D. Dua, Y . Elazar, A. Gottumukkala, N. Gupta, H. Hajishirzi, G. Ilharco, D. Khashabi, K. Lin, J. Liu, N. F. Liu, P. Mulcaire, Q. Ning, S. Singh, N. A. Smith, S. Subramanian, R. Tsarfaty, E. Wallace, A...

    2020

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.