REVIEW 7 cited by
Real-Time Anomaly Detection and Reactive Planning with Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Foundation models, e.g., large language models (LLMs), trained on internet-scale data possess zero-shot generalization capabilities that make them a promising technology towards detecting and mitigating out-of-distribution failure modes of robotic systems. Fully realizing this promise, however, poses two challenges: (i) mitigating the considerable computational expense of these models such that they may be applied online, and (ii) incorporating their judgement regarding potential anomalies into a safe control framework. In this work, we present a two-stage reasoning framework: First is a fast binary anomaly classifier that analyzes observations in an LLM embedding space, which may then trigger a slower fallback selection stage that utilizes the reasoning capabilities of generative LLMs. These stages correspond to branch points in a model predictive control strategy that maintains the joint feasibility of continuing along various fallback plans to account for the slow reasoner's latency as soon as an anomaly is detected, thus ensuring safety. We show that our fast anomaly classifier outperforms autoregressive reasoning with state-of-the-art GPT models, even when instantiated with relatively small language models. This enables our runtime monitor to improve the trustworthiness of dynamic robotic systems, such as quadrotors or autonomous vehicles, under resource and time constraints. Videos illustrating our approach in both simulation and real-world experiments are available on this project page: https://sites.google.com/view/aesop-llm.
Forward citations
Cited by 7 Pith papers
-
Towards General Language-Conditioned Latent Safety Filters
A single Hamilton-Jacobi safety filter conditioned on language constraints reduces violations in simulated pick-and-place, wiping, and stacking, with partial transfer to unseen constraint instances.
-
Adversarial Attacks on Robotic Vision Language Action Models
Text-based adversarial suffixes can make OpenVLA robot policies elicit chosen target actions with over 90% success on one-hot targets and persist across rollout steps.
-
Deferred-Decision Trajectory Optimization
A deterministic optimization framework that maximizes how long multiple candidate targets remain reachable, shown equivalent to a cardinality minimization problem and solved via QCVX, MICP, and SCP methods.
-
Predictive Red Teaming: Breaking Policies Without Breaking Robots
A generative image editing plus anomaly detection pipeline predicts a visuomotor policy's success-rate degradation across off-nominal environmental factors, with an average prediction error below 0.19 in hardware trials.
-
Don't Fool Me Twice: Adapting to Adversity in the Wild with Experience-Driven Reasoning
By detecting trajectory disturbances, attributing them to visual causes with a VLM, and fitting a few-shot spatial disturbance model, robots build personalized danger libraries that improve later navigation.
-
Bounding Distributional Shifts in World Modeling through Novelty Detection
Attaching a VAE novelty detector to the DINO-WM world model and penalizing out-of-distribution predicted states in CEM planning lowers Chamfer distance on small-data robot manipulation benchmarks.
-
Domain Specific Benchmarks for Evaluating Multimodal Large Language Models
A review paper that organizes domain-specific MLLM benchmarks into an eight-discipline taxonomy, with summary tables and performance highlights.
Discussion (0). Continue with ORCID to comment.