{"id":"0df36ef7-44a5-4d51-a94c-7f6896db530a","arxiv_id":"2411.12960","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RONAR is an LLM-based framework that narrates a mobile robot's experiences in natural language, and its user studies show that these narrations help people localize and explain robot failures faster than raw video interfaces.","lead":"This paper introduces RONAR, a system that turns a home robot's sensor data, internal state, and task plan into clear English narrations of what the robot is doing and why. The authors test whether these narrations help people spot and explain robot failures more quickly and accurately than watching raw video.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Failure-analysis results rely on injecting the failure key event and on a key-event threshold (80) tuned on the same RoboNar data; at that threshold 38% of failures are not captured, so the end-to-end 'outperforms and improves recovery efficiency' claim is not yet demonstrated.","rationale":"I read the paper's central claim as an end-to-end claim: RONAR's structured narrations should improve automated failure analysis and human recovery efficiency. The reader's weakest_assumption (threshold 80 tuned on RoboNar) is a real and concrete weakness, and I agree with the CONDITIONAL verdict. My stress-test adds that even on the tuning set, threshold 80 captures only 62% of failures, and the failure-analysis protocol in Appendix C.2 supplies the failure key event for the explanation/recovery tasks. If the Figure 5 evaluation excludes failures missed by the selector or injects the failure key event, the reported advantage over REFLECT/TEM is not an end-to-end system result. This does not mean the method is wrong; the modular framework, dataset, and component-level evaluations are genuine contributions. It does mean the headline statement 'RONAR outperforms state-of-the-art methods and improves failure recovery efficiency' requires either a full end-to-end evaluation or a more modest claim about component-level performance. I therefore keep the verdict CONDITIONAL rather than moving to ACCEPT or REJECT: the concern is specific and testable, and the authors' existing materials could likely address it with additional experiments and releases.","tokens_in":21941,"tokens_out":11474,"duration_ms":150749,"concrete_test":"Run the complete RONAR pipeline on all 70 RoboNar demonstrations using sample rate 0.2s and threshold 80; feed only the automatically selected key events into experience summarization and the four failure-analysis prompts, score all 76 ground-truth failures with missed key events counted as failures, and compare against a baseline that receives all raw frames. Also split the data (e.g., hold out the Collect Dirty Clothes trials plus a random subset of trials from the other tasks), re-tune the threshold on the training split, and check whether threshold 80's capture rate and the failure-analysis accuracies hold on the held-out trials. This would show whether the reported gains survive without oracle key events and without in-sample threshold selection.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires the full pipeline (key-event selection -> summarization -> narration -> failure analysis/recovery) to outperform baselines. That end-to-end chain is not actually measured. Appendix C.2 defines Risk Estimation as 'given previous key events' and Explanation/Recovery as 'given previous key events and the current key event (when the failure happened)'; the failure moment is supplied by the experimenter, not discovered by RONAR. Separately, the key-event selection experiment (Table 5) shows that with the authors' chosen threshold 80 and all modalities, the failure-capture rate is only 0.62 with a +/-1.5s tolerance, meaning roughly 38% of failures produce no key event within tolerance. It is not reported how the 76 failures scored in Figure 5 were filtered, or whether missed failures count as localization/explanation errors. Threshold 80 was also selected by sweeping thresholds on the same RoboNar data used for the reported capture rates (Appendix C.1), so the 0.62 figure itself is an in-sample estimate. Finally, user study 2 measures time to type a failure time/explanation on four chosen failures from one task, not actual recovery actions; the abstract's 'improves failure recovery efficiency' overstates the measured outcome. The framework is plausible, but the headline empirical claim is not yet established end-to-end.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RONAR, a modular LLM-based system that converts multimodal robot data (RGB-D images, joint states, planner states, and robot specifications) into natural-language narrations. The system first aligns raw streams into fixed-rate frames, selects key events via a running sum of z-scored optical flow and joint-state changes plus planner state transitions, summarizes each event into environment/internal/planning text, and then generates progressive narrations in three modes (alert, info, debug). The authors collect a real-robot dataset, RoboNar, with 70 demonstrations and 76 failure cases across four home tasks, and evaluate (1) key event selection by failure capture rate, (2) failure analysis (risk estimation, localization, explanation, recovery recommendation) against BLIP2, REFLECT, and raw-data LLM/VLM baselines, and (3) two user studies on narration quality and failure identification. The central claim is that RONAR outperforms state-of-the-art methods and improves failure recovery efficiency.","tokens_in":22267,"tokens_out":4581,"duration_ms":45460,"significance":"If the main claims held, RONAR would be a genuinely useful step toward transparent, explainable robot systems: it addresses a real problem (grounding high-volume, multi-modal, asynchronous robot data into human-readable narrative), and the released RoboNar dataset with expert failure annotations is a valuable resource for the community. The framework is modular and the progressive-narration design is thoughtful. The paper also reports positive user-study results for narration quality and for the RONAR interface. However, the empirical validation currently has several load-bearing gaps: the key-event threshold is tuned on the same data used for evaluation, the failure-analysis evaluation injects the failure key event rather than measuring the full pipeline, no significance testing accompanies the headline accuracy comparisons, and the user study measures typing time rather than actual recovery actions. These gaps make the abstract's strong claims ('outperforms state-of-the-art methods' and 'improves failure recovery efficiency') not yet demonstrated, so the significance is conditional on the evaluation being strengthened.","major_comments":[{"comment":"The key event selection threshold (80) and frame alignment sample rate (0.2 s) are selected by sweeping thresholds on the same RoboNar data that is later used to report the failure capture rates in Table 5. The resulting 0.62 capture rate at threshold 80 is therefore an in-sample estimate with no held-out validation. Please split demonstrations into a threshold-selection set and an evaluation set (e.g., by cross-validation over the 70 demos), report the failure capture rate on held-out runs at the chosen threshold, and provide a calibration curve over thresholds computed on the selection set only. Without this, the key event selection component's generalizability is not established.","section":"Section 4.1, Appendix C.1, Table 5"},{"comment":"The failure explanation and recovery recommendation tasks are defined as 'given previous key events and the current key event (when the failure happened)', meaning the failure time is injected by the experimenter rather than discovered by RONAR. This does not measure the full pipeline from raw data to failure analysis. Moreover, Table 5 shows that at the chosen threshold 80 with all modalities, only 62% of failures produce a key event within ±1.5 s; the paper does not report how the 76 failures scored in Figure 5 were filtered, nor whether the 38% of uncaptured failures are counted as localization or explanation errors. Please either report end-to-end accuracy that treats uncaptured failures as errors, or state explicitly that Figure 5 reports conditional performance on captured events and discuss how that affects the abstract's claims.","section":"Appendix C.2, Figure 5"},{"comment":"The failure-analysis accuracies (e.g., the reported ~11% gains over REFLECT on risk estimation and explanation) are presented without error bars, confidence intervals, or significance tests. With only 76 failure cases, these differences may not be statistically reliable. Please report per-failure results for all 76 cases, state the tolerance used for localization (the appendix uses ±1.5 s for capture; is the same used for scoring?), and apply a paired test such as McNemar's test on the per-failure correctness vectors for each pairwise comparison.","section":"Section 4.2, Figure 5"},{"comment":"User Study 2 measures the time participants took to type the failure time and explanation for four selected failures from a single task (put cup in sink), using different interfaces. It does not measure actual recovery actions, recovery success, or time-to-recover. Therefore the abstract's statement that RONAR 'improves failure recovery efficiency' is not a direct empirical consequence of the reported experiments. Either add a recovery-action task (e.g., time to issue a teleoperation command or complete a recovery step using the narration) or rephrase the abstract and conclusion to 'improves failure identification efficiency'.","section":"Section 4.1 User Study 2, Figure 6; Abstract"}],"minor_comments":[{"comment":"There are several typos and minor language issues: 'failulre' in Section 4.1, 'Addtionally' in Appendix B.2, 'achive' in Section 3.3, 'uss' in Appendix D.3, and 'trajecory' in Appendix D.3.2. A proofreading pass is needed.","section":"Throughout"},{"comment":"The main-text Table 1 uses the same data as Appendix C.1's Table 5 but does not specify the ±1.5 s tolerance used for the failure capture rate; please state the tolerance in the main-text caption as well.","section":"Table 1 caption"},{"comment":"The definition of Risk Estimation ('the percentage of predicted failures that are actual failures in the actual failure key event') corresponds to precision only. Please clarify how false negatives (failures that are not predicted) are handled, and consider also reporting recall or the full precision-recall trade-off.","section":"Appendix C.2, Risk Estimation definition"},{"comment":"The TEM-LLM baseline is described as sending 'all raw sensory data directly to the LLM'; please specify how the raw data is serialized (e.g., JSON text, concatenated logs) and any length/truncation handling so that the baseline is reproducible.","section":"Appendix C.2, TEM-LLM baseline"},{"comment":"The clarity-score equation is written with an unnormalized discrete Laplacian; please state the kernel used and the variance computation explicitly, since the score is a component of the system's image selection.","section":"Appendix B.1.2, Eq. (3)"},{"comment":"The limitations paragraph does not mention the sensitivity of the key event threshold or the lack of held-out validation; given the paper's reliance on the threshold 80, this limitation should be acknowledged.","section":"Limitations paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea and dataset are solid and likely to interest the CoRL community, but the empirical claims in the abstract exceed what the current experiments establish. The threshold-tuning issue and the injected failure key event are the most serious concerns; both are fixable within the paper's scope by adding held-out validation and reporting end-to-end metrics. The authors should also be advised to soften the 'recovery efficiency' claim unless real recovery actions are measured. I saw no citation-pattern concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a real systems paper, not a toy. The RoboNar dataset—70 real Stretch robot demos across four home tasks, with 76 labeled failures—is a useful asset, and the idea of feeding an LLM a structured mix of environment, internal joint states, and planner history (including prior sub-goal outcomes) is the right direction. On the strength of the narration-quality user study, the system does seem to produce more usable output than REFLECT or raw-data baselines.\n\nBut the abstract's claim that RONAR 'improves failure recovery efficiency' is not backed by the experiments. The failure analysis tasks in Figure 5 are set up so that the model sees the key event at which the failure happened for explanation and recovery; even for localization, the model is given all key events rather than having to discover failures from raw streams. And the key-event selector itself, at the chosen threshold of 80, only captures 62% of failures within ±1.5 seconds. That means roughly 38% of failures would never reach the narration stage at all. The paper does not say whether those missed failures are counted as errors in Figure 5, which makes the headline comparison hard to interpret.\n\nThere are also smaller issues: the threshold and sample rate appear to have been chosen on the same RoboNar data used for the reported capture rates; Figure 5 has no error bars or significance tests; and user study 2 measures time to type a failure time and explanation, not time to recover the robot. Calling that 'recovery efficiency' is an overstatement. Code and dataset are not released, which hurts reproducibility for something that presents itself as a dataset contribution.\n\nNone of this kills the paper. The framework is plausible, the ablations (vision-only vs. full RONAR) are informative, and the user study results are directionally strong. It just needs an honest rewrite of the claims: report end-to-end failure analysis with missed failures counted, add uncertainty quantification, release the data, and drop the recovery-efficiency wording unless actual recovery actions are measured.\n\nWho's this for? Anyone working on robot transparency, LLM-based failure explanation, or dataset collection for assistive robotics. It is worth a serious referee, but I would send it back for major revision rather than accept it as is.","headline":"RONAR is a sensible LLM-based robot narration system with a useful real-robot dataset, but the paper overclaims end-to-end failure recovery gains: the failure key event is largely hand-fed and the key-event threshold is tuned in-sample.","tokens_in":22760,"tokens_out":3485,"would_cite":true,"duration_ms":38012,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM narrations help robots explain failures and recover faster","keywords":["Large Language Model","Explainable AI","Failure Analysis","robot narration","multimodal grounding","key event selection","RoboNar","human-robot interaction"],"falsifier":"Run RONAR's key-event selector with a threshold of 80 on a second robot's logs with different task dynamics and count the failure-capture rate; a large drop from the roughly 62% reported on RoboNar would show the constant is overfit. A stronger test is to repeat the full failure-analysis comparison against REFLECT on a new environment and see whether the ~11% advantage in explanation holds.","tokens_in":21759,"feed_emoji":"🤖","tokens_out":4905,"duration_ms":45281,"temperature":0.7,"pith_summary":"The paper introduces RONAR, a system that converts raw multimodal robot data into natural-language narrations describing what the robot is doing, observing, and planning. The central claim is that these narrations outperform prior summarization methods on failure analysis tasks and help human users localize and explain robot failures more accurately and quickly. The authors build a real-robot dataset, RoboNar, containing four home tasks with 76 expert-labeled failures, and evaluate both automated failure analysis and user studies. A sympathetic reader would care because transparent, searchable robot behavior could make assistive robots safer and easier to debug.","feed_headline":"LLM narrations help robots explain failures and recover faster","feed_subtitle":"RONAR turns messy sensor streams into plain-language stories, helping humans and AI spot and fix robot mistakes.","key_machinery":"The central mechanism is the multimodal key-event selector, which aligns environment, internal, and planning data into 0.2-second frames, normalizes optical-flow magnitudes and joint-state changes, accumulates their positive values, and labels a frame as a key event whenever the cumulative sum reaches a threshold of 80 or the task-planner state changes. This reduces over a thousand frames to roughly thirty per task while capturing 62% of failures at that threshold. The selected key events feed three summarizers—YOLO-World-based environment summaries, numerical internal-state summaries with grounded explanations, and planning summaries that include sub-goal history—which together produce the input for an LLM narration engine with Alert, Info, and Debug modes and progressive generation.","core_discovery":"RONAR claims that grounding raw robot data into natural language works best when done in two stages: first compress the multimodal stream into structured experience summaries, then let an LLM turn those summaries into progressive narrations. In experiments on the RoboNar dataset, RONAR beats REFLECT by about 11% on risk estimation and failure explanation, and beats raw-data TEM-LLM and TEM-VLM by 11% and 29% on explanation. Ablations show internal state and planning history are essential: a vision-only version loses 50% on failure localization and 19% on explanation. In a user study with 24 participants, RONAR-UI produced higher accuracy and shorter time for failure localization and explanation than raw video, raw sensor, or keyframe-only interfaces. The narration quality ratings also favor RONAR over BLIP2, REFLECT, and TEM variants.","pith_inferences":["The threshold 80 and 0.2-second sampling are tuned on RoboNar; a fair test would re-tune or test transfer on another robot log before trusting the failure-capture findings.","The two-step summarization suggests that converting raw multimodal data to text first is what enables LLMs to reason about failures; a single end-to-end VLM may not match this.","Trajectory and system-level summaries could turn robot logs into queryable databases, enabling search-and-retrieval of failure trajectories for imitation learning or debugging.","Latency and LLM cost limit real-time use; event selection reduces frame count but the narration calls remain a bottleneck."],"forward_implications":["If RONAR is correct, robot operators can replace raw video and sensor plots with narrated streams and still catch failures faster.","Including internal joint/base state and planning history is required; vision-only narrations lose roughly half of failure-localization accuracy.","Progressive narration with history enables long-horizon tasks and smooth, non-repetitive stories, which are needed for real deployments.","The RoboNar dataset with expert-labeled failures provides a reusable benchmark for failure analysis in mobile manipulation."],"supporting_citations":[{"why":"Supplies the main state-of-the-art baseline (REFLECT) for failure explanation and correction that RONAR outperforms.","marker":"[26]"},{"why":"YOLO-World open-vocabulary object detection is used to build the environment summary from RGB images.","marker":"[64]"},{"why":"Describes the Stretch SE3 mobile manipulator on which the RoboNar dataset was collected.","marker":"[65]"},{"why":"Provides the common-practice evaluation criteria for human assessment of natural language generation used in the user study.","marker":"[66]"}],"fun_headline_variants":["RONAR turns robot sensor data into plain-language stories","LLM-based RONAR improves robot failure explanation by up to 29%","Two-stage LLM pipeline grounds robot experiences into narrations","Robot's own narration via RONAR boosts failure recovery"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The key-event selector's threshold of 80 and sample rate of 0.2 seconds were chosen by tuning on the same RoboNar dataset, so the assumption is that these constants work across robots and environments without re-tuning.","fun_headline_variants_meta":{"raw":{"variants":["RONAR turns robot sensor data into plain-language stories","LLM-based RONAR improves robot failure explanation by up to 29%","Two-stage LLM pipeline grounds robot experiences into narrations","Robot's own narration via RONAR boosts failure recovery"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001012,"raw_usage":{"total_tokens":4240,"prompt_tokens":874,"completion_tokens":3366,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":3295}},"tokens_in":490,"tokens_out":3366,"duration_ms":25200,"temperature":1.0,"reasoning_tokens":3295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:00:20.255362+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RONAR's key-event selector with a threshold of 80 on a second robot's logs with different task dynamics and count the failure-capture rate; a large drop from the roughly 62% reported on RoboNar would show the constant is overfit. A stronger test is to repeat the full failure-analysis comparison against REFLECT on a new environment and see whether the ~11% advantage in explanation holds.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the Stretch SE3 mobile manipulator on which the RoboNar dataset was collected."}],"review_version":1}