Pith. sign in

REVIEW 4 major objections 6 minor 67 references

I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiences

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM narrations help robots explain failures and recover faster

desk verdict RONAR is a sensible LLM-based robot narration system with a useful real-robot dataset, but the paper overclaims end-to-end failure recovery gains: the failure key event is largely hand-fed and the key-event threshold is tuned in-sample. read the letter →

arxiv 2411.12960 v1 pith:EKJTGO5E submitted 2024-11-20 cs.RO

Zihan Wang , Brian Liang , Varad Dhat , Zander Brumbaugh , Nick Walker , Ranjay Krishna , Maya Cakmak This is my paper · ORCID
classification cs.RO
keywords LargeLanguageModelExplainableAIFailureAnalysisrobotnarrationmultimodalgroundingkeyeventselectionRoboNarhuman-robotinteraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces RONAR, a system that converts raw multimodal robot data into natural-language narrations describing what the robot is doing, observing, and planning. The central claim is that these narrations outperform prior summarization methods on failure analysis tasks and help human users localize and explain robot failures more accurately and quickly. The authors build a real-robot dataset, RoboNar, containing four home tasks with 76 expert-labeled failures, and evaluate both automated failure analysis and user studies. A sympathetic reader would care because transparent, searchable robot behavior could make assistive robots safer and easier to debug.

What carries the argument

The central mechanism is the multimodal key-event selector, which aligns environment, internal, and planning data into 0.2-second frames, normalizes optical-flow magnitudes and joint-state changes, accumulates their positive values, and labels a frame as a key event whenever the cumulative sum reaches a threshold of 80 or the task-planner state changes. This reduces over a thousand frames to roughly thirty per task while capturing 62% of failures at that threshold. The selected key events feed three summarizers—YOLO-World-based environment summaries, numerical internal-state summaries with grounded explanations, and planning summaries that include sub-goal history—which together produce the input for an LLM narration engine with Alert, Info, and Debug modes and progressive generation.

What would settle it

Run RONAR's key-event selector with a threshold of 80 on a second robot's logs with different task dynamics and count the failure-capture rate; a large drop from the roughly 62% reported on RoboNar would show the constant is overfit. A stronger test is to repeat the full failure-analysis comparison against REFLECT on a new environment and see whether the ~11% advantage in explanation holds.

Watch

Extended reading notes

Core claim

RONAR claims that grounding raw robot data into natural language works best when done in two stages: first compress the multimodal stream into structured experience summaries, then let an LLM turn those summaries into progressive narrations. In experiments on the RoboNar dataset, RONAR beats REFLECT by about 11% on risk estimation and failure explanation, and beats raw-data TEM-LLM and TEM-VLM by 11% and 29% on explanation. Ablations show internal state and planning history are essential: a vision-only version loses 50% on failure localization and 19% on explanation. In a user study with 24 participants, RONAR-UI produced higher accuracy and shorter time for failure localization and explanation than raw video, raw sensor, or keyframe-only interfaces. The narration quality ratings also favor RONAR over BLIP2, REFLECT, and TEM variants.

Load-bearing premise

The key-event selector's threshold of 80 and sample rate of 0.2 seconds were chosen by tuning on the same RoboNar dataset, so the assumption is that these constants work across robots and environments without re-tuning.

Editorial extensions

If this is right

  • If RONAR is correct, robot operators can replace raw video and sensor plots with narrated streams and still catch failures faster.
  • Including internal joint/base state and planning history is required; vision-only narrations lose roughly half of failure-localization accuracy.
  • Progressive narration with history enables long-horizon tasks and smooth, non-repetitive stories, which are needed for real deployments.
  • The RoboNar dataset with expert-labeled failures provides a reusable benchmark for failure analysis in mobile manipulation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The threshold 80 and 0.2-second sampling are tuned on RoboNar; a fair test would re-tune or test transfer on another robot log before trusting the failure-capture findings.
  • The two-step summarization suggests that converting raw multimodal data to text first is what enables LLMs to reason about failures; a single end-to-end VLM may not match this.
  • Trajectory and system-level summaries could turn robot logs into queryable databases, enabling search-and-retrieval of failure trajectories for imitation learning or debugging.
  • Latency and LLM cost limit real-time use; event selection reduces frame count but the narration calls remain a bottleneck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces RONAR, a modular LLM-based system that converts multimodal robot data (RGB-D images, joint states, planner states, and robot specifications) into natural-language narrations. The system first aligns raw streams into fixed-rate frames, selects key events via a running sum of z-scored optical flow and joint-state changes plus planner state transitions, summarizes each event into environment/internal/planning text, and then generates progressive narrations in three modes (alert, info, debug). The authors collect a real-robot dataset, RoboNar, with 70 demonstrations and 76 failure cases across four home tasks, and evaluate (1) key event selection by failure capture rate, (2) failure analysis (risk estimation, localization, explanation, recovery recommendation) against BLIP2, REFLECT, and raw-data LLM/VLM baselines, and (3) two user studies on narration quality and failure identification. The central claim is that RONAR outperforms state-of-the-art methods and improves failure recovery efficiency.

Significance. If the main claims held, RONAR would be a genuinely useful step toward transparent, explainable robot systems: it addresses a real problem (grounding high-volume, multi-modal, asynchronous robot data into human-readable narrative), and the released RoboNar dataset with expert failure annotations is a valuable resource for the community. The framework is modular and the progressive-narration design is thoughtful. The paper also reports positive user-study results for narration quality and for the RONAR interface. However, the empirical validation currently has several load-bearing gaps: the key-event threshold is tuned on the same data used for evaluation, the failure-analysis evaluation injects the failure key event rather than measuring the full pipeline, no significance testing accompanies the headline accuracy comparisons, and the user study measures typing time rather than actual recovery actions. These gaps make the abstract's strong claims ('outperforms state-of-the-art methods' and 'improves failure recovery efficiency') not yet demonstrated, so the significance is conditional on the evaluation being strengthened.

major comments (4)
  1. [Section 4.1, Appendix C.1, Table 5] The key event selection threshold (80) and frame alignment sample rate (0.2 s) are selected by sweeping thresholds on the same RoboNar data that is later used to report the failure capture rates in Table 5. The resulting 0.62 capture rate at threshold 80 is therefore an in-sample estimate with no held-out validation. Please split demonstrations into a threshold-selection set and an evaluation set (e.g., by cross-validation over the 70 demos), report the failure capture rate on held-out runs at the chosen threshold, and provide a calibration curve over thresholds computed on the selection set only. Without this, the key event selection component's generalizability is not established.
  2. [Appendix C.2, Figure 5] The failure explanation and recovery recommendation tasks are defined as 'given previous key events and the current key event (when the failure happened)', meaning the failure time is injected by the experimenter rather than discovered by RONAR. This does not measure the full pipeline from raw data to failure analysis. Moreover, Table 5 shows that at the chosen threshold 80 with all modalities, only 62% of failures produce a key event within ±1.5 s; the paper does not report how the 76 failures scored in Figure 5 were filtered, nor whether the 38% of uncaptured failures are counted as localization or explanation errors. Please either report end-to-end accuracy that treats uncaptured failures as errors, or state explicitly that Figure 5 reports conditional performance on captured events and discuss how that affects the abstract's claims.
  3. [Section 4.2, Figure 5] The failure-analysis accuracies (e.g., the reported ~11% gains over REFLECT on risk estimation and explanation) are presented without error bars, confidence intervals, or significance tests. With only 76 failure cases, these differences may not be statistically reliable. Please report per-failure results for all 76 cases, state the tolerance used for localization (the appendix uses ±1.5 s for capture; is the same used for scoring?), and apply a paired test such as McNemar's test on the per-failure correctness vectors for each pairwise comparison.
  4. [Section 4.1 User Study 2, Figure 6; Abstract] User Study 2 measures the time participants took to type the failure time and explanation for four selected failures from a single task (put cup in sink), using different interfaces. It does not measure actual recovery actions, recovery success, or time-to-recover. Therefore the abstract's statement that RONAR 'improves failure recovery efficiency' is not a direct empirical consequence of the reported experiments. Either add a recovery-action task (e.g., time to issue a teleoperation command or complete a recovery step using the narration) or rephrase the abstract and conclusion to 'improves failure identification efficiency'.
minor comments (6)
  1. [Throughout] There are several typos and minor language issues: 'failulre' in Section 4.1, 'Addtionally' in Appendix B.2, 'achive' in Section 3.3, 'uss' in Appendix D.3, and 'trajecory' in Appendix D.3.2. A proofreading pass is needed.
  2. [Table 1 caption] The main-text Table 1 uses the same data as Appendix C.1's Table 5 but does not specify the ±1.5 s tolerance used for the failure capture rate; please state the tolerance in the main-text caption as well.
  3. [Appendix C.2, Risk Estimation definition] The definition of Risk Estimation ('the percentage of predicted failures that are actual failures in the actual failure key event') corresponds to precision only. Please clarify how false negatives (failures that are not predicted) are handled, and consider also reporting recall or the full precision-recall trade-off.
  4. [Appendix C.2, TEM-LLM baseline] The TEM-LLM baseline is described as sending 'all raw sensory data directly to the LLM'; please specify how the raw data is serialized (e.g., JSON text, concatenated logs) and any length/truncation handling so that the baseline is reproducible.
  5. [Appendix B.1.2, Eq. (3)] The clarity-score equation is written with an unnormalized discrete Laplacian; please state the kernel used and the variance computation explicitly, since the score is a component of the system's image selection.
  6. [Limitations paragraph] The limitations paragraph does not mention the sensitivity of the key event threshold or the lack of held-out validation; given the paper's reliance on the threshold 80, this limitation should be acknowledged.

Circularity Check

1 steps flagged · score 2.0 of 10

RONAR is mostly evaluated against external baselines and human judgments, so the central comparison is not circular; the only mild circularity is that the key-event threshold 80 is set and evaluated on the same RoboNar data, making the reported failure-capture rate and the key events used in Figure 5 partially in-sample.

  1. fitted input called prediction [Appendix B.1.1 and Appendix C.1, Table 5]
    "For our data, we aligned frames with a sample rate of 0.2 and set our key event threshold to 80. ... We designed an additional experiment to examine the relationship between the heuristic threshold, the combination of heuristics (modalities), and the failure capture rate."

    The threshold 80 is presented as a fixed setting, but Appendix C.1 shows that the same investigation sweeps thresholds and reports the failure-capture rate at threshold 80 on the same RoboNar dataset (0.62 with all modalities). The capture rate is therefore an in-sample evaluation of a constant chosen after inspecting those very data, not a held-out prediction. Because the failure-analysis results in Figure 5 use key events produced with this threshold on the same demonstrations, any advantage attributable to the threshold choice is not separated from selection on the test set. This is mild and does not collapse the main comparison, which is against external baselines and human judgments.

full rationale

RONAR's central evaluation is not circular in the strong sense. The failure-analysis comparison in Figure 5 pits RONAR against external baselines (BLIP2, REFLECT, TEM-LLM/VLM) and internal ablations, and the user studies compare interfaces against raw video, raw sensors, and keyframes without narration. These comparisons do not reduce to RONAR's own outputs; narration-quality ratings and failure-identification times are external human judgments. No load-bearing self-citation or imported uniqueness theorem is present. The main internal concern is the key-event threshold: Appendix B.1.1 fixes the threshold at 80 for the RoboNar data, and Appendix C.1 reports the failure-capture rate at that same threshold on that same data, so the 0.62 capture rate and the key events used in Figure 5 are in-sample rather than predictive. In addition, the explanation and recovery subtasks are explicitly defined as 'given previous key events and the current key event (when the failure happened)', so those scores measure conditional analysis rather than end-to-end failure discovery; this is an honest task decomposition but should be kept distinct from the abstract's broader 'improves failure recovery efficiency' claim, which the user study only supports for time-to-explain, not for actual recovery actions. These are methodological limitations, not a definitional collapse, so the circularity score is low.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the effectiveness of the key event heuristic and the quality of LLM summarization from structured text, plus the reliability of expert labels and user-study judgments. The main free parameter is the key event threshold, which is set on the dataset used for evaluation. No invented entities are introduced.

free parameters (4)
  • Key event threshold = 80
    Cumulative sum of z-scored optical flow and joint state changes above which a key event is triggered; Appendix B.1.1. Tuned on the evaluation dataset without a separate validation split.
  • Frame alignment sample rate = 0.2 s
    Frames separated by 0.2 seconds; Appendix B.1.1. Chosen by hand; controls temporal resolution of key events.
  • Depth filter distance cd = not specified
    Objects closer than cd are kept in the environment summary (Eq. 1); the value is not reported.
  • Failure time tolerance = not specified
    User study marks failure localization as correct if within a tolerated window; the tolerance value is not reported.
assumptions (5)
  • domain assumption GPT-4o provides faithful narration and failure analysis from structured experience summaries without additional grounding.
    The entire narration engine and failure analysis depend on the proprietary LLM's reliability; prompts are not fully specified.
  • domain assumption YOLO-World plus FastSAM detects and segments the objects relevant to the tasks in home scenes.
    Environment summaries are built from open-vocabulary detections; if detections are wrong, scene descriptions and failure explanations inherit the errors.
  • ad hoc to paper The running sum of z-scored optical flow and joint-state changes is a valid proxy for narratable events, with planner state changes always included.
    Section 3.1 and Appendix B.1.1; this heuristic defines which frames become key events and is not derived from first principles or validated on other datasets.
  • domain assumption Human expert failure labels and evaluation judgments constitute reliable ground truth.
    Section 4.1; failure reasons, timestamps, and recovery instructions are labeled by experts, and failure explanation accuracy is judged by experts without reported inter-annotator agreement.
  • standard math Variance of the Laplacian is used as an image clarity score; z-score normalization makes modalities comparable.
    Appendix B.1; standard signal and image processing, not contested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiences." pith.science (2026). https://pith.science/paper/EKJTGO5E

@misc{pith2026241112960,
  author       = {Pith},
  title        = {Pith review of: I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EKJTGO5E}},
  note         = {Machine review of arXiv:2411.12960}
}
read the original abstract

Understanding robot behaviors and experiences through natural language is crucial for developing intelligent and transparent robotic systems. Recent advancement in large language models (LLMs) makes it possible to translate complex, multi-modal robotic experiences into coherent, human-readable narratives. However, grounding real-world robot experiences into natural language is challenging due to many reasons, such as multi-modal nature of data, differing sample rates, and data volume. We introduce RONAR, an LLM-based system that generates natural language narrations from robot experiences, aiding in behavior announcement, failure analysis, and human interaction to recover failure. Evaluated across various scenarios, RONAR outperforms state-of-the-art methods and improves failure recovery efficiency. Our contributions include a multi-modal framework for robot experience narration, a comprehensive real-robot dataset, and empirical evidence of RONAR's effectiveness in enhancing user experience in system transparency and failure analysis.

Figures

Figures reproduced from arXiv: 2411.12960 by the authors.

Figure 1
Figure 1. Left: Our framework for real-world robot narration, RONAR. It takes in four categories of dynamic inputs and one static input: multimodal environmental observations (E), robot inter￾nal states (I), task planner (TP), and specified conditions (C), along with robot specifications (SP). RONAR then uses its LLM-based narration engine to process these inputs and generate narrations based on the specified narration mode. … view at source ↗
Figure 2
Figure 2. RONAR: Our framework for real-world robot narration. It has three parts, which are key frame selection, experience summarization and narration generation. It takes in the raw multimodal robot data stream and outputs text describing past experiences, current observations, and future plans of the robot. 3.1 Multimodal Key Event Selection When executing robot processes, massive amounts of data are streamed at persisten… view at source ↗
Figure 3
Figure 3. Example of narrations generated by RONAR with different [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: RoboNar Dataset: We design four long-horizon tasks for a Stretch robot in a home environment. Left: the different tasks with base and manipulator trajectories. It also shows states the robot experiences in each task. Right: the number of failure cases under each robot …
Figure 5
Figure 5. Figure 5: Accuracy on failure analysis tasks using different methods. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Left: The accuracy of failure localization and explanation from the user study using different interfaces. Right: The average time taken by participants to localize failure and both localize and explain failure while using different interfaces. mobile robot home tasks.…
Figure 7
Figure 7. Figure 7: The robot detects and localizes the cup using YOLO-World and segments the result with [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Adjacent image selection using clarity score. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: RONAR-UI interfaces. Left: RONAR-UI with offline mode. Right: RONAR-UI with online mode. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Demographics of the participants. Left: Age ranges of the participants of the user study. Right: Highest degrees earned by the participants of the user study. We also create questions on robot familiarity for participants to answer. These questions inlcude: • Expertis…
Figure 11
Figure 11. Figure 11: Expertise of the participants with robot and Stretch. [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Four interfaces used for failure identification in user study. [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Narration examples generated by RONAR. As shown in [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Comparison of Narrations Generated by Different Methods. [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Trajectory summarization The narration generated by RONAR is event-level, meaning each narration cor￾responds to a single key event. It cap￾tures a snapshot of the process but cannot provide an overview of the entire demon￾stration. To address this, we create a higher…
Figure 16
Figure 16. Figure 16: , for a collection of robot demonstrations, RONAR can generate a corresponding trajectory summary for each trajectory. These summaries can then be used for trajectory retrieval purposes. Users can search for and retrieve trajectories with customized queries. These que…
Figure 17
Figure 17. Figure 17: System overview generated by RONAR 28 [PITH_FULL_IMAGE:figures/full_fig_p028_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 36 canonical work pages

  1. [1]

    B ¨armann, R

    L. B ¨armann, R. Kartmann, F. Peller-Konrad, A. Waibel, and T. Asfour. Incremental learning of humanoid robot behavior from natural interaction and large language models. arXiv preprint arXiv:2309.04316, 2023

    arXiv 2023

  2. [2]

    D. Das, S. Banerjee, and S. Chernova. Explainable ai for robot failures: Generating explana- tions that improve user assistance in fault recovery. In Proceedings of the 2021 ACM/IEEE international conference on human-robot interaction, pages 351–360, 2021

    work page 2021

  3. [3]

    Rosenthal, S

    S. Rosenthal, S. P. Selvaraj, and M. M. Veloso. Verbalization: Narration of autonomous robot experience. In International Joint Conference on Artificial Intelligence, 2016

    work page 2016

  4. [4]

    J. Wang, Z. Wu, Y . Li, H. Jiang, P. Shu, E. Shi, H. Hu, C. Ma, Y . Liu, X. Wang, et al. Large language models for robotics: Opportunities, challenges, and perspectives. arXiv preprint arXiv:2401.04334, 2024

    arXiv 2024

  5. [5]

    F. Zeng, W. Gan, Y . Wang, N. Liu, and P. S. Yu. Large language models for robotics: A survey. arXiv preprint arXiv:2311.07226, 2023

    arXiv 2023

  6. [6]

    Zhang, J

    C. Zhang, J. Chen, J. Li, Y . Peng, and Z. Mao. Large language models for human-robot interaction: A review. Biomimetic Intelligence and Robotics, page 100131, 2023

    work page 2023

  7. [7]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022

    arXiv 2022

  8. [8]

    L. Sun, D. K. Jha, C. Hori, S. Jain, R. Corcodel, X. Zhu, M. Tomizuka, and D. Romeres. Interactive planning using large language models for partially observable robotics tasks.arXiv preprint arXiv:2312.06876, 2023

    work page Pith review arXiv 2023

Show all 67 references
  1. [9]

    J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser. Tidybot: Personalized robot assistance with large language models. Au- tonomous Robots, 47(8):1087–1102, 2023

    2023

  2. [10]

    J. P. Macdonald, R. Mallick, A. B. Wollaber, J. D. Pe˜na, N. McNeese, and H. C. Siu. Language, camera, autonomy! prompt-engineered robot control for rapidly evolving deployment. In Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction , pages 717–721, 2024

    2024

  3. [11]

    K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suenderhauf. Sayplan: Ground- ing large language models using 3d scene graphs for scalable task planning. arXiv preprint arXiv:2307.06135, 2023

    2023 arXiv

  4. [12]

    A. Szot, M. Schwarzer, H. Agrawal, B. Mazoure, R. Metcalf, W. Talbott, N. Mackraz, R. D. Hjelm, and A. T. Toshev. Large language models as generalizable policies for embodied tasks. In The Twelfth International Conference on Learning Representations, 2023

    2023

  5. [13]

    Padalkar, A

    A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x mod- els. arXiv preprint arXiv:2310.08864, 2023

    2023 arXiv

  6. [14]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    2023 arXiv

  7. [15]

    Y . Du, O. Watkins, Z. Wang, C. Colas, T. Darrell, P. Abbeel, A. Gupta, and J. Andreas. Guiding pretraining in reinforcement learning with large language models. InInternational Conference on Machine Learning, pages 8657–8677. PMLR, 2023. 9

    2023

  8. [16]

    S. Wang, Z. Zhou, B. Li, Z. Li, and Z. Kan. Multi-modal interaction with transformers: bridging robots and human with natural language. Robotica, 42(2):415–434, 2024. doi: 10.1017/S0263574723001510

    2024 doi

  9. [17]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PMLR, 2021

    2021

  10. [18]

    D. Shah, B. Osi ´nski, S. Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on robot learning , pages 492–504. PMLR, 2023

    2023

  11. [19]

    Driess, F

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  12. [20]

    Y . Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE interna- tional conference on robotics and automation (ICRA), pages 3357–3364. IEEE, 2017

    2017

  13. [21]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  14. [22]

    Crosby, M

    M. Crosby, M. Rovatsos, and R. Petrick. Automated agent decomposition for classical plan- ning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 23, pages 46–54, 2013

    2013

  15. [23]

    B. Xu, Z. Peng, B. Lei, S. Mukherjee, Y . Liu, and D. Xu. Rewoo: Decoupling reasoning from observations for efficient augmented language models.arXiv preprint arXiv:2305.18323, 2023

    2023 arXiv

  16. [24]

    Kojima, S

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022

    2022

  17. [25]

    S. S. Raman, V . Cohen, D. Paulius, I. Idrees, E. Rosen, R. Mooney, and S. Tellex. Cape: Corrective actions from precondition errors using large language models. arXiv preprint arXiv:2211.09935, 2022

    2022 arXiv

  18. [26]

    Z. Liu, A. Bahety, and S. Song. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint arXiv:2306.15724, 2023

    2023 arXiv

  19. [27]

    S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y . Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022

    2022 arXiv

  20. [28]

    N. M. Shafiullah, Z. Cui, A. A. Altanzaya, and L. Pinto. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing systems, 35:22955–22968, 2022

    2022

  21. [29]

    Kolve, R

    E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y . Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai.arXiv preprint arXiv:1712.05474, 2017

    2017 arXiv

  22. [30]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022. 10

    2022 arXiv

  23. [31]

    Matuszek, E

    C. Matuszek, E. Herbst, L. Zettlemoyer, and D. Fox. Learning to parse natural language com- mands to a robot control system. In Experimental robotics: the 13th international symposium on experimental robotics, pages 403–415. Springer, 2013

    2013

  24. [32]

    Chen and R

    D. Chen and R. Mooney. Learning to interpret natural language navigation instructions from observations. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 25, pages 859–865, 2011

    2011

  25. [33]

    Arkin, D

    J. Arkin, D. Park, S. Roy, M. R. Walter, N. Roy, T. M. Howard, and R. Paul. Multimodal estimation and communication of latent semantic knowledge for robust execution of robot instructions. The International Journal of Robotics Research, 39(10-11):1279–1304, 2020

    2020

  26. [34]

    Bucker, L

    A. Bucker, L. Figueredo, S. Haddadin, A. Kapoor, S. Ma, S. Vemprala, and R. Bonatti. Latte: Language trajectory transformer. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7287–7294. IEEE, 2023

    2023

  27. [35]

    T. Wang, R. Zhang, Z. Lu, F. Zheng, R. Cheng, and P. Luo. End-to-end dense video cap- tioning with parallel decoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6847–6857, October 2021

    2021

  28. [36]

    C. Deng, S. Chen, D. Chen, Y . He, and Q. Wu. Sketch, ground, and refine: Top-down dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 234–243, June 2021

    2021

  29. [37]

    Zhang, Y

    Q. Zhang, Y . Song, and Q. Jin. Unifying event detection and captioning as sequence generation via pre-training. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVI, pages 363–379. Springer, 2022

    2022

  30. [38]

    W. Zhu, B. Pang, A. V . Thapliyal, W. Y . Wang, and R. Soricut. End-to-end dense video captioning as sequence generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5651–5665, 2022

    2022

  31. [39]

    A. Yang, A. Nagrani, P. H. Seo, A. Miech, J. Pont-Tuset, I. Laptev, J. Sivic, and C. Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pag...

    2023

  32. [40]

    Armeni, Z.-Y

    I. Armeni, Z.-Y . He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, and S. Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5664–5673, 2019

    2019

  33. [41]

    W. Chen, S. Hu, R. Talak, and L. Carlone. Leveraging large language models for robot 3d scene understanding. arXiv preprint arXiv:2209.05629, 2022

    2022 arXiv

  34. [42]

    B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler. Open-vocabulary queryable scene representations for real world planning. In2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 11509–11522. IEEE, 2023

    2023

  35. [43]

    X. Li, D. Guo, H. Liu, and F. Sun. Embodied semantic scene graph generation. In Conference on robot learning, pages 1585–1594. PMLR, 2022

    2022

  36. [44]

    A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V . Sindhwani, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022

    2022 arXiv

  37. [45]

    Z. Wang, M. Li, R. Xu, L. Zhou, J. Lei, X. Lin, S. Wang, Z. Yang, C. Zhu, D. Hoiem, et al. Lan- guage models with image descriptors are strong few-shot video-language learners. Advances in Neural Information Processing Systems, 35:8483–8497, 2022. 11

    2022

  38. [46]

    Honerkamp, M

    D. Honerkamp, M. Buchner, F. Despinoy, T. Welschehold, and A. Valada. Language-grounded dynamic scene graphs for interactive object search with mobile manipulation. arXiv preprint arXiv:2403.08605, 2024

    2024 arXiv

  39. [47]

    Khanna, E

    P. Khanna, E. Yadollahi, M. Bj ¨orkman, I. Leite, and C. Smith. User study exploring the role of explanation of failures by robots in human robot collaboration tasks. arXiv preprint arXiv:2303.16010, 2023

    2023 arXiv

  40. [48]

    S. Ye, G. Neville, M. Schrum, M. Gombolay, S. Chernova, and A. Howard. Human trust after robot mistakes: Study of the effects of different forms of robot communication. In 2019 28th IEEE international conference on robot and human interactive communication (ro-man), pages 1–7....

    2019

  41. [49]

    Das and S

    D. Das and S. Chernova. Semantic-based explainable ai: Leveraging semantic scene graphs and pairwise ranking to explain robot failures. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3034–3041. IEEE, 2021

    2021

  42. [50]

    Diehl and K

    M. Diehl and K. Ramirez-Amaro. Why did i fail? a causal-based method to find explanations for robot failures. IEEE Robotics and Automation Letters, 7(4):8925–8932, 2022

    2022

  43. [51]

    Inceoglu, E

    A. Inceoglu, E. E. Aksoy, A. C. Ak, and S. Sariel. Fino-net: A deep multimodal sensor fusion framework for manipulation failure detection. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6841–6847. IEEE, 2021

    2021

  44. [52]

    Huang, C

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023

    2023 arXiv

  45. [53]

    Huang, P

    W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Ex- tracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118–9147. PMLR, 2022

    2022

  46. [54]

    Liang, W

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500. IEEE, 2023

    2023

  47. [55]

    A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V . Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726–747. PMLR, 2021

    2021

  48. [56]

    Lynch, A

    C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters , 2023

    2023

  49. [57]

    Z. Wang, Z. Cao, Y . Hao, and D. Sadigh. Weakly supervised correspondence learning. In2022 International Conference on Robotics and Automation (ICRA), pages 469–476. IEEE, 2022

    2022

  50. [58]

    Hellstr ¨om and S

    T. Hellstr ¨om and S. Bensch. Understandable robots - what, why, and how.Paladyn, Journal of Behavioral Robotics, 9(1):110–123, 2018. doi:doi:10.1515/pjbr-2018-0009

    2018 doi

  51. [59]

    R. H. Wortham and A. Theodorou. Robot transparency, trust and utility. Connection Science, 29(3):242–248, 2017. doi:10.1080/09540091.2017.1313816

    2017

  52. [60]

    Mahadevan, J

    K. Mahadevan, J. Chien, N. Brown, Z. Xu, C. Parada, F. Xia, A. Zeng, L. Takayama, and D. Sadigh. Generative expressive robot behaviors using large language models. In Proceed- ings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’24. ACM, Mar. 202...

    2024

  53. [61]

    J. Wang, Y . Liu, T. Yue, C. Wang, J. Mao, Y . Wang, and F. You. Robot transparency and anthropomorphic attribute effects on human–robot interactions. Sensors, 21(17), 2021. ISSN 1424-8220. doi:10.3390/s21175722. 12

    2021 doi

  54. [62]

    Osorio, R

    P. Osorio, R. Sagawa, N. Abe, and G. Venture. A generative model to embed human expres- sivity into robot motions. Sensors, 24(2), 2024. ISSN 1424-8220. doi:10.3390/s24020569

    2024 doi

  55. [63]

    Canal, S

    G. Canal, S. Krivi ´c, P. Luff, and A. Coles. PlanVerb: Domain-Independent Verbalization and Summary of Task Plans. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36(9), pages 9698–9706, 2022. doi:10.1609/aaai.v36i9.21204

    2022 doi

  56. [64]

    Cheng, L

    T. Cheng, L. Song, Y . Ge, W. Liu, X. Wang, and Y . Shan. Yolo-world: Real-time open- vocabulary object detection. arXiv preprint arXiv:2401.17270, 2024

    2024 arXiv

  57. [65]

    C. C. Kemp, A. Edsinger, H. M. Clever, and B. Matulevich. The design of stretch: A com- pact, lightweight mobile manipulator for indoor human environments. In 2022 International Conference on Robotics and Automation (ICRA), pages 3150–3157. IEEE, 2022

    2022

  58. [66]

    Celikyilmaz, E

    A. Celikyilmaz, E. Clark, and J. Gao. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799, 2020. 13 A Dataset Details A.1 Robot Tasks and Implementations We automated four mobile manipulation tasks in a home environment: putting a cup in the sink, microwav...

    2006 arXiv

  59. [80]

    The key event selection can be modeled as a binary classifier, Ckey, which runs across all the multimodal frames in a given task and outputs a binary prediction, 0 (not a key event) or 1 (a key event). It can be represented as following: Ckey(fi) =    1, iX k=c ...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.