Pith. sign in

REVIEW 3 major objections 3 minor 73 references

CoRI: Communication of Robot Intent for Physical Human-Robot Interaction

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CoRI turns a robot's planned motion into spoken intent for the person it touches.

desk verdict CoRI is a genuine task-agnostic intent-to-language pipeline, but its accuracy claims lean on an unreliable NLI metric and a weak baseline—send to review with revisions expected. read the letter →

arxiv 2505.20537 v2 pith:2WWDHCJB submitted 2025-05-26 cs.RO

Junxiang Wang , Emek Bar{i}ş Küçüktabak , Rana Soltani Zarrin , Zackory Erickson This is my paper
classification cs.RO
keywords robotintentcommunicationphysicalhuman-robotinteractionvision-languagemodelsassistiveroboticsnaturallanguagegenerationtrajectorygroundinguserstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoRI is a pipeline that converts a robot's camera view and its planned 3D trajectory into natural-language statements that tell the user what the robot intends to do, how it will move, and what cooperation is needed. The paper argues that this can be done without any task-specific knowledge, using only the visual scene and the motion plan. A user study across bathing, shaving, and feeding tasks on two different robots found that CoRI's communications were rated significantly clearer than scripted or no-communication baselines.

What carries the argument

The key mechanism is the interaction-aware trajectory encoding: a set of operations that segment a planned trajectory by gripper state changes, force onset/termination, and pauses, then project each segment's waypoints onto a whitened, body-pose-annotated image with color-coded speed and force. This visual overlay, together with textual waypoint data, is interpreted by a VLM through structured questions that isolate position, velocity, and force aspects. The VLM's per-segment descriptions, overall intention, and inferred user cooperation are then fed to a reasoning LLM that produces the final user-directed sentences.

What would settle it

Run CoRI on a trajectory where the end-effector is occluded in the camera view for part of the motion, or where the force is applied in a region not visible in the overlay, and check whether the generated statements still correctly identify the force location and the motion timing. If the VLM misreads such an overlay, the communication will be wrong despite the LLM's phrasing.

Watch

Extended reading notes

Core claim

The central claim is that a robot can automatically generate concise, user-directed verbal communication of its intent by visually grounding its planned trajectory in an image of the human and environment, then querying a vision-language model to interpret that grounded representation. CoRI segments the trajectory at interaction events such as gripper changes, force onset, and pauses, and encodes position, velocity, and force into a single overlay image. It then asks a VLM structured questions about where the motion starts and ends, its shape, speed changes, and force application, and finally uses a reasoning LLM to craft one natural-language sentence per segment. The paper reports that generated statements achieve near-oracle entailment consistency with ground-truth descriptions and that users understand the robot's intended actions and required cooperation better than with a scripted baseline.

Load-bearing premise

The whole pipeline depends on the vision-language model correctly reading the trajectory overlay and the textual waypoint data, since its answers are the only source of semantic content for the final statements.

Editorial extensions

If this is right

  • Any robot with a camera and a waypoint trajectory planner can generate understandable intent communication without task-specific labels or retraining.
  • Users can anticipate the robot's next action, its motion dynamics, and any required cooperative behavior, which can reduce ambiguity during physical contact.
  • The same pipeline works across different robot morphologies and assistive tasks, indicating potential for generalization to other pHRI applications.
  • The near-oracle entailment scores suggest that the generated statements are semantically consistent with the planner's actual intent, not just plausible-sounding text.
  • Because each segment gets its own sentence, the communication can be timed to play just before the corresponding motion, keeping the explanation in sync with the action.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • CoRI's reliance on VLM interpretation means that its accuracy is bounded by the VLM's ability to read the overlay and answer the structured questions; a failure in that stage would propagate to the final communication even if the LLM prompt is well-designed.
  • The approach could be extended to closed-loop control by re-running the pipeline after each re-plan, although latency may need to be reduced through lighter models or parallel simulation.
  • The same visual-overlay-plus-questions pattern may transfer to instruction generation for users with limited vision, where audio descriptions generated by this pipeline could serve as assistive feedback.
  • A natural next step would be bidirectional communication, letting the user ask for modifications, since CoRI currently only speaks from robot to human.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces CoRI, a pipeline that converts a robot's camera image and a planned 3D waypoint trajectory (with velocity and force information) into natural-language intent statements for physical human-robot interaction. The method detects body landmarks, segments the trajectory at gripper changes, force onsets/terminations, and pauses, then overlays the trajectory and its dynamics onto the image. A VLM (GPT-4o) answers structured questions about position, velocity, force, overall intention, and required user cooperation, and a reasoning LLM (o3-mini) turns these answers into one concise sentence per segment. The authors evaluate CoRI on three assistive tasks (bathing, shaving, feeding) on two robot platforms. A user study with 16 participants compares CoRI to a scripted landmark-based baseline and a no-communication condition on six Likert items, reporting significant improvements (p < 0.01) for all pairwise comparisons. The paper also reports an entailment-based evaluation against author-written ground-truth paragraphs, where CoRI scores close to oracle summaries (≈0.95 vs ≈0.96–0.98).

Significance. If the results hold, CoRI would be a useful contribution: it is task- and robot-agnostic, uses only perception and a planned trajectory, and can generate user-directed statements about intent, motion dynamics, and needed cooperation without task-specific labels. The authors provide code and video, and the pipeline is evaluated across multiple tasks and robot platforms. The user study is a genuine attempt to measure user comprehension rather than only automatic text quality. However, the strength of the central claim depends on whether the objective faithfulness evaluation is trustworthy and whether the subjective ratings isolate factual correctness from verbosity and naturalness. The entailment metric shows calibration problems, and the baseline condition is minimal, so the evidence for the stronger claims about accurate conveyance of motion details and user cooperation is currently weaker than the abstract suggests.

major comments (3)
  1. [§6.2, Table 1] The entailment evaluation is presented as the objective check that CoRI's statements are faithful to the robot's plan, but the metric's behavior is not calibrated in this domain. A scripted baseline such as 'I'm moving towards your left elbow' should be entailed with near-certainty by the corresponding ground-truth segment, yet Table 1 reports mean entailment probabilities of 0.70 (bathing) and 0.69 (feeding). These values are far from the near-1 scores expected for a true statement, which indicates that the RoBERTa-MNLI model is not functioning as a reliable measure of 'contains no false information' here. Consequently, the near-oracle CoRI scores (0.95 ± 0.03 to 0.95 ± 0.08) do not establish that the VLM is correctly reading the visual overlay. Please add a calibration check (e.g., the baseline script and oracle summaries should also score near 1), or augment the evaluation with human factual-correctness annotations against the executed trajectory.
  2. [§5.2 and §6.1] The user study compares CoRI only against a minimal scripted baseline that says 'I'm moving towards your <LANDMARK>'. CoRI's statements are longer, more natural, and contain more information, so the significantly higher Likert ratings on L3–L6 may reflect information quantity, verbosity, or perceived naturalness rather than factual accuracy of the communicated intent and motion details. This confound matters because the paper's central claim is that CoRI effectively conveys 'crucial details about its motion and any collaborative user action needed', a claim that currently rests on subjective Likert ratings and on the flawed entailment metric. Please add a control condition that matches CoRI in information content and naturalness but varies factual grounding, or separately collect human correctness judgments per statement against the actual planned motion.
  3. [§6.1] The paper reports eight pairwise Wilcoxon signed-rank tests on the same participant responses and states only that all are p < 0.01, without correction for multiple comparisons and without exact p-values or effect sizes. With n = 16, the 'statistically significant difference' claim is underspecified if the threshold is the conventional 0.05 after a Bonferroni or Holm correction. Please report exact p-values, effect sizes, and whether the 0.01 threshold survives multiplicity correction; alternatively, justify why correction is not needed.
minor comments (3)
  1. [§4.1] The pause criterion in the segmentation equation uses exact equality pi = pi+1, which may fail in practice if waypoint positions contain small numerical noise or tiny movements during a pause; a distance threshold would be more robust.
  2. [Figure 4] The significance asterisks in Figure 4 are not keyed to exact p-value ranges in the caption, and the text reports varying thresholds (p < 0.01, p < 0.001). Please make the mapping between asterisks and p-values explicit.
  3. [Appendix B] The ground-truth paragraphs and their one-sentence summaries are author-written and used as the reference for the entailment metric. The paper would benefit from stating whether these references were validated by any second rater or an inter-annotator agreement check, since the metric cannot separate errors in the reference from errors in the generated text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central user-study claim rests on independent human ratings, and no load-bearing step reduces to its own inputs by construction.

full rationale

The paper's central claim is that CoRI, a pipeline taking only an image observation and a planned 3D trajectory, generates natural-language intent communication that users rate as clearer than scripted or no communication. The evaluation that supports this claim is a within-subject user study with 16 participants across three tasks and two robots, using pairwise Wilcoxon signed-rank tests on seven Likert items. That evidence is independent of the authors' own generation pipeline: participant ratings do not reduce to the trajectory inputs or to the VLM/LLM prompts by construction. The secondary entailment evaluation in Section 6.2 compares CoRI statements against author-written 'ground-truth paragraphs' that are themselves derived from the same trajectory plans. This is an internal consistency check rather than a prediction against an independent ground truth, and its evidential value is limited, but it is not circular: the entailment probability is produced by an external NLI model (RoBERTa-large on MNLI), and a CoRI statement could in principle contain information absent from or inconsistent with the oracle paragraph. No parameters are fitted to the evaluation data, and no 'prediction' is a renamed fit. The self-citations in the related work (e.g., RL-VLM-F, Voicepilot) are contextual and not load-bearing: the paper does not invoke any author-specific theorem or uniqueness result to force its design choices. The hand-chosen segmentation thresholds (gripper change, force onset/termination, >2 s pause) and speech-timing rule are design decisions, not hidden dependencies that make the measured outcome equivalent to the inputs. Therefore no specific circular step satisfying the required standard can be identified, and the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the VLM's interpretive ability, the stationarity assumption, the correctness of author-written ground truth, the applicability of an MNLI entailment model, and the completeness of hand-chosen segmentation rules. These are all consequential but stated or implicit in the paper.

free parameters (3)
  • pause threshold = 2 seconds
    In Section 4.1, a pause is defined as a time difference greater than 2 seconds between identical waypoints. This value is hand-chosen, not derived from data.
  • velocity change threshold = 2x
    In Appendix A.2, question 2 defines a notable speed change as a difference greater than two times. This is a hand-chosen criterion for what to mention in the generated statement.
  • speech timing lead factor = 0.5
    In Section 5.1, the statement starts playing at 0.5 times its spoken duration before the corresponding motion begins. This synchronization parameter is hand-chosen.
assumptions (5)
  • domain assumption The pretrained VLM (GPT-4o) can correctly parse the visual overlay (colored waypoints, force gradient, cropped images) and answer structured questions about position, velocity, force, and intention.
    The entire pipeline depends on the VLM's capability to ground trajectory information in the image. If the VLM hallucinates or misreads the overlay, the generated communication will be wrong. This is invoked in Section 4.2.
  • domain assumption The user remains stationary and visible during task execution.
    Stated in Section 3 and in the Limitations. The pipeline assumes no significant user motion; otherwise the trajectory and communication would need to be regenerated.
  • ad hoc to paper The ground-truth paragraphs in Appendix B correctly capture all relevant intent and motion information and contain no false statements.
    The entailment evaluation in Section 6.2 uses these author-written paragraphs as the reference. If they are incomplete or biased, the entailment scores do not measure correctness.
  • domain assumption The MNLI fine-tuned RoBERTa model provides meaningful entailment probabilities for domain-specific robot communication statements.
    Used in Section 6.2 to score statements. The model was not designed for robotics language, so its probabilities may not align perfectly with human judgment.
  • domain assumption The segmentation rules (gripper change, force onset or termination, pause greater than 2 seconds) capture all meaningful interaction events in the evaluated tasks.
    Section 4.1 defines trajectory segments based on these criteria. If a task has interaction events not captured by these rules, the generated statements will not align with user expectations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoRI: Communication of Robot Intent for Physical Human-Robot Interaction." pith.science (2026). https://pith.science/paper/2WWDHCJB

@misc{pith2026250520537,
  author       = {Pith},
  title        = {Pith review of: CoRI: Communication of Robot Intent for Physical Human-Robot Interaction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2WWDHCJB}},
  note         = {Machine review of arXiv:2505.20537}
}
read the original abstract

Clear communication of robot intent fosters transparency and interpretability in physical human-robot interaction (pHRI), particularly during assistive tasks involving direct human-robot contact. We introduce CoRI, a pipeline that automatically generates natural language communication of a robot's upcoming actions directly from its motion plan and visual perception. Our pipeline first processes the robot's image view to identify human poses and key environmental features. It then encodes the planned 3D spatial trajectory (including velocity and force) onto this view, visually grounding the path and its dynamics. CoRI queries a vision-language model with this visual representation to interpret the planned action within the visual context before generating concise, user-directed statements, without relying on task-specific information. Results from a user study involving robot-assisted feeding, bathing, and shaving tasks across two different robots indicate that CoRI leads to statistically significant difference in communication clarity compared to a baseline communication strategy. Specifically, CoRI effectively conveys not only the robot's high-level intentions but also crucial details about its motion and any collaborative user action needed. Video and code of our project can be found on our project website: https://cori-phri.github.io/

Figures

Figures reproduced from arXiv: 2505.20537 by the authors.

Figure 1
Figure 1. Our proposed pipeline, CoRI, generating intent communication during an assisted bathing [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our CoRI pipeline. The pipeline takes as input an image observation [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The three tasks implemented and used in the user study, along with example communica [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Box plots showing distribution of Likert-item responses for each participant, taking the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Example overlay visualization of trajectory 1 in the bathing task, for participant 3. [PITH_FULL_IMAGE:figures/full_fig_p025_5.png]
Figure 6
Figure 6. Figure 6: Example overlay visualization of trajectory 2 in the bathing task, for participant 6. [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]
Figure 7
Figure 7. Figure 7: Example overlay visualization of trajectory 1 in the shaving task, for participant 13. [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]
Figure 8
Figure 8. Figure 8: Example overlay visualization of trajectory 2 in the shaving task, for participant 2. [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]
Figure 9
Figure 9. Figure 9: Example overlay visualization of trajectory 1 in the feeding task, for participant 11. [PITH_FULL_IMAGE:figures/full_fig_p032_9.png]
Figure 10
Figure 10. Figure 10: Example overlay visualization of trajectory 2 in the feeding task, for participant 16. [PITH_FULL_IMAGE:figures/full_fig_p033_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

73 extracted references · 68 canonical work pages

  1. [1]

    Nanavati, V

    A. Nanavati, V . Ranganeni, and M. Cakmak. Physically assistive robots: A systematic review of mobile and manipulator robots that physically assist people with disabilities.Annual Review of Control, Robotics, and Autonomous Systems, 7, 2023

    work page 2023

  2. [2]

    Alhaji, M

    B. Alhaji, M. Prilla, and A. Rausch. Trust dynamics and verbal assurances in human robot physical collaboration. Frontiers in Artificial Intelligence, 4:703504, 2021

    work page 2021

  3. [3]

    Mehrotra, C

    S. Mehrotra, C. Degachi, O. Vereschak, C. M. Jonker, and M. L. Tielman. A systematic review on fostering appropriate trust in human-AI interaction: Trends, opportunities and challenges. ACM Journal on Responsible Computing, 1(4):1–45, 2024

    work page 2024

  4. [4]

    S. Y . Sch¨ott, R. M. Amin, and A. Butz. A literature survey of how to convey transparency in co-located human–robot interaction. Multimodal Technologies and Interaction, 7(3):25, 2023

    work page 2023

  5. [5]

    A. St. Clair and M. Mataric. How robot verbal feedback can improve team performance in human-robot task collaborations. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 213–220, 2015

    work page 2015

  6. [6]

    Fischer, H

    K. Fischer, H. M. Weigelin, and L. Bodenhagen. Increasing trust in human–robot medical interactions: effects of transparency and adaptability.Paladyn, Journal of Behavioral Robotics, 9(1):95–109, 2018

    work page 2018

  7. [7]

    Berg and S

    J. Berg and S. Lu. Review of interfaces for industrial human-robot interaction. Current Robotics Reports, 1(2):27–34, 2020

    work page 2020

  8. [8]

    W. Li, Y . Hu, Y . Zhou, and D. T. Pham. Safe human–robot collaboration for industrial settings: a survey. Journal of Intelligent Manufacturing, 35(5):2235–2261, 2024

    work page 2024

Show all 73 references
  1. [9]

    Huang, Y

    Z. Huang, Y . Shen, J. Li, M. Fey, and C. Brecher. A survey on AI-driven digital twins in industry 4.0: Smart manufacturing and advanced robotics. Sensors, 21(19):6340, 2021

    2021

  2. [10]

    Pascher, U

    M. Pascher, U. Gruenefeld, S. Schneegass, and J. Gerken. How to communicate robot motion intent: A scoping review. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–17, 2023

    2023

  3. [11]

    Domonkos, Z

    M. Domonkos, Z. Dombi, and J. Botzheim. Led strip based robot movement intention signs for human-robot interactions. In Proceedings of the 2020 IEEE 20th International Symposium on Computational Intelligence and Informatics (CINTI), pages 121–126. IEEE, 2020

    2020

  4. [12]

    N. J. Hetherington, E. A. Croft, and H. M. Van der Loos. Hey robot, which way are you going? nonverbal motion legibility cues for human-robot spatial interaction. IEEE Robotics and Automation Letters, 6(3):5010–5015, 2021

    2021

  5. [13]

    Lemasurier, G

    G. Lemasurier, G. Bejerano, V . Albanese, J. Parrillo, H. A. Yanco, N. Amerson, R. Hetrick, and E. Phillips. Methods for expressing robot intent for human–robot collaboration in shared workspaces. ACM Transactions on Human-Robot Interaction (THRI), 10(4):1–27, 2021

    2021

  6. [14]

    R. S. Andersen, O. Madsen, T. B. Moeslund, and H. B. Amor. Projecting robot intentions into human environments. In Proceedings of the 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pages 294–301. IEEE, 2016

    2016

  7. [15]

    Wengefeld, D

    T. Wengefeld, D. H ¨ochemer, B. Lewandowski, M. K ¨ohler, M. Beer, and H.-M. Gross. A laser projection system for robot intention communication and human robot interaction. In Proceedings of the 2020 29th IEEE International Conference on Robot and Human Interactive Communicati...

    2020

  8. [16]

    R. T. Chadalavada, H. Andreasson, M. Schindler, R. Palm, and A. J. Lilienthal. Bi-directional navigation intent communication using spatial augmented reality and eye-tracking glasses for improved safety in human–robot interaction. Robotics and Computer-Integrated Manufactur- i...

    2020

  9. [17]

    M. C. Aubert, H. Bader, and K. Hauser. Designing multimodal intent communication strategies for conflict avoidance in industrial human-robot teams. In Proceedings of the 2018 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pages 1018–1025, 2018

    2018

  10. [18]

    Matsumaru

    T. Matsumaru. Mobile robot with preliminary-announcement and indication function of forth- coming operation using flat-panel display. In Proceedings of the 2007 IEEE International Conference on Robotics and Automation (ICRA), pages 1774–1781. IEEE, 2007

    2007

  11. [19]

    Walker, H

    M. Walker, H. Hedayati, J. Lee, and D. Szafir. Communicating robot motion intent with aug- mented reality. In Proceedings of the 2018 ACM/IEEE International Conference on Human- Robot Interaction (HRI), pages 316–324, 2018

    2018

  12. [20]

    Gruenefeld, L

    U. Gruenefeld, L. Pr ¨adel, J. Illing, T. Stratmann, S. Drolshagen, and M. Pfingsthorn. Mind the ARm: realtime visualization of robot motion intent in head-mounted augmented reality. In Proceedings of Mensch und Computer 2020, pages 259–266, 2020

    2020

  13. [21]

    Tsamis, G

    G. Tsamis, G. Chantziaras, D. Giakoumis, I. Kostavelis, A. Kargakos, A. Tsakiris, and D. Tzo- varas. Intuitive and safe interaction in multi-user human robot collaboration environments through augmented reality displays. In Proceedings of the 2021 30th IEEE international con- ...

    2021

  14. [22]

    M. Gu, A. Cosgun, W. P. Chan, T. Drummond, and E. Croft. Seeing thru walls: Visualiz- ing mobile robots in augmented reality. In Proceedings of the 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN), pages 406–411. IEEE, 2021

    2021

  15. [23]

    Rosen, D

    E. Rosen, D. Whitney, E. Phillips, G. Chien, J. Tompkin, G. Konidaris, and S. Tellex. Commu- nicating and controlling robot arm motion intent through mixed-reality head-mounted displays. The International Journal of Robotics Research, 38(12-13):1513–1526, 2019

    2019

  16. [24]

    J. He, A. van Maris, and P. Caleb-Solly. Investigating the effectiveness of different interac- tion modalities for spatial human-robot interaction. In Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 239–241, 2020

    2020

  17. [25]

    Mikawa, Y

    M. Mikawa, Y . Yoshikawa, and M. Fujisawa. Expression of intention by rotational head move- ments for teleoperated mobile robot. In Proceedings of the 2018 IEEE 15th International Workshop on Advanced Motion Control (AMC), pages 249–254. IEEE, 2018

    2018

  18. [26]

    Lohse, R

    M. Lohse, R. Rothuis, J. Gallego-P ´erez, D. E. Karreman, and V . Evers. Robot gestures make difficult tasks easier: the impact of gestures on perceived workload and task performance. In Proceedings of the SIGCHI conference on human factors in computing systems , pages 1459– 1...

    2014

  19. [27]

    Schreiter, L

    T. Schreiter, L. Morillo-Mendez, R. T. Chadalavada, A. Rudenko, E. Billing, M. Magnusson, K. O. Arras, and A. J. Lilienthal. Advantages of multimodal versus verbal-only robot-to- human communication with an anthropomorphic robotic mock driver. In Proceedings of the 2023 32nd I...

    2023

  20. [28]

    A. D. Dragan, K. C. Lee, and S. S. Srinivasa. Legibility and predictability of robot motion. In Proceedings of the 2013 8th ACM/IEEE International Conference on Human-Robot Interac- tion (HRI), pages 301–308. IEEE, 2013. 12

    2013

  21. [29]

    Dragan and S

    A. Dragan and S. Srinivasa. Generating legible motion. In Proceedings of Robotics: Science and Systems, June 2013

    2013

  22. [30]

    Szafir, B

    D. Szafir, B. Mutlu, and T. Fong. Communication of intent in assistive free flyers. In Proceed- ings of the 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 358–365, 2014

    2014

  23. [31]

    Capelli, C

    B. Capelli, C. Secchi, and L. Sabattini. Communication through motion: Legibility of multi- robot systems. In Proceedings of the 2019 International Symposium on Multi-Robot and Multi- Agent Systems (MRS), pages 126–132. IEEE, 2019

    2019

  24. [32]

    V . V . Unhelkar, S. Li, and J. A. Shah. Decision-making for bidirectional communication in se- quential human-robot collaborative tasks. InProceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, pages 329–341, 2020

    2020

  25. [33]

    Nikolaidis, M

    S. Nikolaidis, M. Kwon, J. Forlizzi, and S. Srinivasa. Planning with verbal communication for human-robot collaboration. ACM Transactions on Human-Robot Interaction (THRI), 7(3): 1–21, 2018

    2018

  26. [34]

    K. M. Lee, A. Krishna, Z. Zaidi, R. Paleja, L. Chen, E. Hedlund-Botti, M. Schrum, and M. Gombolay. The effect of robot skill level and communication in rapid, proximate human- robot collaboration. In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Inte...

    2023

  27. [35]

    S ¨oderlund

    M. S ¨oderlund. Service robot verbalization in service processes with moral implications and its impact on satisfaction. Technological Forecasting and Social Change, 196:122831, 2023

    2023

  28. [36]

    Rosenthal, S

    S. Rosenthal, S. P. Selvaraj, and M. Veloso. Verbalization: narration of autonomous robot experience. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI), pages 862–868, 2016

    2016

  29. [37]

    Canal, S

    G. Canal, S. Krivic, P. Luff, and A. Coles. Task plan verbalizations with causal justifications. In ICAPS 2021 Workshop on Explainable AI Planning (XAIP), 2021

    2021

  30. [38]

    W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, et al. Language to rewards for robotic skill synthesis. In Proceed- ings of the 7th Conference on Robot Learning (CoRL), pages 374–404. PMLR, 2023

    2023

  31. [39]

    Y . J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y . Zhu, L. Fan, and A. Anandkumar. Eureka: Human-level reward design via coding large language models. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR) , 2024

    2024

  32. [40]

    T. Xie, S. Zhao, C. H. Wu, Y . Liu, Q. Luo, V . Zhong, Y . Yang, and T. Yu. Text2reward: Re- ward shaping with language models for reinforcement learning. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024

    2024

  33. [41]

    Y . Wang, Z. Sun, J. Zhang, Z. Xian, E. Biyik, D. Held, and Z. Erickson. RL-VLM-F: Rein- forcement learning from vision language foundation model feedback. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024

    2024

  34. [42]

    Brohan, Y

    A. Brohan, Y . Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as I can, not as I say: Grounding language in robotic affordances. In Proceedings of the 6th Conference on Robot Learning (CoRL), pages 287–318. PMLR, 2022

    2022

  35. [43]

    Huang, P

    W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. InProceedings of the 39th International Conference on Machine Learning (ICML), pages 9118–9147. PMLR, 2022. 13

    2022

  36. [44]

    Singh, V

    I. Singh, V . Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. ProgPrompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 11523– 11530. IEEE, 2023

    2023

  37. [45]

    Zitkovich, T

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. InPro- ceedings of the 7th Conference on Robot Learning (CoRL), pages 2165–2183. PMLR, 2023

    2023

  38. [46]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. OpenVLA: An open-source vision-language-action model. In Proceedings of...

    2024

  39. [47]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024

    2024 arXiv

  40. [48]

    M. J. Kian, M. Zong, K. Fischer, A. Singh, A.-M. Velentza, P. Sang, S. Upadhyay, A. Gupta, M. A. Faruki, W. Browning, et al. Can an LLM-powered socially assistive robot effectively and safely deliver cognitive behavioral therapy? a study with university students. arXiv preprin...

    2024 arXiv

  41. [49]

    Mahmood, J

    A. Mahmood, J. Wang, B. Yao, D. Wang, and C.-M. Huang. User interaction patterns and breakdowns in conversing with LLM-powered voice assistants. International Journal of Human-Computer Studies, 195:103406, 2025

    2025

  42. [50]

    Padmanabha, J

    A. Padmanabha, J. Yuan, J. Gupta, Z. Karachiwalla, C. Majidi, H. Admoni, and Z. Erickson. V oicepilot: Harnessing LLMs as speech interfaces for physically assistive robots. InProceed- ings of the 37th Annual ACM Symposium on User Interface Software and Technology , pages 1–18, 2024

    2024

  43. [51]

    H. Wang, K. Kedia, J. Ren, R. Abdullah, A. Bhardwaj, A. Chao, K. Y . Chen, N. Chin, P. Dan, X. Fan, G. Gonzalez-Pumariega, A. Kompella, M. A. Pace, Y . Sharma, X. Sun, N. Sunkara, and S. Choudhury. Mosaic: A modular system for assistive and interactive cooking. InProceedings o...

    2024

  44. [52]

    Grannen, S

    J. Grannen, S. Karamcheti, B. Wulfe, and D. Sadigh. Provox: Personalization and proactive planning for situated human-robot collaboration. arXiv preprint arXiv:2506.12248, 2025

    2025 arXiv

  45. [53]

    Mandi, S

    Z. Mandi, S. Jain, and S. Song. Roco: Dialectic multi-robot collaboration with large language models. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 286–299. IEEE, 2024

    2024

  46. [54]

    Z. Liu, A. Bahety, and S. Song. REFLECT: Summarizing robot experiences for failure expla- nation and correction. In Proceedings of the 7th Conference on Robot Learning (CoRL), pages 3468–3484. PMLR, 2023

    2023

  47. [55]

    Z. Wang, B. Liang, V . Dhat, Z. Brumbaugh, N. Walker, R. Krishna, and M. Cakmak. I can tell what i am doing: Toward real-world natural language grounding of robot experiences. In Proceedings of the 8th Conference on Robot Learning (CoRL). PMLR, 2024

    2024

  48. [56]

    Lugaresi, J

    C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. Yong, J. Lee, et al. MediaPipe: A framework for perceiving and processing reality. InThird Workshop on Computer Vision for AR/VR at IEEE Computer Vision and Pattern Recognition (CVPR),...

    2019

  49. [57]

    Mahadevan, J

    K. Mahadevan, J. Chien, N. Brown, Z. Xu, C. Parada, F. Xia, A. Zeng, L. Takayama, and D. Sadigh. Generative expressive robot behaviors using large language models. InProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (HRI) , pages 482–491, 2024

    2024

  50. [58]

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  51. [59]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  52. [60]

    Understanding your planned trajectory

    M. Deitke, C. Clark, S. Lee, R. Tripathi, Y . Yang, J. S. Park, M. Salehi, N. Muennighoff, K. Lo, L. Soldaini, et al. Molmo and PixMo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2024. 15 A Example Prompts and Responses G...

    2024 arXiv

  53. [61]

    Answer the following primarily from look- ing at the images, but refer to human body landmarks in the exact name (left shoulder, right elbow, etc.)

    Consider the position of the robot end-effector. Answer the following primarily from look- ing at the images, but refer to human body landmarks in the exact name (left shoulder, right elbow, etc.). When describing left and right body parts, always reference them from the per- ...

  54. [62]

    Recall that a brighter-green waypoint indi- cates higher speed, and darker-green point means lower speed

    Consider the velocity of the robot end-effector. Recall that a brighter-green waypoint indi- cates higher speed, and darker-green point means lower speed. Length of line segments con- necting two waypoints is also directly proportional to the speed. Base your response primaril...

  55. [63]

    ## Segment description

    Consider the force of the robot end effector. Recall that the line segments between two waypoints are colored according to the external forces as the end effector travels through that segment. In a gradient from blue to red, blue means lower force and red means higher force. 3...

  56. [64]

    1b) The red square waypoint is at the left wrist, indicating the end of the motion

    Position Analysis 1a) The blue square waypoint is near the robot’s gripper, starting close to the left wrist. 1b) The red square waypoint is at the left wrist, indicating the end of the motion. 1c) The trajectory is a straight line. 1d) The trajectory ends at the left wrist. 1...

  57. [65]

    2b) The speed near the end is 2.0 cm/s

    Velocity Analysis 2a) The speed near the start is 3.0 cm/s. 2b) The speed near the end is 2.0 cm/s. 2c) The starting speed is the same as the ending speed of the last segment, as this is the first segment. 2d) The ending speed is notably slower than the starting speed. 2e) Not...

  58. [66]

    ## Segment Description The robot moves in a straight line from its initial position towards the person’s left wrist

    Force Analysis 3a) There are no forces involved in this segment. ## Segment Description The robot moves in a straight line from its initial position towards the person’s left wrist. The speed decreases as it approaches the wrist, indicating a careful approach. The trajectory e...

  59. [67]

    1b) The red square waypoint ends near the left elbow

    Position Analysis 1a) The blue square waypoint starts at the left wrist. 1b) The red square waypoint ends near the left elbow. 1c) The trajectory is a straight line. 1d) The trajectory passes close to the left elbow. 1e) The motion is directed towards the left elbow

  60. [68]

    2b) The speed near the end is 1.0 cm/s

    Velocity Analysis 2a) The speed near the start is 1.0 cm/s. 2b) The speed near the end is 1.0 cm/s. 2c) The starting speed is slower than the ending speed of the last segment. 2d) The ending speed is the same as the starting speed in this segment. 2e) There is no notable chang...

  61. [69]

    ## Overall intention

    Force Analysis 3a) Yes, there are forces involved. 3b) The force is applied throughout the motion from the left wrist to the left elbow. 3c) The force gradually increases throughout the segment. 3d) The held towel might be used to apply gentle pressure or cleaning action along...

  62. [70]

    1b) The red square waypoint ends near the left wrist

    Position Analysis 1a) The blue square waypoint starts near the left elbow. 1b) The red square waypoint ends near the left wrist. 1c) The trajectory is a straight line. 1d) The trajectory gets close to the left elbow and ends near the left wrist. 1e) The motion is directed down...

  63. [71]

    2b) The speed near the end is 4.0 cm/s

    Velocity Analysis 2a) The speed near the start is 2.0 cm/s. 2b) The speed near the end is 4.0 cm/s. 2c) The starting speed is slower than the ending speed of the last segment. 2d) The ending speed is notably faster than the starting speed in this segment. 2e) Notable change in...

  64. [72]

    ## Segment Description

    Force Analysis 3a) There are no forces involved in this segment. ## Segment Description The robot moves in a straight line from the left elbow towards the left wrist. The speed increases as it moves downward, indicating a swift motion. No force is applied during this segment, ...

  65. [73]

    an example trajectory, shown in the same overlay fashion as what the VLM reasons on, 2) the ground-truth paragraph for each segment in the trajectory, 3) the one-sentence summary for each paragraph, generated by a reasoning LLM (o3-mini), and 4) sample communications generated...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.