Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

ROSAnnotator: A Web Application for ROSBag Data Analysis in Human-Robot Interaction

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces ROSAnnotator, a web application that brings manual and multimodal-LLM-assisted qualitative coding directly to ROSBag data on synchronized time axes.

desk verdict An honest systems paper that fills a real niche for ROSBag-based qualitative analysis, but its headline efficiency benefit is unvalidated and the authors themselves concede the LLM auto-annotation is not robust. read the letter →

arxiv 2501.07051 v1 pith:YTMHPF2H submitted 2025-01-13 cs.RO cs.HC

Yan Zhang , Haoqi Li , Ramtin Tabatabaei , Wafa Johal This is my paper · ORCID
classification cs.ROcs.HC
keywords human-robotinteractionROSBagqualitativedataanalysisannotationtoolmultimodallargelanguagemodelvideoaudiotranscriptionwebapplication
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ROSBags collect time-synchronized multimodal data in human-robot interaction experiments, but qualitative coding of that data typically happens in separate tools that do not understand the format. The paper argues that a web application can close this gap by parsing ROSBags, replaying the video and audio, transcribing speech, and presenting video, audio, and transcripts on shared annotation time axes. It also embeds a multimodal large language model that can create codebook-based annotations on those time axes from instructions typed in a chatbox, with manual editing afterward. The paper's claim is that this integration streamlines qualitative analysis, creates a more cohesive pipeline, and gives researchers statistical summaries and CSV export for further analysis.

What carries the argument

The central mechanism is the synchronized time axis combined with the multimodal LLM chatbox. The backend parses ROSBags and stores processed video, audio, and transcriptions; the frontend renders a video player and user-created annotation tiers on a shared timeline. The multimodal LLM (a model that can take video, audio, and text as input) receives user instructions and produces annotations with time intervals, which users can then edit on the time axes. A supporting mechanism is the local face-removal algorithm that filters frames before external upload.

What would settle it

A controlled evaluation in which two independent coders annotate the same set of ROSBags manually and with ROSAnnotator's auto-annotation, comparing inter-rater agreement and total time spent; if the LLM-generated codes agree with manual codes only at chance level, or if correcting them takes longer than manual coding from scratch, the central efficiency claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a working system can be built that treats ROSBag files as the hub of qualitative HRI analysis, rather than forcing researchers to extract recordings and code them in separate tools. ROSAnnotator extracts video, audio, and transcriptions, shows them with a synchronized video player and multiple user-defined codebook tiers, and—the novel part—lets a multimodal large language model create annotation intervals directly on those tiers after receiving instructions in a chatbox. The authors also add a local face-detection step that removes frames containing human faces before video is sent to the external model, and an open backend interface that users can extend for custom robot messages.

Load-bearing premise

The load-bearing premise is that the multimodal LLM's automated annotations are accurate enough to reduce, not increase, total human coding effort; the paper itself concedes that the automated annotation is not robust.

Editorial extensions

If this is right

  • An HRI researcher can go from a raw ROSBag to coded annotation tiers without switching between separate video, audio, and transcript tools.
  • Automated transcription places each speaker's utterances on its own tier, so verbal, gestural, and emotional coding can be done against the same synchronized timeline.
  • The multimodal LLM produces draft annotations on time axes that users can drag, edit, or delete, making automation an assist rather than a final judgment.
  • Per-tier statistics such as counts, durations, and time occupancy, together with CSV export, give researchers a direct route to reliability checks and dataset labeling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a controlled study comparing manual-only coding with manual-plus-auto coding on the same ROSBags would test the efficiency claim; the paper reports no such evaluation.
  • Beyond the paper, the synchronized time-axis and codebook design is not robot-specific and could generalize to other multimodal corpora, such as classroom or clinical interaction recordings.
  • Beyond the paper, the privacy protection is partial: face removal happens before video is sent to the external model, but audio transcription still depends on an external service, so fully private processing would require local transcription and local model hosting.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents ROSAnnotator, an open-source web application for analyzing ROSBag data in human-robot interaction (HRI) qualitative research. The application extracts video, audio, and transcription messages from ROSBags, displays them on synchronized time axes, supports manual codebook-based annotation, provides multimodal-LLM-based automated annotation with a privacy-preserving face-removal step, computes statistical summaries of annotations, and exports annotations and statistics to CSV. The authors motivate the tool by pointing to a lack of tools that integrate qualitative coding with ROSBag data and claim that ROSAnnotator streamlines the qualitative analysis process and increases the efficiency of HRI data analysis.

Significance. If the tool works as described, it would fill a genuine practical gap: HRI researchers often collect synchronized multimodal data in ROSBags, but the standard qualitative annotation tools (ELAN, ATLAS.ti) do not integrate with that format. The paper's strengths are its concrete feature descriptions, its open-source repository link, and the fact that the core workflow (import, synchronized playback, tiered manual annotation, export) is plausible and directly relevant to the community. The principal weakness is that the central benefit claim—efficiency gains from automated multimodal-LLM annotation—is not evaluated in any way, and the manuscript's own limitation section concedes that automated annotation 'is not robust.' There is no derivation or quantitative claim whose correctness could be checked, so the concern is not circularity but missing evidence for the paper's advertised contribution.

major comments (3)
  1. [Abstract; §VI; §VII] The abstract and §VII state that ROSAnnotator 'streamline[s] the qualitative analysis process' and 'enhanc[es] the overall efficiency of HRI data analysis,' but no evidence is provided for these claims. There is no user study, no inter-rater reliability or ground-truth comparison for LLM-generated codes, and no measurement of annotation time or effort relative to manual coding in ELAN/ATLAS.ti or in ROSAnnotator itself. §VI explicitly concedes that 'due to the constraints of the multimodal LLM's capabilities, the performance of the automated annotation is not robust.' Since the automated annotation is the only mechanism that differentiates the tool from existing manual-coding workflows, the headline efficiency benefit is currently an assertion. Please either add an evaluation of annotation quality and time savings or reframe the claims as potential benefits awaiting validation.
  2. [§II Automated Annotation; §IV] The privacy-preserving preprocessing step is described only as 'a local algorithm has been implemented to detect and remove frames containing human faces before the video data is uploaded to the LLM.' It is not specified whether faces are masked in place or whether the entire frames are dropped. If frames are dropped, the multimodal LLM cannot see facial expressions, gaze, or other face-based cues that are central to many HRI qualitative coding schemes, including the warmth ratings mentioned in §V. Please clarify the behavior and, if frames are dropped, assess and disclose how this affects the availability of video evidence for automated annotation.
  3. [§III Installation; §IV Code Architecture] The automated-annotation component is not reproducible from the paper alone. §III requires an OpenAI API key and a Hugging Face token, but no default multimodal model name, video sampling policy, or prompt template is specified; §IV only says that users 'can modify these prompts or switch to a different model if necessary.' Since the central efficiency claim depends on this component, the paper should document the default model identifier and configuration (or state clearly that these are user-supplied choices), so that the automated-annotation behavior can be independently reproduced and evaluated.
minor comments (5)
  1. [Figure 1 caption] The caption spells the tool name as 'ROSAnnotation' while the paper consistently uses 'ROSAnnotator'; please align the spelling.
  2. [§II Data Visualisation] The phrase 'a toolbar *see Figure 2b and c)' appears to have an omitted separator or formatting marker; it should read 'a toolbar (see Figure 2b and c)'.
  3. [§II Data Import] No JSON schema or example codebook is provided; a small example in the paper or appendix would help users prepare a valid codebook before importing data.
  4. [§II Statistical Summary; Table I] The definition of 'Latency' in Table I ('the time delay before the first annotation occurs') is ambiguous: please specify whether this is measured from the start of the ROSBag, from the start of the selected observation window, and whether it is per tier or per observation session.
  5. [§VI Limitations and Future Works] The text says 'we will open-source ROSAnnotator,' while the abstract already provides a public GitHub URL; please make the release status consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: ROSAnnotator is a systems paper with no fitted parameters, equations, or predictive claims, so no circular loop can form.

full rationale

This paper presents a software tool rather than a derivation. It contains no equations, no fitted parameters, no quantitative predictions, and no uniqueness theorems. The claimed contribution is a web application that parses ROSBags, synchronizes modalities, supports manual and LLM-assisted annotation, and exports statistics. None of these claims reduce to their own inputs by construction: the LLM auto-annotation is a feature invoked through an external commercial API, not a quantity derived from the paper's own definitions. The efficiency claim in the abstract ('streamline the qualitative analysis process ... enhancing the overall efficiency of HRI data analysis') is an assertion, not a prediction, and it is not backed by evaluation. Section VI explicitly concedes that 'due to the constraints of the multimodal LLM's capabilities, the performance of the automated annotation is not robust,' and there is no inter-rater reliability, ground-truth comparison, or user study. That missing evidence is a correctness and validity risk, but it is not circular reasoning: the tool's functionality is not defined in terms of the efficiency benefit, and no fitted input is later relabeled as an output. The references to prior annotation tools (ELAN, ATLAS.ti) and prior HRI datasets are contextual, not load-bearing self-citations. The authors' own limitation statement is flagged and weighed here: it undermines the strength of the efficiency claim, but it does not indicate that any claimed result is equivalent to its inputs by construction. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This is a software description paper, so the ledger contains no fitted parameters and no invented entities. The central claim rests on domain assumptions about the deployment environment (Docker, ROS topic conventions, cloud APIs) and on the unvalidated behavior of the LLM annotation feature. The paper itself flags the weakest of these in its limitations section, which I mirror in the axiom list.

assumptions (3)
  • domain assumption ROSBags used with the tool contain the default topics /image_raw and /audio, or users edit topic names in data_util.py.
    Section IV states that the default functions support video and audio extraction from these topics and that users must modify the topic names to match their ROSBags; the out-of-box behavior depends on this convention.
  • ad hoc to paper The multimodal LLM produces annotation codes of sufficient quality for qualitative analysis.
    The automated annotation feature is the paper's headline novelty, yet Section VI concedes that 'the performance of the automated annotation is not robust,' and no evaluation or ground-truth comparison is provided anywhere in the paper.
  • domain assumption Commercial cloud services (OpenAI API, HuggingFace) remain accessible and affordable.
    Section III lists the required OpenAI API key and HuggingFace access token; the LLM annotation and audio transcription features depend on these external paid services.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ROSAnnotator: A Web Application for ROSBag Data Analysis in Human-Robot Interaction." pith.science (2026). https://pith.science/paper/YTMHPF2H

@misc{pith2026250107051,
  author       = {Pith},
  title        = {Pith review of: ROSAnnotator: A Web Application for ROSBag Data Analysis in Human-Robot Interaction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YTMHPF2H}},
  note         = {Machine review of arXiv:2501.07051}
}
read the original abstract

Human-robot interaction (HRI) is an interdisciplinary field that utilises both quantitative and qualitative methods. While ROSBags, a file format within the Robot Operating System (ROS), offer an efficient means of collecting temporally synched multimodal data in empirical studies with real robots, there is a lack of tools specifically designed to integrate qualitative coding and analysis functions with ROSBags. To address this gap, we developed ROSAnnotator, a web-based application that incorporates a multimodal Large Language Model (LLM) to support both manual and automated annotation of ROSBag data. ROSAnnotator currently facilitates video, audio, and transcription annotations and provides an open interface for custom ROS messages and tools. By using ROSAnnotator, researchers can streamline the qualitative analysis process, create a more cohesive analysis pipeline, and quickly access statistical summaries of annotations, thereby enhancing the overall efficiency of HRI data analysis. https://github.com/CHRI-Lab/ROSAnnotator

Figures

Figures reproduced from arXiv: 2501.07051 by the authors.

Figure 1
Figure 1. This figure illustrates the workflow of ROSAnnotation. (a) The process begins with users collecting ROSBags during an empirical study. (b) Once the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Code structure of the ROSAnnotator WebApp [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 2
Figure 2. The interface of ROSAnnotator WebApp. (a) shows the import page, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Conversation to Orchestration: HCI Challenges and Opportunities in Interactive Multi-Agentic Systems

    cs.HC 2025-06 conditional novelty 5.0 of 10

    A position paper identifies six HCI design challenges for end-user interaction with hierarchical multi-agentic systems and proposes a research agenda.

Reference graph

Works this paper leans on

20 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qualitative research in hri: A review and taxonomy,

    L. Veling and C. McGinn, “Qualitative research in hri: A review and taxonomy,” International Journal of Social Robotics , vol. 13, pp. 1689– 1709, 2021

    work page 2021

  2. [2]

    Review of human studies methods in hri and recommendations,

    C. L. Bethel and R. R. Murphy, “Review of human studies methods in hri and recommendations,” International Journal of Social Robotics , vol. 2, no. 4, pp. 347–359, 2010

    work page 2010

  3. [3]

    Ros: an open-source robot operating system,

    M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, A. Y . Ng, et al. , “Ros: an open-source robot operating system,” in ICRA workshop on open source software , vol. 3, p. 5, Kobe, Japan, 2009

    2009

  4. [4]

    Hritk: the human-robot interaction toolkit rapid de- velopment of speech-centric interactive systems in ros,

    I. Lane, V . Prasad, G. Sinha, A. Umuhoza, S. Luo, A. Chandrashekaran, and A. Raux, “Hritk: the human-robot interaction toolkit rapid de- velopment of speech-centric interactive systems in ros,” in NAACL- HLT Workshop on Future directions and needs in the Spoken Dialog Community: Tools and Data (SDCTD 2012) , pp. 41–44, 2012

    work page 2012

  5. [5]

    Ros for human-robot interaction,

    Y . Mohamed and S. Lemaignan, “Ros for human-robot interaction,” in 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 3020–3027, IEEE, 2021

    work page 2021

  6. [6]

    Participant observation and field notes,

    U. Papen, “Participant observation and field notes,” in The Routledge handbook of linguistic ethnography , pp. 141–153, Routledge, 2019

    work page 2019

  7. [7]

    Dynamics of collective group affect: Group-level anno- tations and the multimodal modeling of convergence and divergence,

    N. R. Prabhu, M. Tsfasman, C. Oertel, T. Gerkmann, and N. Lehmann- Willenbrock, “Dynamics of collective group affect: Group-level anno- tations and the multimodal modeling of convergence and divergence,” arXiv preprint arXiv:2409.08578 , 2024

    arXiv 2024

  8. [8]

    Elan: A professional framework for multimodality research,

    P. Wittenburg, H. Brugman, A. Russel, A. Klassmann, and H. Sloetjes, “Elan: A professional framework for multimodality research,” in 5th international conference on language resources and evaluation (LREC 2006), pp. 1556–1559, 2006

    work page 2006

Show all 20 references
  1. [9]

    Atlas/ti—a prototype for the support of text interpretation,

    T. Muhr, “Atlas/ti—a prototype for the support of text interpretation,” Qualitative sociology, vol. 14, no. 4, pp. 349–371, 1991

    1991

  2. [10]

    Interrater reliability: the kappa statistic,

    M. L. McHugh, “Interrater reliability: the kappa statistic,” Biochemia medica, vol. 22, no. 3, pp. 276–282, 2012

    2012

  3. [11]

    Toward the markerless and automatic analysis of kinematic features: A toolkit for gesture and movement research,

    J. P. Trujillo, J. Vaitonyte, I. Simanova, and A. ¨Ozy¨urek, “Toward the markerless and automatic analysis of kinematic features: A toolkit for gesture and movement research,” Behavior Research Methods , vol. 51, pp. 769–777, 2019

    2019

  4. [12]

    A survey on facial emotion recognition techniques: A state-of-the-art literature review,

    F. Z. Canal, T. R. M ¨uller, J. C. Matias, G. G. Scotton, A. R. de Sa Junior, E. Pozzebon, and A. C. Sobieranski, “A survey on facial emotion recognition techniques: A state-of-the-art literature review,” Information Sciences, vol. 582, pp. 593–617, 2022

    2022

  5. [13]

    Supporting qualitative analysis with large language models: Combining codebook with gpt-3 for deductive coding,

    Z. Xiao, X. Yuan, Q. V . Liao, R. Abdelghani, and P.-Y . Oudeyer, “Supporting qualitative analysis with large language models: Combining codebook with gpt-3 for deductive coding,” in Companion proceedings of the 28th international conference on intelligent user interfaces , pp...

    2023

  6. [14]

    Docker: lightweight linux containers for consistent development and deployment,

    D. Merkel et al. , “Docker: lightweight linux containers for consistent development and deployment,” Linux j, vol. 239, no. 2, p. 2, 2014

    2014

  7. [15]

    Gackenheimer, Introduction to React

    C. Gackenheimer, Introduction to React . Apress, 2015

    2015

  8. [16]

    Forcier, P

    J. Forcier, P. Bissex, and W. J. Chun, Python web development with Django. Addison-Wesley Professional, 2008

    2008

  9. [17]

    See what i see: Enabling user-centric robotic assistance using first-person demonstrations,

    Y . Wang, G. Ajaykumar, and C.-M. Huang, “See what i see: Enabling user-centric robotic assistance using first-person demonstrations,” inPro- ceedings of the 2020 ACM/IEEE International Conference on Human- Robot Interaction, pp. 639–648, 2020

    2020

  10. [18]

    From human-human to human-robot: how social psychology research methods can inform hri evaluation,

    I. Kuyucu, A. Dogan, S. Akay, S. C. Bagci, and J. Kanero, “From human-human to human-robot: how social psychology research methods can inform hri evaluation,” in Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction , pp. 637–640, 2024

    2024

  11. [19]

    Human mobile robot interaction in the retail environment,

    Y . Chen, Y . Luo, C. Yang, M. O. Yerebakan, S. Hao, N. Grimaldi, S. Li, R. Hayes, and B. Hu, “Human mobile robot interaction in the retail environment,” Scientific Data, vol. 9, no. 1, p. 673, 2022

    2022

  12. [20]

    The vernissage corpus: A conversational human-robot-interaction dataset,

    D. B. Jayagopi, S. Sheiki, D. Klotz, J. Wienke, J.-M. Odobez, S. Wrede, V . Khalidov, L. Nyugen, B. Wrede, and D. Gatica-Perez, “The vernissage corpus: A conversational human-robot-interaction dataset,” in 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (...

    2013

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.