{"id":"dc61141a-4cf8-472d-ba75-a4465f72c503","arxiv_id":"2501.07051","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ROSAnnotator is an open-source web application that combines manual and LLM-assisted annotation of multimodal ROSBag recordings for qualitative human-robot interaction research.","lead":"This paper presents ROSAnnotator, a web application for labeling video, audio, and transcripts recorded from robot experiments, supporting both manual coding and AI-assisted annotation. It targets a practical gap in human-robot interaction research: qualitative analysis of multimodal data stored in the ROSBag format.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The efficiency claim rests on unvalidated LLM auto-annotation; Section VI concedes it is 'not robust,' yet no evaluation measures accuracy or time savings.","rationale":"The reader and I identify the same load-bearing weak point: the tool's stated purpose is to streamline qualitative analysis, and the only mechanism distinguishing it from existing annotation software is the multimodal-LLM automated annotation. That mechanism is explicitly conceded to be 'not robust' in Section VI and is never evaluated. I considered whether the more basic concern is reproducibility of the Docker build; while that is relevant to the existence claim, it is secondary because the paper's own contribution statement is about efficiency, and a non-running build would be a packaging defect rather than an invalidation of the design. The concrete test I propose measures both reliability and time savings; either metric could falsify the efficiency claim. I therefore keep the reader's CONDITIONAL verdict: the tool may be useful, but the headline benefit cannot be accepted without evaluation.","tokens_in":6122,"tokens_out":3725,"duration_ms":40648,"concrete_test":"Run one controlled comparison on a realistic ROSBag HRI session: have two trained coders manually annotate a fixed 10-minute video using the codebook, and have the same coders correct ROSAnnotator's LLM-generated annotations. Measure (1) Cohen's kappa between the LLM output and the corrected gold standard, and (2) total annotation time per minute of data under manual versus LLM-assisted conditions. If kappa is below 0.6 or LLM-assisted correction time does not fall below manual coding time, the efficiency claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's advertised benefit—that researchers can 'streamline the qualitative analysis process' and 'enhance the overall efficiency of HRI data analysis'—depends on the automated multimodal-LLM annotation producing usable codes. This is never measured. There is no inter-rater reliability analysis, no ground-truth comparison, no user study, and no benchmark against manual coding in tools such as ELAN or ATLAS.ti. Section VI explicitly states: 'due to the constraints of the multimodal LLM's capabilities, the performance of the automated annotation is not robust.' If LLM-generated codes require extensive manual correction, the main differentiator of ROSAnnotator collapses into a convenience wrapper around existing manual-coding workflows. The manual-annotation, import, and export features may be functional, but they do not by themselves support the efficiency claim. This is not merely a missing optional comparison: the authors assert the efficiency conclusion as a factual benefit while their own limitation section undermines the only novel mechanism that could deliver it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ROSAnnotator, an open-source web application for analyzing ROSBag data in human-robot interaction (HRI) qualitative research. The application extracts video, audio, and transcription messages from ROSBags, displays them on synchronized time axes, supports manual codebook-based annotation, provides multimodal-LLM-based automated annotation with a privacy-preserving face-removal step, computes statistical summaries of annotations, and exports annotations and statistics to CSV. The authors motivate the tool by pointing to a lack of tools that integrate qualitative coding with ROSBag data and claim that ROSAnnotator streamlines the qualitative analysis process and increases the efficiency of HRI data analysis.","tokens_in":6193,"tokens_out":4493,"duration_ms":47974,"significance":"If the tool works as described, it would fill a genuine practical gap: HRI researchers often collect synchronized multimodal data in ROSBags, but the standard qualitative annotation tools (ELAN, ATLAS.ti) do not integrate with that format. The paper's strengths are its concrete feature descriptions, its open-source repository link, and the fact that the core workflow (import, synchronized playback, tiered manual annotation, export) is plausible and directly relevant to the community. The principal weakness is that the central benefit claim—efficiency gains from automated multimodal-LLM annotation—is not evaluated in any way, and the manuscript's own limitation section concedes that automated annotation 'is not robust.' There is no derivation or quantitative claim whose correctness could be checked, so the concern is not circularity but missing evidence for the paper's advertised contribution.","major_comments":[{"comment":"The abstract and §VII state that ROSAnnotator 'streamline[s] the qualitative analysis process' and 'enhanc[es] the overall efficiency of HRI data analysis,' but no evidence is provided for these claims. There is no user study, no inter-rater reliability or ground-truth comparison for LLM-generated codes, and no measurement of annotation time or effort relative to manual coding in ELAN/ATLAS.ti or in ROSAnnotator itself. §VI explicitly concedes that 'due to the constraints of the multimodal LLM's capabilities, the performance of the automated annotation is not robust.' Since the automated annotation is the only mechanism that differentiates the tool from existing manual-coding workflows, the headline efficiency benefit is currently an assertion. Please either add an evaluation of annotation quality and time savings or reframe the claims as potential benefits awaiting validation.","section":"Abstract; §VI; §VII"},{"comment":"The privacy-preserving preprocessing step is described only as 'a local algorithm has been implemented to detect and remove frames containing human faces before the video data is uploaded to the LLM.' It is not specified whether faces are masked in place or whether the entire frames are dropped. If frames are dropped, the multimodal LLM cannot see facial expressions, gaze, or other face-based cues that are central to many HRI qualitative coding schemes, including the warmth ratings mentioned in §V. Please clarify the behavior and, if frames are dropped, assess and disclose how this affects the availability of video evidence for automated annotation.","section":"§II Automated Annotation; §IV"},{"comment":"The automated-annotation component is not reproducible from the paper alone. §III requires an OpenAI API key and a Hugging Face token, but no default multimodal model name, video sampling policy, or prompt template is specified; §IV only says that users 'can modify these prompts or switch to a different model if necessary.' Since the central efficiency claim depends on this component, the paper should document the default model identifier and configuration (or state clearly that these are user-supplied choices), so that the automated-annotation behavior can be independently reproduced and evaluated.","section":"§III Installation; §IV Code Architecture"}],"minor_comments":[{"comment":"The caption spells the tool name as 'ROSAnnotation' while the paper consistently uses 'ROSAnnotator'; please align the spelling.","section":"Figure 1 caption"},{"comment":"The phrase 'a toolbar *see Figure 2b and c)' appears to have an omitted separator or formatting marker; it should read 'a toolbar (see Figure 2b and c)'.","section":"§II Data Visualisation"},{"comment":"No JSON schema or example codebook is provided; a small example in the paper or appendix would help users prepare a valid codebook before importing data.","section":"§II Data Import"},{"comment":"The definition of 'Latency' in Table I ('the time delay before the first annotation occurs') is ambiguous: please specify whether this is measured from the start of the ROSBag, from the start of the selected observation window, and whether it is per tier or per observation session.","section":"§II Statistical Summary; Table I"},{"comment":"The text says 'we will open-source ROSAnnotator,' while the abstract already provides a public GitHub URL; please make the release status consistent.","section":"§VI Limitations and Future Works"}],"recommendation":"major_revision","confidential_remarks":"The paper is best read as a systems/demonstration contribution, not as a validated tool study. For a tools venue, the missing evaluation is substantial because the abstract's main benefit is the efficiency of automated annotation, and Section VI concedes that this feature is not robust. I would require either a modest evaluation (e.g., a case study with a real ROSBag, inter-rater agreement or time-to-annotate comparison) or a clear revision that narrows the claims to the manual workflow plus a preliminary automated-annotation capability. The reviewer was unable to inspect the linked repository during review; the authors should ensure the public repository matches the paper's described feature set."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ROSAnnotator is a genuinely useful systems contribution: a web app that parses ROSBags, syncs video/audio/transcripts, and gives you tier-based manual coding plus LLM-assisted auto-annotation, all open-source. The niche is real—ROSBags are standard in HRI, but qualitative analysis of them still means exporting to ELAN or doing it in spreadsheets.\n\nWhat's new is the integration. ELAN handles tiers, ROS4HRI handles conventions, and prior work applies LLMs to deductive coding, but I don't know of another tool that puts all three in one web interface with a chatbox that writes to annotation tiers. The architecture is clean (React/Django, Docker), and the little touches, like removing face frames before sending video to the LLM, show practical design thinking.\n\nThe soft spot is exactly what the stress-test flags. The abstract promises that researchers 'can streamline the qualitative analysis process' and 'enhance the overall efficiency,' but no measurement supports that. There is no user study, no comparison against ELAN or manual coding, no inter-rater reliability, not even a single worked example with a small annotated dataset. Section VI concedes that 'the performance of the automated annotation is not robust' due to multimodal LLM limitations. That concession is honest but damaging: the one novel mechanism that could justify the efficiency claim is the auto-annotation, and the authors themselves say it doesn't work reliably. Without that, the tool is a convenience wrapper around standard practice.\n\nI want to be clear about what this does and doesn't mean. As a systems description, the paper is plausible and the repo is verifiable. I didn't run the code, but the installation steps are concrete. As an evidence-based claim about efficiency, it is unsupported. The paper's framing in the conclusion is more measured than the abstract; the limitations section is candid. So CONDITIONAL is the right call, not rejection.\n\nThe fix is straightforward: give a small group of annotators the same ROSBag, compare time and agreement with ELAN-plus-spreadsheet, and publish an example annotation file. That would convert this from a promising demo to a real tool.\n\nWho is this for? HRI researchers who do qualitative coding on robot logs and want a single interface. It deserves a serious referee; I'd send it to review with a request for evaluation, or accept it as a systems/demo paper with the limitations clearly framed. Don't desk-reject.","headline":"An honest systems paper that fills a real niche for ROSBag-based qualitative analysis, but its headline efficiency benefit is unvalidated and the authors themselves concede the LLM auto-annotation is not robust.","tokens_in":6798,"tokens_out":3037,"would_cite":true,"duration_ms":28786,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces ROSAnnotator, a web application that brings manual and multimodal-LLM-assisted qualitative coding directly to ROSBag data on synchronized time axes.","keywords":["human-robot interaction","ROSBag","qualitative data analysis","annotation tool","multimodal large language model","video annotation","audio transcription","web application"],"falsifier":"A controlled evaluation in which two independent coders annotate the same set of ROSBags manually and with ROSAnnotator's auto-annotation, comparing inter-rater agreement and total time spent; if the LLM-generated codes agree with manual codes only at chance level, or if correcting them takes longer than manual coding from scratch, the central efficiency claim fails.","tokens_in":5842,"feed_emoji":"🤖","tokens_out":9411,"duration_ms":79031,"temperature":0.7,"pith_summary":"ROSBags collect time-synchronized multimodal data in human-robot interaction experiments, but qualitative coding of that data typically happens in separate tools that do not understand the format. The paper argues that a web application can close this gap by parsing ROSBags, replaying the video and audio, transcribing speech, and presenting video, audio, and transcripts on shared annotation time axes. It also embeds a multimodal large language model that can create codebook-based annotations on those time axes from instructions typed in a chatbox, with manual editing afterward. The paper's claim is that this integration streamlines qualitative analysis, creates a more cohesive pipeline, and gives researchers statistical summaries and CSV export for further analysis.","feed_headline":"One web app handles ROSBag video, audio, and transcript coding","feed_subtitle":"It puts video, audio, and transcripts on one timeline, with LLM-generated auto-codes to trim manual coding time.","key_machinery":"The central mechanism is the synchronized time axis combined with the multimodal LLM chatbox. The backend parses ROSBags and stores processed video, audio, and transcriptions; the frontend renders a video player and user-created annotation tiers on a shared timeline. The multimodal LLM (a model that can take video, audio, and text as input) receives user instructions and produces annotations with time intervals, which users can then edit on the time axes. A supporting mechanism is the local face-removal algorithm that filters frames before external upload.","core_discovery":"On the paper's own terms, the central discovery is that a working system can be built that treats ROSBag files as the hub of qualitative HRI analysis, rather than forcing researchers to extract recordings and code them in separate tools. ROSAnnotator extracts video, audio, and transcriptions, shows them with a synchronized video player and multiple user-defined codebook tiers, and—the novel part—lets a multimodal large language model create annotation intervals directly on those tiers after receiving instructions in a chatbox. The authors also add a local face-detection step that removes frames containing human faces before video is sent to the external model, and an open backend interface that users can extend for custom robot messages.","pith_inferences":["Beyond the paper, a controlled study comparing manual-only coding with manual-plus-auto coding on the same ROSBags would test the efficiency claim; the paper reports no such evaluation.","Beyond the paper, the synchronized time-axis and codebook design is not robot-specific and could generalize to other multimodal corpora, such as classroom or clinical interaction recordings.","Beyond the paper, the privacy protection is partial: face removal happens before video is sent to the external model, but audio transcription still depends on an external service, so fully private processing would require local transcription and local model hosting."],"forward_implications":["An HRI researcher can go from a raw ROSBag to coded annotation tiers without switching between separate video, audio, and transcript tools.","Automated transcription places each speaker's utterances on its own tier, so verbal, gestural, and emotional coding can be done against the same synchronized timeline.","The multimodal LLM produces draft annotations on time axes that users can drag, edit, or delete, making automation an assist rather than a final judgment.","Per-tier statistics such as counts, durations, and time occupancy, together with CSV export, give researchers a direct route to reliability checks and dataset labeling."],"supporting_citations":[{"why":"Establishes the importance of qualitative methods in HRI, motivating the need for annotation tools.","marker":"[1]"},{"why":"Defines ROSBag as the standard format for recording synchronized multimodal robot data.","marker":"[3]"},{"why":"Represents prior open-source ROS tooling for HRI data that the paper's analysis-gap argument builds on.","marker":"[5]"},{"why":"Names the established manual annotation tool whose workflow ROSAnnotator aims to subsume.","marker":"[8]"},{"why":"Names the established qualitative-analysis software that motivates the need for a ROSBag-native alternative.","marker":"[9]"},{"why":"Shows that codebook-based deductive coding can be automated with a large language model, the capability ROSAnnotator extends to multimodal data.","marker":"[13]"}],"fun_headline_variants":["ROSBag coding gets a dedicated web app with LLM auto-annotation","ROSAnnotator: a web tool for coding ROSBag video, audio, and text","Multimodal LLM helps code ROSBag data in a single web interface","Forget separate tools: ROSAnnotator codes ROSBag multimodal data","Streamline HRI analysis: ROSAnnotator integrates LLM coding for ROSBags"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the multimodal LLM's automated annotations are accurate enough to reduce, not increase, total human coding effort; the paper itself concedes that the automated annotation is not robust.","fun_headline_variants_meta":{"raw":{"variants":["ROSBag coding gets a dedicated web app with LLM auto-annotation","ROSAnnotator: a web tool for coding ROSBag video, audio, and text","Multimodal LLM helps code ROSBag data in a single web interface","Forget separate tools: ROSAnnotator codes ROSBag multimodal data","Streamline HRI analysis: ROSAnnotator integrates LLM coding for ROSBags"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1657,"prompt_tokens":855,"completion_tokens":802,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":696}},"tokens_in":471,"tokens_out":802,"duration_ms":7243,"temperature":1.0,"reasoning_tokens":696,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:49:12.560267+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled evaluation in which two independent coders annotate the same set of ROSBags manually and with ROSAnnotator's auto-annotation, comparing inter-rater agreement and total time spent; if the LLM-generated codes agree with manual codes only at chance level, or if correcting them takes longer than manual coding from scratch, the central efficiency claim fails.","supporting_citations":[{"cited_title":"Qualitative research in hri: A review and taxonomy,","cited_arxiv_id":null,"evidence_quote":"Establishes the importance of qualitative methods in HRI, motivating the need for annotation tools."},{"cited_title":"Ros for human-robot interaction,","cited_arxiv_id":null,"evidence_quote":"Represents prior open-source ROS tooling for HRI data that the paper's analysis-gap argument builds on."},{"cited_title":"Elan: A professional framework for multimodality research,","cited_arxiv_id":null,"evidence_quote":"Names the established manual annotation tool whose workflow ROSAnnotator aims to subsume."},{"cited_title":"Atlas/ti—a prototype for the support of text interpretation,","cited_arxiv_id":null,"evidence_quote":"Names the established qualitative-analysis software that motivates the need for a ROSBag-native alternative."},{"cited_title":"Supporting qualitative analysis with large language models: Combining codebook with gpt-3 for deductive coding,","cited_arxiv_id":null,"evidence_quote":"Shows that codebook-based deductive coding can be automated with a large language model, the capability ROSAnnotator extends to multimodal data."}],"review_version":1}