{"id":"24512c77-c14d-42d7-81fa-225c380cb45b","arxiv_id":"2509.00660","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CARIS is a modular Wizard-of-Oz web interface that combines teleoperation, perception, LLM dialogue, and data logging, tested with small usability studies in tour guide and mental health check scenarios.","lead":"CARIS is an open-source web dashboard that lets a human operator remotely drive a robot, see through its camera, track people, and talk through a large language model; the authors tested it in two roles, a lab tour guide and a mental health check-in. The tool is plausibly useful for human-robot interaction labs, but the reported usability evidence is thin.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-robot adaptability claim in Sec III-E is unsupported: only one TIAGo integration is shown, so the system's 'context-adaptable' central contribution is not established for robots beyond the tested stack.","rationale":"The reader identified the same load-bearing weak point: cross-robot adaptability is asserted but not tested. I agree because the paper's contribution is framed as context-adaptable and scalable, with the abstract explicitly mentioning adaptation across 'robotic platforms.' The pilot studies only show two contexts on the same TIAGo stack, so they support reconfigurability of dialogue/prompts, not robot-level adaptability. The architecture may well be modular, but the paper provides no evidence that hardware-specific details are isolated. The proposed port to TurtleBot3 (or a simulated equivalent) would directly probe whether CARIS meets its central claim. I considered whether the low SUS scores or small sample are more load-bearing, but those affect the strength of the usability evidence, not the core adaptability logic; even perfect usability would not validate the cross-robot claim. The lack of a repository URL is a serious reproducibility issue but secondary to the unverified generalization. Thus, the reader's CONDITIONAL verdict stands, and we recommend no change.","tokens_in":9340,"tokens_out":5506,"duration_ms":67266,"concrete_test":"Obtain the promised public CARIS repository and port it to a second ground robot with a different ROS distribution and sensor set, e.g., TurtleBot3 (with its own LIDAR and camera) in Gazebo simulation. The integration should be done by only editing configuration files and the localization module's hardware interface; if any change to the core FastAPI endpoints, ROSBridge message handling, or wizard UI is required, the cross-robot adaptability claim is falsified. Additionally, grep the code for hardcoded TIAGo-specific topic names, frame IDs, or camera URLs. If no repository is available, this test cannot be run, which itself undermines the 'publicly available' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CARIS is adaptable across robotic platforms ('different settings, users, and robotic platforms' in the abstract). The only evidence is a single integration with a TIAGo Base, an Oak-D camera on a Raspberry Pi 4, and a FastAPI backend on Apple Silicon (Sec III-E). Section III-E states 'CARIS can be adapted to most ground-based mobile robots with basic localization and speech,' but no cross-robot test or architectural analysis is provided. The modularity depends on ROSBridge for movement/SLAM and FastAPI for perception/dialogue; the paper does not show that these interfaces are robot-agnostic. The two pilot studies vary the conversational context but keep the same hardware/software stack, so they demonstrate reconfigurability of prompts and roles, not adaptability across robots. If the ROSBridge topics (e.g., /cmd_vel), LIDAR frame IDs, or camera drivers are TIAGo-specific, the adaptability claim collapses. This is the load-bearing gap: the 'context-adaptable' label is credible only if the module boundaries isolate hardware dependencies, and that is never demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CARIS, a modular Wizard-of-Oz (WoZ) platform for teleoperating social robots. CARIS combines a web interface (Svelte) with a FastAPI backend for perception (YOLO v8, DeepSORT) and LLM-based dialogue, plus a ROSBridge link to robot localisation and movement. The authors report two pilot studies: a mental-health companion scenario and a tour-guide scenario, with 7 participants in the pilot and 4 in a follow-up 'main study'. They report SUS scores and thematic analysis of interviews, and they argue that CARIS is context-adaptable, modular, and publicly available for HRI researchers.","tokens_in":9527,"tokens_out":5674,"duration_ms":65034,"significance":"If correctly supported, CARIS would be a useful open-source contribution: it combines teleoperation, perception, LLM dialogue, and multimodal data logging in a single web interface, and Table I suggests it offers a broader feature set than prior WoZ tools. The modular architecture (Localization, Perception, Conversation, Wizard) is a sensible design. However, the evidence is thin: the two pilots use the same hardware stack, the SUS means are far below the standard acceptability threshold, and no inferential statistics are provided. The central claim of adaptability across 'robotic platforms' is not demonstrated by the reported experiments.","major_comments":[{"comment":"The claim that CARIS 'can be adapted to most ground-based mobile robots with basic localization and speech' is unsupported by the evidence. Only one hardware stack (TIAGo Base, Oak-D camera on Raspberry Pi 4, FastAPI backend on Apple Silicon) was integrated and tested; both pilot scenarios use this same stack. The modules rely on specific ROSBridge topics (e.g., /cmd_vel) and camera drivers, but the paper does not demonstrate that these interfaces are robot-agnostic. Because cross-platform adaptability is a central contribution, either integrate a second mobile platform or provide a detailed mapping of generic module interfaces with justification of portability.","section":"Sec. III-E and Abstract"},{"comment":"The SUS results do not support the stated conclusions. Pilot SUS scores (M=40, SD=37.36) and main-study scores (M=43.75, SD=19.63) are both below the conventional 68-point threshold for acceptable usability, yet Sec. VII concludes 'These results demonstrate moderate usability CARIS.' No inferential statistics, confidence intervals, or effect sizes are reported, and no comparison between pilot and main study is made, so the claimed iterative improvement is not demonstrated. Please report appropriate analyses and explicitly interpret the low scores as a significant limitation.","section":"Secs. V and VII"},{"comment":"The participant flow is unclear and likely underpowered for the claims. The text says 'we aimed to collect eight data points (two users and two wizards for each scenario),' but only four SUS scores are reported from the pilot. It is not stated how many sessions used a researcher as the wizard, nor whether the four SUS responses correspond to four distinct participants or to four wizard sessions. In the main study, n=4 is extremely small. Please clarify the exact number of participant-wizard sessions and report demographics for all participants, not only the 4 of 7 who completed the survey.","section":"Sec. IV (Participants)"},{"comment":"The abstract and Sec. I state that CARIS is 'publicly available' and 'open-source,' but no repository URL, package name, or license appears anywhere in the manuscript. This is a load-bearing omission for a systems contribution: readers cannot verify or reuse the tool. Provide a code-availability statement with a stable link (e.g., Zenodo) and license, or remove the claim.","section":"Abstract and Sec. I"}],"minor_comments":[{"comment":"Several typographical and grammatical errors: 'Futhermore' should be 'Furthermore'; 'inheritantly' should be 'inherently'; 'It’s suitability' should be 'its suitability'; 'ages aged' should be 'aged'; 'is HRI scenarios' likely means 'in HRI scenarios'; the abstract's 'enhance the usability wizard control' should be 'wizard control usability'.","section":"Sec. IV"},{"comment":"Scenario naming is inconsistent: 'Room tour', 'lab tour', and 'tour guide' are used interchangeably. Use a single consistent name throughout.","section":"Secs. IV and VII"},{"comment":"In the SUS quote, 'I'II needed to learn a lot of things...' appears to be a typo; should likely be 'I'd' or 'I'll'. Also, the discussion of 'average positive SUS scores' and 'average negative SUS range' is confusing; clarify that these are item-level means and explain reverse scoring for negative items.","section":"Sec. VII"},{"comment":"The IRB approval number is given as '#XX'; this placeholder should be replaced with the actual approval number before publication.","section":"Sec. IV"},{"comment":"The statement 'CARIS can be adapted to most ground-based mobile robots' is presented without any architectural analysis or reference to external interface standards. At minimum, provide a list of the ROS topics and services that the Localization Module actually subscribes to/publishes, and note which of these are TIAGo-specific.","section":"Sec. III-E"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is the adaptability claim, but the evidence is too weak: only one physical robot is integrated, the SUS scores are below the standard threshold, and no code link is provided despite the open-source claim. The authors should either substantially temper the claims to a single-platform WoZ tool with preliminary usability feedback, or add a second robot integration and a proper comparison. Editors should require the code repository to be available at revision time."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a plausible systems contribution: CARIS bundles teleoperation, SLAM-based localization, person tracking, LLM dialogue, and multimodal logging into one web-based Wizard-of-Oz interface, with modules separated via ROSBridge and FastAPI. The feature list is broader than most cited WoZ tools, and the two pilot scenarios (tour guide, mental health check) show the interface can actually drive a robot through both. The authors are appropriately modest in the abstract — they claim \"potential,\" not proven generalizability — and the qualitative feedback is honestly reported and used to iterate. That is real credit for an engineering paper.\n\nThe soft spots are in exact proportion to the reader's report, and the stress-test note lands. The abstract promises adaptability \"across different settings, users, and robotic platforms,\" but only one platform was ever integrated: a TIAGo Base with an Oak-D camera on a Raspberry Pi and an Apple Silicon backend. The two scenarios vary the conversational context and prompts, not the robot or the hard interfaces. Section III-E's assertion that CARIS can adapt to \"most ground-based mobile robots with basic localization and speech\" is the load-bearing claim, and it is unsupported by any cross-robot test or architectural analysis. That paragraph needs to be rewritten as a conjecture or supported with a second integration.\n\nThe evaluation is also thin: pilot n=7, main n=4, SUS means of 40 and 43.75 with large variance, well below the common 68 threshold, no inferential statistics, convenience sampling, and interviews coded by the authors. The SUS numbers actually suggest the system is not yet easy enough to use, which is consistent with the participants' complaints about movement control and layout. The paper would be stronger if it owned that failure rather than leaning on \"moderate usability.\" Also, the public-availability claim needs a repository URL, and Table I's \"uniquely combines\" wording should be softened without comparative experiments.\n\nThese are fixable. The architecture is coherent, the modules are sensibly decoupled, and the tool is genuinely useful for HRI researchers who want one interface instead of rebuilding WoZ stacks. A serious referee should see it, but with the request that the adaptability claim be either demonstrated or explicitly scoped down. I would not desk-reject this; I would send it to review with a clear \"major revision\" posture.","headline":"A workmanlike WoZ system paper with a genuinely useful integration but an overextended adaptability claim and a too-thin evaluation; send to review but expect major revision.","tokens_in":10093,"tokens_out":2083,"would_cite":true,"duration_ms":26251,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CARIS is a single open-source Wizard-of-Oz web interface that bundles teleoperation, perception, LLM-driven dialogue, and multimodal logging so HRI researchers can steer a robot through very different tasks—here a room tour and a mental-hea","keywords":["Wizard-of-Oz","human-robot interaction","modular robot interface","teleoperation","LLM dialogue","person tracking","multimodal data logging","context adaptability"],"falsifier":"Run CARIS on a second ground robot with different LIDAR, camera, and driver software; if the Wizard Module cannot issue movement commands or receive perception data without changing module internals, the adaptability claim is false. Alternatively, in a mental-health scenario, if the wizard cannot simultaneously move and converse, the ease-of-use claim weakens.","tokens_in":9175,"feed_emoji":"🤖","tokens_out":8235,"duration_ms":87562,"temperature":0.7,"pith_summary":"This paper introduces CARIS, a web-based Wizard-of-Oz control panel designed to be reused across human-robot interaction contexts without rebuilding the system. It bundles four modules—localization, perception, conversation, and the wizard interface—so that a human operator can teleoperate the robot, watch live camera and LIDAR data, track and name people, converse through an LLM with speech input and output, and log every interaction in real time. The authors show the same setup driving a mobile robot in two pilot scenarios, a room tour and a mental-health check, and use wizard feedback to refine the interface. If the adaptability claim holds, HRI teams could stop building bespoke WoZ rigs and instead configure one general-purpose platform per study.","feed_headline":"One control panel runs both a tour guide and a mental-health companion","feed_subtitle":"Combines teleoperation, perception, LLM dialogue, and logging so one HRI setup switches contexts without a rebuild.","key_machinery":"The core mechanism is the four-module architecture: Localization, Perception, Conversation, and the Wizard interface. Localization uses a 2D LIDAR and SLAM with movement commands relayed through ROSBridge; Perception runs YOLO-based detection and DeepSORT tracking in a backend separate from the robot; Conversation connects speech-to-text, LLMs (cloud or local), and text-to-speech; and the Wizard Module ties them together as a web-based command center that also logs multimodal data. The separation of robot-side and backend-side processing is what lets each module be swapped or reconfigured for a new context.","core_discovery":"The central claim is that CARIS's four-module decomposition—Localization, Perception, Conversation, and Wizard—decouples the robot's hardware from the higher-level Wizard-of-Oz decision loop, making the same interface adaptable to different contexts and robots. The Wizard Module acts as the command center: it sends movement commands, displays live video and LIDAR maps, lets the operator name and group tracked individuals, edits LLM prompts, and writes text, image, and JSON logs. The system was evaluated in a tour-guide scenario and a mental-health-check scenario with seven pilot participants and four main-study participants. In both contexts, CARIS successfully WoZ-controlled the robot, and","pith_inferences":["If the module boundaries hold, CARIS could extend to telepresence and remote caregiving, where a remote operator needs the same mix of movement, perception, and dialogue.","The pairing of person re-identification with interaction history suggests a path to longitudinal HRI studies, but privacy-sensitive contexts would need on-device storage and consent controls.","A direct stress-test of the adaptability claim would be deploying CARIS on a different ground robot base with different LIDAR and camera hardware and verifying that only configuration changes are needed.","The 'recommended prompts' and one-click communication suggestions point toward a semi-autonomous wizard where the LLM proposes actions and the human approves, cutting wizard workload in future studies."],"forward_implications":["A researcher could take CARIS from a mental-health companion study to a tour-guide study by changing prompts and role settings instead of writing a new WoZ system.","The built-in multimodal logging—movement commands, TTS usage, photos, and LLM exchanges—would let a single experiment produce synchronized data for training data-driven HRI models.","Because perception and conversation run in a backend separate from the robot's control loop, swapping in a different camera or language model is a module change, not a system rewrite.","The study feedback—keyboard control, dropdown model selection, clearer notes—shows the interface can be iterated on quickly between pilot and main study."],"supporting_citations":[{"why":"Establishes the Wizard-of-Oz design methodology that CARIS is built on and that the pilot procedure follows.","marker":"[2]"},{"why":"Provides the systematic review and reporting guidelines for WoZ studies that motivate the usability evaluation.","marker":"[3]"},{"why":"Baseline configurable WoZ interface; its single-robot focus defines the adaptability gap CARIS targets.","marker":"[4]"},{"why":"Runtime-configurable WoZ framework compared against CARIS's fixed module architecture.","marker":"[13]"},{"why":"ROS-based reproducible HRI architecture compared as a broader module-integration approach.","marker":"[14]"},{"why":"Supplies the YOLO object-detection backbone used in the Perception Module.","marker":"[18]"},{"why":"Supplies the tracking and re-identification algorithm used by the Perception Module.","marker":"[19]"},{"why":"The ground robot platform on which CARIS was deployed in both study scenarios.","marker":"[23]"},{"why":"The usability questionnaire used to measure wizard-perceived usability in pilot and main studies.","marker":"[25]"}],"fun_headline_variants":["CARIS: one WoZ panel for tour guide and mental-health robot","Same CARIS panel runs a tour guide and a therapy companion","CARIS swaps robot contexts without touching hardware","Context-adaptable WoZ: CARIS pilots tour guide and mental care","This interface controls both a guide bot and a therapy bot"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The claim that CARIS adapts to most ground-based robots rests on the assumption that its modules—especially the movement relay and the separate backend for perception and conversation—are portable beyond the single TIAGo Base setup with its specific camera and computer; that portability was never tested.","fun_headline_variants_meta":{"raw":{"variants":["CARIS: one WoZ panel for tour guide and mental-health robot","Same CARIS panel runs a tour guide and a therapy companion","CARIS swaps robot contexts without touching hardware","Context-adaptable WoZ: CARIS pilots tour guide and mental care","This interface controls both a guide bot and a therapy bot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000361,"raw_usage":{"total_tokens":1781,"prompt_tokens":729,"completion_tokens":1052,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":966}},"tokens_in":473,"tokens_out":1052,"duration_ms":10652,"temperature":1.0,"reasoning_tokens":966,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:20:53.698335+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CARIS on a second ground robot with different LIDAR, camera, and driver software; if the Wizard Module cannot issue movement commands or receive perception data without changing module internals, the adaptability claim is false. Alternatively, in a mental-health scenario, if the wizard cannot simultaneously move and converse, the ease-of-use claim weakens.","supporting_citations":[{"cited_title":"Wizard of Oz support throughout an iterative design process,","cited_arxiv_id":null,"evidence_quote":"Establishes the Wizard-of-Oz design methodology that CARIS is built on and that the pilot procedure follows."},{"cited_title":"Wizard of Oz studies in HRI: a systematic review and new reporting guidelines,","cited_arxiv_id":null,"evidence_quote":"Provides the systematic review and reporting guidelines for WoZ studies that motivate the usability evaluation."},{"cited_title":"WoZ4U: An Open- Source Wizard-of-Oz Interface for Easy, Efficient and Robust HRI Experiments,","cited_arxiv_id":null,"evidence_quote":"Baseline configurable WoZ interface; its single-robot focus defines the adaptability gap CARIS targets."},{"cited_title":"OpenWoZ: A Runtime-Configurable Wizard-of-Oz Framework for Human-Robot Interaction,","cited_arxiv_id":null,"evidence_quote":"Runtime-configurable WoZ framework compared against CARIS's fixed module architecture."},{"cited_title":"RISE: an open-source archi- tecture for interdisciplinary and reproducible human–robot interaction research,","cited_arxiv_id":null,"evidence_quote":"ROS-based reproducible HRI architecture compared as a broader module-integration approach."},{"cited_title":"Jocher, A","cited_arxiv_id":null,"evidence_quote":"Supplies the YOLO object-detection backbone used in the Perception Module."},{"cited_title":"Simple Online and Realtime Tracking with a Deep Association Metric,","cited_arxiv_id":null,"evidence_quote":"Supplies the tracking and re-identification algorithm used by the Perception Module."},{"cited_title":"TIAGo Base | Customisable AMR for Logistics and Research","cited_arxiv_id":null,"evidence_quote":"The ground robot platform on which CARIS was deployed in both study scenarios."},{"cited_title":"SUS: A quick and dirty usability scale,","cited_arxiv_id":null,"evidence_quote":"The usability questionnaire used to measure wizard-perceived usability in pilot and main studies."}],"review_version":1}