{"id":"ca5a5aa9-8788-4f83-9352-f4dc84931c20","arxiv_id":"2507.13052","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"IVS is a dual-LLM virtual sonographer in XR that mediates physician-robot-patient communication during robotic ultrasound, with a 14-participant pilot showing high command accuracy and positive ratings.","lead":"This paper presents IVS, a virtual sonographer avatar that uses two local AI language models to talk with doctors and patients during robotic ultrasound scans, and to control the robot by voice. A small pilot study with 14 participants reports high command accuracy and positive user ratings, but no comparison to existing workflows.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy metrics in §3.2 lack a defined scoring protocol and independent audit; the admitted hallucination failures may be excluded from the 90.48% figure, so the reliability claim is not yet supported.","rationale":"The reader's weakest assumption was the simulated setting (scripted patients, role-playing physicians) limiting generalizability to clinical practice. That is a legitimate external-validity concern. My concern is more internal: even within the reported simulation, the accuracy and execution rates are not defined or audited. The paper itself concedes that hallucination occurred, yet does not explain how those failures were treated in the headline percentages. Since the strongest claim is that IVS 'reliably mediates' communication, the evidence must first be internally valid; external validity is a second step. This does not move the verdict: the paper remains a plausible systems contribution whose feasibility claim is real but needs stronger evaluation. A CONDITIONAL verdict is appropriate, with the condition being release of auditable logs and a pre-registered scoring protocol. Hence the reader's verdict is unchanged, but the justification shifts from external realism to internal metric validity.","tokens_in":7135,"tokens_out":3918,"duration_ms":50773,"concrete_test":"Publish the per-participant transcripts, LLM prompts/API logs, and robot command/state logs from the 14-session study. Pre-register a scoring rubric (e.g., each relayed name/age/history item is correct, incorrect, or hallucinated; each action is executed only if a corresponding robot state change or API call is logged). Have two independent annotators blind to the reported outcomes score all trials, then recompute the three percentages with exact denominators. If the recomputed rates deviate by more than 5 percentage points from the paper's numbers, or if the hallucinated responses were excluded, the quantitative support for the central reliability claim does not hold as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('reliably mediates triadic communication') rests on three numbers in §3.2: 90.48% patient-information relay accuracy, 85.71% patient-requested action execution, and 92.86% physician-requested action execution. The paper never specifies how these rates were computed: what counts as a 'patient-specific information' item, whether name/age/medical-history are scored separately, who labeled an action as correctly executed, and whether failures were included or excluded. This is not a cosmetic gap: the same section admits that 'Occasionally, the LLM hallucinated patient-specific information, generating responses from example patients in the prompt instead of accurately retrieving the correct participant information.' If those hallucinations occurred during the scored trials and were omitted from the numerator/denominator, the reported accuracy is not a valid estimate. If they were included, the paper should report the error taxonomy rather than one aggregate percentage. Additionally, the patient role was played by the authors with pre-defined personas, and the person scoring 'execution' may have been the experimenter who issued the commands. Without inter-rater reliability, blinded annotation, or robot-command logs, the execution rates are unverifiable even as a feasibility demonstration. Because the paper's own discussion acknowledges LLM reliability as a 'critical aspect,' the evaluation protocol is the weakest load-bearing element: a different scoring rule could change the conclusion from 'reliably mediates' to 'works in a majority of scripted exchanges under favorable scoring.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Intelligent Virtual Sonographer (IVS), an XR-embodied conversational agent built from two locally deployed Llama 3.1 8B instances, one physician-facing and one patient-facing, connected to a robotic ultrasound system (KUKA LBR iiwa, Siemens ultrasound, Unity/Quest 3 visualization). The IVS is designed to mediate triadic communication: physicians issue verbal commands that are converted into robotic control API calls, patients can request pauses or pressure adjustments, and the agent relays patient-specific information between parties. A user study with 14 participants (7 medical doctors and 7 biomedical engineers) is reported, with authors playing scripted patients. The paper reports 90.48% accuracy in relaying patient-specific information, 85.71% execution of patient-requested actions, 92.86% execution of physician-requested actions, and positive subjective ratings on perceived intelligence, interaction quality, usability, and satisfaction.","tokens_in":7415,"tokens_out":2993,"duration_ms":36156,"significance":"If the reported accuracy and usability results hold, the paper would provide a useful feasibility demonstration of a fully local, privacy-preserving dual-LLM embodied agent for robotic ultrasound, a genuinely underexplored interaction paradigm. The manuscript has concrete strengths: the code is publicly released, the system is deployed on real robotic hardware, the two-LLM architecture is a sensible way to handle asynchronous physician-patient conversations, and the authors openly acknowledge the hallucination risk in the discussion. However, the evidence is pilot-level: the evaluation is a small simulated study with no control condition, no statistical inference, and no verified scoring protocol. The central feasibility claim is defensible, but the reliability claim ('reliably mediates triadic communication') is not yet supported by the presented metrics.","major_comments":[{"comment":"The three headline accuracy values (90.48% patient-information relay, 85.71% patient-requested actions, 92.86% physician-requested actions) are reported without any scoring protocol. The manuscript does not state how many items or trials each rate is based on, what counts as one 'patient-specific information' item, whether name, age, and medical history are scored separately, who decided whether an action was correctly executed, or whether the acknowledged hallucination failures were included in or excluded from the numerators and denominators. This is load-bearing because the same section admits that 'the LLM hallucinated patient-specific information, generating responses from example patients in the prompt.' Please provide an error taxonomy with per-category counts, confidence intervals or exact binomial intervals, and either robot-command logs or blinded annotation with inter-rater reliability to make these rates interpretable.","section":"§3.2 Quantitative Evaluation"},{"comment":"The evaluation is a simulated pilot in which the authors themselves acted as patients using predefined personas, and there is no baseline or control condition. The study can support only the narrow claim that trained participants can operate the system in a controlled, scripted setting; it cannot support the stronger comparative claims in the Abstract and Discussion that IVS 'enhances efficiency, clarity, and accessibility' or 'improves patient experience and acceptance.' Please either temper the wording throughout to a feasibility claim, or add a comparison condition such as direct physician-robot interaction without the IVS, and report quantitative differences with appropriate uncertainty.","section":"§3.1 User Study"},{"comment":"The subjective ratings come from 14 participants split into groups of 7, reported only as means and standard deviations on a 5-point Likert scale. No statistical tests, confidence intervals, or pre-specified questionnaires are provided, and the observed differences (e.g., physician usability 3.82 vs. novice 4.21) are not analyzed. The statement that the system 'received positive ratings' is defensible at face value, but no conclusions about comparative usability or generalizable acceptance can be drawn from these data.","section":"§3.3 Subjective Ratings"},{"comment":"The dual-LLM architecture is the core mechanism for relaying patient-specific information and converting commands to robotic API calls, yet the prompt templates, API-call schema, and failure-handling logic are not included or even summarized in pseudocode. This makes the accuracy results non-reproducible and leaves unclear how the LLM is constrained to retrieve information only from the current patient's persona rather than from examples embedded in the prompt. Please include the prompts (or at least the full interaction protocol in an appendix) and describe the exact mechanism that prevents cross-patient leakage.","section":"§2.2 Intelligent Virtual Sonographer"}],"minor_comments":[{"comment":"Phrases such as 'enhances the efficiency, clarity, and accessibility' and 'improving patient experience' are not supported by any comparative measurement; consider rewording to 'supports' or 'may facilitate' to match the pilot-level evidence.","section":"Abstract"},{"comment":"The term 'accuracy' for action execution is ambiguous; consider using 'command execution success rate' or 'completion rate' to distinguish from information-relay accuracy.","section":"§3.2 Quantitative Evaluation"},{"comment":"The claim that 1.09 seconds of conversational latency 'remains within an acceptable range for interactive medical applications' cites a systematic review on cognitively disabled users; this reference does not directly support the acceptance threshold for robotic ultrasound, and the statement should be qualified.","section":"§4 Discussion and Conclusion"},{"comment":"The description of the dual-LLM relay says the physician-facing LLM 'relayed the query to the patient-facing LLM if it lacked the required data,' but it is not explained what data the physician-facing LLM is initialized with or how the two instances share state; please clarify the memory and context-management design.","section":"§2.2 Intelligent Virtual Sonographer"}],"recommendation":"major_revision","confidential_remarks":"The paper is a promising feasibility demonstration, but the evaluation protocol is the main barrier to publication in its current form. The accuracy metrics in §3.2 are unverifiable without a scoring protocol and per-item counts, and the simulated setting with authors as patients limits the generality of the claims. I recommend major revision with a request for a detailed evaluation appendix; if the authors cannot provide the scoring protocol, the paper might be better framed as a late-breaking results report rather than a full validation of the reliability claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about arXiv:2507.13052. The design is genuinely new: two role-specific Llama 3.1 8B instances, one professional for the physician, one empathetic for the patient, running in parallel and coordinating through XR with the robot. That is a real advance over prior work that did dyads or used a single LLM for control. The paper is also honest: it flags hallucination, describes the local-deployment rationale, and ships code. But the evaluation numbers in §3.2 do not yet support the word \"reliably.\" The 90.48% relay accuracy, 85.71% patient-action, and 92.86% physician-action rates have no defined scoring protocol: we don't know what counted as a patient-specific information point, who judged an action as executed, or whether the admitted hallucinations were in the denominator. Given the same paragraph admits the LLM periodically invented example-patient data, the aggregate percentages could change meaningfully under a stricter scoring rule. That is a load-bearing flaw, not cosmetic.\n\nWhat else is soft: 14 participants, authors acting as scripted patients, no baseline or control condition, no confidence intervals, and Likert ratings without any inferential statistics. The paper claims \"efficiency, clarity, and accessibility\" improvements but no comparative data support that. These are standard pilot-study limitations and are not fatal to the idea; they undermine the specific accuracy claims as stated.\n\nWhat holds up: the dual-LLM architecture, the parallel conversation handling, the full pipeline from speech to robot control, and the honest discussion of latency and hallucination. It is a credible feasibility prototype for LLM-mediated tele-ultrasound.\n\nWho is this for: researchers building LLM-driven robotic ultrasound or embodied agents for clinician-patient interaction. It deserves a serious referee; a hypothetical reviewer should ask for a precise evaluation protocol, a baseline (even a no-agent condition), and logged command ground truth before accepting the accuracy claims. I'd send it out, with the expectation of major revision.","headline":"A genuinely new dual-LLM triadic architecture for robotic ultrasound, but the accuracy numbers in §3.2 lack a defined scoring protocol and should not be taken at face value yet.","tokens_in":7960,"tokens_out":1987,"would_cite":true,"duration_ms":26195,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two local LLMs let a virtual sonographer run robotic ultrasound through live conversation.","keywords":["robotic ultrasound","extended reality","large language models","embodied conversational agent","physician-robot-patient communication","telemedicine","virtual sonographer","human-robot interaction"],"falsifier":"A replication with real patients speaking freely, independent clinicians, and the same hardware would falsify the claim if information-relay accuracy or action-execution accuracy falls clearly below the pilot's 90.48%, 85.71%, and 92.86% figures, or if the physician-facing instance fabricates patient details that would alter management decisions.","tokens_in":6969,"feed_emoji":"🤖","tokens_out":6707,"duration_ms":66521,"temperature":0.7,"pith_summary":"This paper argues that a single conversational agent can bridge the three-way communication gap among physician, robot, and patient in robotic ultrasound, a role previous systems left to separate patient-facing and physician-facing tools. The authors build IVS, an extended-reality avatar backed by two independently prompted local large language models: one speaks professionally with the physician and translates spoken commands into robotic-control API calls, the other speaks empathetically with the patient and relays concerns. In a 14-participant pilot with scripted patients played by the authors, the agent relayed patient-specific information with 90.48% accuracy, executed 85.71% of patient-requested actions and 92.86% of physician-requested actions, and received positive usability ratings. If the pilot generalizes, the system shows a feasible path to privacy-preserving, locally deployed LLM agents for tele-ultrasound.","feed_headline":"Two local LLMs run robotic ultrasound through live talk","feed_subtitle":"Pilot: 90% accuracy relaying patient info, 93% of physician commands executed in real time","key_machinery":"The load-bearing mechanism is the dual independent LLM architecture: two local 8-billion-parameter large language models run in parallel without fine-tuning, one prompted for professional physician interaction and one for empathetic patient interaction. Each instance is connected to a speech-to-text module, a text-to-speech module, and an extended-reality avatar pipeline, and the physician-facing instance maps verbal commands into robotic-control API calls, following the approach of cited prior work. The parallel design lets the physician and patient conversations proceed asynchronously without blocking each other, while the physician-facing instance relays patient information obtained from the patient-facing instance. The robotic ultrasound system itself uses hand-eye calibration, body-feature tracking for trajectory planning, and a vessel-segmentation network to provide real-time feedback that the IVS relays to the physician.","core_discovery":"The central claim is that a dual-instance LLM agent in extended reality can act as an intelligent virtual sonographer: it maintains two parallel conversations—professional with the physician, empathetic with the patient—while converting physician instructions into executable robot commands and narrating those actions back to the patient. The paper reports that this architecture, without fine-tuning, achieved 90.48% accuracy in relaying patient-specific information, 85.71% accuracy in executing patient-requested adjustments, and 92.86% accuracy in executing physician-requested actions, with a total conversational latency of 1.09 seconds per turn. It also reports positive subjective ratings for perceived intelligence, interaction quality, usability, and satisfaction, with physicians rating usability lower (3.82 out of 5) than novices (4.21 out of 5). The authors interpret this as first evidence that a moderate-sized, locally hosted LLM can support trust and transparency in robotic ultrasound.","pith_inferences":["Because the pilot used authors as scripted patients, the reported accuracy likely overstates performance with unscripted real patients; a test with naive patients reporting their own history would show whether the 90.48% relay figure holds.","The same dual-instance architecture could be extended to higher-level clinical commands such as 'scan the carotid bifurcation' if the physician-facing instance is connected to a motion planner rather than low-level API calls.","The observed hallucination of patient details suggests a concrete fix: query the electronic patient record before the agent speaks, so fabricated information is checked against the source of truth.","Adding physiological sensors or facial-expression analysis to the patient-facing instance could let the IVS infer discomfort that the patient does not verbalize, which the paper lists as future work but does not test."],"forward_implications":["The pilot's measurements imply that one local agent can handle both physician and patient conversations simultaneously, so clinics would not need separate systems for each side.","Running the LLM locally means patient data need not leave the hospital, which addresses a barrier to cloud-based tele-ultrasound.","The 92.86% execution rate for physician commands supports spoken natural language as a usable control mode for robotic ultrasound rather than a research-only interface.","The IVS's practice of narrating robot actions to the patient gives patients an opportunity to request pressure changes or pauses, which is a concrete mechanism for improving acceptance of robotic imaging.","A per-turn latency of 1.09 seconds, while slower than human turn-taking, is within the range the paper argues is acceptable for interactive medical use, so the system is near practical responsiveness."],"supporting_citations":[{"why":"Supplies the 8-billion-parameter LLM backbone that powers both agent instances.","marker":"[8]"},{"why":"Provides the method for converting spoken physician commands into robotic control API calls.","marker":"[28]"},{"why":"Prior patient-facing conversational agent work that this study extends to triadic communication.","marker":"[24]"},{"why":"Framework used to extract body features for automatic scan-path generation.","marker":"[20]"},{"why":"Hand-eye calibration routine that maps the camera pose to robot coordinates.","marker":"[13]"},{"why":"Vessel-segmentation model that supplies real-time anatomical feedback relayed to the physician.","marker":"[22]"}],"fun_headline_variants":["One agent, two voices: professional for doctor, warm for patient in robot ultrasound","LLM virtual sonographer speaks doctor and patient languages in robot ultrasound","LLM agent chats to doctor and patient as it steers ultrasound robot","Virtual sonographer LLM talks doctor and patient, controls ultrasound robot","Dual LLM agent keeps doctor, robot, patient in sync during ultrasound"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pilot's simulated setting—with the authors playing scripted patients and 14 participants acting as physicians—accurately represents how real physicians, patients, and robots interact in clinical robotic ultrasound.","fun_headline_variants_meta":{"raw":{"variants":["One agent, two voices: professional for doctor, warm for patient in robot ultrasound","LLM virtual sonographer speaks doctor and patient languages in robot ultrasound","LLM agent chats to doctor and patient as it steers ultrasound robot","Virtual sonographer LLM talks doctor and patient, controls ultrasound robot","Dual LLM agent keeps doctor, robot, patient in sync during ultrasound"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000869,"raw_usage":{"total_tokens":3759,"prompt_tokens":932,"completion_tokens":2827,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":2730}},"tokens_in":548,"tokens_out":2827,"duration_ms":21544,"temperature":1.0,"reasoning_tokens":2730,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:31:21.218386+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication with real patients speaking freely, independent clinicians, and the same hardware would falsify the claim if information-relay accuracy or action-execution accuracy falls clearly below the pilot's 90.48%, 85.71%, and 92.86% figures, or if the physician-facing instance fabricates patient details that would alter management decisions.","supporting_citations":[{"cited_title":"In: International Conference on Medical Image Computing and Computer-Assisted Intervention","cited_arxiv_id":null,"evidence_quote":"Provides the method for converting spoken physician commands into robotic control API calls."},{"cited_title":"The international journal of robotics research 14(3), 195–210 (1995)","cited_arxiv_id":null,"evidence_quote":"Hand-eye calibration routine that maps the camera pose to robot coordinates."}],"review_version":1}