REVIEW 4 major objections 4 minor 28 references
Intelligent Virtual Sonographer (IVS): Enhancing Physician-Robot-Patient Communication
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Two local LLMs let a virtual sonographer run robotic ultrasound through live conversation.
desk verdict A genuinely new dual-LLM triadic architecture for robotic ultrasound, but the accuracy numbers in §3.2 lack a defined scoring protocol and should not be taken at face value yet. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the dual independent LLM architecture: two local 8-billion-parameter large language models run in parallel without fine-tuning, one prompted for professional physician interaction and one for empathetic patient interaction. Each instance is connected to a speech-to-text module, a text-to-speech module, and an extended-reality avatar pipeline, and the physician-facing instance maps verbal commands into robotic-control API calls, following the approach of cited prior work. The parallel design lets the physician and patient conversations proceed asynchronously without blocking each other, while the physician-facing instance relays patient information obtained from the patient-facing instance. The robotic ultrasound system itself uses hand-eye calibration, body-feature tracking for trajectory planning, and a vessel-segmentation network to provide real-time feedback that the IVS relays to the physician.
What would settle it
A replication with real patients speaking freely, independent clinicians, and the same hardware would falsify the claim if information-relay accuracy or action-execution accuracy falls clearly below the pilot's 90.48%, 85.71%, and 92.86% figures, or if the physician-facing instance fabricates patient details that would alter management decisions.
Extended reading notes
Core claim
The central claim is that a dual-instance LLM agent in extended reality can act as an intelligent virtual sonographer: it maintains two parallel conversations—professional with the physician, empathetic with the patient—while converting physician instructions into executable robot commands and narrating those actions back to the patient. The paper reports that this architecture, without fine-tuning, achieved 90.48% accuracy in relaying patient-specific information, 85.71% accuracy in executing patient-requested adjustments, and 92.86% accuracy in executing physician-requested actions, with a total conversational latency of 1.09 seconds per turn. It also reports positive subjective ratings for perceived intelligence, interaction quality, usability, and satisfaction, with physicians rating usability lower (3.82 out of 5) than novices (4.21 out of 5). The authors interpret this as first evidence that a moderate-sized, locally hosted LLM can support trust and transparency in robotic ultrasound.
Load-bearing premise
The pilot's simulated setting—with the authors playing scripted patients and 14 participants acting as physicians—accurately represents how real physicians, patients, and robots interact in clinical robotic ultrasound.
Editorial extensions
If this is right
- The pilot's measurements imply that one local agent can handle both physician and patient conversations simultaneously, so clinics would not need separate systems for each side.
- Running the LLM locally means patient data need not leave the hospital, which addresses a barrier to cloud-based tele-ultrasound.
- The 92.86% execution rate for physician commands supports spoken natural language as a usable control mode for robotic ultrasound rather than a research-only interface.
- The IVS's practice of narrating robot actions to the patient gives patients an opportunity to request pressure changes or pauses, which is a concrete mechanism for improving acceptance of robotic imaging.
- A per-turn latency of 1.09 seconds, while slower than human turn-taking, is within the range the paper argues is acceptable for interactive medical use, so the system is near practical responsiveness.
Reading between the lines
- Because the pilot used authors as scripted patients, the reported accuracy likely overstates performance with unscripted real patients; a test with naive patients reporting their own history would show whether the 90.48% relay figure holds.
- The same dual-instance architecture could be extended to higher-level clinical commands such as 'scan the carotid bifurcation' if the physician-facing instance is connected to a motion planner rather than low-level API calls.
- The observed hallucination of patient details suggests a concrete fix: query the electronic patient record before the agent speaks, so fabricated information is checked against the source of truth.
- Adding physiological sensors or facial-expression analysis to the patient-facing instance could let the IVS infer discomfort that the patient does not verbalize, which the paper lists as future work but does not test.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the Intelligent Virtual Sonographer (IVS), an XR-embodied conversational agent built from two locally deployed Llama 3.1 8B instances, one physician-facing and one patient-facing, connected to a robotic ultrasound system (KUKA LBR iiwa, Siemens ultrasound, Unity/Quest 3 visualization). The IVS is designed to mediate triadic communication: physicians issue verbal commands that are converted into robotic control API calls, patients can request pauses or pressure adjustments, and the agent relays patient-specific information between parties. A user study with 14 participants (7 medical doctors and 7 biomedical engineers) is reported, with authors playing scripted patients. The paper reports 90.48% accuracy in relaying patient-specific information, 85.71% execution of patient-requested actions, 92.86% execution of physician-requested actions, and positive subjective ratings on perceived intelligence, interaction quality, usability, and satisfaction.
Significance. If the reported accuracy and usability results hold, the paper would provide a useful feasibility demonstration of a fully local, privacy-preserving dual-LLM embodied agent for robotic ultrasound, a genuinely underexplored interaction paradigm. The manuscript has concrete strengths: the code is publicly released, the system is deployed on real robotic hardware, the two-LLM architecture is a sensible way to handle asynchronous physician-patient conversations, and the authors openly acknowledge the hallucination risk in the discussion. However, the evidence is pilot-level: the evaluation is a small simulated study with no control condition, no statistical inference, and no verified scoring protocol. The central feasibility claim is defensible, but the reliability claim ('reliably mediates triadic communication') is not yet supported by the presented metrics.
major comments (4)
- [§3.2 Quantitative Evaluation] The three headline accuracy values (90.48% patient-information relay, 85.71% patient-requested actions, 92.86% physician-requested actions) are reported without any scoring protocol. The manuscript does not state how many items or trials each rate is based on, what counts as one 'patient-specific information' item, whether name, age, and medical history are scored separately, who decided whether an action was correctly executed, or whether the acknowledged hallucination failures were included in or excluded from the numerators and denominators. This is load-bearing because the same section admits that 'the LLM hallucinated patient-specific information, generating responses from example patients in the prompt.' Please provide an error taxonomy with per-category counts, confidence intervals or exact binomial intervals, and either robot-command logs or blinded annotation with inter-rater reliability to make these rates interpretable.
- [§3.1 User Study] The evaluation is a simulated pilot in which the authors themselves acted as patients using predefined personas, and there is no baseline or control condition. The study can support only the narrow claim that trained participants can operate the system in a controlled, scripted setting; it cannot support the stronger comparative claims in the Abstract and Discussion that IVS 'enhances efficiency, clarity, and accessibility' or 'improves patient experience and acceptance.' Please either temper the wording throughout to a feasibility claim, or add a comparison condition such as direct physician-robot interaction without the IVS, and report quantitative differences with appropriate uncertainty.
- [§3.3 Subjective Ratings] The subjective ratings come from 14 participants split into groups of 7, reported only as means and standard deviations on a 5-point Likert scale. No statistical tests, confidence intervals, or pre-specified questionnaires are provided, and the observed differences (e.g., physician usability 3.82 vs. novice 4.21) are not analyzed. The statement that the system 'received positive ratings' is defensible at face value, but no conclusions about comparative usability or generalizable acceptance can be drawn from these data.
- [§2.2 Intelligent Virtual Sonographer] The dual-LLM architecture is the core mechanism for relaying patient-specific information and converting commands to robotic API calls, yet the prompt templates, API-call schema, and failure-handling logic are not included or even summarized in pseudocode. This makes the accuracy results non-reproducible and leaves unclear how the LLM is constrained to retrieve information only from the current patient's persona rather than from examples embedded in the prompt. Please include the prompts (or at least the full interaction protocol in an appendix) and describe the exact mechanism that prevents cross-patient leakage.
minor comments (4)
- [Abstract] Phrases such as 'enhances the efficiency, clarity, and accessibility' and 'improving patient experience' are not supported by any comparative measurement; consider rewording to 'supports' or 'may facilitate' to match the pilot-level evidence.
- [§3.2 Quantitative Evaluation] The term 'accuracy' for action execution is ambiguous; consider using 'command execution success rate' or 'completion rate' to distinguish from information-relay accuracy.
- [§4 Discussion and Conclusion] The claim that 1.09 seconds of conversational latency 'remains within an acceptable range for interactive medical applications' cites a systematic review on cognitively disabled users; this reference does not directly support the acceptance threshold for robotic ultrasound, and the statement should be qualified.
- [§2.2 Intelligent Virtual Sonographer] The description of the dual-LLM relay says the physician-facing LLM 'relayed the query to the patient-facing LLM if it lacked the required data,' but it is not explained what data the physician-facing LLM is initialized with or how the two instances share state; please clarify the memory and context-management design.
Circularity Check
No significant circularity: the central claims are empirical and based on user-study measurements, not derived from the system's inputs.
full rationale
The paper's central claim—that the IVS can mediate physician-robot-patient communication—is supported by direct user-study measurements: patient-provided information compared with physician-received information, and requested actions checked against executed actions. These are empirical outcomes, not quantities derived from fitted parameters or from the definitions of the system components. The self-citation to Song et al. [24] in Section 3.1 is used only to position the present work relative to prior patient-facing-agent research, not as load-bearing evidence for the reported accuracy or usability ratings. The LLM, MediaPipe, U-Net, and other components are cited from external or prior-art sources and are not used to force the evaluation results. While the evaluation protocol is underspecified (e.g., what counts as a correctly relayed item or a correctly executed action), that is a measurement-validity concern, not circularity. No equation or definition reduces a claimed prediction to its own input, so the circularity score is 0.
Assumptions & free parameters
free parameters (2)
- UNet vessel segmentation weights =
trained on 3000 ultrasound images; Dice=0.954 ± 0.012
- Dual LLM prompt templates =
hand-designed, not quantified
assumptions (5)
- domain assumption MediaPipe skin segmentation and hand landmark detection reliably identify the patient's arm and wrist from a single RGB image.
- domain assumption Hand-eye calibration provides accurate transformation between camera and robot frames.
- domain assumption A fine-tuned Llama 3.1 8B, without fine-tuning, reliably maps natural language commands to robotic API calls and does not generate unsafe actions.
- domain assumption Impedance control maintains safe probe-skin contact under variable patient anatomy.
- domain assumption Role-play with authors as patients and physicians/biomedical engineers as physicians approximates real clinical interactions.
Cite this review
Pith. "Pith review of Intelligent Virtual Sonographer (IVS): Enhancing Physician-Robot-Patient Communication." pith.science (2026). https://pith.science/paper/PS67PXIP
@misc{pith2026250713052,
author = {Pith},
title = {Pith review of: Intelligent Virtual Sonographer (IVS): Enhancing Physician-Robot-Patient Communication},
year = {2026},
howpublished = {\url{https://pith.science/paper/PS67PXIP}},
note = {Machine review of arXiv:2507.13052}
}
read the original abstract
The advancement and maturity of large language models (LLMs) and robotics have unlocked vast potential for human-computer interaction, particularly in the field of robotic ultrasound. While existing research primarily focuses on either patient-robot or physician-robot interaction, the role of an intelligent virtual sonographer (IVS) bridging physician-robot-patient communication remains underexplored. This work introduces a conversational virtual agent in Extended Reality (XR) that facilitates real-time interaction between physicians, a robotic ultrasound system(RUS), and patients. The IVS agent communicates with physicians in a professional manner while offering empathetic explanations and reassurance to patients. Furthermore, it actively controls the RUS by executing physician commands and transparently relays these actions to the patient. By integrating LLM-powered dialogue with speech-to-text, text-to-speech, and robotic control, our system enhances the efficiency, clarity, and accessibility of robotic ultrasound acquisition. This work constitutes a first step toward understanding how IVS can bridge communication gaps in physician-robot-patient interaction, providing more control and therefore trust into physician-robot interaction while improving patient experience and acceptance of robotic ultrasound.
Figures
Reference graph
Works this paper leans on
-
[1]
https://assetstore.unity.com/packages/tools/animation/final-ik-14290, accessed: 2025-02-26
Final IK. https://assetstore.unity.com/packages/tools/animation/final-ik-14290, accessed: 2025-02-26
work page 2025
-
[2]
SALSALipSyncSuite.https://assetstore.unity.com/packages/tools/animation/salsa- lipsync-suite-148442, accessed: 2025-02-26
work page 2025
-
[3]
Annual Review of Control, Robotics, and Autonomous Systems7 (2024)
Bi, Y., Jiang, Z., Duelmer, F., Huang, D., Navab, N.: Machine learning in robotic ultrasound imaging: Challenges and perspectives. Annual Review of Control, Robotics, and Autonomous Systems7 (2024)
work page 2024
-
[4]
Patient education and counseling 59(1), 21–30 (2005)
Bickmore, T., Gruber, A., Picard, R.: Establishing the computer–patient working alliance in automated health behavior change interventions. Patient education and counseling 59(1), 21–30 (2005)
work page 2005
-
[5]
In: Proceedings of the SIGCHI conference on human factors in computing systems
Bickmore, T.W., Pfeifer, L.M., Jack, B.W.: Taking the time to care: empowering low health literacy hospital patients with virtual nurse agents. In: Proceedings of the SIGCHI conference on human factors in computing systems. pp. 1265–1274 (2009)
work page 2009
-
[6]
Cassell,J.:Embodiedconversationalinterfaceagents.CommunicationsoftheACM 43(4), 70–78 (2000)
work page 2000
-
[7]
AI magazine22(4), 67–67 (2001)
Cassell, J.: Embodied conversational agents: representation and intelligence in user interfaces. AI magazine22(4), 67–67 (2001)
work page 2001
-
[8]
arXiv preprint arXiv:2407.21783 (2024)
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)
arXiv 2024
Show all 28 references
-
[9]
Ap- plied Sciences 13(18), 10199 (2023)
Eck, U., Wechner, M., Pankratz, F., Yu, K., Lazarovici, M., Navab, N.: Real-time 3d reconstruction pipeline for room-scale, immersive, medical teleconsultation. Ap- plied Sciences 13(18), 10199 (2023)
2023
-
[10]
International Journal of Computer Assisted Radiology and Surgery18(7), 1261–1267 (2023) 10 T
Eilers, C., van Kemenade, R., Busam, B., Navab, N.: On the importance of pa- tient acceptance for medical robotic imaging. International Journal of Computer Assisted Radiology and Surgery18(7), 1261–1267 (2023) 10 T. Song and F. Li et al
2023
-
[11]
Frontiers in virtual reality 1, 561558 (2020)
Gonzalez-Franco, M., Ofek, E., Pan, Y., Antley, A., Steed, A., Spanlang, B., Maselli, A., Banakou, D., Pelechano, N., Orts-Escolano, S., et al.: The rocket- box library and the utility of freely available rigged avatars. Frontiers in virtual reality 1, 561558 (2020)
2020
-
[12]
https://doi.org/10.57967/hf/4329, https://huggingface.co/hexgrad/Kokoro-82M
Hexgrad: Kokoro-82m (2025). https://doi.org/10.57967/hf/4329, https://huggingface.co/hexgrad/Kokoro-82M
2025 doi
-
[13]
The international journal of robotics research 14(3), 195–210 (1995)
Horaud, R., Dornaika, F.: Hand-eye calibration. The international journal of robotics research 14(3), 195–210 (1995)
1995
-
[14]
Disability and Rehabilitation: Assistive Technology19(3), 1059–1078 (2024)
Huq, S.M., Maskeli¯ unas, R., Damaševičius, R.: Dialogue agents for artificial intelligence-based conversational systems for cognitively disabled: A systematic re- view. Disability and Rehabilitation: Assistive Technology19(3), 1059–1078 (2024)
2024
-
[15]
The International Journal of Robotics Research43(7), 981–1002 (2024)
Jiang, Z., Bi, Y., Zhou, M., Hu, Y., Burke, M., Navab, N.: Intelligent robotic sono- grapher: Mutual information-based disentangled reward learning from few demon- strations. The International Journal of Robotics Research43(7), 981–1002 (2024)
2024
-
[16]
IEEE Transac- tions on Industrial Electronics68(11), 11200–11211 (2020)
Jiang, Z., Grimm, M., Zhou, M., Hu, Y., Esteban, J., Navab, N.: Automatic force- based probe positioning for precise robotic ultrasound acquisition. IEEE Transac- tions on Industrial Electronics68(11), 11200–11211 (2020)
2020
-
[17]
Medical image analysis p
Jiang, Z., Salcudean, S.E., Navab, N.: Robotic ultrasound imaging: State-of-the-art and future perspectives. Medical image analysis p. 102878 (2023)
2023
-
[18]
International journal of computer assisted radiology and surgery19(6), 1165–1173 (2024)
Killeen, B.D., Chaudhary, S., Osgood, G., Unberath, M.: Take a shot! natural lan- guage control of intelligent robotic x-ray systems in surgery. International journal of computer assisted radiology and surgery19(6), 1165–1173 (2024)
2024
-
[19]
arXiv preprint arXiv:2412.08020 (2024)
Killeen,B.D.,Suresh,A.,Gomez,C.,Inigo,B.,Bailey,C.,Unberath,M.:Intelligent control of robotic x-ray devices using a language-promptable digital twin. arXiv preprint arXiv:2412.08020 (2024)
2024 arXiv
-
[20]
arXiv preprint arXiv:1906.08172 (2019)
Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.L., Yong, M.G., Lee, J., et al.: Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172 (2019)
2019 arXiv
-
[21]
Journal of medical systems 41(9), 135 (2017)
Martínez-Miranda, J.: Embodied conversational agents for the detection and pre- vention of suicidal behaviour: current applications and open challenges. Journal of medical systems 41(9), 135 (2017)
2017
-
[22]
In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18....
2015
-
[23]
In: 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)
Roth, D., Yu, K., Pankratz, F., Gorbachev, G., Keller, A., Lazarovici, M., Wilhelm, D., Weidert, S., Navab, N., Eck, U.: Real-time mixed reality teleconsultation for intensive care units in pandemic situations. In: 2021 IEEE Conference on Virtual Reality and 3D User Interfaces...
2021
-
[24]
IEEE Transactions on Visualization and Computer Graphics31(5), 2901–2911 (2025)
Song,T.,Pabst,F.,Eck,U.,Navab,N.:Enhancingpatientacceptanceofroboticul- trasound through conversational virtual agent and immersive visualizations. IEEE Transactions on Visualization and Computer Graphics31(5), 2901–2911 (2025). https://doi.org/10.1109/TVCG.2025.3549181
2025
-
[25]
Proceedings of the National Academy of Sciences 106(26), 10587–10592 (2009)
Stivers, T., Enfield, N.J., Brown, P., Englert, C., Hayashi, M., Heinemann, T., Hoy- mann, G., Rossano, F., De Ruiter, J.P., Yoon, K.E., et al.: Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences 106(26), 10587–105...
2009
-
[26]
Inter- national Journal of Human-Computer Studies138, 102409 (2020) Intelligent Virtual Sonographer (IVS) 11
Ter Stal, S., Kramer, L.L., Tabak, M., op den Akker, H., Hermens, H.: Design features of embodied conversational agents in ehealth: a literature review. Inter- national Journal of Human-Computer Studies138, 102409 (2020) Intelligent Virtual Sonographer (IVS) 11
2020
-
[27]
Dementia19(4), 1173–1188 (2020)
Walker, T., Christensen, H., Mirheidari, B., Swainston, T., Rutten, C., Mayer, I., Blackburn, D., Reuber, M.: Developing an intelligent virtual agent to stratify people with cognitive complaints: a comparison of human–patient and intelligent virtual agent–patient interaction. ...
2020
-
[28]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Xu, H., Wu, J., Cao, G., Chen, Z., Lei, Z., Liu, H.: Transforming surgical in- terventions with embodied intelligence for ultrasound robotics. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 703–713. Springer (2024)
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.