Pith. sign in

REVIEW 4 major objections 5 minor 6 references

A Custom-Built Ambient Scribe Reduces Cognitive Load and Documentation Burden for Telehealth Clinicians

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The authors report that a custom-built AI scribe in a telehealth EHR reduced self-reported cognitive load and documentation burden for surveyed clinicians.

desk verdict A useful production case study with real adoption and latency data, but the abstract overstates the survey and LLM-judge evidence; send to peer review with demands for response-rate reporting, human evaluation, and data-split clarity. read the letter →

arxiv 2507.17754 v1 pith:3W2GPJA5 submitted 2025-05-02 cs.HC cs.AIcs.CLcs.CY

Justin Morse , Kurt Gilbert , Kyle Shin , Rick Cooke , Peyton Rose , Jack Sullivan , Angelo Sisante This is my paper · ORCID
classification cs.HCcs.AIcs.CLcs.CY
keywords ambientscribeSOAPnotestelehealthcognitiveloaddocumentationburdenin-contextlearningWhispertranscriptionBARTfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Inside a telehealth practice, the authors built an ambient scribe that records visits and automatically drafts SOAP notes and patient instructions, and they integrated it directly into the electronic health record clinicians already use. The paper's central claim is that the tool lightens clinicians' administrative load: 94% of surveyed clinicians said it reduced cognitive load during visits and 97% said it lowered documentation burden. Supporting evidence includes LLM-judge comparisons on mock visits where generated notes outperformed expert-written notes, production notes requiring only minor clinician edits, and a fine-tuned post-processor that shortens notes while mostly preserving meaning. If the claim holds, ambient scribes assembled from off-the-shelf speech recognition and large language models could ease a known driver of clinician burnout without forcing a workflow change.

What carries the argument

The load-bearing mechanism is a modular chain-of-thought in-context learning pipeline: rather than prompting one large language model call to write an entire SOAP note, the system sends three parallel prompts—Subjective/History of Present Illness, Past Medical Encounters and Vitals, and Assessment and Plan—plus a separate prompt for patient instructions, and then chains each generated subsection through a verification prompt. That totals six GPT-4o calls per visit. This modular design is what the paper credits for note quality, consistency, and the clinically useful features like documented pertinent negatives; the separate fine-tuned BART post-processor then compresses notes by learning clinician editing patterns. Whisper, prompted with domain terms, supplies the transcript input at a word error rate of 0.21 on mock medical audio.

What would settle it

An audit of the service line's EHR event logs could settle the burden claim: if clinicians who use the scribe spend no less time on documentation than their pre-rollout baseline, the 97% documentation-burden reduction is contradicted. For the quality claim, a blinded panel of practicing clinicians judging the same 32 Primock57 notes against the expert-written versions would provide a direct test of whether the LLM judges' preference for the generated notes reflects real clinical quality.

Watch

Extended reading notes

Core claim

The authors report that a production ambient scribe—Whisper for transcription plus a modular in-context learning pipeline using GPT-4o for note generation—produces SOAP notes that, on 32 mock telehealth visits, were judged by multiple LLMs to be higher quality than the expert-written notes in the dataset (win rates 0.84 with GPT-4o judge, 0.97 with Claude 3.7-sonnet, and 1.0 with o1-preview). In live use, over 540 clinicians tried the tool within three months, and 94% (n=63) and 97% (n=66) of survey respondents respectively reported reduced cognitive load and documentation burden. Objectively, clinician-edited production notes were only about 4% shorter than the generated versions with a BERTScore F1 of 0.97, suggesting minor edits; fine-tuning BART on 130,000 pairs of generated and clinician-edited notes shrank the History of Present Illness section by 17% with a BERTScore drop to 0.92. These results are the paper's evidence that AI-generated documentation in telehealth can match or exceed human note quality while reducing the cognitive overhead of charting.

Load-bearing premise

The central claim assumes that the surveyed clinicians' retrospective self-reports—from a self-selected subset of over 540 users, with no baseline or control group—correspond to an actual reduction in cognitive load and documentation burden rather than to enthusiasm for a new tool.

Editorial extensions

If this is right

  • Clinicians using the scribe can produce SOAP notes that require only minor edits, since production notes differed from clinician-submitted versions by only 4% in length and 0.97 BERTScore F1.
  • Prompting Whisper with the top 200 TF-IDF medical terms from an out-of-sample corpus lowers word error rate on medical audio by 19%, which would likely transfer to other medical transcription deployments.
  • A fine-tuned BART post-processor can shrink AI-generated History of Present Illness sections by 17% while losing only 5% semantic similarity, suggesting a viable compression step before clinician review.
  • The modular parallel-subsection prompting strategy keeps note generation latency low enough for production use (p50 14.4 seconds), making real-time ambient scribing feasible in telehealth.
  • Widespread adoption within three months—nearly 70% of virtual primary care and 40% of urgent care visits—indicates clinicians will use such a tool when it is embedded in the existing EHR workflow.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's survey response rate and non-responder characteristics are not reported, so the 94% and 97% figures may overstate the effect; a randomized or time-motion comparison with objective documentation-time measurements would test whether perceived relief matches actual workload reduction.
  • The fine-tuned BART step's removal of pertinent negatives indicates a conciseness/completeness trade-off that could matter clinically; a clinician-preference study with the shorter notes would determine whether the compression is acceptable in practice.
  • The modular prompting design is not tied to SOAP in principle; the same parallel-chain-of-thought pattern could generate other structured documents, but the paper does not test that.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a production ambient scribe system deployed at Included Health. The system uses Whisper for transcription (with a TF-IDF-derived medical term prompt), a modular in-context learning pipeline with GPT-4o to generate SOAP sections and patient instructions, and a fine-tuned BART model to shorten the generated History of Present Illness text. The authors report that prompted Whisper reduces WER, that LLM judges prefer the ICL-generated SOAP notes over expert-written Primock57 notes, that over 540 clinicians adopted the system, and that surveyed clinicians self-report reduced cognitive load (94%, n=63) and reduced documentation burden (97%, n=66). They further report that BART post-processing reduces HPI length by 17% with only a 5% drop in BERTScore.

Significance. If the claims were fully supported, this would be a valuable real-world deployment study: it provides a concrete system architecture, adoption curves across service lines, latency data, and a practical post-processing approach to shorten AI-generated notes. The Primock57 WER comparison is a straightforward, testable result, and the production-scale adoption data are informative for the CS/HCI community. However, the central burden-reduction claim rests almost entirely on a voluntary, retrospective, uncontrolled survey with no response-rate reporting, and the note-quality claim relies on LLM-as-a-judge comparisons without human validation. These are load-bearing weaknesses because they support the title, abstract, and conclusion. The paper's engineering contribution is real but the evidence for its headline causal claims is currently insufficient.

major comments (4)
  1. [Section 3.3 and Abstract] The claim that the scribe 'reduces cognitive load and documentation burden' is supported only by a voluntary survey with no reported response rate, no baseline, no control group, and no non-responder analysis. The two survey items also have different denominators (63 and 66), which suggests item-level non-response that is not discussed. Since the items ask whether the application 'reduced' load or burden, the responses require retrospective comparison to pre-tool experience, and they are vulnerable to social desirability bias in a workplace where the tool was built by colleagues. The data support 'respondents reported reduced burden' but not 'the application reduces burden.' The abstract and conclusion should be re-scoped accordingly unless additional evidence (e.g., baseline workload measurements, response-rate information, or a comparison group) is provided.
  2. [Section 3.2 and Table 2] The LLM-as-a-judge evaluation reports win rates of 0.84 (GPT-4o), 0.97 (Claude 3.7-sonnet), and 1.00 (o1-preview) for ICL-generated notes over expert-written notes, but no human evaluation is provided. Because the generator is GPT-4o and the judges are also frontier LLMs with overlapping training data, the high win rates may partly reflect style self-preference or rubric-satisfying verbosity rather than clinical superiority. The claim that the notes 'exceed the quality of expert-written notes' would require either human expert validation or at least a demonstrated correlation between the LLM judges' preferences and human expert preferences.
  3. [Section 2.4 and Section 3.4, Table 3] The BART post-processing evaluation does not state whether the 130,000-note fine-tuning dataset was split into training and test sets, nor whether the evaluation pairs in Table 3 overlap with training data. If the BART model was evaluated on notes it was trained to mimic, the reported 17% length reduction and 0.92 BERTScore are not trustworthy. Additionally, the authors acknowledge that post-processing eliminates some pertinent negatives, which conflicts with the 'little impact on semantics' claim when BERTScore is the only semantic metric; BERTScore with bert-base-uncased is a weak proxy for clinical adequacy.
  4. [Section 3.1] The description of the Whisper prompt terms is ambiguous: the paper says the top-200 TF-IDF terms are extracted from an 'out-of-sample corpus of human-written Primock57 transcripts' but does not specify how this corpus is disjoint from the 32 test recordings. If the TF-IDF terms are estimated from transcripts of the same visits used for evaluation, the reported 19% WER improvement could be inflated by information leakage. The authors should clarify the exact split and how 'out-of-sample' is defined.
minor comments (5)
  1. [Table 1] The row label 'GPT4o-transcribe-mini0.29 +/- 0.06' is missing a delimiter; the model name and WER value should be separated.
  2. [Table 2] The o1-preview row shows '1' without a standard deviation, unlike the other rows; the entry should be formatted consistently or an explanation should be given for the absence of a variance estimate.
  3. [Appendix C] The rubric for Chronology contains a typographical error: 'the 0 0 - HPI needs improvement' should read 'the HPI needs improvement.'
  4. [Section 3.3] The sentence 'Among respondents, 94% (n = 63) agreed that the application reduced their cognitive load during patient visits' would be clearer if the exact survey question wording and the full item text were included, because the interpretation of 'reduced' depends on the reference point.
  5. [Section 5] The Limitations section mentions data privacy and employment conflicts but omits the survey validity threats and the LLM-judge circularity discussed above; these should be acknowledged explicitly.

Circularity Check

1 steps flagged · score 6.0 of 10

BART post-processing evaluation is circular: the model is trained to reproduce clinician-submitted notes and then scored against those same notes as reference, with no reported held-out split.

  1. fitted input called prediction [Section 2.4 (training data) and Section 3.4/Table 3 (evaluation)]
    "We trained the model using a proprietary dataset consisting of ~130,000 ICL-generated notes and the corresponding clinician-edited version submitted to the EHR. ... After processing the HPI section with the fine-tuned BART model, character length decreases by 17% (p < 1e-10, Table 3). In comparison, the F1 BERTscore, again calculated using the submitted documents as the reference, drops by only 5% (p < 1e-10, Table 3)."

    The BART post-processor is fit to map ICL-generated notes onto clinician-submitted notes (Section 2.4). Section 3.4 then reports the post-processed output's length reduction and F1 BERTscore using 'the submitted documents as the reference.' No train/test split is reported, so as presented the model is scored against the very documents that define its training targets. A model that simply copies the clinician-edited reference would trivially produce high BERTScore and the observed length pattern; the claimed 'improves conciseness' is therefore an artifact of the fitting target rather than an out-of-sample prediction.

full rationale

The title claim about reduced cognitive load and documentation burden rests on a self-reported volunteer survey; that is a validity/sampling limitation, not a circular derivation, since the survey responses are independent of the paper's model inputs. The ICL-vs-expert SOAP-note comparison on Primock57 uses external expert notes and an external judge rubric, so it does not reduce to the paper's own outputs. The one concrete circularity is the BART post-processing benchmark: the model is trained on (ICL-generated note, clinician-edited note) pairs and then evaluated with the clinician-edited notes as the reference, with no held-out split stated. Because this is an ancillary 'Additionally' contribution rather than the paper's headline claim, the overall circularity score is moderate rather than severe.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its claims rest on four domain assumptions and on several fitted components (TF-IDF term list, few-shot examples, fine-tuned BART weights) that are either not shared or selected from the same datasets used for evaluation.

free parameters (3)
  • Top-200 TF-IDF medical terms for Whisper prompt = not listed
    Section 3.1 selects the top 200 terms by TF-IDF from human-written Primock57 transcripts; if these are from the same 32 visits used for WER evaluation, the 19% improvement may be inflated by in-distribution term selection.
  • ICL few-shot examples = not provided (omitted for brevity)
    Appendix A says multi-shot examples are omitted; the examples constrain output style and content and are a fitted component of the pipeline.
  • BART fine-tuned weights = not shared (proprietary)
    Section 2.4 trains on 130k clinician-edited pairs; the resulting weights are a fitted function from ICL notes to clinician edits, used as the post-processor.
assumptions (4)
  • domain assumption LLM-as-a-judge preferences are a valid measure of clinical note quality
    Section 3.2 uses GPT-4o, Claude 3.7, and o1 to declare ICL notes better than expert notes; no human clinician evaluation is provided.
  • domain assumption Clinician-edited notes submitted to the EHR are ground truth for note quality
    Section 3.3 treats final clinician notes as ground truth for BERTScore and length comparisons, assuming all edits improve quality and all unedited content is correct.
  • domain assumption Self-reported cognitive load and documentation burden changes reflect actual changes
    Section 3.3 survey asks clinicians post-hoc whether burden decreased; there is no baseline measurement, control group, or objective workload data.
  • domain assumption Primock57 mock visits are representative of production telehealth visits
    The quality evaluation uses 32 mock primary-care visits; production visits at Included Health include urgent care, VPC, psychiatry, and therapy with possibly different case mixes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Custom-Built Ambient Scribe Reduces Cognitive Load and Documentation Burden for Telehealth Clinicians." pith.science (2026). https://pith.science/paper/3W2GPJA5

@misc{pith2026250717754,
  author       = {Pith},
  title        = {Pith review of: A Custom-Built Ambient Scribe Reduces Cognitive Load and Documentation Burden for Telehealth Clinicians},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3W2GPJA5}},
  note         = {Machine review of arXiv:2507.17754}
}
read the original abstract

Clinician burnout has motivated the growing adoption of ambient medical scribes in the clinic. In this work, we introduce a custom-built ambient scribe application integrated into the EHR system at Included Health, a personalized all-in-one healthcare company offering telehealth services. The application uses Whisper for transcription and a modular in-context learning pipeline with GPT-4o to automatically generate SOAP notes and patient instructions. Testing on mock visit data shows that the notes generated by the application exceed the quality of expert-written notes as determined by an LLM-as-a-judge. The application has been widely adopted by the clinical practice, with over 540 clinicians at Included Health using the application at least once. 94% (n = 63) of surveyed clinicians report reduced cognitive load during visits and 97% (n = 66) report less documentation burden when using the application. Additionally, we show that post-processing notes with a fine-tuned BART model improves conciseness. These findings highlight the potential for AI systems to ease administrative burdens and support clinicians in delivering efficient, high-quality care.

Figures

Figures reproduced from arXiv: 2507.17754 by the authors.

Figure 1
Figure 1. A modular chain-of-thought prompting approach to generate SOAP sections in parallel We use in-context learning with GPT-4o to generate SOAP notes. Whisper visit transcripts are fed with a multi-shot prompt to GPT-4o via OpenAI's Chat Completion API. Instead of relying on a single prompt to create the entire SOAP note, we use a modular approach to generate the following mutually exclusive subsections in parallel ( [… view at source ↗
Figure 2
Figure 2. System architecture of an ambient medical scribe for use in telehealth applications [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Histogram of note generation latency User adoption, feedback, and quality at scale Included Health provides healthcare services across urgent care (UC), virtual primary care (VPC), psychiatry, and therapy. The ambient scribe application was integrated into Included Health's EHR in November 2024. Within three months, the application was used to document nearly 70% of all VPC visits and 40% of UC visits ( [PITH_FULL_… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Adoption of an ambient medical scribe application across clinical service lines at Included Health To assess the application's impact on burnout we conducted a survey of all clinicians at Included Health. Among respondents, 94% (n = 63) agreed that the application redu…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 2 canonical work pages

  1. [1]

    G., Beasley, J

    Arndt, B. G., Beasley, J. W., Watkinson, M. D., Temte, J. L., Tuan, W. J., Sinsky, C. A., & Gilchrist, V. J. (2017). Tethered to the EHR: Primary care physician workload assessment using EHR event log data and time-motion observations. Annals of Family Medicine, 15(5). https://doi.org/10.1370/afm.2121 Duggan, M. J., Gervase, J., Schoenbaum, A., Hanson, W....

    arXiv 2017

  2. [2]

    https://doi.org/10.18653/v1/2022.acl-short.65 Krishna, K., Khosla, S., Bigham, J., & Lipton, Z. C. (2021). Generating SOAP notes from doctor- patient conversations using modular summarization techniques. ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pr...

    arXiv 2021

  3. [3]

    (2025, March 20)

    https://doi.org/10.18653/v1/2022.acl-long.394 OpenAI. (2025, March 20). Introducing next-generation audio models in the API. https://openai.com/index/introducing-our-next-generation-audio-models/ Peng, P., Yan, B., Watanabe, S., & Harwath, D. (2023). Prompting the Hidden Talent of Web- Scale Speech Models for Zero-Shot Task Generalization. Proceedings of ...

    doi:10.18653/v1/2022.acl-long.394 2023

  4. [202]

    Ramprasad, S., Ferracane, E., & Selvaraj, S. P. (2023). Generating more faithful and consistent SOAP notes using attribute-specific parameters. Proceedings of Machine Learning Research,

    work page 2023

  5. [219]

    pajama time

    Sara Berg. (2024). Burnout on the way down, but “pajama time” stands still. AMA. Sinsky, C., Colligan, L., Li, L., Prgomet, M., Reynolds, S., Goeders, L., Westbrook, J., Tutty, M., & Blike, G. (2016). Allocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties. Annals of Internal Medicine, 165(11), 753–760. https://doi.or...

    doi:10.7326/m16-0961 2024

  6. [2492]

    A., Gayre, G., Hoberman, B., Mattern, B., Ballesca, M., Kipnis, P., Liu, V., & Lee, K

    https://doi.org/10.21437/Interspeech.2022-10935 Tierney, A. A., Gayre, G., Hoberman, B., Mattern, B., Ballesca, M., Kipnis, P., Liu, V., & Lee, K. (2024). Ambient Artificial Intelligence Scribes to Alleviate the Burden of Clinical Documentation. NEJM Catalyst, 5(3). https://doi.org/10.1056/CAT.23.0404 Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & A...

    work page doi:10.21437/interspeech.2022-10935 2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.