BehaviorSFT: Behavioral Token Conditioning for Clinical Agents Across the Proactivity Spectrum

Chanwoo Park; Cynthia Breazeal; Daniel McDuff; Eugene Park; Hae Won Park; Hyeonhoon Lee; Hyewon Jeong; Mingyu Lu; Samir Tulebaev; Shiyun Xiong

REVIEW 4 major objections 5 minor 13 references

BehaviorSFT: Behavioral Token Conditioning for Clinical Agents Across the Proactivity Spectrum

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BehaviorSFT claims that prefixing clinical LLM responses with <reactive> or <proactive> tokens during fine-tuning raises proactive task scores (e.g., from 95.0% to 96.5%) without hurting reactive performance.

desk verdict Useful benchmark and a simple conditioning trick, but the headline F1 gains are smaller than the label noise in the LLM-generated ground truth, so treat the claims as promising rather than proven. read the letter →

arxiv 2505.21757 v1 pith:4FL2B2U2 submitted 2025-05-27 cs.CL

Yubin Kim , Zhiyuan Hu , Hyewon Jeong , Eugene Park , Shuyue Stella Li , Chanwoo Park , Shiyun Xiong , MingYu Lu

show 6 more authors

Hyeonhoon Lee Xin Liu Daniel McDuff Cynthia Breazeal Samir Tulebaev Hae Won Park

This is my paper · ORCID

classification cs.CL

keywords clinicalagentsproactivityspectrumbehavioraltokensBehaviorSFTBenchsupervisedfine-tuningreactiveandproactive

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that clinical AI agents need to adapt their behavior along a reactive–proactive spectrum, and that the right way to induce this is to condition fine-tuning on explicit behavior tokens rather than on prompts. To make the case, it introduces BehaviorBench, a dataset of 142,496 clinical tasks derived from 6,876 real NEJM case reports, spanning reactive, balanced, and proactive behaviors such as omission detection and consistency checking. The proposed method, BehaviorSFT, prepends a or token to the target response during supervised fine-tuning, teaching the model to infer from the input context which stance is appropriate. On BehaviorBench, BehaviorSFT achieves up to 97.3% overall Macro F1, and it lifts proactive task scores from 95.0% to 96.5% for Qwen2.5-7B-Ins, while reactive and balanced scores remain essentially flat. In a blind clinician evaluation, the BehaviorSFT agent received the best mean rank (1.80), indicating that behavior-conditioned outputs strike a balance between helpful initiative and restraint that clinicians prefer.

What carries the argument

The machinery is a pair of special tokens—<reactive> and <proactive>—inserted as the first token of the target response during supervised fine-tuning, using the standard causal language modeling objective. The tokens act as behavior control signals: <reactive> signals a direct, concise answer that stays within the query, while <proactive> signals responses that volunteer warnings, flag omissions, or apply external clinical knowledge. BehaviorBench supplies the training data, with each of the 142,496 task instances annotated for the appropriate token, including context-dependent annotations for balanced tasks. The model learns both to assess whether a situation calls for proactivity and to generate in the chosen mode, since the first token is predicted from the input context.

What would settle it

Run a prospective clinical evaluation on a held-out set of real patient records (not NEJM case reports), where clinicians label each response as appropriate, over-proactive, or under-proactive; if BehaviorSFT-trained agents do not outperform general SFT on the appropriateness rate—or if their proactive gains are offset by increased alert fatigue—the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that a simple training-time conditioning signal—prefixing the target sequence with <reactive> or <proactive>—lets a single LLM dynamically choose its level of initiative across a clinical assistance spectrum. The paper demonstrates this with BehaviorSFT, which uses the standard causal language modeling loss but prepends the behavior token to every target, so the model learns to predict the token from the clinical context and then generate in the corresponding mode. The result is a model that outperforms general SFT on proactive reasoning tasks (e.g., Qwen2.5-7B-Ins proactive scores go from 95.0% to 96.5%) while preserving reactive and balanced performance, and that clinicians rank highest in a blind comparison against general SFT and explicit-instructed agents. The paper positions this as a step toward Level 3 ('Conditional Proactive Assistance') autonomy in its proposed six-level healthcare AI autonomy taxonomy.

Load-bearing premise

The load-bearing premise is that BehaviorBench's LLM-generated task scenarios and answer labels correctly encode what counts as appropriate clinical proactivity, since only ten of the generated cases received physician review before the data was used for training and evaluation.

Editorial extensions

If this is right

BehaviorSFT raises proactive task Macro F1 from 95.0% to 96.5% on Qwen2.5-7B-Ins and from 94.2% to 94.7% on Llama3.1-8B-Ins, while reactive and balanced scores stay essentially flat.
The method requires no architectural change beyond adding two special tokens, so it can be applied to any autoregressive LLM for which behavior-annotated training data exists.
Because the behavior token is predicted from the input context, the trained model performs contextual behavior assessment as part of generation, rather than needing a separate classifier to route queries.
Clinician evaluators ranked BehaviorSFT outputs best (mean rank 1.80), indicating the behavior-conditioned balance between initiative and restraint is preferred over general SFT (2.08) and explicit instruction (2.12).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If BehaviorBench's generated notion of proactivity aligns with real clinician expectations, the same token-conditioning recipe is directly portable to other high-stakes agent domains—such as radiology triage or legal document review—where both over- and under-intervention are costly.
A sharper test of the mechanism would be to ablate the training data's proactive skew (73,810 highly proactive instances vs. 2,491 highly reactive) to see whether the F1 gains persist on a balanced distribution; the paper does not report this experiment.
The small clinician panel (three doctors, 24 comparative cases) leaves open whether the observed ranking generalizes across specialties and care settings; a multi-institution study would be needed to convert 'more realistic' into a robust preference estimate.
One could extend the approach to multi-label behavior tokens (e.g., <clarify_info>, <flag_safety>, <escalate_critical>)—which the authors list as future work—to test whether finer-grained conditioning reduces the occasional over-firing that the binary token produces.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this one for the benchmark, not for the headline numbers. BehaviorBench is a genuinely new artifact: 142K tasks from NEJM-derived vignettes, covering reactive, balanced, and proactive clinical behaviors with multi-turn dialogue and multimodal inputs. That fills a real gap. BehaviorSFT is a simple and plausible trick—prefix tokens conditioning the model to stay reactive or go proactive—and the authors are honest that control tokens and initiative-shifting are not new. What is new is applying it to this spectrum and showing a consistent directional gain on proactive tasks (Qwen 95.0 to 96.5, Llama 94.2 to 94.7) without hurting reactive or balanced performance.

The soft spots are not minor. The benchmark's ground truth is generated by Gemini-2.5 Flash, with only 10 cases physician-reviewed, and training and evaluation happen on splits of the same generated distribution. The appendix's own validation says clinicians agree with the generated answer only 83.3% of the time and are "Unsure" about plausibility for 20% of tasks. A 1.5-point F1 shift is smaller than that disagreement rate, so the headline improvement could easily be an artifact of fitting to noisy LLM labels. The clinician study is too small and internally inconsistent: Phase 1 says N=30, results say 60 tasks; Phase 2 says N=10, results say 24; the agent labels A/B/C don't even match between design and results. I would not lean on the clinician-preference claims at all. LLM-as-judge G-Eval on the same generated data is not an independent check.

What holds up: the effect is consistent across two backbones, the largest gains are on the hardest proactive tasks (omission detection, predictive next action), and the training method is straightforward enough to reproduce once the data is released. But no code or artifacts are available, which matters for a benchmark paper.

This paper deserves a serious referee, because the benchmark and task taxonomy are useful to the clinical-agent community. I'd send it to review with a clear request: error bars or significance tests, a human-validated held-out set, artifact release, and a clinician study that is internally coherent. As is, I read it as promising but conditional, not as an accepted demonstration.

Referee Report

4 major / 5 minor

Summary. The paper introduces BehaviorBench, a large-scale benchmark of 142,496 clinical agent tasks derived from NEJM case reports, and proposes BehaviorSFT, a fine-tuning method that prepends binary behavior tokens (<reactive>, <proactive>) to the target sequence to condition model responses along a reactive–proactive spectrum. The benchmark covers 13 task categories grouped into reactive, balanced, and proactive sets, with tasks generated by Gemini-2.5 Flash and light clinician review (N=10). The authors report that BehaviorSFT improves Macro F1 on BehaviorBench, especially on proactive tasks (Qwen2.5-7B-Ins: 95.0→96.5; Llama3.1-8B-Ins: 94.2→94.7), and that G-Eval and a small clinician study indicate better behavioral quality. The paper also proposes a six-level autonomy taxonomy for healthcare AI.

Significance. If the reported gains are reliable, the paper would offer a simple, practical mechanism—prefix behavioral tokens—for shifting clinical agents along the reactive–proactive spectrum, a direction of growing importance for safe deployment of LLM-based clinical assistants. The scale of BehaviorBench and its grounding in real NEJM cases are assets, and the training recipe (LoRA, single epoch, clear hyperparameters) is reproducible. However, the central evaluation is compromised by the fact that the benchmark's ground truth is generated by the same LLM family used to create the training data, with only cursory clinician validation, and the reported F1 differences are smaller than the measured human–generator disagreement. The clinician study, while a valuable attempt at independent assessment, has unresolved internal inconsistencies and partially contradicts the headline claim. The paper's own limitations section acknowledges that the binary token is too coarse and can over-fire, which tempers the claim of a 'superior balance.' These issues make the current evidence insufficient to establish the central claim, though the methodological ingredients are promising.

major comments (4)

[Section 2 and Appendix I.4.1] The benchmark's ground-truth validity is the load-bearing premise, and it is not established. Section 2 states that Gemini-2.5 Flash generates all tasks and that only N=10 cases receive physician review. The independent clinician validation in Appendix I.4.1 reports only 83.3% agreement with the generated ground-truth answers and 20% of tasks rated 'Unsure' for clinical plausibility. The headline proactive F1 gains in Table 2 (Qwen: 95.0 to 96.5; Llama: 94.2 to 94.7) are 0.5–1.5 points, far below the 16.7% label-disagreement rate. Since training and testing both use splits of the same Gemini-generated benchmark, the reported improvements may reflect adaptation to the generator's annotation quirks rather than to clinically appropriate proactivity. To support the claim, the authors should report performance on a larger clinician-validated holdout set, or provide agreement-adjusted metrics or confidence intervals that account for label noise.
[Section 4.2, Table 2] No statistical significance or variance reporting is provided. The proactive F1 improvements of 0.5–1.5 points are small, and no error bars, bootstrap intervals, or paired significance tests are given. Moreover, the claimed 'clear advantage' is not consistent across proactive tasks: BehaviorSFT is worse than Gen. SFT on standard_of_care (Qwen: 93.3 vs 94.8; Llama: 88.4 vs 91.5), on explicit_error_correction for Llama (97.2 vs 98.4), and on data_conflict_resolution for Llama (98.6 vs 99.2). The paper should report per-task variance and significance tests, and temper the claim of a consistent proactive advantage.
[Appendix I.2–I.4] The clinician study contains internal inconsistencies that undermine its use as independent confirmation. Phase 1 is designed with N=30 tasks but the results report 60 evaluated tasks; Phase 2 is designed with N=10 tasks but the results report N=24. The agent labeling also differs: the design defines (1) BehaviorSFT, (2) General SFT, (3) ZS + Explicit Instr., while the results refer to 'Agent A (BEHAVIORBENCH)' and 'Agent C (ZS + Explicit Instr.)'. Furthermore, the appropriateness Likert results are only partially reported: the text says Agent C scored highest (4.20) and Agent B scored 4.08, but does not report BehaviorSFT's score, which is necessary for comparing the ranking claim with the appropriateness claim. The conclusion that clinicians confirmed BehaviorSFT's superiority is not supported by a fully consistent, fully reported analysis.
[Section 4.3, Table 3] The ablation of prompting variants in Section 4.3 appears disconnected from the main method. It evaluates behavior classification with prompt recipes (BT, BC, OC, OP) rather than the BehaviorSFT fine-tuned models, and the reported Macro F1 scores are markedly lower (e.g., 42.6–58.2 for the five-class balanced category) than the main results. The text does not explain how this experiment relates to BehaviorSFT training, and the results are not used in the conclusion. Either the connection should be made explicit, or this section should be removed or reframed to avoid confusing the reader about the main contribution.

minor comments (5)

[Abstract/Table 2] The model name is written as 'Qwen2.5-7B-Ins' in the abstract and Table 2 but as 'Qwen-2.5-7B-Instruct' in Section 4.1; please standardize the notation across the paper.
[Section 4.2] The paragraph describing the 'Proactive-Reactive Scale of 0.0-0.4' appears without a prior definition of this scale in the main text; it reads like an excerpt from a different section and should be integrated or moved.
[Appendix I.1–I.4] Several cross-references are unresolved, including 'shown in Figure ??' for the plausibility results and the reference to 'Figure 16' for the feedback panel; please update all figure references.
[Appendix I.2] The Phase 2 description states that clinicians are shown the correct answer highlighted for reference, which conflicts with the abstract's use of the term 'blind' for the clinician evaluation; clarify what was blinded.
[References] Some reference entries are incomplete, with placeholder author lists such as 'and 1 others'; these should be completed for a journal submission.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BehaviorSFT's benchmark gains are evaluated on a held-out split of BehaviorBench; the auto-generated labels raise validity concerns, but no claim reduces by construction to its inputs.

full rationale

The paper's central empirical claim is that BehaviorSFT improves Macro F1 on BehaviorBench relative to baselines. This is supported by a fixed 6,776/110/977 train–val–test split (Section 4.1), with BehaviorSFT trained only on the training split and evaluated on the unseen test split. The improvement is therefore not statistically forced by construction: the test labels are not inputs to training, and the method could in principle have underperformed. The use of Gemini-2.5 Flash to draft all 142,496 tasks, with only N=10 physician-reviewed cases in Section 2 and 83.3% physician MCQ agreement and 20% 'Unsure' plausibility ratings in Appendix I.4.1, is a legitimate concern about label validity and the strength of the claim that gains reflect clinically appropriate proactivity. But that is a data-quality/external-validity issue, not circularity: the benchmark labels are not a fitted parameter renamed as a prediction, and no equation or definition reduces the reported F1 to the training labels. The clinician study, although under-powered and with internal inconsistencies (Phase 1 N=30 vs. 60; Phase 2 N=10 vs. 24), provides an independent, if limited, external check. No load-bearing self-citation or uniqueness theorem is invoked. I therefore find no circular step under the requested definitions.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The central claim rests on the validity of an LLM-generated benchmark, a binary token-based training signal, and a very small clinician study. These are powerful assumptions that are not independently verified at scale.

assumptions (4)

domain assumption LLM-generated tasks and labels from Gemini-2.5 Flash are clinically valid ground truth.
Section 2: tasks are generated by prompting Gemini-2.5 Flash with NEJM cases; only 10 cases were reviewed by two physicians. The remaining 142K task labels were not systematically human-validated.
domain assumption BehaviorBench test split is an independent measure of clinical behavior.
Test split sampled from the same LLM-generated distribution as the training split; BehaviorSFT is trained and evaluated on splits of this one benchmark, so in-distribution fit may be measured rather than external clinical quality.
ad hoc to paper Binary reactive/proactive tokens can capture the needed behavior spectrum.
Appendix B: authors acknowledge the binary switch cannot express nuances such as anticipatory clarification versus high-urgency escalation and occasionally over-fires.
domain assumption The clinician study sample is representative of real clinical judgment.
Appendix B: the study involves three medical doctors and is described as "under-powered"; Phase 2 uses only 10 to 24 tasks, which is far too small for robust conclusions about behavior.

invented entities (1)

Binary behavior tokens (<reactive>, <proactive>)
purpose: Condition the LLM's generation mode during SFT and inference
Introduced as the core training mechanism; no external validation that a binary toggle is sufficient for the clinical behavior spectrum. The authors themselves note in Appendix B that the binary switch cannot express nuances.

how reviews work

0 comments

Cite this review

Pith. "Pith review of BehaviorSFT: Behavioral Token Conditioning for Clinical Agents Across the Proactivity Spectrum." pith.science (2026). https://pith.science/paper/4FL2B2U2

@misc{pith2026250521757,
  author       = {Pith},
  title        = {Pith review of: BehaviorSFT: Behavioral Token Conditioning for Clinical Agents Across the Proactivity Spectrum},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4FL2B2U2}},
  note         = {Machine review of arXiv:2505.21757}
}

read the original abstract

Large Language Models (LLMs) as clinical agents require careful behavioral adaptation. While adept at reactive tasks (e.g., diagnosis reasoning), LLMs often struggle with proactive engagement, like unprompted identification of critical missing information or risks. We introduce BehaviorBench, a comprehensive dataset to evaluate agent behaviors across a clinical assistance spectrum, ranging from reactive query responses to proactive interventions (e.g., clarifying ambiguities, flagging overlooked critical data). Our BehaviorBench experiments reveal LLMs' inconsistent proactivity. To address this, we propose BehaviorSFT, a novel training strategy using behavioral tokens to explicitly condition LLMs for dynamic behavioral selection along this spectrum. BehaviorSFT boosts performance, achieving up to 97.3% overall Macro F1 on BehaviorBench and improving proactive task scores (e.g., from 95.0% to 96.5% for Qwen2.5-7B-Ins). Crucially, blind clinician evaluations confirmed BehaviorSFT-trained agents exhibit more realistic clinical behavior, striking a superior balance between helpful proactivity (e.g., timely, relevant suggestions) and necessary restraint (e.g., avoiding over-intervention) versus standard fine-tuning or explicit instructed agents.

Figures

Figures reproduced from arXiv: 2505.21757 by the authors.

**Figure 1.** Six representative tasks from BEHAVIORBENCH, showcasing the spectrum of agent behaviors in clinical settings. The figure illustrates (a-c, f) proactive tasks where the LLM agent identifies issues or offers insights without direct prompting, and (b, d, e) reactive tasks responding to explicit clinician queries. these autonomy levels, its capacity and responsibility to engage in sophisticated proactive behaviors, rat… view at source ↗

**Figure 2.** [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** G-Eval with gpt-4o-mini as evaluator of Qwen-2.5-7B-Ins responses across four key metrics. We compare the average scores for the Baseline model, our proposed BehaviorSFT, and GeneralSFT. BehaviorSFT consistently outperforms the Baseline across all metrics and demonstrates competitive or superior performance compared to GeneralSFT. 4 Experiments and Results 4.1 Setup All experiments use BEHAVIORBENCH with a fixed 6… view at source ↗

Figures from the paper (16 more)

**Figure 4.** Figure 4: The Landscape of Healthcare AI Systems and Enabling Frameworks. Systems are positioned based on their primary Task Scope (Narrow, Medium, or Broad) and their demonstrated level of System Autonomy. The autonomy levels are derived from the Six-Level Taxonomy for Healthca…

**Figure 5.** Figure 5: Performance comparison on BEHAVIORBENCH for Few-Shot (k=3); Gen. SFT, and our proposed BehaviorSFT. Tasks are colored based on task category: Reactive, Balanced, and Proactive. The radar plot illustrates that our BehaviorSFT achieves best or second-best performance acr…

**Figure 6.** Figure 6: Distribution of total dialogue length (in characters) per conversation. This metric captures the overall verbosity of clinical discussions. Most conversations range between 3000 and 5000 characters in length, indicating substantial detail per case [PITH_FULL_IMAGE:fig…

**Figure 7.** Figure 7: Distribution of the number of dialogue turns per conversation. Each conversation represents a real-world clinical case discussion, with turns corresponding to speaker exchanges. The majority of cases fall between 15 and 30 turns [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

**Figure 8.** Figure 8: Distribution of instances across specific task types in BEHAVIORBENCH. Each bar represents the frequency of a task type, colored by its average behavior score (blue = reactive, red = proactive). This illustrates the diversity of evaluation scenarios, spanning a wide ra…

**Figure 9.** Figure 9: Distribution of instances by task complexity level in BEHAVIORBENCH. Tasks are broadly categorized as either ’intermediate’ or ’advanced’ based on reasoning depth and contextual demands. The dataset skews toward advanced tasks, aligning with the goal of evaluating high…

**Figure 10.** Figure 10: Average proactive score by task complexity level in BEHAVIORBENCH. Tasks labeled as ‘advanced’ exhibit a significantly higher average proactive score (above 0.8) compared to ‘intermediate’ tasks (around 0.4), highlighting the alignment between task complexity and expe…

**Figure 11.** Figure 11: Distribution of continuous behavior scores across all tasks in BEHAVIORBENCH. The behavior score ranges from 0.0 (fully reactive) to 1.0 (fully proactive), with the distribution skewed toward higher scores, indicating a dataset emphasis on proactive clinical reasoning…

**Figure 12.** Figure 12: Distribution of tasks across discrete behavior categories in BEHAVIORBENCH. Tasks are grouped into five categories, ranging from ‘highly reactive’ to ‘highly proactive’ to support structured evaluation of agent behavior along the autonomy spectrum. Phase 2: Comparativ…

**Figure 13.** Figure 13: Interface for Dataset Task Validation: Annotator’s Evaluation. Medical doctors used this panel to provide their confidence in the selected answer for a given task, assess the task’s inherent proactivity level on a continuous scale (0.0 Reactive to 1.0 Proactive), and …

**Figure 14.** Figure 14: Interface for Dataset Task Validation: Task Presentation. This view provided clinicians with the ‘Task Context‘ (relevant excerpts from the case), the specific ‘Question‘ being posed for the BehaviorBench task, and the multiple-choice ‘Options‘, one of which was the g…

**Figure 15.** Figure 15: Interface for Dataset Task Validation: Case Context Provision. To ensure comprehensive understanding, clinicians had access to the broader ‘Case Context‘, including a ‘Case Presentation Summary‘ (if available from the task file), the ‘Full Conversation‘ transcript le…

**Figure 16.** Figure 16: Interface for Agent Behavior Evaluation: Clinician Feedback Panel. After reviewing the task and agent responses (shown in Figure ??), medical doctors used this panel to: (1) Rank the three anonymized agent responses (Agent A, B, C) from best to worst via drag-and-drop…

**Figure 17.** Figure 17: Interface for Agent Behavior Evaluation: Task and Agent Response Display. For each evaluation scenario, clinicians were presented with the ‘Question Posed to AI‘ and the ‘Task Options‘ (with the correct answer highlighted for reference). Below this, the distinct respo…

**Figure 18.** Figure 18: (a) Over half (55.0%) of the responses were marked as ‘High’ confidence, while ‘Moderate’ confidence accounted for 36.7%. ‘Low’ confidence was the least frequent category, representing only 8.3% of responses. (b) The vast majority (80.0%) of responses affirmed the cli…

**Figure 19.** Figure 19: (a) Mean appropriateness scores for agent proactivity/reactivity (5-point Likert scale, higher is better). (b) BehaviorSFT received the lowest (best) mean rank (1.80), suggesting it was most frequently ranked highest by evaluators. Gen. SFT had a mean rank of 2.08, wh…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages

[6]

Allison B McCoy, Eric J Thomas, Marie Krousel-Wood, and Dean F Sittig

Wearable ai to enhance patient safety and clini- cal decision-making.npj Digital Medicine, 8(1):176. Allison B McCoy, Eric J Thomas, Marie Krousel-Wood, and Dean F Sittig. 2014. Clinical decision support alert appropriateness: a review and proposal for im- provement.Ochsner journal, 14(2):195–202. Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy...

work page 2014
[7]

arXiv preprint arXiv:2404.18416

Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mah- davi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, and 1 others. 2023. Large language models encode clinical knowledge.Nature, 620(7972):172–180. Reed T Sutton, David Pincock, Daniel...

arXiv 2023
[8]

Bethany A Van Dort, Wu Yi Zheng, Vivek Sundar, and Melissa T Baysari

Towards conversational diagnostic ai.arXiv preprint arXiv:2401.05654. Bethany A Van Dort, Wu Yi Zheng, Vivek Sundar, and Melissa T Baysari. 2021. Optimizing clinical deci- sion support alerts in electronic medical records: a systematic review of reported strategies adopted by hospitals.Journal of the American Medical Informat- ics Association, 28(1):177–1...

arXiv 2021
[9]

Evalagent: Discovering implicit evaluation cri- teria from the web.arXiv preprint arXiv:2504.15219. Maria Beatriz Walter Costa, Mark Wernsdorfer, Alexan- der Kehrer, Markus V oigt, Carina Cundius, Martin Federbusch, Felix Eckelt, Johannes Remmler, Maria Schmidt, Sarah Pehnke, and 1 others. 2021. The clinical decision support system ampel for laboratory di...

work page Pith review arXiv 2021
[13]

Behavioral Chasm,

or ClinicBench (Chen et al., 2024). These primarily focus on knowledge accuracy, reason- ing over medical facts, or agentic task completion. While some, like MediQ (Li et al., 2024), touch upon aspects of proactivity (information-seeking), there is a lack of systematic frameworks to eval- uate and train LLMs specifically on their ability to dynamically ad...

work page arXiv 2024
[2014]

In AMIA Annual Symposium Proceedings, volume 2014, page 845

Medical alert management: a real-time adap- tive decision support tool to reduce alert fatigue. In AMIA Annual Symposium Proceedings, volume 2014, page 845. Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan Ilgen, Emma Pierson, Pang Wei W Koh, and Yulia Tsvetkov. 2024. Mediq: Question- asking llms and a benchmark for reliable interactive cl...

arXiv 2014
[2015]

InMayo Clinic Proceedings, vol- ume 90, pages 469–480

Digital health interventions for the preven- tion of cardiovascular disease: a systematic review and meta-analysis. InMayo Clinic Proceedings, vol- ume 90, pages 469–480. Elsevier. Adam Wright, Skye Aaron, Diane L Seger, Lipika Samal, Gordon D Schiff, and David W Bates. 2018. Reduced effectiveness of interruptive drug-drug in- teraction alerts after conve...

work page 2018
[2019]

Yixing Jiang, Kameron C Black, Gloria Geng, Danny Park, Andrew Y Ng, and Jonathan H Chen

Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review.Journal of the American Medical Informatics Association, 26(10):1141–1149. Yixing Jiang, Kameron C Black, Gloria Geng, Danny Park, Andrew Y Ng, and Jonathan H Chen. 2025. Medagentbench: Dataset for benchmarking llms as agents in medical a...

arXiv 2025

Show all 13 references

[2021]

Surface Vehicle Recommended Practice

Taxonomy and definitions for terms related to driving automation systems for on-road motor vehi- cles. Surface Vehicle Recommended Practice. Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Pre- ston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan...

2025 arXiv
[2022]

Michael JQ Zhang and Eunsol Choi

Linkbert: Pretraining language models with document links.arXiv preprint arXiv:2203.15827. Michael JQ Zhang and Eunsol Choi. 2023. Clarify when necessary: Resolving ambiguity through interaction with lms.arXiv preprint arXiv:2311.09469. 10 A Related Works The Evolving Role of ...

2023 arXiv
[2023]

Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shen- gran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha

Autogen: Enabling next-gen llm applica- tions via multi-agent conversation.arXiv preprint arXiv:2308.08155. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shen- gran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. 2025. The ai scientist-v2: Workshop-level automated scien...

2025 arXiv
[2024]

In2024 12th International Symposium on Digital Forensics and Security (IS- DFS), pages 1–7

Llm-based framework for administrative task automation in healthcare. In2024 12th International Symposium on Digital Forensics and Security (IS- DFS), pages 1–7. IEEE. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Feli...

2025 arXiv
[2025]

alert fatigue

Understanding “alert fatigue” in primary care: Qualitative systematic review of general practitioners attitudes and experiences of clinical alerts, prompts, and reminders.Journal of Medical Internet Research, 27:e62763. Senay A Gebreab, Khaled Salah, Raja Jayaraman, Muhammad H...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [6]

Allison B McCoy, Eric J Thomas, Marie Krousel-Wood, and Dean F Sittig

Wearable ai to enhance patient safety and clini- cal decision-making.npj Digital Medicine, 8(1):176. Allison B McCoy, Eric J Thomas, Marie Krousel-Wood, and Dean F Sittig. 2014. Clinical decision support alert appropriateness: a review and proposal for im- provement.Ochsner journal, 14(2):195–202. Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy...

work page 2014

[2] [7]

arXiv preprint arXiv:2404.18416

Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mah- davi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, and 1 others. 2023. Large language models encode clinical knowledge.Nature, 620(7972):172–180. Reed T Sutton, David Pincock, Daniel...

arXiv 2023

[3] [8]

Bethany A Van Dort, Wu Yi Zheng, Vivek Sundar, and Melissa T Baysari

Towards conversational diagnostic ai.arXiv preprint arXiv:2401.05654. Bethany A Van Dort, Wu Yi Zheng, Vivek Sundar, and Melissa T Baysari. 2021. Optimizing clinical deci- sion support alerts in electronic medical records: a systematic review of reported strategies adopted by hospitals.Journal of the American Medical Informat- ics Association, 28(1):177–1...

arXiv 2021

[4] [9]

Evalagent: Discovering implicit evaluation cri- teria from the web.arXiv preprint arXiv:2504.15219. Maria Beatriz Walter Costa, Mark Wernsdorfer, Alexan- der Kehrer, Markus V oigt, Carina Cundius, Martin Federbusch, Felix Eckelt, Johannes Remmler, Maria Schmidt, Sarah Pehnke, and 1 others. 2021. The clinical decision support system ampel for laboratory di...

work page Pith review arXiv 2021

[5] [13]

Behavioral Chasm,

or ClinicBench (Chen et al., 2024). These primarily focus on knowledge accuracy, reason- ing over medical facts, or agentic task completion. While some, like MediQ (Li et al., 2024), touch upon aspects of proactivity (information-seeking), there is a lack of systematic frameworks to eval- uate and train LLMs specifically on their ability to dynamically ad...

work page arXiv 2024

[6] [2014]

In AMIA Annual Symposium Proceedings, volume 2014, page 845

Medical alert management: a real-time adap- tive decision support tool to reduce alert fatigue. In AMIA Annual Symposium Proceedings, volume 2014, page 845. Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan Ilgen, Emma Pierson, Pang Wei W Koh, and Yulia Tsvetkov. 2024. Mediq: Question- asking llms and a benchmark for reliable interactive cl...

arXiv 2014

[7] [2015]

InMayo Clinic Proceedings, vol- ume 90, pages 469–480

Digital health interventions for the preven- tion of cardiovascular disease: a systematic review and meta-analysis. InMayo Clinic Proceedings, vol- ume 90, pages 469–480. Elsevier. Adam Wright, Skye Aaron, Diane L Seger, Lipika Samal, Gordon D Schiff, and David W Bates. 2018. Reduced effectiveness of interruptive drug-drug in- teraction alerts after conve...

work page 2018

[8] [2019]

Yixing Jiang, Kameron C Black, Gloria Geng, Danny Park, Andrew Y Ng, and Jonathan H Chen

Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review.Journal of the American Medical Informatics Association, 26(10):1141–1149. Yixing Jiang, Kameron C Black, Gloria Geng, Danny Park, Andrew Y Ng, and Jonathan H Chen. 2025. Medagentbench: Dataset for benchmarking llms as agents in medical a...

arXiv 2025