Specialty-Specific Medical Language Model for Immune-Mediated Diseases

Ace Vo; David Talby; Gursev Pirge; Veysel Kocaman; Yigit Gul; Zhenya Nargizyan

Specialty-Specific Medical Language Model for Immune-Mediated Diseases

T0 review · 3 major / 0 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Tool-calling LLM agents reliably pick the same tools in the same order, yet vary in arguments; only the tool sequence predicts whether the task succeeds.

desk verdict Solid empirical agent-reliability study: structural tool sequences stay stable while arguments wander, and only the structural layer tracks success—worth reading if you ship tool-calling agents, with the usual T=1.0 / simulated-tools caveats. read the letter →

arxiv 2605.28838 v1 pith:CSBANBSI submitted 2026-04-12 cs.CL cs.AI

Veysel Kocaman , Gursev Pirge , Yigit Gul , Ace Vo , Zhenya Nargizyan , David Talby This is my paper · ORCID

classification cs.CLcs.AI

keywords LLMagentstoolcallingbehavioralconsistencyreproducibilitysequencesimilarityargumentagentreliability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks a basic reliability question about production LLM agents that call tools: if you run the same agent on the same task twice, do you get the same behavior? Across six models, 19 tasks, and 1,140 traces, the authors show a clear split: agents are highly consistent at the structural level (which tools, in which order) but far less consistent at the argument level (the concrete parameters they fill in). Structural consistency strongly predicts task correctness; argument-level variance does not. Ambiguous task wording hurts consistency more than model choice, and most divergence appears in the first one or two steps. The practical upshot is a lightweight, label-free signal—tool-sequence similarity—that can be used for testing, monitoring, and routing without needing ground-truth answers.

What carries the argument

Tool Sequence Similarity (TSS): a normalized Levenshtein measure of how often repeated runs produce the same ordered list of tool names. It is the central reliability proxy because it alone predicts correctness without needing labels, while the companion Argument Consistency (AC) metric does not.

What would settle it

Re-run the same 19-task suite at temperature 0 (or a full temperature sweep) with non-deterministic real APIs and human-scored correctness; if the TSS–correctness link (high-TSS ~90% vs low-TSS ~61%) disappears or reverses, the reliability-proxy claim fails.

Watch

Extended reading notes

Core claim

Multi-step tool-calling agents exhibit structural consistency with parametric variance: mean tool-sequence similarity (TSS) is 0.87 while mean argument consistency (AC) is 0.69 (large, highly significant gap). High-TSS conditions reach 90.2% correctness versus 61.2% for low-TSS, whereas argument consistency shows no reliable link to success. Task ambiguity reduces argument consistency more than switching models does, and roughly 60% of first divergences occur in the first two pipeline steps.

Load-bearing premise

The entire consistency picture is measured only at temperature 1.0 with fully deterministic simulated tools and a pattern-based correctness rubric, so it may not hold under other temperatures, real APIs whose outputs vary, or full human semantic judgment.

Editorial extensions

If this is right

Unit tests and production monitors should assert on tool names and early tool calls, not on free-text final answers (exact-match rates stay under 5%).
Clarifying ambiguous task wording yields larger consistency gains than swapping models.
A cheap early-step check (first 1–2 tool calls against a reference) catches the majority of behavioral divergence.
TSS can serve as an online, correctness-free reliability signal for routing or human review.
Consistency-critical workflows can prefer models that score higher on TSS (e.g., certain smaller proprietary models over open 70B models in this suite).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If structural schemas are learned more cleanly than argument instantiation, future tool-use fine-tuning could explicitly regularize argument formats without sacrificing the procedural consistency that already predicts success.
The same structural/parametric split may appear in multi-turn web or coding agents; measuring early-step TSS there would test whether the monitoring recipe generalizes beyond the simulated tool suite.
Consistency-aware routing that predicts expected TSS from task features alone could cut cost by sending low-stakes or high-ambiguity work to cheaper models.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The metadata points at a medical NER paper, but the manuscript we actually got is the agent-consistency study (tool-calling reproducibility). I’m writing about that text.

What’s new is not “agents are inconsistent”—Mehta et al. already showed that for ReAct/search. What’s useful is the split: same tools in the same order (TSS ≈ 0.87) but arguments jump around (AC ≈ 0.69), with a large paired gap, and only TSS tracks task success (high-TSS ~90% correct vs low-TSS ~61%; AC uncorrelated). That makes TSS a cheap, label-free reliability signal, plus the practical bits: ambiguity hurts more than model choice, ~60% of first divergences land in the first two steps, and final text almost never exact-matches even when the tool plan is fixed.

They did the experiment carefully for what it is. Deterministic tools isolate model variance. Six models, 19 tasks, 1,140 traces. Metrics are defined cleanly; correctness is a separate rubric, not circular with TSS. Stats are reported with CIs, effect sizes, and basic corrections. Full code/traces release is real credit.

Soft spots are scope, not internal collapse. Everything is at temperature 1.0; tools never nondeterministically fail; correctness is pattern/regex-style, not human semantic judgment; N=10 per cell and only 3–4 tasks per category. So treat “use TSS in production” as a well-supported hypothesis inside this setup, not a settled ops standard. The structural/parametric story still holds on the data they show.

Who it’s for: people building or evaluating multi-step tool agents who need testing and monitoring invariants. Not a theory paper. I’d send it to peer review; it’s honest, reproducible, and actionable enough to deserve referee time. I’d bring it to reading group if the group cares about agent reliability; I’d cite the TSS/AC split and the early-divergence monitoring idea if I were writing on agent eval or production guardrails.

Referee Report

3 major / 0 minor

Summary. This paper presents a systematic empirical study of behavioral consistency in multi-step tool-calling LLM agents. Using 19 tasks across five categories, 10 deterministic simulated tools, and six models (1,140 traces), the authors formalize Tool Sequence Similarity (TSS) and Argument Consistency (AC) and report a robust “structural consistency, parametric variance” pattern: mean TSS = 0.87 vs. mean AC = 0.69 (d = 0.75, p < 10^{-13}). They further show that TSS predicts task success under a structured rubric (high-TSS 90.2% vs. low-TSS 61.2%, d = 0.81) while AC does not (r = 0.12, n.s.), that ambiguity reduces consistency more than model choice, and that ~60% of first divergences occur in the first two steps. Code, traces, and analysis scripts are released.

Significance. The work addresses a practically important and understudied reliability dimension for production tool-calling agents—reproducibility under identical inputs—rather than only capability. Strengths include a clear layered metric framework, isolation of model variance via deterministic tools, multi-provider evaluation, effect sizes with CIs and multiplicity control, an explicit correctness analysis that separates structural from parametric effects, and a full public release of code and traces. If the TSS–success link generalizes beyond the current setup, TSS is a useful, label-free monitoring signal. The structural/parametric distinction and early-divergence result are actionable for testing and deployment even if some external-validity caveats remain.

major comments (3)

§4.3 and Limitations §6.5: All main results are at temperature 1.0 only, with no T=0 (or other-T) baseline. The central deployment claim that TSS is a practical reliability proxy (§5.5, §6.2–6.3) depends on how much of the observed variance is sampling noise versus model/policy structure. A T=0 (or low-T) condition is load-bearing for interpreting the structural/parametric gap and for the production monitoring recommendations; without it, the causal attribution to sampling-time argument variance remains incomplete.
§4.5 and §5.5: Correctness is defined by required-tool coverage, key argument patterns, and regex-style output checks. Because component (1) is tool-set coverage, the large TSS–correctness association (d = 0.81) may partly reflect that runs which mutually agree on tool sequences also tend to cover the required tools—i.e., a mechanical link between the structural metric and the dominant correctness criterion—rather than full semantic task success. The paper correctly notes that high TSS is not “consistently wrong,” but it should quantify how much of the TSS–success correlation is driven by required-tool coverage alone versus argument/output failures, and/or validate a human-judged subset as promised in §6.5.
§3.2 Definition 4 (AC) and §6.5: AC is step-index aligned and assigns AC = 0 when tools differ at a step, so AC partially absorbs structural divergence by construction. The claim that “argument-level variance is benign” (r = 0.12, n.s.) is therefore harder to interpret on misaligned steps. A disentangled AC (e.g., computed only on steps with matching tool names, or after sequence alignment) is needed to support the structural-vs-parametric separation as cleanly as Hypothesis 1 states.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical measurement study with independently defined metrics; no definitional or self-citation circularity in the structural/parametric claim.

full rationale

This paper is a controlled empirical study of behavioral reproducibility in multi-step tool-calling agents, not a first-principles derivation. TSS and AC are defined independently via Levenshtein distance on tool-name sequences and Jaccard overlap on argument key-value sets (Definitions 3–4, §3.2). Correctness is scored with a separate three-component rubric (required tool coverage, argument validity patterns, output completeness; §4.5), so high TSS does not force high correctness by construction—consistently wrong tool sequences would yield high TSS and low correctness, a case the authors explicitly rule out. The reported TSS–correctness link (high-TSS 90.2% vs low-TSS 61.2%, d=0.81) and the null AC–correctness correlation (r=0.12, n.s.) are therefore empirical findings, not tautologies. There is no fitted parameter renamed as a prediction, no uniqueness theorem imported from the same authors, and no load-bearing self-citation chain. Scope limits (T=1.0 only, deterministic simulated tools, structured correctness proxy) are methodological caveats, not circularity. Score 0 is appropriate.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

Load-bearing content is empirical, not axiomatic physics. The claim rests on chosen metrics, a fixed temperature, deterministic tools, a hand-built 19-task suite, and a structured correctness rubric. No new physical entities; free design choices and domain modeling assumptions dominate.

free parameters (5)

temperature
Fixed at 1.0 for all runs; consistency–capability trade-off at other temperatures (including T=0) is unmeasured and acknowledged as a limitation.
runs_per_cell N
N=10 traces per model×task; split-half reliability only moderate (r=0.66); aggregate power is high but per-cell estimates are noisy.
max_tool_loop_iterations
Agent loop capped at 10 iterations; could truncate some hard multi-tool behaviors.
task_suite_composition
19 hand-designed tasks in five categories with 3–4 tasks each; category effects (especially ambiguous vs structured) depend on this small, author-chosen sample.
correctness_rubric_patterns
Required tools and regex/pattern argument checks define success; different patterns could re-rank models or weaken the TSS–correctness correlation.

assumptions (4)

domain assumption Identical inputs with deterministic tools imply that residual trace variance is attributable to the model’s generation process.
Stated in §4.2; isolates LLM variance but may not transfer to real APIs with nondeterministic outputs.
ad hoc to paper Tool-name Levenshtein similarity (TSS) and step-aligned argument Jaccard (AC) are appropriate layers for behavioral consistency.
Definitions 3–4 in §3.2; reasonable but design choices (edit distance, step-index alignment) affect measured gaps.
ad hoc to paper A trace is correct iff required tools, key argument patterns, and output completeness checks all pass.
§4.5 rubric; proxy for semantic task success, not human judgment.
domain assumption Provider-native tool-calling APIs with a minimal shared system prompt are a fair cross-model comparison setting.
§4.3–4.4; prompt intentionally minimal to avoid anchoring strategies.

invented entities (2)

Tool Sequence Similarity (TSS)
purpose: Scalar measure of agreement of tool-name sequences across repeated runs.
Defined in Def. 3; operational metric, not an external natural kind. Independent evidence is only the paper’s own correlations with correctness.
Argument Consistency (AC)
purpose: Scalar measure of argument key-value agreement on step-aligned calls.
Defined in Def. 4; same status as TSS—paper-internal measurement construct.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Specialty-Specific Medical Language Model for Immune-Mediated Diseases." pith.science (2026). https://pith.science/paper/CSBANBSI

@misc{pith2026260528838,
  author       = {Pith},
  title        = {Pith review of: Specialty-Specific Medical Language Model for Immune-Mediated Diseases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CSBANBSI}},
  note         = {Machine review of arXiv:2605.28838}
}

read the original abstract

Extracting detailed clinical information from free-text medical narratives remains a practical challenge for researchers and healthcare systems. Terminology for immune-mediated and infectious diseases is especially inconsistent across sources, which often limits the ability of general-purpose Natural Language Processing (NLP) systems to capture the relevant biomedical concepts with sufficient granularity. We developed a domain-specific Named Entity Recognition (NER) model tailored to identify disease-related entities occurring in immunology and infectious disease contexts. We assembled and manually annotated a dataset of 371 case reports in collaboration with two clinical specialists, defining twelve entity classes covering immune-mediated and infectious conditions as well as related symptoms and clinical descriptors. We evaluated several modeling strategies, including the MedicalNER architecture with multiple healthcare-specific embeddings, a BERT-based token classification model, and zero-shot NER systems. The strongest performance was obtained with a transformer-based model trained on clinical-domain embeddings, which reached an F1 score of 0.89, consistently outperforming baseline and zero-shot approaches. The combination of specialized embeddings and expert annotation proved particularly valuable for capturing nuanced disease terminology and improving generalization across heterogeneous biomedical text. The prompted LLM baseline achieved substantially lower performance under the same evaluation protocol, reflecting difficulties in producing span-consistent outputs for fine-grained entity boundaries despite detailed prompting. The resulting model provides a structured way to analyze case reports and can support downstream tasks such as cohort identification, disease monitoring, and clinical decision support.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 5 linked inside Pith

[1]

and Narayanan, A

Kapoor, S. and Narayanan, A. AI Agents That Matter.arXiv preprint arXiv:2407.01502,

arXiv
[2]

When Agents Disagree With Themselves: Measuring Behavioral Consistency in LLM-Based Agents.arXiv preprint arXiv:2602.11619,

Mehta, A., Ramesh, A., and Singla, A. When Agents Disagree With Themselves: Measuring Behavioral Consistency in LLM-Based Agents.arXiv preprint arXiv:2602.11619,

arXiv
[3]

G., Zhang, T., Wang, X., and Gonzalez, J

Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla: Large Language Model Connected with Massive APIs.arXiv preprint arXiv:2305.15334,

arXiv
[4]

and Guven, E

Renze, M. and Guven, E. Self-Reflection in LLM Agents: Effects on Problem-Solving Performance.arXiv preprint arXiv:2405.06682,

arXiv
[5]

Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design.arXiv preprint arXiv:2310.11324,

Sclar, M., Choi, Y., Tsvetkov, Y., and Suhr, A. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design.arXiv preprint arXiv:2310.11324,

arXiv
[6]

Ethical and Social Risks of Harm from Language Models.arXiv preprint arXiv:2112.04359,

Weidinger, L., Mellor, J., Rauh, M., et al. Ethical and Social Risks of Harm from Language Models.arXiv preprint arXiv:2112.04359,

arXiv
[7]

Meeting moved to 3pm tomorrow

12 Appendix A. Full Task Benchmark Table 3 lists all 19 benchmark tasks used in the study. Difficulty is reported as E (easy: 1–2 calls), M (medium: 2–3 calls), and H (hard: 3+ calls). Expected tools denote the minimal correct solution pattern. Table 3: All 19 benchmark tasks used in the evaluation. ID Diff. Task Instruction Expected Tools Data Retrieval ...

2026

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.

[1] [1]

and Narayanan, A

Kapoor, S. and Narayanan, A. AI Agents That Matter.arXiv preprint arXiv:2407.01502,

arXiv

[2] [2]

When Agents Disagree With Themselves: Measuring Behavioral Consistency in LLM-Based Agents.arXiv preprint arXiv:2602.11619,

Mehta, A., Ramesh, A., and Singla, A. When Agents Disagree With Themselves: Measuring Behavioral Consistency in LLM-Based Agents.arXiv preprint arXiv:2602.11619,

arXiv

[3] [3]

G., Zhang, T., Wang, X., and Gonzalez, J

Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla: Large Language Model Connected with Massive APIs.arXiv preprint arXiv:2305.15334,

arXiv

[4] [4]

and Guven, E

Renze, M. and Guven, E. Self-Reflection in LLM Agents: Effects on Problem-Solving Performance.arXiv preprint arXiv:2405.06682,

arXiv

[5] [5]

Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design.arXiv preprint arXiv:2310.11324,

Sclar, M., Choi, Y., Tsvetkov, Y., and Suhr, A. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design.arXiv preprint arXiv:2310.11324,

arXiv

[6] [6]

Ethical and Social Risks of Harm from Language Models.arXiv preprint arXiv:2112.04359,

Weidinger, L., Mellor, J., Rauh, M., et al. Ethical and Social Risks of Harm from Language Models.arXiv preprint arXiv:2112.04359,

arXiv

[7] [7]

Meeting moved to 3pm tomorrow

12 Appendix A. Full Task Benchmark Table 3 lists all 19 benchmark tasks used in the study. Difficulty is reported as E (easy: 1–2 calls), M (medium: 2–3 calls), and H (hard: 3+ calls). Expected tools denote the minimal correct solution pattern. Table 3: All 19 benchmark tasks used in the evaluation. ID Diff. Task Instruction Expected Tools Data Retrieval ...

2026