Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Fact-Controlled Diagnosis of Hallucinations in Medical Text Summarization

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that fact alignment, which extracts atomic facts from a clinical summary and its source transcript and counts the unsupported ones, is the most reliable and explainable way to detect hallucinations in medical…

desk verdict Useful datasets and a cautionary transfer result; the generalization claim runs ahead of the evidence. read the letter →

arxiv 2506.00448 v1 pith:LCHIMLNG submitted 2025-05-31 cs.CL

Suhas BN , Han-Chin Shing , Lei Xu , Mitch Strong , Jon Burnsky , Jessica Ofor , Jordan R. Mason , Susan Chen
show 4 more authors
Sundararajan Srinivasan Chaitanya Shivade Jack Moriarty Joseph Paul Cohen
This is my paper
classification cs.CL
keywords hallucinationdetectionclinicalsummarizationfactalignmentLeave-N-OutdatasetnaturalLLMevaluationmetricsSOAPnotesexplainability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether existing hallucination detectors, built for general summarization, can be trusted on clinical summaries of patient-clinician dialogues. To make the question testable, the authors build two expert-annotated datasets: a controlled Leave-N-Out set, where facts are systematically removed from transcripts while summaries keep them, and a Natural Hallucination set drawn from real LLM summaries of clinical visits. They report that most general-domain metrics correlate weakly with the number of injected or naturally occurring hallucinations, and that success on the controlled set does not reliably predict success on natural errors. Their central claim is that an LLM-based fact-alignment detector, which extracts atomic facts from both summary and transcript and counts unsupported summary facts, generalizes across datasets and outperforms existing methods, reaching correlations up to 0.43 on synthetic and 0.37 on real-world data. If right, such counting methods provide a more explainable and clinically useful signal than scalar quality scores.

What carries the argument

The central object is the fact-alignment pipeline: an LLM extracts concise atomic facts separately from the transcript and the summary, then a second LLM pass aligns the summary facts against the transcript facts and returns the list of unsupported facts, which is counted to produce a hallucination score. Atomic facts are individual clinically meaningful claims such as age, sex, symptoms, or diagnosis, with statements like 'Patient is a 35-year-old male with hypertension' decomposed into three separate facts. The Leave-N-Out (LNO) construction is the controlled-evaluation machinery: an LLM rewrites the transcript to delete all occurrences of N selected summary facts while the summary is left unchanged, so the number of injected hallucinations is known exactly. The Natural Hallucination (NH) dataset supplies organically occurring errors annotated by clinical experts, and Pearson correlation between detector output and the known or annotated count N is the sensitivity measure.

What would settle it

Run the fact-alignment detector on a new collection of clinician-patient dialogues with independently adjudicated hallucination annotations; if its Pearson correlation with those expert counts does not exceed existing baselines such as FENICE or SummaC, the paper's generalization claim is false.

Watch

Extended reading notes

Core claim

The paper's central claim is that decomposing hallucination evaluation into atomic fact verification provides a more robust framework for detection than existing general-domain metrics. Concretely, the authors find that general-domain detectors such as FENICE, SummaC, and AlignScore struggle on clinical data: FENICE reaches 0.45 correlation on the controlled Leave-N-Out set but drops to 0.10 on natural hallucinations, while the proposed fact-alignment method maintains correlations of 0.43 on LNO and 0.36-0.37 on the Natural Hallucination dataset. The method works by having an LLM extract concise atomic facts from the transcript and the summary separately, align them, and return the list of unsupported summary facts; counting that list gives an interpretable hallucination score. The paper also reports that this approach performs better on high-severity clinical errors and shows partial transfer to the XSum news summarization domain. The authors conclude that fact-based counting, despite its modest absolute correlations, generalizes better than classical, entailment-based, and single-prompt LLM baselines.

Load-bearing premise

The controlled Leave-N-Out dataset assumes that rewriting a transcript to delete all occurrences of a chosen fact, with scribe correction, leaves every other aspect of the dialogue unchanged; if the rewrite also alters tone, emphasis, or unrelated facts, the injected count N no longer isolates hallucination-detection quality.

Editorial extensions

If this is right

  • General-domain metrics, including ROUGE-1, BERTScore, SummaC, AlignScore, and FENICE, should not be used alone as clinical hallucination screens because their correlations are weak on natural hallucinations.
  • Counting unsupported atomic facts gives a reference-free, explainable hallucination score, allowing a clinician to inspect exactly which summary claims are unsupported by the source transcript.
  • High performance on fact-controlled synthetic data does not guarantee performance on natural hallucinations, so synthetic benchmarks are useful for developing detectors but not sufficient for validating them.
  • Fact alignment's correlation strength is modest in absolute terms, up to 0.43 with 0.37 on natural data, so it should be treated as a screening signal rather than a complete guarantee of clinical safety.
  • The approach transfers partially beyond medicine, as shown by its correlation on the XSum news summarization dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the Leave-N-Out construction could be reused as a general recipe for building controlled hallucination benchmarks in any domain where natural errors are rare, but the paper's own mismatch result implies such benchmarks should always be paired with a small natural-error validation set.
  • Beyond the paper, because fact alignment returns a list of unsupported claims rather than a single number, it could serve as an explainable audit trail inside clinical documentation systems, showing clinicians which summary statements need verification before sign-off.
  • Beyond the paper, the paper's stated limitation that LLM fact extraction often undercounts atomic facts suggests the correlation ceiling is partly set by the fact-decomposition step, so improved fact splitting could push detection performance above the reported 0.43 and 0.37.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses hallucination detection in clinical dialogue summarization. It constructs two datasets: Leave-N-Out (LNO), where facts are removed from transcripts to create controlled hallucinations, and Natural Hallucination (NH), where hallucinated summaries are human-annotated. It evaluates a wide range of existing metrics and proposes several fact-based detectors, reporting that LLM-based fact alignment achieves the best correlations (up to 0.43 on LNO and 0.37 on NH) and generalizes to the XSum news dataset. The conclusion claims these methods outperform existing metrics and generalize well to real-world clinical hallucinations.

Significance. If the central claims hold, the paper contributes two useful datasets and an explainable fact-alignment approach for a clinically important task. The strengths include the construction of a controlled synthetic dataset, the use of human expert annotations for natural hallucinations, evaluation on an external cross-domain benchmark (XSum), and a broad comparison of existing detectors. However, the reported correlations are modest, and several methodological gaps—particularly around the isolation of edits in LNO, lack of significance testing, and missing annotation reliability—currently limit the strength of the conclusions. The proposed fact-alignment method does offer better explainability than existing black-box metrics, which is a concrete contribution even if the performance advantage is not yet fully established.

major comments (4)
  1. [§3.1] The LNO dataset construction assumes that rewriting the transcript to remove a selected atomic fact does not alter any other fact, tone, or detail. The paper states that medical scribes verified and corrected 428 lines, but it does not report any validation that the rewrites preserved all non-target facts. Removing a clinical fact such as 'hypertension' often requires editing related medications, vitals, or history, so non-target edits are likely. This assumption is load-bearing for every LNO correlation in Table 1: if rewrites introduce or delete other facts, the correlations may reflect general textual mismatch or unrelated factual inconsistencies rather than detection of the intended hallucination. The limitations section concedes that atomic fact extraction 'often didn't work,' but it does not quantify how often or whether such failures affected the LNO edits. I request a human evaluation of edit isolation (e.g., proportion of rewrites with unintended fact changes) and an analysis of whether detector scores correlate with unintended edits.
  2. [Table 1] No significance tests are reported between methods. The table gives means and bootstrap standard deviations, but the conclusion (Section 5) claims that fact alignment 'outperform[s] existing metrics.' Given the large standard deviations—e.g., NH fact alignment is 0.36 ± 0.11 while Single Prompt Counting is 0.19 ± 0.06—the observed differences may not be statistically reliable. The authors should report paired significance tests across the three trials or bootstrap confidence intervals for the differences, particularly for the NH and High Severity comparisons.
  3. [§3.2] No inter-annotator agreement is reported for the NH dataset annotations or for the LNO scribe verification. Section 3.2 describes how clinical experts categorized statements into Hallucination, Inference, Misunderstanding, and No Factual Error, but without agreement metrics (e.g., Cohen's kappa) the reliability of the aggregated N counts used in the NH correlations is unknown. This is especially important because the NH correlations are the main evidence for generalization to real-world hallucinations.
  4. [§4.3] The fact-alignment detectors use the same model family (Claude Sonnet 3.5) that was used to select and remove facts in the LNO dataset (§3.1). Consequently, the LNO correlations may partly measure the model's self-consistency rather than its general ability to detect unsupported facts. The NH evaluation uses independent human annotations, so this concern does not directly invalidate the NH results, but it does weaken the claim that LNO development estimates real-world performance. I suggest quantifying this by generating a subset of LNO edits with a different model (e.g., Llama) and re-measuring the correlations.
minor comments (5)
  1. [Table 1] The entry for 'Sum. & Transcript Fact Extract + Emb. Alignment (bioBERT)' on NH reports 0.37 ± 0.00; a zero standard deviation is suspicious and likely a typo, please verify.
  2. [§4.3] The embedding alignment threshold of 0.75 is said to be determined on a held-out dataset split, but the size and composition of this split, the criterion for choosing the threshold, and the variance of the resulting performance are not described.
  3. [§1] Research question RQ2 asks whether performance on synthetic data transfers to real-world data, but the paper only compares correlations qualitatively in Table 1 and does not provide a formal statistical test of transfer.
  4. [§3.2] The paper mentions 'varying levels of prompt complexity (simple, medium, and complex)' used to induce hallucinations, but does not provide the actual prompts or describe how complexity was varied, which is needed for reproducibility.
  5. [Figure 1] The caption says references to knee pain are 'seamlessly replaced,' but the figure only shows a schematic; please include a concrete example of original and rewritten transcript text so readers can see the nature of the edits.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the generalization claim is anchored by human-annotated NH and XSum evaluations, not by the LNO construction alone.

full rationale

The paper's derivation chain is not circular under the stated standards. The LNO dataset is constructed by removing atomic facts with Claude Sonnet 3.5 and scribe verification (§3.1), and the proposed fact-alignment detector also uses Claude Sonnet 3.5 (§4.2, §4.3), so LNO correlations could in principle reflect same-model self-consistency. However, this is a confound, not a definitional reduction: the detector must still extract facts and judge support, and no equation forces its output to equal N by construction. Indeed FENICE achieves a comparable LNO correlation (0.45) but fails on NH (0.10), showing that high LNO correlation is not automatically produced by the evaluation setup. The central generalization claim is independently supported: the NH ground truths are clinical scribe annotations of natural model outputs (§3.2), and XSum uses human faithfulness annotations (§3.3). The embedding threshold was set on a held-out split (§4.3), not fitted to the reported correlations. The limitations section concedes imperfect atomic-fact extraction and undercounting, which is an accuracy or validity limitation rather than a circular step. No load-bearing self-citation or imported uniqueness claim appears in the reference list. Therefore the paper merits a circularity score of 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. Its assumptions are about dataset construction validity, annotation reliability, and the use of correlation as an evaluation proxy. The main free parameter is the embedding threshold, tuned on held-out data. The same-LLM coupling between LNO generation and detection is the most notable circularity burden.

free parameters (1)
  • Embedding alignment cosine threshold = 0.75
    Threshold for declaring a summary fact supported by a transcript fact; determined using a held-out dataset split (§4.3). It affects all embedding-based alignment and semantic similarity scores.
assumptions (4)
  • domain assumption Removing all occurrences of a selected atomic fact from a transcript, with human correction, simulates a hallucination without introducing other changes.
    Load-bearing for the LNO dataset; §3.1 describes LLM rewriting plus scribe verification.
  • domain assumption Atomic fact decomposition is a reliable unit: complex statements can be split into independent facts and a hallucination count equals the number of unsupported facts.
    Used by fact extraction and alignment in §4.3; the limitations section admits undercounting due to the definition of a fact.
  • domain assumption Clinical scribe annotations of natural hallucinations and severity classes are accurate, with no reported inter-annotator agreement.
    The NH dataset's N and high-severity labels come from expert annotation (§3.2); reliability is not quantified.
  • domain assumption Pearson correlation between a detector's output and the number of annotated errors measures detection quality.
    Used throughout Table 1 to rank methods; assumes a monotonic linear relation and that aggregated counts are meaningful.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fact-Controlled Diagnosis of Hallucinations in Medical Text Summarization." pith.science (2026). https://pith.science/paper/LCHIMLNG

@misc{pith2026250600448,
  author       = {Pith},
  title        = {Pith review of: Fact-Controlled Diagnosis of Hallucinations in Medical Text Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LCHIMLNG}},
  note         = {Machine review of arXiv:2506.00448}
}
read the original abstract

Hallucinations in large language models (LLMs) during summarization of patient-clinician dialogues pose significant risks to patient care and clinical decision-making. However, the phenomenon remains understudied in the clinical domain, with uncertainty surrounding the applicability of general-domain hallucination detectors. The rarity and randomness of hallucinations further complicate their investigation. In this paper, we conduct an evaluation of hallucination detection methods in the medical domain, and construct two datasets for the purpose: A fact-controlled Leave-N-out dataset -- generated by systematically removing facts from source dialogues to induce hallucinated content in summaries; and a natural hallucination dataset -- arising organically during LLM-based medical summarization. We show that general-domain detectors struggle to detect clinical hallucinations, and that performance on fact-controlled hallucinations does not reliably predict effectiveness on natural hallucinations. We then develop fact-based approaches that count hallucinations, offering explainability not available with existing methods. Notably, our LLM-based detectors, which we developed using fact-controlled hallucinations, generalize well to detecting real-world clinical hallucinations. This research contributes a suite of specialized metrics supported by expert-annotated datasets to advance faithful clinical summarization systems.

Figures

Figures reproduced from arXiv: 2506.00448 by the authors.

Figure 1
Figure 1. Dataset creation of the Leave-N-Out dataset, where a conversation transcript and its summary are shown side-by￾side. The original transcript contains a dialogue about knee pain (in green). In the edited transcript, all references to knee pain are seamlessly replaced (in gray), while the summary re￾mains unchanged and includes the specific detail ‘knee pain’ (highlighted). This illustrates how the dataset systematica… view at source ↗
Figure 2
Figure 2. details what this looks like. The lists of transcript and summary facts are included in the prompt and the output is list of unsupported facts which can be counted. In [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A comprehensive taxonomy of hallucinations in Large Language Models

    cs.CL 2025-08 conditional novelty 2.0 of 10

    A survey that organizes LLM hallucination types, causes, benchmarks, and mitigations, and restates the theorem that hallucination is inevitable for computable LLMs.

Reference graph

Works this paper leans on

31 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    hallucination

    Introduction The ability to generate precise summaries of patient-clinician dialogues and medical documentation is fundamental to health- care delivery, enabling rapid comprehension of patient histo- ries and informed clinical decision-making while reducing clin- ician’s burnout [1, 2, 3]. Language models have enabled the development of automated clinical...

    work page

  2. [2]

    Related Work Traditional lexical and semantic overlap metrics such as ROUGE [4], BLEU [5], and BERT-Score [6] offer straightfor- ward implementation but struggle with semantic understanding. More sophisticated approaches include entailment-based met- rics (e.g., FactCC [7], SummaC [8], AlignScore [9]), question- answering based metrics (e.g., FEQA [10], Q...

    arXiv 2025

  3. [3]

    Patient is a 35-year-old male with hypertension

    Hallucination Datasets We focus our research on the ACI-Bench dataset [16], which comprises clinician-patient conversation transcripts and their corresponding clinical summaries (SOAP Notes [17]). A SOAP note is a structured way for healthcare providers to document patient information, including Subjective (patient reports), Ob- jective (observable data),...

    work page

  4. [4]

    Baseline Methods Our goal is to identify metrics that reliably quantify halluci- nations in clinical text summaries

    Experiments 4.1. Baseline Methods Our goal is to identify metrics that reliably quantify halluci- nations in clinical text summaries. A hallucination-sensitive metric should show a monotonic relationship (increasing or de- creasing) with the number of removed facts (N) in the LNO dataset. We use Pearson correlation between the model outputs and the number...

    work page

  5. [5]

    This approach makes it tractable to generate domain specific evaluation data when nat- ural hallucinations are difficult to find in large numbers

    Conclusion We presented a novel fact-controlled approach to generate do- main specific evaluation benchmarks using fact removal to rewrite the input in the LNO dataset. This approach makes it tractable to generate domain specific evaluation data when nat- ural hallucinations are difficult to find in large numbers. While this dataset was helpful in develop...

    work page

  6. [6]

    Addi- tionally, our research focuses solely on SOAP notes, excluding other clinical documents like discharge summaries or progress reports

    Limitations The study primarily uses the ACI-Bench dataset, which may not represent the full complexity of clinical language. Addi- tionally, our research focuses solely on SOAP notes, excluding other clinical documents like discharge summaries or progress reports. Fact based analysis is limited by the definition of what is a fact. While we tried to promp...

    work page

  7. [7]

    Evaluating the factual consistency of abstractive text summarization,

    W. Kry ´sci´nski, B. McCann, C. Xiong, and R. Socher, “Evaluating the factual consistency of abstractive text summarization,”arXiv preprint arXiv:1910.12840, 2019

    arXiv 1910

  8. [8]

    Measurement of clinical documentation bur- den among physicians and nurses using electronic health records: a scoping review,

    A. J. Moy, J. M. Schwartz, R. Chen, S. Sadri, E. Lucas, K. D. Cato, and S. C. Rossetti, “Measurement of clinical documentation bur- den among physicians and nurses using electronic health records: a scoping review,”Journal of the American Medical Informatics Association, vol. 28, no. 5, 2021

    work page 2021

Show all 31 references
  1. [9]

    Challenges of developing a digital scribe to reduce clinical documentation burden,

    J. C. Quiroz, L. Laranjo, A. B. Kocaballi, S. Berkovsky, D. Reza- zadegan, and E. Coiera, “Challenges of developing a digital scribe to reduce clinical documentation burden,”NPJ digital medicine, vol. 2, no. 1, 2019

    2019

  2. [10]

    Association of electronic health record design and use factors with clinician stress and burnout,

    P. J. Kroth, N. Morioka-Douglas, S. Veres, S. Babbott, S. Poplau, F. Qeadan, C. Parshall, K. Corrigan, and M. Linzer, “Association of electronic health record design and use factors with clinician stress and burnout,”JAMA Network Open, vol. 2, no. 8, 08 2019. [Online]. Availab...

    2019

  3. [11]

    Rouge: A package for automatic evaluation of sum- maries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of sum- maries,” inText summarization branches out, 2004

    2004

  4. [12]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” inProceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002

    2002

  5. [13]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,”arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  6. [14]

    Fenice: Factuality evalu- ation of summarization based on natural language inference and claim extraction,

    A. Scir `e, K. Ghonim, and R. Navigli, “Fenice: Factuality evalu- ation of summarization based on natural language inference and claim extraction,”arXiv preprint arXiv:2403.02270, 2024

    2024 arXiv

  7. [15]

    Summac: Re-visiting nli-based models for inconsistency detection in sum- marization,

    P. Laban, T. Schnabel, P. N. Bennett, and M. A. Hearst, “Summac: Re-visiting nli-based models for inconsistency detection in sum- marization,”Transactions of the Association for Computational Linguistics, vol. 10, 2022

    2022

  8. [16]

    Alignscore: Evaluating fac- tual consistency with a unified alignment function,

    Y . Zha, Y . Yang, R. Li, and Z. Hu, “Alignscore: Evaluating fac- tual consistency with a unified alignment function,”arXiv preprint arXiv:2305.16739, 2023

    2023 arXiv

  9. [17]

    Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization,

    E. Durmus, H. He, and M. Diab, “Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization,”arXiv preprint arXiv:2005.03754, 2020

    2005 arXiv

  10. [18]

    Questeval: Summarization asks for fact-based evaluation,

    T. Scialom, P.-A. Dray, P. Gallinari, S. Lamprier, B. Piwowarski, J. Staiano, and A. Wang, “Questeval: Summarization asks for fact-based evaluation,”arXiv preprint arXiv:2103.12693, 2021

    2021 arXiv

  11. [19]

    Q2: Evaluating factual consistency in knowledge- grounded dialogues via question generation and question answer- ing,

    O. Honovich, L. Choshen, R. Aharoni, E. Neeman, I. Szpektor, and O. Abend, “Q2: Evaluating factual consistency in knowledge- grounded dialogues via question generation and question answer- ing,”arXiv preprint arXiv:2104.08202, 2021

    2021 arXiv

  12. [20]

    Infolm: A new metric to evaluate summarization & data2text generation,

    P. J. A. Colombo, C. Clavel, and P. Piantanida, “Infolm: A new metric to evaluate summarization & data2text generation,” inPro- ceedings of the AAAI conference on artificial intelligence, vol. 36, no. 10, 2022

    2022

  13. [21]

    On faithfulness and factuality in abstractive summarization,

    J. Maynez, S. Narayan, B. Bohnet, and R. McDonald, “On faithfulness and factuality in abstractive summarization,”arXiv preprint arXiv:2005.00661, 2020

    2005 arXiv

  14. [22]

    Doclens: Multi-aspect fine-grained medical text evaluation,

    Y . Xie, S. Zhang, H. Cheng, P. Liu, Z. Gero, C. Wong, T. Nau- mann, H. Poon, and C. Rose, “Doclens: Multi-aspect fine-grained medical text evaluation,” inProceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), 2024

    2024

  15. [23]

    While BioBERT offers domain- specific understanding, MiniLM provides comparable perfor- mance with faster inference and a smaller memory footprint

    (12 layers, 400MB). While BioBERT offers domain- specific understanding, MiniLM provides comparable perfor- mance with faster inference and a smaller memory footprint. This allows us to investigate the trade-off between efficiency and domain expertise in capturing semantic sim...

  16. [24]

    Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation,

    W.-w. Yim, Y . Fu, A. Ben Abacha, N. Snider, T. Lin, and M. Yetis- gen, “Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation,”Scientific Data, vol. 10, no. 1, 2023

    2023

  17. [25]

    Soap notes.[updated 2023 aug 28],

    V . Podder, V . Lew, and S. Ghassemzadeh, “Soap notes.[updated 2023 aug 28],”In: StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing; Available from: https://www.ncbi.nlm.nih.gov/books/NBK482263/, 2023

    2023

  18. [26]

    Claude Sonnet 3.5,

    Anthropic, “Claude Sonnet 3.5,” https://www.anthropic.com, 2024, https://www.anthropic.com/news/claude-3-family

    2024

  19. [27]

    Mistral Large,

    Mistral, “Mistral Large,” https://mistral.ai, 2024, https://mistral. ai/news/mistral-large/

    2024

  20. [28]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Let- man, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  21. [29]

    Minilm: Deep self-attention distillation for task-agnostic com- pression of pre-trained transformers,

    W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “Minilm: Deep self-attention distillation for task-agnostic com- pression of pre-trained transformers,”Advances in Neural Infor- mation Processing Systems, vol. 33, 2020

    2020

  22. [30]

    Biobert: a pre-trained biomedical language representation model for biomedical text mining,

    J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “Biobert: a pre-trained biomedical language representation model for biomedical text mining,”Bioinformatics, vol. 36, no. 4, 2020

    2020

  23. [31]

    spaCy: Industrial-strength Natural Language Processing in Python,

    M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd, “spaCy: Industrial-strength Natural Language Processing in Python,” 2020

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.