Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

Automatic Evaluation of Healthcare LLMs Beyond Question-Answering

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces Relaxed Perplexity, a metric that scores an open-ended healthcare answer by the probability that the model would ever sample the target text, and argues this measures factuality without regard to exact wording.

desk verdict Useful empirical evaluation study and a new benchmark, but the headline metric's derivation doesn't support the probability claim, so the paper needs revision before the central results can be trusted. read the letter →

arxiv 2502.06666 v1 pith:WWOVGXVN submitted 2025-02-10 cs.CL cs.AI

Anna Arias-Duart , Pablo Agustin Martin-Torres , Daniel Hinjos , Pablo Bernabeu-Perez , Lucia Urcelay Ganzabal , Marta Gonzalez Mallo , Ashwin Kumar Gururajan , Enrique Lopez-Cuena
show 2 more authors
Sergio Alvarez-Napagao Dario Garcia-Gasulla
This is my paper · ORCID
classification cs.CLcs.AI
keywords healthcareLLMevaluationopen-endedgenerationRelaxedPerplexityCareQAbenchmarkmedicalfactualityquestionansweringmetriccorrelationLLM-as-a-judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper takes on a practical problem: automatic evaluation of medical language models in open-ended settings, where the same answer can be phrased in many ways. It reports that open-ended and close-ended benchmarks barely correlate in healthcare, so neither can replace the other. It introduces CareQA, a bilingual multiple-choice and open-ended benchmark drawn from Spanish medical exams, as a fresh sanity check. The central proposal is Relaxed Perplexity, which estimates the probability that the model would ever sample the target answer during a completion, ignoring exact wording and rewarding intermediate reasoning tokens. If the proposal holds, healthcare-fine-tuned models can be ranked above general models on open-ended factuality without human grading.

What carries the argument

The load-bearing object is the Relaxed Perplexity identity, defined as $\exp\left(-\frac{1}{n + \text{len(target)}} \sum_{i=0}^{n} \log P(A_i \mid B_i)\right)$ with $A_i = \{\text{target sampled after question} + \text{seq}_i\}$ and $B_i = \{\text{seq}_i \sim \text{model(question)}\}$. To make this tractable, the derivation approximates $P(A_n \mid B_n)$ by an unweighted sum over the $\ell$ most likely $n$-token continuations sampled from the model, $P(A_n \mid \text{seq}^1_n) + \cdots + P(A_n \mid \text{seq}^\ell_n)$, after a length-normalization step that discards the exponentially collapsing weights $P(\text{seq}^j_n)$. The implementation uses top-$p$ sampling with $\ell = 5$ candidate sequences, a search space of $s = 10$, and a stride of 8 or 16 tokens. This approximation is what lets the metric be computed at all, and it is also the step on which the argument's validity rests.

What would settle it

One concrete check: compute Relaxed Perplexity on the same open-ended questions with $\ell$ raised from 5 to 20 or 50 and with a larger search space; if the model rankings change materially, the unweighted-sum approximation is unstable. Another: ask blinded clinicians to rate the factuality of the same model answers and compare with the metric's ranking—if the metric ranks as more factual answers that clinicians consider less factual, the claim that it measures factuality is falsified.

Watch

Extended reading notes

Core claim

Stated on the paper's own terms, the central discovery is that factuality in open-ended answers can be estimated by the probability that the target is sampled from the model at any point in the completion, written $P(\text{target} \sim \text{model} \mid \text{question})$. Relaxed Perplexity implements this as $\exp\left(-\frac{1}{n + \text{len(target)}} \sum_{i=0}^{n} \log P(A_i \mid B_i)\right)$, where $A_i$ is the event that the target follows the question plus an $i$-token continuation $\text{seq}_i$ and $B_i$ is the event that $\text{seq}_i$ is sampled from the model. Because it averages over alternative continuations rather than scoring one reference string, it is insensitive to phrasing and can credit chain-of-thought tokens that ordinary perplexity punishes. On the OLAPH medical factuality dataset the metric ranks healthcare-fine-tuned models above general-purpose models, reversing the ranking given by standard perplexity, and the paper offers this as filling the gap in automatic open-ended factuality evaluation.

Load-bearing premise

The approximation drops the probabilities of the candidate intermediate sequences and treats each of the $\ell$ sampled continuations as equally likely; if that simplification does not track the true probability that the model would produce the target, then the ranking the metric produces on OLAPH is not evidence of factuality.

Editorial extensions

If this is right

  • Open-ended factuality can be scored automatically without reference-text overlap, reducing dependence on n-gram metrics and human judges.
  • Perplexity-based, n-gram/semantic, and LLM-judge metrics form three distinct clusters, so evaluations should sample across all three rather than rely on one.
  • Multiple-choice accuracy barely correlates with open-ended performance, so strong MCQA results do not imply strong open-ended medical answers.
  • CareQA provides a fresh, contamination-resistant bilingual benchmark with paired closed and open formats for future evaluators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The length-normalization step in the appendix effectively reweights the target probability toward earlier positions; a testable question is whether the OLAPH ranking is driven mainly by brevity rather than factuality.
  • The metric is essentially an expected-reward estimate over model sample paths, so importance-sampling or beam-marginalization versions might give a less heuristic approximation with provable consistency.
  • The correlation results imply that a single leaderboard score will mislead for clinical deployment; evaluation suites should combine one MCQA, one open-ended generation, and one judge-based metric.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents a multi-axis evaluation study of healthcare LLMs, comparing closed-ended (MCQA) and open-ended benchmarks and metrics, releasing a new bilingual closed-ended and English open-ended dataset (CareQA), and proposing a novel metric called Relaxed Perplexity for open-ended factuality evaluation. The empirical analysis reports correlations across tasks, datasets, and metrics, as well as resilience-to-rephrasing and self-consistency experiments. The central methodological contribution is Relaxed Perplexity, defined as an exponentiated average of log-probabilities that aims to estimate the probability that a target answer is sampled by the model at some point during an open-ended completion, and is validated mainly through rankings on the OLAPH medical factuality dataset.

Significance. If valid, the correlation study would provide a useful mapping of evaluation methodology in healthcare NLP, and CareQA would be a practical resource for the community; the paper also deserves credit for releasing the dataset and integrating tasks into the lm-evaluation-harness framework, and for running a human evaluation of the benchmark construction. However, the load-bearing claim is Relaxed Perplexity, and the derivation and validation of that metric are not sound. Since the metric is advertised as the main novelty that 'fills a gap' in open-ended evaluation, the paper's central contribution requires substantial rework, while the correlation analyses are also underpowered by the lack of inferential statistics.

major comments (5)
  1. [Appendix D] The derivation of Relaxed Perplexity is mathematically unjustified. The law of total probability requires P(A_n|B_n) = sum_i P(A_n|seq_i_n) P(seq_i_n|B_n). After initially writing the weighted sum, the authors introduce a length-dependent constant (c_n+d_n)/(2 c_n d_n) and then remove the entire weight factor, resulting in the unweighted approximation P(A_n|B_n) ≈ P(A_n|seq_i1_n)+...+P(A_n|seq_iℓ_n). The per-sequence probabilities P(seq_i_n) are not constant within the same length n; top-beam sequences can be orders of magnitude more probable than the ℓ-th beam. Dropping the weights allows unlikely continuations to contribute equally, so the sum no longer estimates any probability of the target being sampled. This directly undermines the claim in Section 4 that the metric 'allows to evaluate correctness in the model's answers probability distribution.'
  2. [Section 4, Table 10, Figure 17] The validation of Relaxed Perplexity as a factuality metric is insufficient. Table 10 reports ranked values from a single dataset without any comparison to human factuality labels or to an existing factuality-aware baseline. The observation that some healthcare-tuned models receive low Relaxed Perplexity is not a demonstration that the metric tracks factuality. Moreover, the manuscript refers to Figure 17 as a correlation between OLAPH Relaxed Perplexity and other benchmarks, but the actual figure and any numerical results are not present in the text, so this evidence cannot be inspected. A metric intended to measure factuality should be validated against ground-truth factuality judgments across models and prompts.
  3. [Sections 3.1–3.3 and Appendix B] The correlation analyses are based on at most 10–12 models and report only point estimates of correlation coefficients, with no confidence intervals, bootstrap, or permutation tests. For example, the claim that 'no consistently high correlation is observed' for any benchmark or task is a negative claim that is highly fragile at this sample size. Similarly, the claimed three-cluster structure of metrics should be accompanied by a test of cluster stability. These inferential gaps affect the paper's empirical conclusions about the complementarity of open- and closed-ended evaluation.
  4. [Section 2.1, Table 2] CareQA-Open uses reference answers that were rephrased from MCQA and then filtered through human review; after the second iteration, only 73.6% of the rephrased QA pairs were unanimously judged correct by three annotators. This means about a quarter of the open-ended references are not unanimously correct, and the paper does not analyze how this label noise affects the open-ended metric scores and correlations. Since CareQA-Open is used in several experiments, the high noise level in its reference answers should be addressed as a validity threat.
  5. [Section 4 and Table 10] The claim that 'Relaxed Perplexity assigns higher scores to models fine-tuned on healthcare datasets' is not cleanly supported by the presented data. In Table 10, the healthcare-tuned BioMistral-MedMNX obtains a Relaxed Perplexity of 81.532, which is worse than general-purpose models such as Mistral-7B-Instruct-v0.3 (27.64), Mixtral-8x7B-Instruct-v0.1 (23.045), and Phi-3-medium-4k-instruct (44.207). Only two of the three healthcare-tuned models rank among the best, so the qualitative pattern is mixed. Without a statistical comparison or a factuality ground truth, the metric's superiority claim is unsubstantiated.
minor comments (5)
  1. [Appendix D] The inequality 'd_n ≤ n' appears to be a typo; based on the context, the intended bound should relate d_n to the inverse probability scale, not to n.
  2. [Table 3] The table header contains a duplicated 'Option 3' column label, which should be corrected to 'Option 4'.
  3. [Appendix B.1] The list of small models includes 'Starling-LM-7B-beta' twice, which looks like a copy-paste error.
  4. [Section 2.1] The abbreviation 'MIR' is used without expansion at first mention; it should be defined as the Spanish Specialised Healthcare Training exam.
  5. [Figure 17] The figure caption appears without any actual figure or data in the manuscript; either include the figure with axis labels and numerical values or remove the reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Relaxed Perplexity directly measures target likelihood, and the OLAPH validation is external to the metric's construction.

full rationale

The paper's central derivation, the Relaxed Perplexity metric in Section 4 and Appendix D, is not circular. It is defined directly as exp(-1/(n+len(target)) * sum_i log P(A_i|B_i)), i.e., the normalized negative log-likelihood of the target under the model's own sampled continuations. The target answer is the object of measurement, not a fitted output: no parameter of the metric is estimated from the target answers or from the OLAPH rankings to force a result. The Appendix D law-of-total-probability derivation is an approximation, not an identity; the paper explicitly says 'Notice this step may be omitted depending on the evaluation goal,' acknowledging that the unweighted sum is an estimation choice. The choice of l=5, s=10, stride and max_tokens is a robustness and hyperparameter selection ('We observe similar results with all combinations'), not a fit to the benchmark. The validation on OLAPH is external to the metric's construction and is reported as an observed ranking, not as a consequence of the definition. The only identifiable weakness is the unjustified dropping of the P(seq_i) weights, which threatens the correctness of the estimate but does not make the derivation equivalent to its inputs. CareQA is a fresh benchmark whose open-ended filtering for unique answers is dataset construction, not a metric fitting step. There are no load-bearing self-citations or imported uniqueness arguments. Hence no circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The metric's core approximation depends on several hand-selected hyperparameters and two unproven probabilistic assumptions. CareQA open-ended questions rely on the assumption of unique correct answers after filtering, which is only partially supported by human agreement. No new physical or conceptual entities are introduced.

free parameters (4)
  • l (number of sampled sequences) = 5
    Appendix D: authors tested l in {5, 10} and fixed l = 5 for better performance.
  • s (search space size) = 10
    Appendix D: tested s in {10, 100} and fixed s = 10.
  • stride = 8 or 16
    Appendix D: introduced for efficiency, selected from {8, 16} after finding them 'as effective'.
  • max_tokens = [128, 256]
    Appendix D: completion length cap used to justify P(B_n) approximately 1.
assumptions (4)
  • ad hoc to paper The l most likely n-token sequences dominate the sum over all sequences, so P(A_n | B_n) can be approximated without the P(seq_i_n) weights.
    Appendix D: the paper asserts this after noting the weights collapse exponentially, but gives no bound that justifies dropping them.
  • domain assumption P(B_n) is approximately 1 because stop tokens rarely appear before n tokens in the completion.
    Appendix D: stated directly with max_tokens in [128, 256].
  • domain assumption Top-p sampling or diverse beam search with l = 5 and s = 10 yields a representative set of likely completions for the probability estimate.
    Appendix D: no theoretical or empirical demonstration that the sampled set is representative.
  • domain assumption Ground truth answers in CareQA-Open are uniquely correct and correctly transcribed, based on the human review.
    Appendix A.3: after two iterations, only 73.6% of rephrased questions were labeled correct by all three evaluators.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Evaluation of Healthcare LLMs Beyond Question-Answering." pith.science (2026). https://pith.science/paper/WWOVGXVN

@misc{pith2026250206666,
  author       = {Pith},
  title        = {Pith review of: Automatic Evaluation of Healthcare LLMs Beyond Question-Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WWOVGXVN}},
  note         = {Machine review of arXiv:2502.06666}
}
read the original abstract

Current Large Language Models (LLMs) benchmarks are often based on open-ended or close-ended QA evaluations, avoiding the requirement of human labor. Close-ended measurements evaluate the factuality of responses but lack expressiveness. Open-ended capture the model's capacity to produce discourse responses but are harder to assess for correctness. These two approaches are commonly used, either independently or together, though their relationship remains poorly understood. This work is focused on the healthcare domain, where both factuality and discourse matter greatly. It introduces a comprehensive, multi-axis suite for healthcare LLM evaluation, exploring correlations between open and close benchmarks and metrics. Findings include blind spots and overlaps in current methodologies. As an updated sanity check, we release a new medical benchmark --CareQA-- with both open and closed variants. Finally, we propose a novel metric for open-ended evaluations -- Relaxed Perplexity -- to mitigate the identified limitations.

Figures

Figures reproduced from arXiv: 2502.06666 by the authors.

Figure 1
Figure 1. Correlation between the weighted average [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Mean variance distributions across different runs and averaged across models using the CareQA￾Open dataset. Closer to 0 means more self-consistent. However, as seen before, this does not necessarily entail good performance on open or close-ended downstream tasks. Additionally, perplexity can be greatly impacted by instruct-tuning and alignment techniques (Lee et al., 2024). On the other hand, it has been widely note… view at source ↗
Figure 3
Figure 3. CareQA example from Medicine category. While CareQA shares its source with HeadQA in the Spanish Specialised Healthcare Training (MIR) exams, there is no overlap between the datasets. CareQA expands upon its predecessor, covering the years 2020 to 2024 and comprising 5,621 question￾answer test pairs, compared to HeadQA’s 2,742 test pairs from 2013 to 2017. The dataset’s composi￾tion is illustrated in [PITH_FULL_IMA… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Iterations with human evaluators to create the [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Category distribution per Category and Year ( [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Category distribution per Category and Year ( [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Correctness distribution per Category CareQA (open-ended). 12 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Comparison of correlations between MCQA benchmarks and ELO results for small models [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Comparison of correlations between MCQA benchmarks and ELO results for medium models. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: This correlation matrix illustrates the rela [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 12
Figure 12. Figure 12: Correlation of Prometheus scores across benchmarks. C Resilience to rephrasing and self-consistency C.1 Resilience As described earlier, we conducted this experiment by rephrasing the model outputs six times and re￾computing the metrics. We used both Qwen2.5- 72B-Inst…
Figure 13
Figure 13. Figure 13: Mean variance distributions across different rephrasings and models using the MEDIQA2019 dataset. Each metric is represented by a different color [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Mean variance distributions across different rephrasings using the Phi-3-mini-4k-instruct model and the MEDIQA2019 dataset. Each metric is represented by a different color. shows the results for the Yi-1.5-9B-Chat model. In [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 16
Figure 16. Figure 16: Mean coefficient of variation distributions across different runs and averaged across models for self￾consistency. Each metric is represented by a different color. D Novel Metric: Relaxed Perplexity As mentioned before, we define Relaxed Perplexity as Relaxed-Perplexi…
Figure 17
Figure 17. Figure 17: Correlation between OLAPH - Relaxed Perplexity and the rest of benchmarks. That is, An is the event that target is sampled from the model inputted with question + seqn, for any seqn of n tokens. Thus, in order to compute P(An | Bn) we need to take into account the pro…
Figure 18
Figure 18. Figure 18: Ranking results for all models on the OLAPH medical factuality dataset for all metrics. The top position [PITH_FULL_IMAGE:figures/full_fig_p018_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Essential-Web v1.0: 24T tokens of organized web data

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 24T-token web corpus with automatic document-level taxonomy labels enables competitive domain-specific datasets via simple filters.

  2. SynapseRoute: An Auto-Route Switching Framework on Dual-State Large Language Model

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A learned router for a dual-mode LLM raises medical QA accuracy from 0.827 to 0.839 while cutting inference time by 36.8% and tokens by 39.7% versus always using thinking mode.

Reference graph

Works this paper leans on

46 extracted references · 18 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Muhammad Aurangzeb Ahmad, Ilker Yaramis, and Taposh Dutta Roy. 2023. Creating trustworthy llms: Dealing with hallucinations in healthcare ai. arXiv preprint arXiv:2311.01463

    arXiv 2023

  4. [4]

    AI@Meta. 2024. https://github.com/meta-llama/llama3blob/main/MODEL_CARD.md Llama 3 model card

    work page 2024

  5. [5]

    Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, et al. 2024. When benchmarks are targets: R evealing the sensitivity of large language model leaderboards. arXiv preprint arXiv:2402.01781

    arXiv 2024

  6. [6]

    Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason A Fries, Michael Wornow, Akshay Swaminathan, Lisa Soleymani Lehmann, et al. 2024. A systematic review of testing and evaluation of healthcare applications of large language models (llms). medRxiv, pages 2024--04

    work page 2024

  7. [7]

    Asma Ben Abacha , Chaitanya Shivade, and Dina Demner - Fushman. 2019. Overview of the mediqa 2019 shared task on textual inference, question entailment and question answering. In ACL-BioNLP 2019

    work page 2019

  8. [8]

    Asma Ben Abacha, Wen-wai Yim, Yadan Fan, and Thomas Lin. 2023. https://aclanthology.org/2023.eacl-main.168 An empirical study of clinical note generation from doctor-patient encounters . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2291--2302, Dubrovnik, Croatia. Association for Comp...

    work page 2023

Show all 46 references
  1. [9]

    Qingyu Chen, Jingcheng Du, Yan Hu, Vipina Kuttichi Keloth, Xueqing Peng, Kalpana Raja, Rui Zhang, Zhiyong Lu, and Hua Xu. 2023. Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations. arXiv preprint arXiv:2305.16326

    2023 arXiv

  2. [10]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot A rena: A n O pen P latform for E valuating LLM s by H uman P reference. In Forty-first Internatio...

    2024

  3. [11]

    Cl \'e ment Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, and Marco AF Pimentel. 2024. Med42-v2: A suite of clinical llms. arXiv preprint arXiv:2408.06142

    2024 arXiv

  4. [12]

    Amin Dada, Marie Bauer, Amanda Butler Contreras, Osman Alperen Kora s , Constantin Marc Seibold, Kaleb E Smith, and Jens Kleesiek. 2024. Clue: A clinical language understanding evaluation for llms. arXiv preprint arXiv:2404.04067

    2024

  5. [13]

    Ankit Pal et al. 2022. https://proceedings.mlr.press/v174/pal22a.html Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering . In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Lea...

    2022

  6. [14]

    Dan Hendrycks et al. 2020 a . Measuring M assive M ultitask L anguage U nderstanding. In International Conference on Learning Representations

    2020

  7. [15]

    Di Jin et al. 2020 b . What disease does this patient have? a large-scale open domain question answering dataset from medical exams. arXiv preprint arXiv:2009.13081

    2020 arXiv

  8. [16]

    Qiao Jin et al. 2019. Pub M ed QA : A D ataset for B iomedical R esearch Q uestion A nswering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pa...

    2019

  9. [17]

    Scott L Fleming, Alejandro Lozano, William J Haberkorn, Jenelle A Jindal, Eduardo Reis, Rahul Thapa, Louis Blankemeier, Julian Z Genkins, Ethan Steinberg, Ashwin Nayak, et al. 2024. Medalign: A clinician-generated dataset for instruction following with electronic medical recor...

    2024

  10. [18]

    Evan Frick, Peter Jin, Tianle Li, Karthik Ganesan, Jian Zhang, Jiantao Jiao, and Banghua Zhu. 2024. https://nexusflow.ai/blogs/athene Athene-70b: Redefining the boundaries of post-training for open models

    2024

  11. [19]

    Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, et al. 2024. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Natu...

    2024

  12. [20]

    Kai He, Rui Mao, Qika Lin, Yucheng Ruan, Xiang Lan, Mengling Feng, and Erik Cambria. 2025. A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics. Information Fusion, page 102963

    2025

  13. [21]

    Smith, Iz Beltagy, and Hannaneh Hajishirzi

    Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. https://arxiv.org/abs/2311.10702 Camels in a changing climate: Enhancing lm adaptation with tulu 2...

    2023 arXiv

  14. [22]

    Minbyul Jeong, Hyeon Hwang, Chanwoong Yoon, Taewhoo Lee, and Jaewoo Kang. 2024. Olaph: Improving factuality in biomedical long-form question answering. arXiv preprint arXiv:2405.12701

    2024 arXiv

  15. [23]

    Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1--9

    2016

  16. [24]

    Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. 2023. Evaluating open-domain question answering in the era of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5591--5606

    2023

  17. [25]

    Praveen K Kanithi, Cl \'e ment Christophe, Marco AF Pimentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, and Shadab Khan. 2024. Medic: Towards a comprehensive framework for evaluating llms in clinical applications. arXiv preprint a...

    2024 arXiv

  18. [26]

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535

    2024 arXiv

  19. [27]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  20. [28]

    Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. 2024. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint arXiv:2401.01967

    2024 arXiv

  21. [29]

    Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. 2024. Can multiple-choice questions really be useful in detecting the abilities of llms? arXiv preprint arXiv:2403.17752

    2024 arXiv

  22. [30]

    Ling Luo, Po-Ting Lai, Chih-Hsuan Wei, Cecilia N Arighi, and Zhiyong Lu. 2022. Biored: a rich biomedical relation extraction dataset. Briefings in Bioinformatics, 23(5):bbac282

    2022

  23. [31]

    Iftitahu Nimah, Meng Fang, Vlado Menkovski, and Mykola Pechenizkiy. 2023. Nlg evaluation metrics beyond correlation analysis: An empirical metric preference checklist. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2023

  24. [32]

    Pouya Pezeshkpour and Estevam Hruschka. 2023. Large language models sensitivity to the order of options in multiple-choice questions. arXiv preprint arXiv:2308.11483

    2023 arXiv

  25. [33]

    Tim Schopf, Daniel Braun, and Florian Matthes. 2023. https://doi.org/10.1145/3582768.3582795 Evaluating unsupervised text classification: Zero-shot and similarity-based approaches . In Proceedings of the 2022 6th International Conference on Natural Language Processing and Info...

    2023

  26. [34]

    Ofir Ben Shoham and Nadav Rappoport. 2024. Medconceptsqa--open source medical concepts qa benchmark. arXiv preprint arXiv:2405.07348

    2024 arXiv

  27. [35]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261

    2022 arXiv

  28. [36]

    Gemma Team. 2024. https://doi.org/10.34740/KAGGLE/M/3301 Gemma

    2024 doi

  29. [37]

    Logesh Kumar Umapathi, Ankit Pal, and Malaikannan Sankarasubbu. 2023. Med-halt: Medical domain hallucination test for large language models. arXiv preprint arXiv:2307.15343

    2023 arXiv

  30. [38]

    Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv:1610.02424

    2016 arXiv

  31. [39]

    Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023. Towards understanding chain-of-thought prompting: An empirical study of what matters. In The 61st Annual Meeting Of The Association For Computational Linguistics

    2023

  32. [40]

    Xuezhi Wang and Denny Zhou. 2024. Chain-of-thought reasoning without prompting. arXiv preprint arXiv:2402.10200

    2024 arXiv

  33. [41]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  34. [42]

    Wen - wai Yim, Yujuan Fu, Asma Ben Abacha , Neal Snider, Thomas Lin, and Meliha Yetisgen. 2023. https://www.nature.com/articles/s41597-023-02487-3 Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation . Nature Scientific Data, 10

    2023

  35. [43]

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652

    2024 arXiv

  36. [44]

    Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang, Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoyu Zhang, et al. 2020. Meddialog: Large-scale medical dialogue datasets. In Proceedings of the 2020 conference on empirical methods in natural language processi...

    2020

  37. [45]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations

    2023

  38. [46]

    Hongjian Zhou, Boyang Gu, Xinyu Zou, Yiru Li, Sam S Chen, Peilin Zhou, Junling Liu, Yining Hua, Chengfeng Mao, Xian Wu, et al. 2023. A survey of large language models in medicine: Progress, application, and challenge. arXiv preprint arXiv:2311.05112

    2023 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.