Overview of the ClinIQLink 2025 Shared Task on Medical Question-Answering

Brandon Colelough; Davis Bartels; Dina Demner-Fushman

REVIEW 3 major objections 5 minor 23 references

Overview of the ClinIQLink 2025 Shared Task on Medical Question-Answering

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces ClinIQLink, a benchmark of 4,978 expert-verified, source-grounded general-practitioner-level medical QA pairs in seven formats, and argues that embedding-based semantic similarity, with a step penalty for multi-hop…

desk verdict A genuinely useful medical QA benchmark whose own Section 4.2 undermines the headline evaluation claim: the paper admits its automated metrics were not effective, then builds the leaderboard and conclusions on them. read the letter →

arxiv 2506.21597 v1 pith:SEGXEEHR submitted 2025-06-18 cs.CL cs.AIcs.IR

Brandon Colelough , Davis Bartels , Dina Demner-Fushman This is my paper

classification cs.CLcs.AIcs.IR

keywords medicalquestionansweringLLMevaluationsemanticsimilarityembedding-basedmetricshallucinationdetectionmulti-hopreasoningsharedtaskbenchmarkgeneralpractitionerknowledge

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents ClinIQLink, a shared-task benchmark of 4,978 expert-verified, source-grounded medical question-answer pairs aimed at general-practitioner knowledge, spanning true/false, multiple choice, unordered list, short answer, short-inverse, multi-hop, and multi-hop-inverse formats. Its central claim is that the benchmark stress-tests modern LLMs: closed-ended accuracy looks high (0.75-0.82 on multiple choice and true/false) yet models still emit invalid options, and list questions expose hallucinated extras. For open-ended answers, the paper argues that the ClinIQLink embedding-based semantic-similarity metric, which blends token, sentence, and paragraph cosine layers and applies a step-distance penalty to multi-hop inverse items, tracks clinical accuracy better than BLEU, ROUGE, and METEOR. Who should care: anyone evaluating medical QA systems, because n-gram metrics reward verbatim copying and penalize lexically novel but faithful paraphrases, so leaderboard rankings built on them can mislead.

What carries the argument

The load-bearing machinery is the ClinIQLink semantic-similarity score, a three-tier cosine metric computed as $S_{\mathrm{raw}} = 0.4 C_{\mathrm{tok}} + 0.4 C_{\mathrm{sent}} + 0.2 C_{\mathrm{para}}$, where $C_{\mathrm{tok}}$ is an IDF-weighted greedy token-alignment F1, $C_{\mathrm{sent}}$ is the cosine similarity of sentence-level CLS embeddings, and $C_{\mathrm{para}}$ is cosine similarity of the raw answer strings. After subtracting a baseline offset of $\beta = 0.25$ and snapping scores at or above 0.95 to 1, the score is multiplied by a step-distance penalty for multi-hop inverse items, $\alpha(d) = 1, 0.7, 0.3, 0.3 \cdot 2^{-(d-2)}$ for predicted-versus-gold step distances $d = 0, 1, 2, \ge 3$. This graded combination is what lets the metric reward lexically novel but semantically faithful answers while punishing wrong reasoning steps; it is the object whose behavior the paper contrasts with BLEU, ROUGE, and METEOR across all free-text modalities. The dataset-generation pipeline, a neuro-symbolic system that produced roughly 20,000 source-linked QA pairs later reduced to 4,978 maximally relevant expert-verified items, supplies the evaluation surface on which the metric comparison rests.

What would settle it

The metric claim could be settled by correlating the ClinIQLink semantic-similarity score with the Task-2 physician panel's per-item quality ratings on the open-ended modalities; if BLEU, ROUGE, or METEOR matches or beats the embedding score in that correlation, especially on heavily paraphrased items or multi-hop-inverse questions, the paper's central evaluation claim would be refuted.

Watch

Extended reading notes

Core claim

ClinIQLink's central finding is that contemporary LLMs saturate simple closed-ended medical questions while failing in ways those questions conceal. Across the frozen 4,978-item test set, the strongest systems score about 0.79-0.82 on true/false and 0.75-0.80 on multiple choice, yet still occasionally output out-of-range or invalid answers; unordered-list questions, because they require selecting all correct elements and rejecting distractors, spread F1 scores from roughly 0.30 to 0.68 and thereby reveal hallucinated extras. On open-ended questions, the paper finds that embedding-based semantic similarity separates meaningful paraphrase from surface overlap: high-ranked models frequently score above 0.60 on the semantic metric while BLEU sits below 0.05, whereas runs that recycle reference wording earn better n-gram scores but lower embedding scores. The multi-hop-inverse format, with its step-distance penalty, is the most discriminative open-ended task and exposes brittle reasoning chains that n-gram metrics miss. The conclusion is that evaluation of medical QA should use embedding-level similarity as a minimum standard, with explicit reasoning validation still an open problem.

Load-bearing premise

The central claim stands on the assumption that the 4,978 scored question-answer pairs fairly represent general-practitioner medical knowledge rather than inheriting the generating LLM's phrasing or fact biases; if the generator's artifacts or the reviewers' exclusions systematically skewed the retained items, the metric comparison and system rankings would reflect data-construction artifacts rather than medical competence.

Editorial extensions

If this is right

Future medical QA benchmarks should report embedding-based semantic similarity for open-ended items, since n-gram metrics systematically mis-score lexically novel yet faithful paraphrases.
High closed-ended accuracy alone is misleading: the paper's results imply that strong true/false and multiple-choice scores hide residual invalid answers, so evaluation should include formats like unordered list that expose hallucinated extras.
The multi-hop-inverse format, with its step-distance penalty, should be part of medical QA evaluation because it is the most discriminative open-ended task and reveals brittle reasoning chains that overlap metrics miss.
System rankings on open-ended medical QA will shift when re-scored with semantic similarity: models that recycle reference wording rank higher under BLEU, ROUGE, and METEOR, while paraphrasing models rank higher under the embedding metric.
A composite metric combining semantic similarity with explicit reasoning and entailment validation is the stated next step; the paper's results imply that purely distributional similarity is necessary but not sufficient for factual grounding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Implicit in the paper: because the full evaluation set is private, the leaderboard cannot be reproduced from the public sample alone; publishing item-level semantic scores for the open-ended items would let third parties check whether the metric's edge over n-gram scores persists outside the shared-task harness.
A testable extension would be to measure how the snap-to-one threshold behaves: near-perfect answers are treated as identical, so one could ask physicians whether fine-grained distinctions within that band correlate with patient-relevant answer quality.
The finding that critique (short-inverse) is easier than generation implies a cheap benchmark design: pairing each forward question with an inverse item separates factual recall from the ability to detect errors, and could be applied to non-medical domains.
Because the model used to draft the data is also a top-scoring baseline, a neutral control would compare system rankings on a fully human-written subset to test for home-field advantage.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The ClinIQLink dataset is the real contribution here: 4,978 expert-verified, source-grounded GP-level QA pairs across seven formats, with a documented curation workflow, double annotation on roughly half the items, and held-out disputes. That is a solid, reusable resource for the medical QA community, and the observation that unordered-list questions expose hallucinated extras better than true/false or multiple choice is practically useful and well supported by the data. The step-penalized multi-hop-inverse scoring is a sensible idea worth copying, even if its specific weights are hand-chosen.

The soft spots are real and one is load-bearing. Section 4.2 states plainly that the automated metrics used for open-ended evaluation — which include the ClinIQLink semantic score — were “not effective for evaluation of model responses, nor were they effective in discriminating top-ranking model responses from mediocre model responses.” That statement covers the paper’s own embedding metric, not just BLEU/ROUGE/METEOR. Yet Table 2 ranks systems using only those automatic scores, and the conclusion asserts that embedding-level similarity is “now required as the minimum standard.” No Task 2 physician preference results are reported anywhere, so there is no human-grounded validation of the metric in the paper. This is an internal inconsistency, not a minor caveat. The authors seem aware of the limitation and even call for better metrics, but they do not reconcile that with the strength of their findings.

Also worth noting: Llama 3.3-70B-Instruct generated the QA pairs and then ranks first on the leaderboard. This is disclosed, but it is a genuine contamination risk that the paper does not resolve. The metric weights and threshold parameters are ad hoc, and the private test set prevents full replication. And with only one participating team (three runs), the leaderboard is essentially a baseline sweep, so the ranking comparisons are weaker than the presentation suggests.

None of this kills the paper. It is a shared-task overview, and the dataset is the deliverable, not the metric validation. But the claims about the semantic metric need to be dialed back until Task 2 results or other human correlations are provided, or the contradiction with Section 4.2 needs to be explicitly addressed. As is, the paper deserves a serious referee — the resource is valuable and the authors are transparent about much of the methodology — but it needs major revision before the evaluation findings can be taken at face value.

Recommendation: send to peer review, with instructions to the authors to either report the Task 2 audit or soften the conclusions about the semantic metric. I would bring the dataset to a reading group, and I would cite it if I were working on medical QA evaluation.

Referee Report

3 major / 5 minor

Summary. The paper presents ClinIQLink, a shared task for evaluating LLMs on general-practitioner-level medical question answering. The authors describe a dataset of 4,978 expert-verified, source-grounded QA pairs in seven formats, an automated evaluation harness (Task 1) that uses exact match for closed-ended items and a custom three-tier embedding-based semantic similarity score for open-ended items, and a planned physician preference study (Task 2) that is announced but for which no results are reported. The paper reports baseline and participant results in a leaderboard (Table 2) and draws conclusions about the relative merits of embedding-based versus n-gram metrics, culminating in the claim that embedding-level similarity is 'now required as the minimum standard' for LLM evaluation (Section 8.5).

Significance. If the dataset is as carefully curated as described, ClinIQLink could be a useful resource for medical QA evaluation, particularly the multi-hop-inverse format, and the open-source evaluation script and sample dataset are concrete reproducibility strengths. However, the paper's central evaluative claim — that the custom semantic similarity score tracks answer quality better than BLEU/ROUGE/METEOR — is not supported by the evidence presented, because the paper itself states in Section 4.2 that the automated metrics were not effective for open-ended evaluation, because no Task 2 physician-judgment results are reported, and because the metric's parameters are hand-chosen without human validation. The disclosed overlap between the data-generation model (Llama 3.3-70B-Instruct) and the top-ranked baseline further undermines confidence in the leaderboard as a measure of medical competence. These issues are load-bearing for the paper's main claims, though the dataset contribution and task infrastructure remain potentially valuable.

major comments (3)

[§4.2, §7 (Table 2), §8.5] Section 4.2 states that "the automated metrics employed for analysis of the open-ended QA pairs were not effective for evaluation of model responses, nor were they effective in discriminating top-ranking model responses from mediocre model responses." Since this sentence refers to the automated metrics broadly, it includes the ClinIQLink semantic similarity score that is the basis of the open-ended column of Table 2 and of the leaderboard rankings. Yet Section 8.5 concludes that "embedding-level similarity scores for LLM evaluation tasks are now required as the minimum standard." This is a direct internal inconsistency: the paper's own text negates the validity of the metric on which its ranking and open-ended conclusions rest. The authors should either report per-item human judgments (from Task 2) that correlate with the semantic scores, or substantially weaken the claims in Sections 8.5 and 9. Without such evidence, the claim that the embedding metric reflects answer quality better than n-gram metrics is not established.
[§3.1, §3.2, §5, §7 (Table 2)] The dataset was generated with a neuro-symbolic pipeline whose "neuro" component was Llama 3.3-70B-Instruct (disclosed in Section 5), and the same model ranks first on the leaderboard (Table 2, overall 0.541). This creates a contamination risk: the model may have imprinted its own phrasing or fact-selection preferences into the QA pairs, inflating its own score relative to models that did not contribute to data generation. The disclosure is transparent, but no analysis is provided to estimate the magnitude of this effect — for example, a comparison of scores on human-written versus model-generated items, or a per-item analysis of whether Llama 3.3's answers are systematically closer to the reference strings. Without such a control, the leaderboard ranking cannot be interpreted as a measure of general medical competence.
[§4.1, §8.5] The semantic similarity score is defined with hand-chosen parameters — w_tok = 0.4, w_sent = 0.4, w_para = 0.2, the SBERT baseline offset beta = 0.25, the near-perfect snap at S >= 0.95, and the multi-hop-inverse penalty alpha(d) — but no evidence is presented that these values were validated against human judgments or that the results are robust to reasonable variations in them. The claim in Section 8.5 that embedding similarity "aligns far more closely with clinical accuracy" is therefore unsupported. The authors should provide an ablation or a sensitivity analysis, or at minimum report Task 2 correlations if they become available, before asserting the metric's superiority over n-gram measures.

minor comments (5)

[Abstract] The abstract contains a typo: "CLIN-IQLINKa" should be "ClinIQLink," and "4 978" uses a nonstandard spacing; please use consistent formatting for numbers.
[Table 1] The per-format counts of items with two independent reviews (369, 346, 341, 339, 353, 331, 318) sum to 2,397, but the table reports a total of 2,497–2,505 double-annotated items; please reconcile this discrepancy.
[§3.2] The sentence "The 1062 QA Pairs that had been flagged as feedback or disputes were not used for testing and are presently still being held for later review" should clarify whether these 1,062 items are disjoint from the 140 lower-relevance items set aside in §3.3, and whether any items were both disputed and rated below 5.
[§4.2, §8] Task 2 is listed in the Introduction as a contribution and described in Section 4.2, but no Task 2 results are presented anywhere in the paper. If the study is ongoing, state this explicitly and describe the planned analysis; otherwise, the physician audit should be reported or removed from the list of contributions.
[§8.3] The description of Phi-4-Reasoning-Plus mentions "624 malformed list entries and 813 invalid True/False lines" without explaining how these counts relate to the dataset sizes in Table 1; please provide the denominator or a brief clarification.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dataset is externally grounded and the metric is not fitted to the conclusions, though Section 4.2 raises a serious validity concern.

full rationale

The paper's central derivation chain is self-contained: the ClinIQLink dataset is grounded in open-source medical texts and human expert verification, not defined in terms of the evaluated models, and the semantic-similarity metric is a stated composition of token, sentence, and paragraph cosine layers with fixed weights rather than a parameter fitted to the leaderboard or to the conclusions. No equation in Section 4.1 reduces to an input, and no fitted quantity is renamed as a prediction. The self-citation to Singhal et al. (MultiMedQA) is background context and is independent published work, so it is not load-bearing. The disclosed fact that Llama 3.3-70B-Instruct generated the QA pairs and also ranks first is a contamination risk, but it is a validity concern, not circularity. However, the paper's own Section 4.2 states that the automated metrics for open-ended QA pairs "were not effective for evaluation of model responses, nor were they effective in discriminating top-ranking model responses from mediocre model responses," and no Task 2 physician-preference results are reported; this directly weakens the later Section 8.5 claim that embedding-level similarity scores are "now required as the minimum standard." That is an internal inconsistency and a missing evidence problem, not a circular derivation, so the circularity score remains 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim's load-bearing choices are the manually weighted semantic metric, the offset and snap thresholds, and the step-penalty schedule. No new physical or conceptual entities are introduced. The dataset validity rests on unquantified annotator reliability and on the neutrality of the Llama-based generator.

free parameters (4)

semantic metric weights (w_tok, w_sent, w_para) = 0.4, 0.4, 0.2
Chosen by the authors in Section 4.1; no external validation is reported for these weights.
SBERT baseline offset beta = 0.25
Subtracted from the raw semantic score in Section 4.1; claimed as SBERT baseline similarity for unrelated pairs.
near-perfect snap threshold = 0.95
Scores at or above 0.95 are set to 1 in Section 4.1.
multi-hop inverse step penalty alpha(d) = 1, 0.7, 0.3, then decaying
Weights in Section 4.1 for wrong reasoning step distance d; chosen by hand.

assumptions (4)

domain assumption Paid medical students are reliable expert annotators for factual accuracy and GP relevance.
Section 3.2; the entire dataset validity rests on this, but no inter-annotator agreement statistic is reported.
domain assumption The open-source medical texts used as source grounding are authoritative, and the QA pairs correctly extract facts from them.
Section 3.1; the neuro-symbolic pipeline links each pair to a supporting passage, but the extraction correctness is not independently audited.
ad hoc to paper The custom semantic similarity metric is a valid measure of answer correctness.
Section 4.1; weights and offset are hand-set, and the paper later says automated metrics were not effective, so this assumption is internally weakened.
standard math Standard mathematical formulations of cosine similarity, F1, BLEU, ROUGE, and METEOR are correct.
Used without proof throughout Section 4.1; this is standard background.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Overview of the ClinIQLink 2025 Shared Task on Medical Question-Answering." pith.science (2026). https://pith.science/paper/SEGXEEHR

@misc{pith2026250621597,
  author       = {Pith},
  title        = {Pith review of: Overview of the ClinIQLink 2025 Shared Task on Medical Question-Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SEGXEEHR}},
  note         = {Machine review of arXiv:2506.21597}
}

read the original abstract

In this paper, we present an overview of ClinIQLink, a shared task, collocated with the 24th BioNLP workshop at ACL 2025, designed to stress-test large language models (LLMs) on medically-oriented question answering aimed at the level of a General Practitioner. The challenge supplies 4,978 expert-verified, medical source-grounded question-answer pairs that cover seven formats: true/false, multiple choice, unordered list, short answer, short-inverse, multi-hop, and multi-hop-inverse. Participating systems, bundled in Docker or Apptainer images, are executed on the CodaBench platform or the University of Maryland's Zaratan cluster. An automated harness (Task 1) scores closed-ended items by exact match and open-ended items with a three-tier embedding metric. A subsequent physician panel (Task 2) audits the top model responses.

Figures

Figures reproduced from arXiv: 2506.21597 by the authors.

**Figure 2.** Distributions of individual n-gram scores [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Comprehensive dashboard for FLAN-UL2 showing boxplots, jitter plots and histograms across semantic and n-gram metrics. tra choices (e.g. falcon3_10b_instruct) underperformed relative to their multiple-choice score. The tight inter-quartile ranges on True/False and multiple-choice further suggest that most contemporary LLMs share a common ceiling on purely factual one-shot classification, leaving little room for ar… view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Comprehensive dashboard for Phi-4- reasoning-plus showing boxplots, jitter plots and histograms across semantic and n-gram metrics. closed-ended performance. • LLaMA-3-3-70B [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 1 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, Piero Kauffmann, Yash Lara, Caio César Teodoro Mendes, Arindam Mitra, Besmira Nushi, Dimitris Papailiopoulos, Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, and 4 others. 2025. https://arxi...

arXiv 2025
[4]

Hewett, Mojan Javaheripi, Piero Kauffmann, James R

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 others. 2024. https://arxiv.org/abs/2412.08905 Phi-4 technic...

arXiv 2024
[5]

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. https://arxiv.org/abs/2311.16867 The falcon series of open language models . Preprint, arXiv:2...

arXiv 2023
[6]

Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909/ METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Compu...

2005
[7]

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. https://arxiv.org/abs/2202.07646 Quantifying memorization across neural language models . Preprint, arXiv:2202.07646

arXiv 2023
[8]

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, and 16 others. 2022. https://arxiv.org/abs/2210.11416 Scaling instruction-fin...

arXiv 2022

Show all 23 references

[9]

Emma Croxford, Yanjun Gao, Nicholas Pellegrino, Karen Wong, Graham Wills, Elliot First, Frank Liao, Cherodeep Goswami, Brian Patterson, and Majid Afshar. 2025. https://doi.org/10.1038/s44401-024-00011-2 Current and future state of evaluation of large language models for medica...

2025 doi
[10]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

2024 arXiv
[11]

Takumi Ito, Kees van Deemter, and Jun Suzuki. 2025. https://doi.org/10.48550/ARXIV.2501.12011 Reference-free evaluation metrics for text generation: A survey . arXiv preprint

2025 doi arXiv
[12]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

2023 arXiv
[13]

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

2024 arXiv
[14]

Omar Khattab and Matei Zaharia. 2020. https://arxiv.org/abs/2004.12832 Colbert: Efficient and effective passage search via contextualized late interaction over bert . Preprint, arXiv:2004.12832

2020 arXiv
[15]

Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

2004
[16]

Manning, Prabhakar Raghavan, and Hinrich Schütze

Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press

2008
[17]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

2002
[18]

Stephen Robertson, Hugo Zaragoza, and 1 others. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389

2009
[19]

Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Sch\" a rli, Aakanksha Chowdhery, Philip M...

2023 doi
[20]

Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler

Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. https://arxiv.org/abs/2205.05131 Ul2: Unifying language learning ...

2023 arXiv
[21]

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://arxiv.org/abs/2505.09388 Qw...

2025 arXiv
[22]

Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. 2023. https://arxiv.org/abs/2305.06311 Automatic evaluation of attribution by large language models . Preprint, arXiv:2305.06311

2023 arXiv
[23]

Hye Sun Yun and Timothy Bickmore. 2025. https://doi.org/10.2196/68560 Online health information--seeking in the era of large language models: Cross-sectional web-based survey study . J Med Internet Res, 27:e68560

2025 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, Piero Kauffmann, Yash Lara, Caio César Teodoro Mendes, Arindam Mitra, Besmira Nushi, Dimitris Papailiopoulos, Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, and 4 others. 2025. https://arxi...

arXiv 2025

[4] [4]

Hewett, Mojan Javaheripi, Piero Kauffmann, James R

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 others. 2024. https://arxiv.org/abs/2412.08905 Phi-4 technic...

arXiv 2024

[5] [5]

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. https://arxiv.org/abs/2311.16867 The falcon series of open language models . Preprint, arXiv:2...

arXiv 2023

[6] [6]

Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909/ METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Compu...

2005

[7] [7]

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. https://arxiv.org/abs/2202.07646 Quantifying memorization across neural language models . Preprint, arXiv:2202.07646

arXiv 2023

[8] [8]

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, and 16 others. 2022. https://arxiv.org/abs/2210.11416 Scaling instruction-fin...

arXiv 2022