Pith. sign in

REVIEW 3 major objections 4 minor 84 references

When Retrieval Helps and Distracts: Evaluating Evidence-Generating LLMs for Biomedical Claim Verification

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper establishes that PubMed retrieval helps verify claims aligned with scientific abstracts but actively distracts on broader public-health claims, and that its gold-reference-normalized diagnostic, Bio-GRACE, exposes when retrieval

desk verdict Careful evaluation and a transparent protocol, but Bio-GRACE's central finding rests on subtracting probabilities from separately trained classifiers, so the 'retrieval distracts' result needs a same-verifier recomputation before it can be trusted. read the letter →

arxiv 2608.01409 v1 pith:53GCAVSV submitted 2026-08-02 cs.CL

Pritam Deka , Prabhjot Singh This is my paper
classification cs.CL
keywords biomedicalclaimverificationevidencegenerationretrieval-augmentedPubMedretrievalBio-GRACEselectiveCARE-XAIfact-checking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks what it takes for a biomedical fact-checking system to do more than pick a verdict: it must also produce evidence that is faithful, complete, and useful. To answer, it builds CARE-XAI, a unified benchmark of 17,803 claims from five fact-checking sources, and compares base LLMs, PubMed retrieval-augmented LLMs, fine-tuned LLMs, label-only LLMs, and biomedical classifiers under one protocol. The central result is that verdict-only classifiers are still the strongest label predictors, fine-tuned LLMs are the most reliable evidence generators, and PubMed retrieval is source-dependent: it recovers much of the reference-evidence benefit on PubMed-aligned sources but actively distracts on public-health and misinformation claims. If correct, this means always-on PubMed RAG is not a safe default and should be replaced by selective retrieval, and it motivates a new diagnostic, Bio-GRACE, that measures decision utility rather than topical relevance.

What carries the argument

Bio-GRACE (Biomedical Gold-Reference Assessment of Contextual Evidence) is the paper's central diagnostic. For each claim it records the verifier's probability of the true label under three inputs: claim-only $p^C_i(y_i)$, retrieved context $p^R_i(y_i)$, and gold reference evidence $p^G_i(y_i)$. Oracle utility is $U^\star_i = p^G_i(y_i) - p^C_i(y_i)$; retrieved utility is $U^R_i = p^R_i(y_i) - p^C_i(y_i)$. On evidence-sensitive examples $H = \{i : U^\star_i > 0\}$, utility recovery is $UR = \frac{1}{|H|}\sum_{i \in H}\mathrm{clip}\!\left(\frac{U^R_i}{U^\star_i + \epsilon}; -1, 1\right)$. Positive UR means retrieval recovers some of the reference-evidence benefit; negative UR means retrieval

What would settle it

On a stratified sample of PUBHEALTH rows with negative UR, have two experts judge whether each retrieved PubMed abstract actually addresses the claim; if the abstracts address the claims and UR remains negative, retrieval distraction is confirmed, while if the abstracts are off-topic the negative UR reflects source mismatch rather than retrieval-induced distraction.

Watch

Extended reading notes

Core claim

On CARE-XAI, reference/gold evidence is decision-useful: supplying it lifts classifier macro-F1 from 0.547 (claim-only) to 0.692, so the task is evidence-sensitive. Retrieved PubMed context does not supply that benefit: it leaves classifier macro-F1 at 0.386. Bio-GRACE makes the pattern explicit by comparing, on evidence-sensitive examples, how much retrieved context raises the verifier's true-label probability relative to gold evidence. The score is strongly positive for PubMedQA (UR=0.676) and SciFact (0.435), near zero for HealthVer, and negative for HealthFC (-0.222) and PUBHEALTH (-0.378); all three encoder verifiers have negative aggregate UR. The paper concludes that retrieval utility

Load-bearing premise

The load-bearing premise is that CARE-XAI's reference evidence is a valid gold standard and that the decision benefit of that evidence is faithfully captured by the change in a supervised classifier's probability of the true label; the paper itself notes the reference evidence may be incomplete or heterogeneous.

Editorial extensions

If this is right

  • Deployers of biomedical fact-checking should treat always-on PubMed RAG as a liability on public-health and misinformation claims; the paper's source-router diagnostic raises label-only LLM macro-F1 from 0.343 to 0.381 by disabling retrieval on negative-utility sources.
  • Retrieval evaluation should include a decision-utility metric like Bio-GRACE alongside recall and nDCG, because the paper shows high query success (0.871) with low source-PMID recall (0.224 at 10) can coexist with negative utility.
  • Fine-tuning on the target evidence format is a more reliable adaptation than retrieval for evidence-generating LLMs: mean macro-F1 across complete fine-tuned runs is 0.447 versus 0.385 for RAG runs.
  • Reference evidence is decision-useful (classifier macro-F1 rises from 0.547 claim-only to 0.692 with gold evidence), so the observed RAG failures are not evidence-insensitivity in the verifier but a supply-side source-matching problem.
  • The UNADDRESSED verdict class is a distinct challenge: systems that retrieve topically related abstracts must avoid over-predicting SUPPORTED, and Bio-GRACE's evidence-sensitive set makes this explicit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same source-gating logic could be applied per claim rather than per source, using retriever confidence or verifier uncertainty as a gate; the paper's no-new-inference router already shows headroom over always-retrieval.
  • Editorial inference: adding a non-PubMed retriever (news, guidelines, policy documents) for public-health sources is the most direct testable extension; if it turns PUBHEALTH and HealthFC Bio-GRACE UR positive, the paper's source-mismatch reading is confirmed.
  • Editorial inference: because Bio-GRACE uses supervised verifier probabilities, its sign should be checked under a calibrated verifier; if negative UR flips sign with calibration, part of the distraction effect is attributable to miscalibrated confidence rather than retrieval content.
  • Editorial inference: the low inter-annotator agreement in the paper's 100-output human evaluation suggests that evidence-faithfulness standards vary even among experts; a practical extension would be adjudicated multi-rater evidence labels that can serve as a higher-confidence test bed for Bio-GRACE.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper evaluates evidence-generating LLMs for biomedical claim verification on CARE-XAI, a unified benchmark of 17,803 examples from PubMedQA, SciFact, HealthVer, PUBHEALTH, and HealthFC. It compares base instruction LLMs, PubMed RAG LLMs, fine-tuned LLMs, label-only LLMs, and biomedical encoder classifiers under a shared protocol with a frozen retrieval cache, explicit failure manifest, and leakage audit. The main empirical conclusions are: (i) biomedical classifiers remain strongest for verdict-only prediction; (ii) fine-tuned LLMs are the most reliable evidence generators; and (iii) PubMed retrieval helps on PubMed-aligned sources (PubMedQA, SciFact) but distracts on public-health sources (PUBHEALTH, HealthFC). The paper introduces Bio-GRACE, a gold-reference-normalized diagnostic intended to measure how much of the decision benefit of reference evidence is recovered by retrieved context, and uses it to argue that retrieval utility is source-dependent and that retrieval should be selective rather than always-on.

Significance. If the central claims hold, the paper makes a useful contribution to biomedical fact-checking evaluation. The controlled evaluation design is a genuine strength: a frozen retrieval cache decouples retrieval from answer generation, prompts and output contracts are shared across families, failed runs are retained in a manifest rather than discarded, and a leakage audit is reported. The finding that verdict-only biomedical classifiers outperform evidence-generating LLMs on label prediction is credible and important, as is the observation that retrieval metrics such as Recall@10 do not imply decision utility. Bio-GRACE is an interesting diagnostic idea, but its current implementation has a load-bearing internal inconsistency: the three probabilities it subtracts come from separately trained classifier families, not from a single verifier under different evidence conditions. The paper also lacks error bars and significance tests for its main LLM comparisons. These issues are fixable within the scope of the manuscript, but they must be addressed before the source-dependent retrieval-utility claim is accepted.

major comments (3)
  1. [§5, Eqs. (4)–(6); §4/§5, Tables 7–8] Bio-GRACE is defined as a counterfactual intervention on a verifier: p^C_i, p^R_i, and p^G_i should be probabilities from the same verifier under claim-only, retrieved, and gold-evidence inputs. The implementation, however, uses three separately trained 9-run classifier families, labeled 'Classifier claim-only,' 'Classifier Retrieved PubMed,' and 'Classifier Gold evidence' in Table 7. Each family is trained on a different input condition. Consequently, U^R_i = p^R_i − p^C_i and U*_i = p^G_i − p^C_i subtract probabilities across different models, conflating the evidence intervention with training-condition, model-capacity, and seed differences. The large gaps in Table 7 (0.547, 0.386, 0.692) are exactly what one would expect from separately trained models and do not establish what a fixed verifier would do. This is load-bearing because the headline 'retrieval utility is source-dependent'
  2. [§6, Tables 5–6, Figure 3; Table 9] The main LLM comparisons are single runs: no seeds are reported for LLM rows, and the mean macro-F1 values in Table 6 are presented without confidence intervals or significance tests. The best fine-tuned LLM (0.483) and best base LLM (0.450) differ by 0.033, and the RAG-vs-base differences vary by prompt; without multiple seeds or paired bootstrap tests over examples, 'fine-tuning is the most reliable adaptation strategy' is not statistically established. The same issue applies to the source-level UR estimates in Table 9, where the effective sample sizes are only 90–105 unique rows per source. Please provide seed-level variation or bootstrap intervals, and paired significance tests for the matched prompt conditions, especially the zero-shot comparisons that the Limitations section identifies as the cleanest retrieval-effect estimate.
  3. [§7, Table 11; Appendix H] The leakage audit reports only regime-level macro-F1 sensitivity (Table 11), but the paper's central claim is that retrieval utility is source-dependent. Given that 588 test rows have exact evidence overlap and the stricter evidence-group-safe filter removes 591 rows, the source-level Bio-GRACE and NRI values could change substantially after filtering, particularly for PUBHEALTH, which dominates the test set. The sentence in §7 that 'retrieval remains source-dependent' is not supported by the table actually presented. Please report leakage-safe Bio-GRACE/NRI by source, or at least show that the source-level ordering in Table 9 is preserved on the evidence-group-safe subset.
minor comments (4)
  1. [§6.3, Table 10] The source-router simulation is transparently labeled 'no-new-inference,' but because the router is constructed from the same test-set UR values it is meant to predict, the modest gains partly reflect the diagnostic signal by construction. The paper already cautions this; I would make that caution more prominent in the main text and avoid wording such as 'motivates selective biomedical RAG' being read as a validated routing result.
  2. [Table 9] SciFact has Hit@10 = 0.000 but positive UR = 0.435. This is not contradictory, since retrieval can help without recovering source PMIDs, but it is surprising enough to deserve an explicit explanation in the text.
  3. [Appendix P, Table 34] The human-consensus denominators are small (21–29 per regime after exact agreement), so the regime percentages in Table 34 have wide uncertainties. Reporting bootstrapped intervals or including the raw counts more prominently would prevent over-reading.
  4. [References and formatting] There are typographical issues in the references, e.g., 'V olume' in the ARES entry and 'F oundations' in Robertson and Zaragoza (2009). The spelling of 'synthesised' vs. 'synthesized' is inconsistent. These are minor.

Circularity Check

1 steps flagged · score 4.0 of 10

No central circularity: Bio-GRACE and main verdict comparisons are independent evaluations. One secondary diagnostic, the Table 10 source-router simulation, is circular because its routing rule is read from test-set Bio-GRACE values and then evaluated on the same test set.

  1. fitted input called prediction [Section 6.3, Table 10 caption and surrounding text]
    "The source router uses retrieval for sources with positive Bio-GRACE UR and claim-only prediction otherwise. ... The routing simulation in Table 10 is not a deployed model and does not use a validation-trained threshold. It tests whether the Bio-GRACE source signal identifies where retrieval should be used."

    The router's decision rule is derived from source-level Bio-GRACE UR values computed on the same 1,752-row test set (Table 9: PubMedQA and SciFact positive; HealthVer, HealthFC, PUBHEALTH negative). Table 10 then reports source-router macro-F1 on that exact test set. The reported gains (classifier 0.547→0.567; label-only LLM 0.331→0.381) are therefore a selection of test-set condition means using test-set-derived signals, not an out-of-sample prediction of selective-retrieval benefit. The paper discloses that the router is a 'no-new-inference diagnostic' and does not claim a deployed system, but the 'motivates selective retrieval' conclusion still relies on this in-sample gain, which is forced by construction rather than independently validated.

full rationale

The central derivation is not circular. Bio-GRACE (Eqs. 4–6) is defined as a comparison of true-label probabilities under claim-only, retrieved, and gold-evidence inputs, normalized by gold-evidence benefit; the gold/reference evidence is used only for evaluation, and the source-dependent signs of UR are empirical outcomes rather than consequences of the definition. The main verdict comparisons (Tables 5–7) are standard macro-F1 evaluations against external CARE-XAI labels and do not fit parameters to the quantities they predict. The CARE-XAI self-citation (Singh, Prabhjot, 2026) is a dataset-artifact citation, not a load-bearing uniqueness theorem, and it does not by itself make the paper's results circular. The paper's own limitation that reference evidence 'may itself be incomplete or heterogeneous' is an external gold-standard caveat, not a by-construction reduction. One genuine circular step exists: the Table 10 source-router simulation uses test-set Bio-GRACE values to choose which sources get retrieval and then evaluates on the same test set, so its gains are partly manufactured by the evaluation protocol. That step is disclosed and peripheral rather than the central claim. Separately, the same-verifier premise of Bio-GRACE is not evidently honored in the implementation: Table 7 lists separate 9-run classifier families for claim-only, retrieved, and gold conditions, so U^R and U^* subtract probabilities from different models. This is a validity/correctness risk for interpreting UR as a counterfactual intervention, but it is not a circularity because the metric's values are not equivalent to its inputs by construction. Overall, the paper's central results are self-contained and externally grounded; the circularity score is low-moderate due to the in-sample router diagnostic.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central empirical claims rest on the CARE-XAI labels/reference evidence being correct, on PubMed retrieval being the right intervention to test, and on classifier probabilities being a meaningful measure of decision benefit. The fine-tuning hyperparameters and Bio-GRACE numeric choices are standard modeling decisions, not fitted parameters that bake in the conclusion.

free parameters (3)
  • Bio-GRACE epsilon = not stated
    Small additive constant in the denominator of UR to avoid division by zero; hand-chosen stabilizer, not fitted to data.
  • Bio-GRACE clip bound = 1
    Clip range [-1, 1] is a design choice to bound the score; not fitted.
  • Retrieval top-k = not stated in main text
    Number of PubMed contexts inserted into the prompt is selected but the exact k is not given in the main text, harming reproducibility and constituting a hand-chosen modeling parameter.
assumptions (4)
  • domain assumption CARE-XAI reference labels and evidence are ground truth
    All verdict metrics and Bio-GRACE normalize against the reference evidence and labels; if these are incomplete or noisy, conclusions shift.
  • domain assumption PubMed is the appropriate retrieval source for the tested claims
    The study only tests PubMed+Entrez+BGE-M3; public-health claims may need news/policy evidence, so 'retrieval distracts' is specific to PubMed.
  • domain assumption Classifier probability shifts measure decision benefit
    Bio-GRACE defines oracle and retrieved utility via p^G, p^C, p^R from supervised verifiers; this assumes the verifier's probabilities are a faithful measure of evidence usefulness.
  • ad hoc to paper Source router rule (retrieve for positive-UR sources) is a valid diagnostic
    The router is constructed from test-set UR and is explicitly not a validation-trained model; it is an illustrative diagnostic.
invented entities (1)
  • Bio-GRACE (UR score)
    purpose: Gold-reference-normalized diagnostic measuring how much retrieved context recovers the decision benefit of reference evidence
    It is a new metric defined and evaluated within this paper; it has no external falsifiable handle and depends on the choice of verifier.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Retrieval Helps and Distracts: Evaluating Evidence-Generating LLMs for Biomedical Claim Verification." pith.science (2026). https://pith.science/paper/53GCAVSV

@misc{pith2026260801409,
  author       = {Pith},
  title        = {Pith review of: When Retrieval Helps and Distracts: Evaluating Evidence-Generating LLMs for Biomedical Claim Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/53GCAVSV}},
  note         = {Machine review of arXiv:2608.01409}
}
read the original abstract

Biomedical fact-checking systems must do more than predict whether a claim is supported, contradicted, or unaddressed: they should also produce evidence that is faithful, complete, and useful for verification. We study this evidence-generation setting on CARE-XAI, a unified benchmark spanning five biomedical and health fact-checking sources. We compare base instruction LLMs, PubMed retrieval-augmented LLMs, fine-tuned LLMs, label-only LLMs, and biomedical encoder classifiers under a shared evaluation protocol. Biomedical classifiers remain strongest for verdict-only prediction, while fine-tuned LLMs are the strongest evidence-generating systems. PubMed retrieval is mixed: it helps PubMed-aligned sources such as PubMedQA and SciFact, but can distract models on broader public-health claims. We introduce Bio-GRACE, a gold-reference-normalized diagnostic for measuring whether retrieved evidence recovers the decision benefit of reference evidence. Bio-GRACE shows that retrieval utility is source-dependent, motivates selective retrieval, and exposes why retrieval recall and lexical evidence overlap are insufficient for biomedical fact-checking.

Figures

Figures reproduced from arXiv: 2608.01409 by the authors.

Figure 1
Figure 1. Overview of the evaluation framework. CARE-XAI claims are processed by base, PubMed RAG, and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Frozen PubMed retrieval diagnostics by source. Query success is high, but source-PMID recov￾ery is uneven and often weak for public-health sources. stracts for a claim without retrieving the source evidence that determines the CARE-XAI verdict. This motivates separating retrieval success, down￾stream verdict accuracy, and Bio-GRACE utility instead of treating retrieved context as automati￾cally beneficial. Bio-GRACE… view at source ↗
Figure 3
Figure 3. Best macro-F1 for LLMs with complete base, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (11 more)
Figure 5
Figure 5. Figure 5: Bio-GRACE utility by source. PubMedQA and SciFact benefit from PubMed retrieval, while PUB￾HEALTH and HealthFC show negative retrieval utility. Family Condition Runs Macro-F1 Classifier claim-only 9 0.547 Classifier always-retrieved 9 0.386 Classifier source-router 9 0…
Figure 6
Figure 6. Figure 6: Retrieval distraction and rescue rates by source. Public-health sources show higher distraction than rescue, [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: Classifier macro-F1 under claim-only, retrieved-evidence, and gold-evidence protocols. Gold evidence is [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: Label-only LLM protocol comparison. Removing evidence generation isolates whether LLMs can use [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: Full-test versus leakage-filtered performance. The qualitative ordering remains stable after evidence [PITH_FULL_IMAGE:figures/full_fig_p029_9.png]
Figure 10
Figure 10. Figure 10: Macro-F1 versus accuracy for paper-facing runs. Macro-F1 is emphasized because class imbalance [PITH_FULL_IMAGE:figures/full_fig_p030_10.png]
Figure 14
Figure 14. Figure 14: Classifier calibration error for representative runs. Calibration is used as an uncertainty diagnostic rather [PITH_FULL_IMAGE:figures/full_fig_p030_14.png]
Figure 15
Figure 15. Figure 15: Selective risk–coverage behavior for classifier confidence. Higher confidence improves risk on retained [PITH_FULL_IMAGE:figures/full_fig_p030_15.png]
Figure 11
Figure 11. Figure 11: NLI entailment behavior by scorer and regime. Biomedical NLI scorers are more permissive than the [PITH_FULL_IMAGE:figures/full_fig_p031_11.png]
Figure 12
Figure 12. Figure 12: NLI length sensitivity. Evidence length and template-like outputs can affect entailment rates, reinforcing [PITH_FULL_IMAGE:figures/full_fig_p032_12.png]
Figure 13
Figure 13. Figure 13: Mean entailment rate by NLI scorer and generation regime. The spread across scorers is larger than [PITH_FULL_IMAGE:figures/full_fig_p032_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

84 extracted references · 44 canonical work pages

  1. [1]

    Journal of Data Intelligence , volume=

    Improved methods to aid unsupervised evidence-based fact checking for online health news , author=. Journal of Data Intelligence , volume=

  2. [2]

    International conference on health information science , pages=

    Evidence extraction to validate medical claims in fake news detection , author=. International conference on health information science , pages=. 2022 , organization=

    2022

  3. [3]

    The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks , pages=

    Multiple evidence combination for fact-checking of health-related information , author=. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks , pages=

  4. [4]

    , booktitle=

    DeYoung, Jay and Jain, Sarthak and Rajani, Nazneen Fatema and Lehman, Eric and Xiong, Caiming and Socher, Richard and Wallace, Byron C. , booktitle=

  5. [5]

    Towards Faithfully Interpretable

    Jacovi, Alon and Goldberg, Yoav , booktitle=. Towards Faithfully Interpretable

  6. [6]

    Proceedings of the 18th conference of the european chapter of the association for computational linguistics: system demonstrations , pages=

    Ragas: Automated evaluation of retrieval augmented generation , author=. Proceedings of the 18th conference of the european chapter of the association for computational linguistics: system demonstrations , pages=

  7. [7]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    Ares: An automated evaluation framework for retrieval-augmented generation systems , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    work page 2024

  8. [8]

    International Conference on Learning Representations , volume=

    Making retrieval-augmented language models robust to irrelevant context , author=. International Conference on Learning Representations , volume=

    work page

Show all 84 references
  1. [9]

    Advances in Neural Information Processing Systems , volume=

    Worse than zero-shot? a fact-checking dataset for evaluating the robustness of rag against misleading retrievals , author=. Advances in Neural Information Processing Systems , volume=

  2. [10]

    Jain, Siddharth and Vedam, Venkat Narayan , journal=

  3. [11]

    arXiv preprint arXiv:2605.01302 , year=

    Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generation , author=. arXiv preprint arXiv:2605.01302 , year=

    arXiv

  4. [12]

    Pubmedqa: A dataset for biomedical research question answering , author=. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages=

    2019

  5. [13]

    Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=

    HealthFC: Verifying health claims with evidence-based medical fact-checking , author=. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=

    2024

  6. [14]

    2026 , howpublished=

    2026

  7. [15]

    MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims , author=. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages=

    2019

  8. [16]

    arXiv preprint arXiv:2601.06519 , year=

    MedRAGChecker: Claim-Level Verification for Biomedical Retrieval-Augmented Generation , author=. arXiv preprint arXiv:2601.06519 , year=

  9. [17]

    arXiv preprint arXiv:2604.08549 , year=

    VerifAI: A Verifiable Open-Source Search Engine for Biomedical Question Answering , author=. arXiv preprint arXiv:2604.08549 , year=

    arXiv

  10. [18]

    arXiv preprint arXiv:2512.10996 , year=

    MedBioRAG: Semantic Search and Retrieval-Augmented Generation with Large Language Models for Medical and Biological QA , author=. arXiv preprint arXiv:2512.10996 , year=

  11. [19]

    arXiv preprint arXiv:2502.03004 , year=

    MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation , author=. arXiv preprint arXiv:2502.03004 , year=

    arXiv

  12. [20]

    Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    Combining evidence and reasoning for biomedical fact-checking , author=. Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

  13. [21]

    arXiv preprint arXiv:2606.12854 , year=

    Small LLMs for Biomedical Claim Verification: Cost-Effective Fine-Tuning, Structural Dataset Shortcuts, and Cross-Domain Generalization , author=. arXiv preprint arXiv:2606.12854 , year=

    arXiv

  14. [22]

    arXiv preprint arXiv:2509.12440 , year=

    MedFact: Benchmarking the Fact-Checking Capabilities of Large Language Models on Chinese Medical Texts , author=. arXiv preprint arXiv:2509.12440 , year=

    arXiv

  15. [23]

    arXiv preprint arXiv:2509.17436 , year=

    MedFact: A Large-scale Chinese Dataset for Evidence-based Medical Fact-checking of LLM Responses , author=. arXiv preprint arXiv:2509.17436 , year=

  16. [24]

    International Conference on Machine Learning , pages=

    On Calibration of Modern Neural Networks , author=. International Conference on Machine Learning , pages=

  17. [25]

    Advances in Neural Information Processing Systems , year=

    Selective Classification for Deep Neural Networks , author=. Advances in Neural Information Processing Systems , year=

  18. [26]

    Advances in Neural Information Processing Systems , year=

    Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , author=. Advances in Neural Information Processing Systems , year=

  19. [27]

    International Conference on Learning Representations , year=

    Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation , author=. International Conference on Learning Representations , year=

  20. [28]

    arXiv preprint arXiv:2403.08295 , year=

    Gemma: Open Models Based on Gemini Research and Technology , author=. arXiv preprint arXiv:2403.08295 , year=

    arXiv

  21. [29]

    arXiv preprint arXiv:2503.19786 , year=

    Gemma 3 Technical Report , author=. arXiv preprint arXiv:2503.19786 , year=

    arXiv

  22. [30]

    arXiv preprint arXiv:2412.15115 , year=

    Qwen2.5 Technical Report , author=. arXiv preprint arXiv:2412.15115 , year=

    arXiv

  23. [31]

    arXiv preprint arXiv:2403.05530 , year=

    Gemini 1.5: Unlocking Multimodal Understanding across Millions of Tokens of Context , author=. arXiv preprint arXiv:2403.05530 , year=

    arXiv

  24. [32]

    Chen, Jianlv and Xiao, Shitao and Zhang, Peitian and Luo, Kun and Lian, Defu and Liu, Zheng , journal=

  25. [33]

    Lee, Jinhyuk and Yoon, Wonjin and Kim, Sungdong and Kim, Donghyeon and Kim, Sunkyu and So, Chan Ho and Kang, Jaewoo , journal=

  26. [34]

    ACM Transactions on Computing for Healthcare , volume=

    Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing , author=. ACM Transactions on Computing for Healthcare , volume=

  27. [35]

    Association for Computational Linguistics , pages=

    LinkBERT: Pretraining Language Models with Document Links , author=. Association for Computational Linguistics , pages=

  28. [36]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=

    Explainable Automated Fact-Checking for Public Health Claims , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=

    2020

  29. [37]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=

    Fact or Fiction: Verifying Scientific Claims , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=

    2020

  30. [38]

    Findings of the Association for Computational Linguistics: EMNLP , pages=

    Evidence-Based Fact-Checking of Health-Related Claims , author=. Findings of the Association for Computational Linguistics: EMNLP , pages=

  31. [39]

    The Annals of Statistics , volume=

    Bootstrap Methods: Another Look at the Jackknife , author=. The Annals of Statistics , volume=

  32. [40]

    2026 , url=

    Gemma 4 Technical Report , author=. 2026 , url=

    2026

  33. [41]

    arXiv preprint arXiv:2312.11805 , year=

    Gemini: a family of highly capable multimodal models , author=. arXiv preprint arXiv:2312.11805 , year=

    arXiv

  34. [42]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    arXiv

  35. [43]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. Iclr , volume=

  36. [44]

    Journal of Educational Psychology , volume=

    Estimating causal effects of treatments in randomized and nonrandomized studies , author=. Journal of Educational Psychology , volume=

  37. [45]

    Statistics in Medicine , volume=

    Surrogate endpoints in clinical trials: definition and operational criteria , author=. Statistics in Medicine , volume=

  38. [46]

    Journal of the American Statistical Association , volume=

    Strictly proper scoring rules, prediction, and estimation , author=. Journal of the American Statistical Association , volume=

  39. [47]

    IEEE Transactions on Systems Science and Cybernetics , volume=

    Information value theory , author=. IEEE Transactions on Systems Science and Cybernetics , volume=

  40. [48]

    Retrieval-Augmented Generation for Knowledge-Intensive

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and Kuttler, Heinrich and Lewis, Mike and Yih, Wen-tau and Rocktaschel, Tim and Riedel, Sebastian and Kiela, Douwe , booktitle=. Retrieval-Augmented Generation fo...

  41. [49]

    Proceedings of EMNLP , pages=

    Dense Passage Retrieval for Open-Domain Question Answering , author=. Proceedings of EMNLP , pages=

  42. [50]

    Proceedings of EACL , pages=

    Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering , author=. Proceedings of EACL , pages=

  43. [51]

    Proceedings of ICML , year=

    Improving Language Models by Retrieving from Trillions of Tokens , author=. Proceedings of ICML , year=

  44. [52]

    Asai, Akari and Wu, Zeqiu and Wang, Yizhong and Sil, Avirup and Hajishirzi, Hannaneh , booktitle=. Self-

  45. [53]

    Proceedings of EMNLP , pages=

    Active Retrieval Augmented Generation , author=. Proceedings of EMNLP , pages=

  46. [54]

    arXiv preprint arXiv:2312.10997 , year=

    Retrieval-Augmented Generation for Large Language Models: A Survey , author=. arXiv preprint arXiv:2312.10997 , year=

    arXiv

  47. [55]

    Advances in Neural Information Processing Systems , volume=

    Ragchecker: A fine-grained framework for diagnosing retrieval-augmented generation , author=. Advances in Neural Information Processing Systems , volume=

  48. [56]

    Gao, Tianyu and Yen, Hsin-Yuan and Yu, Jiatong and Chen, Danqi , booktitle=

  49. [57]

    Wu, Jinhao and Zhang, Zhaoheng and Zhang, Yuan and others , booktitle=

  50. [58]

    Min, Sewon and Krishna, Kalpesh and Lyu, Xinxi and Lewis, Mike and Yih, Wen-tau and Koh, Pang and Iyyer, Mohit and Zettlemoyer, Luke and Hajishirzi, Hannaneh , booktitle=

  51. [59]

    Manakul, Potsawee and Liusie, Adian and Gales, Mark , booktitle=

  52. [60]

    Proceedings of ACL , pages=

    On Faithfulness and Factuality in Abstractive Summarization , author=. Proceedings of ACL , pages=

  53. [61]

    ACM Computing Surveys , volume=

    Survey of Hallucination in Natural Language Generation , author=. ACM Computing Surveys , volume=

  54. [62]

    and Artzi, Yoav , booktitle=

    Zhang, Tianyi and Kishore, Varsha and Wu, Felix and Weinberger, Kilian Q. and Artzi, Yoav , booktitle=

  55. [63]

    Lin, Chin-Yew , booktitle=

  56. [64]

    Thorne, James and Vlachos, Andreas and Christodoulopoulos, Christos and Mittal, Arpit , booktitle=

  57. [65]

    Proceedings of EMNLP , pages=

    Lessons from Natural Language Inference in the Clinical Domain , author=. Proceedings of EMNLP , pages=

  58. [66]

    Herlihy, Christine and Rudinger, Rachel , booktitle=. Med

  59. [67]

    Beltagy, Iz and Lo, Kyle and Cohan, Arman , booktitle=

  60. [68]

    and Weissenborn, Dirk and Krithara, Anastasia and Petridis, Sergios and Polychronopoulos, Dimitris and others , journal=

    Tsatsaronis, George and Balikas, Georgios and Malakasiotis, Prodromos and Partalas, Ioannis and Zschunke, Matthias and Alvers, Michael R. and Weissenborn, Dirk and Krithara, Anastasia and Petridis, Sergios and Polychronopoulos, Dimitris and others , journal=. An Overview of the

  61. [69]

    The Probabilistic Relevance Framework:

    Robertson, Stephen and Zaragoza, Hugo , journal=. The Probabilistic Relevance Framework:

  62. [70]

    Findings of EMNLP , pages=

    Document Ranking with a Pretrained Sequence-to-Sequence Model , author=. Findings of EMNLP , pages=

  63. [71]

    Sentence-

    Reimers, Nils and Gurevych, Iryna , booktitle=. Sentence-

  64. [72]

    Advances in Neural Information Processing Systems , volume=

    Training Language Models to Follow Instructions with Human Feedback , author=. Advances in Neural Information Processing Systems , volume=

  65. [73]

    Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke , booktitle=

  66. [74]

    arXiv preprint arXiv:2207.05221 , year=

    Language Models (Mostly) Know What They Know , author=. arXiv preprint arXiv:2207.05221 , year=

    arXiv

  67. [75]

    Transactions on Machine Learning Research , year=

    Teaching Models to Express Their Uncertainty in Words , author=. Transactions on Machine Learning Research , year=

  68. [76]

    arXiv preprint arXiv:2107.07511 , year=

    A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification , author=. arXiv preprint arXiv:2107.07511 , year=

    arXiv

  69. [77]

    Psychometrika , volume=

    Note on the Sampling Error of the Difference between Correlated Proportions or Percentages , author=. Psychometrika , volume=

  70. [78]

    Scandinavian Journal of Statistics , volume=

    A Simple Sequentially Rejective Multiple Test Procedure , author=. Scandinavian Journal of Statistics , volume=

  71. [79]

    Educational and Psychological Measurement , volume=

    A Coefficient of Agreement for Nominal Scales , author=. Educational and Psychological Measurement , volume=

  72. [80]

    Content Analysis: An Introduction to Its Methodology , author=

  73. [81]

    Proceedings of ACL , pages=

    The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing , author=. Proceedings of ACL , pages=

  74. [82]

    Journal of Machine Learning Research , volume=

    Statistical Comparisons of Classifiers over Multiple Data Sets , author=. Journal of Machine Learning Research , volume=

  75. [83]

    2026 , type =

    Claude Opus 4.8 System Card , institution =. 2026 , type =

    2026

  76. [84]

    2026 , type =

    GPT-5.6 Preview System Card , institution =. 2026 , type =

    2026

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.