Pith. sign in

REVIEW 5 major objections 6 minor 57 references

Retrieval Augmented Generation Evaluation for Health Documents

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that careful retrieval-augmented generation can minimize most common LLM failures in health-document processing, with a full-document retrieval variant reaching 0.85 precision on yes/no questions.

desk verdict Useful benchmark and candid write-up, but the headline scores are not independent: hyperparameters and SHy were chosen on the same data used for scoring. read the letter →

arxiv 2505.04680 v1 pith:HUXSJZPK submitted 2025-05-07 cs.IR

Mario Ceresa , Lorenzo Bertolini , Valentin Comte , Nicholas Spadaro , Barbara Raffael , Brigitte Toussaint , Sergio Consoli , Amalia Muñoz Piñeiro
show 3 more authors
Alex Patak Maddalena Querci Tobias Wiesenthal
This is my paper · ORCID
classification cs.IR
keywords retrieval-augmentedgenerationlargelanguagemodelshealthdocumentsquestionansweringevaluationbenchmarkhallucinationpolicysupport
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether retrieval-augmented generation (RAG) can make large language models safe enough to process health documents and scientific papers. It builds a reference pipeline, RAGEv, with several retrieval variants, plus a benchmark suite, RAGEv-Bench, covering both automatic and manual evaluations. The central result is that a variant called SHy, which interrogates each document in the collection rather than retrieving only top-k chunks, reaches an average precision of 0.85 on binary yes/no medical questions and an average BERTScore F1 of 0.83 on long answers. The authors conclude that careful RAG implementations remove most common LLM errors in this domain, making the approach useful for first-pass policy support, but not yet consistent enough for unsupervised use.

What carries the argument

The load-bearing mechanism is a retrieval unit choice: SHy (for single hybrid) treats each document in a collection as a single unit and interrogates every document, rather than returning the top-k chunks from a global corpus. It combines full-text and vector search with reciprocal rank fusion, a formula for merging the two ranking lists, and feeds the expanded context to the LLM. This design is what allows the system to answer broad, cross-document summary questions and to miss fewer relevant passages. The companion evaluation machinery is RAGEv-Bench, a set of four datasets pairing automatic and manual scoring so that pipeline choices can be compared against both machine metrics and human expert judgment.

What would settle it

Inspect the version history of the manual question sets: if any question or gold answer was edited after viewing the system's output, or if SHy's design choices were made on those same collections, the reported usability scores are inflated and the policy-readiness conclusion collapses. Re-running the evaluation with frozen questions, a frozen pipeline, and independent annotators would show whether the scores reproduce.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the main quality bottleneck in health-document question answering is retrieval coverage, not generation. Adding any RAG component dramatically improves over a no-RAG LLM, and the SHy pipeline, which treats each document as a single source and scans all documents before composing an answer, scores highest on both short and long answers. The reported numbers are 0.85 average precision on yes/no questions and 0.83 average BERTScore F1 on long answers. This holds for text-based questions; tables, figures, subtitles, numerical values, and very long answers remain recurring failure modes. The authors therefore position RAG as a high-potential aid for day-to-day policy-support synthesis, with final verification by a domain expert always required.

Load-bearing premise

The manual usability scores stand only if the test questions and gold answers were independent of the system's outputs, and if the SHy pipeline was not tuned on the same collections used to score it.

Editorial extensions

If this is right

  • A RAG configuration that scans each document individually can answer broad, cross-document questions about a health collection without missing large parts of the relevant context.
  • Text-based questions, both yes/no and long-form, reach high scores, but questions whose answers live in tables, figures, or subheadings remain unreliable.
  • Retrieval grounding, not model choice alone, is what separates usable answers from unusable ones, since no-RAG baselines score markedly worse on the automatic benchmark.
  • Deploying the tool in policy-support workflows is promising only if every statement is verified against cited sources, because the answers are not exhaustive and the provided references can include irrelevant chunks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the manual usability scores are taken at face value, policy-support teams could use a SHy-style RAG pipeline for first-pass literature scanning, but should treat every answer as a pointer to sources rather than as a verified conclusion.
  • A cleaner test of the central claim would freeze the SHy pipeline and pre-register the manual questions and gold answers before any system output is seen; without that, tuning effects cannot be separated from genuine capability.
  • The benchmark's long-answer scoring could be extended from corpus-level semantic similarity to statement-level factuality checking, which would catch subtle factual shifts that BERTScore can miss.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents RAGEv, a proof-of-concept retrieval-augmented generation (RAG) pipeline for processing health-related documents, together with a benchmark suite (RAGEv-Bench) and a two-part evaluation. In the automatic performance evaluation (APE), a factorial design of 720 runs compares five retrieval pipelines (Vanilla, Vector Search, Full-text Search, Hybrid with Reranking, SHy, ColBERTv2) on a subset of 88 public-access PubMedQA documents, using accuracy/precision for binary yes/no answers and ROUGE/BERTScore for long answers. In the manual usability checks, domain experts scored answers on three policy-oriented collections (Horizon Research, Virtual Human Twins, Bacteriophages) using a 0–5 scale. The report's central claim is that careful RAG implementations can minimize most common LLM problems in the health domain, achieving very high scores on both short and long answers. The SHy pipeline is reported as the best, with an average precision of 0.85 on yes/no questions and an average BERTScore F1 of 0.83.

Significance. The paper's intended contribution is a practical, reproducible assessment of RAG for health document synthesis, including a new benchmark and a reference implementation. The 720-run factorial comparison is a substantial empirical effort, and the automatic PubMedQA component provides an independent, public anchor for comparing RAG against vanilla LLMs, a comparison that is not compromised by the authors' specific use cases. If the reported quantitative claims were unbiased, the paper would be a useful practical guide. However, as detailed below, the headline numbers are not reliable estimates of performance on new documents, and the manual evaluation contains several design issues that undermine the strength of the central claim. The most durable contributions are likely the release of RAGEv-Bench, the detailed description of the RAG pipeline variants, and the candid documentation of usability issues.

major comments (5)
  1. [§4.4 and §5.1] The APE evaluation uses the same 88 PubMedQA documents both to define hyperparameters and to score the final systems; Section 4.4 states that this step 'has also been used to define many of the hyperparameters of the RAG pipelines,' and Section 4.4.1 describes the evaluation set as only 88 documents. No train/validation/test split is reported. Because the SHy pipeline was selected after inspecting results on these same instances, the reported average precision of 0.85 and BERTScore F1 of 0.83 are optimistic upper bounds under selection, not unbiased estimates. This is load-bearing because the abstract and Section 6 repeat these numbers as evidence for the central claim that RAG 'minimizes most of the common problems.' The authors should either provide a proper held-out test set or clearly reframe the numbers as in-sample tuning results, and adjust the conclusions accordingly.
  2. [§4.4.2 and §5.3] The manual usability evaluation is partly circular. Section 4.4.2 says that the HR questions, 'based on the answer provided by the system, were adapted in the different phases on the test,' and Section 5.3 acknowledges that the SHy pipeline's good manual results 'might be explained by the fact that such pipeline was developed under the interaction between and the author who developed the pipeline.' The HR collection yields the highest human score (4.4), so the claim that the system is usable for policy-support tasks is confounded by the system's exposure to the evaluation questions. This does not invalidate the qualitative observations, but it means the manual scores cannot be used as independent evidence of usability.
  3. [§5.2.1 and Figure 7] The manual usability analysis removes questions targeting tables, figures, and subheadings; Section 5.2.1 states that 'Figure 7 shows the main type of questions evaluated during the usability test performed in the development stages for each of the collections, after removal of those targeting tables, figures, and subheadings.' Yet the same section and Section 5.2.4 document that the system fails on exactly these question types. Excluding known failure modes from the reported scores while the abstract claims that RAG 'minimizes most of the common problems' is not justified. The claim should be qualified to text-based questions, and the removal should be disclosed prominently in the abstract and conclusions.
  4. [§5.1 and §6] The claim that the pipeline achieves 'state-of-the-art results' is not substantiated. The evaluation on PubMedQA reports precision on binary yes/no questions and BERTScore for long answers, but does not report the standard PubMedQA accuracy over the full yes/no/maybe label set, and no comparison with published PubMedQA results is provided. Without a standard metric and a comparison baseline, the 'state-of-the-art' statement in Section 6 is unsupported. The authors should report the standard three-class accuracy or clearly specify how the maybe labels were handled, and either remove or substantiate the state-of-the-art claim.
  5. [§5.1 / Figure 5] The pipeline comparison in Figure 5 shows no confidence intervals or error bars, yet the text states that SHy 'consistently score the highest results.' Given the small evaluation set (n=88) and the factorial structure, the observed differences among pipelines may be within noise. The authors should report uncertainty quantification (e.g., bootstrap CIs or standard errors) for the primary metrics to support the claim of consistent superiority.
minor comments (6)
  1. [§4.5.2] The text mentions a 'four-way ANOVA' but the experimental design is described as a 7-factor structure; it is unclear which factors were entered into the ANOVA and how the design was reduced.
  2. [§4.5.2 and §5.3] The LMM formula 'Human Score ~ Pipeline * Collection * Type of qQestion + BERTScore F1 + (Question id | 1))' contains a typo ('qQestion') and the random-effects term is likely meant to be '(1 | Question id)'. The GLRT result is reported, but no regression coefficients or effect sizes are presented, making it difficult to interpret the magnitude of the BERTScore effect.
  3. [§4.4.1] 'Altogether, the PubMedQA dataset contains up to two 200 k instances' should read 'up to 200k instances'.
  4. [§5.3] The sentence 'developed under the interaction between and the author who developed the pipeline' is missing a noun and should be completed.
  5. [General] There are numerous typos throughout, including 'preparedperformed', 'repost', 'imporant', 'corret', and 'qQestion'; a careful proofreading pass is needed.
  6. [Figure 10 caption] The caption states 'Error bars repost standard error of the mean' — 'repost' should be 'report'.

Circularity Check

3 steps flagged · score 6.0 of 10

APE hyperparameters tuned on the same 88 PubMedQA items and HR questions adapted to system answers; SHy manual scores partly reflect a development loop.

  1. fitted input called prediction [Section 4.4 (RAGEv-Bench), Section 5.1 (APE results), Section 6 (Conclusions)]
    "First, we ran an automatic evaluation based on public datasets and on classical automatic metrics. This part of the evaluation, called in subsequent sections the “Automatic performance evaluation” (APE), sets the baseline and allows comparison with existing methods and implementation on public and well-known datasets. This step has also been used to define many of the hyperparameters of the RAG pipelines that are then used in the second evaluation."

    The APE benchmark was used to define many hyperparameters of the pipelines, and the same APE set (reduced to 88 public-access PubMedQA documents in Section 4.4.1) is then used for the headline results: SHy average precision 0.85 on yes/no questions and average BERTScore F1 0.83 (Section 5.1, Figure 5). No train/validation/test split is described. The reported numbers are therefore in-sample evaluations of configurations fitted to the same instances, not out-of-sample predictions; Section 6 repeats the same numbers as evidence that RAG 'minimizes most of the common problems.' The predicted performance is statistically forced by the tuning procedure.

  2. self definitional [Section 4.4.2, Horizon research (HR)]
    "For this, 10 questions ranging from very general questions, that is covering multiple documents, to very specific ones together with the ground truth answer, were prepared. The questions, based on the answer provided by the system, were adapted in the different phases on the test."

    The HR evaluation questions, and the ground-truth answers used for scoring, were adapted after seeing the answer provided by the system. The gold standard is therefore co-defined by the system's output: the manual score measures agreement with a target that was in part derived from the very system being evaluated. This directly supports the usability conclusion, since HR is the collection where the system obtained its highest manual score (average 4.4 in Section 5.3).

1 more flagged steps
  1. other [Section 5.3 (Quantitative analysis), Section 5.2.2 (SHy development), Section 6 (Conclusions)]
    "This might be explained by the fact that such pipeline was developed under the interaction between and the author who developed the pipeline."

    The paper itself explains SHy's unusually high human-evaluation/ BERTScore alignment on the HR collection by the fact that SHy was developed in interaction with the author who developed it. SHy was created in response to issues found during the use-case tests (Section 5.2.2: 'To this end, the SHy pipeline was developed'), and it was then evaluated on the same collections. The manual scores for SHy therefore reflect an iterative development loop in which the evaluation signal was used to modify the system. Section 6 nevertheless selects SHy as 'the best pipeline' using these coupled results, so the selection is not based on an independent comparison.

full rationale

The central claim—that careful RAG implementations achieve very high scores on yes/no and long answers—is supported by two evidence threads. The automatic thread (PubMedQA/APE) rests on an externally grounded public benchmark, so the conclusion that RAG outperforms vanilla LLMs is not circular in the self-citation sense. However, its quantitative headline numbers (SHy: 0.85 average precision, 0.83 BERTScore F1) are weakened because Section 4.4 states APE 'has also been used to define many of the hyperparameters' and no held-out split is described; the reported scores are in-sample estimates rather than predictions on new data. The manual thread is more directly circular: Section 4.4.2 says the HR questions were adapted based on the system's answers, so gold-standard agreement is partly a construction; Section 5.3 explicitly notes that SHy's good manual results on HR 'might be explained by the fact that such pipeline was developed under the interaction between and the author who developed the pipeline.' These admissions appear in the paper's own text. No load-bearing self-citation chain or imported uniqueness theorem is present. Overall, the derivation is not equivalent to its inputs, but a material part of the reported evidence is fitted to or co-defined with the evaluation data, giving partial circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities are postulated. The central claims rest on standard RAG components, hand-set hyperparameters, and the reliability of small human-scored evaluation sets. The benchmark and SHy pipeline are software artifacts, not invented entities.

free parameters (4)
  • retrieval top-k = not reported (top 3 or 5 implied)
    The number of chunks fed to the LLM is hand-set and affects completeness; SHy was introduced because top-k misses relevant content (Section 5.2.2).
  • chunk size and chunking strategy = not reported
    Chunking splits tables and subtitles, which the paper identifies as a cause of failure on table/figure questions (Section 5.2.1).
  • embedding model choice = selected from MTEB leaderboard (e.g. E5-Mistral, SFR, NV-Embed)
    The encoder is chosen by leaderboard score, not derived; results depend on this choice (Section 4.2.2).
  • prompt template = not reported
    Prompt wording is described as crucial to answer quality (Section 5.2.3), but exact prompts are not included in the report.
assumptions (4)
  • domain assumption The answer to every benchmark question is present in the text chunks of the uploaded collections.
    The evaluation relies on retrieval finding the answer in text; the paper acknowledges table, figure, and subtitle content is not processed (Sections 5.2.1 and 6.1.2), so questions whose answer lives there cannot be answered.
  • domain assumption Human expert scores on the 0-5 scale are a reliable and unbiased ground truth.
    Manual usability conclusions rest entirely on scores by domain experts, with no inter-annotator reliability reported and with question wording affecting outputs (Section 5.2.3).
  • domain assumption The 88-document PubMedQA subset is representative of health-document policy workloads.
    The automatic evaluation uses only 88 public-access PubMedQA papers, and PubMedQA questions are generated from paper titles rather than from policy-style queries (Section 4.4.1).
  • domain assumption The LLM and retrieval components behave consistently enough for the factorial experiment to be meaningful.
    The paper states identical questions give identical answers for HR, but also says two replies to the same question will not necessarily be identical (Section 5.2.3); output stochasticity is not quantified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Retrieval Augmented Generation Evaluation for Health Documents." pith.science (2026). https://pith.science/paper/HUXSJZPK

@misc{pith2026250504680,
  author       = {Pith},
  title        = {Pith review of: Retrieval Augmented Generation Evaluation for Health Documents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HUXSJZPK}},
  note         = {Machine review of arXiv:2505.04680}
}
read the original abstract

Safe and trustworthy use of Large Language Models (LLM) in the processing of healthcare documents and scientific papers could substantially help clinicians, scientists and policymakers in overcoming information overload and focusing on the most relevant information at a given moment. Retrieval Augmented Generation (RAG) is a promising method to leverage the potential of LLMs while enhancing the accuracy of their outcomes. This report assesses the potentials and shortcomings of such approaches in the automatic knowledge synthesis of different types of documents in the health domain. To this end, it describes: (1) an internally developed proof of concept pipeline that employs state-of-the-art practices to deliver safe and trustable analysis for healthcare documents and scientific papers called RAGEv (Retrieval Augmented Generation Evaluation); (2) a set of evaluation tools for LLM-based document retrieval and generation; (3) a benchmark dataset to verify the accuracy and veracity of the results called RAGEv-Bench. It concludes that careful implementations of RAG techniques could minimize most of the common problems in the use of LLMs for document processing in the health domain, obtaining very high scores both on short yes/no answers and long answers. There is a high potential for incorporating it into the day-to-day work of policy support tasks, but additional efforts are required to obtain a consistent and trustworthy tool.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 9 canonical work pages

  1. [1]

    Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases,

    L. Bornmann, R. Haunschild, and R. Mutz, “Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases,” Humanit Soc Sci Commun , vol. 8, no. 1, pp. 1 –15, Oct. 2021, doi: 10.1057/s41599-021-00903-w

    doi:10.1057/s41599-021-00903-w 2021

  2. [2]

    A Survey of Large Language Models,

    W. X. Zhao et al. , “A Survey of Large Language Models,” Nov. 24, 2023, arXiv: arXiv:2303.18223. doi: 10.48550/arXiv.2303.18223

    arXiv doi:10.48550/arxiv.2303.18223 2023

  3. [3]

    Large language models in medicine,

    A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting, “Large language models in medicine,” Nat Med, vol. 29, no. 8, pp. 1930 – 1940, Aug. 2023, doi: 10.1038/s41591-023-02448-8

    doi:10.1038/s41591-023-02448-8 1930

  4. [4]

    A Comprehensive Overview of Large Language Models,

    H. Naveed et al., “A Comprehensive Overview of Large Language Models,” Apr. 09, 2024, arXiv: arXiv:2307.06435. doi: 10.48550/arXiv.2307.06435

    arXiv doi:10.48550/arxiv.2307.06435 2024

  5. [5]

    A Survey of Transformers,

    T. Lin, Y. Wang, X. Liu, and X. Qiu, “A Survey of Transformers,” Jun. 2021, [Online]. Available: http://arxiv.org/abs/2106.04554

    arXiv 2021

  6. [6]

    Attention Is All You Need,

    A. Vaswani et al., “Attention Is All You Need,” Dec. 05, 2017, arXiv: arXiv:1706.03762. Accessed: Aug. 12, 2022. [Online]. Available: http://arxiv.org/abs/1706.03762

    arXiv 2017

  7. [7]

    Language Models are Few -Shot Learners,

    T. B. Brown et al., “Language Models are Few -Shot Learners,” Jul. 22, 2020, arXiv: arXiv:2005.14165. Accessed: Nov. 23, 2023. [Online]. Available: http://arxiv.org/abs/2005.14165

    arXiv 2020

  8. [8]

    A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly,

    Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang, “A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly,” High-Confidence Computing, vol. 4, no. 2, p. 100211, Jun. 2024, doi: 10.1016/j.hcc.2024.100211

    arXiv 2024

Show all 57 references
  1. [9]

    Analyzing Leakage of Personally Identifiable Information in Language Models,

    N. Lukas, A. Salem, R. Sim, S. Tople, L. Wutschitz, and S. Zanella-Béguelin, “Analyzing Leakage of Personally Identifiable Information in Language Models,” in 2023 IEEE Symposium on Security and Privacy (SP) , May 2023, pp. 346 –363. doi: 10.1109/SP46215.2023.10179300

    2023

  2. [10]

    Are Large Pre -Trained Language Models Leaking Your Personal Information?,

    J. Huang, H. Shao, and K. C. -C. Chang, “Are Large Pre -Trained Language Models Leaking Your Personal Information?,” Oct. 20, 2022, arXiv: arXiv:2205.12628. doi: 10.48550/arXiv.2205.12628

    2022 doi arXiv

  3. [11]

    Bias and Fairness in Lar ge Language Models: A Survey,

    I. O. Gallegos et al., “Bias and Fairness in Lar ge Language Models: A Survey,” Computational Linguistics, pp. 1–79, Jun. 2024, doi: 10.1162/coli_a_00524

    2024 doi

  4. [12]

    Explainability for Large Language Models: A Survey,

    H. Zhao et al., “Explainability for Large Language Models: A Survey,” ACM Trans. Intell. Syst. Technol. , vol. 15, no. 2, p. 20:1 -20:38, Feb. 202 4, doi: 10.1145/3639372

    doi

  5. [13]

    Rethinking Interpretability in the Era of Large Language Models,

    C. Singh, J. P. Inala, M. Galley, R. Caruana, and J. Gao, “Rethinking Interpretability in the Era of Large Language Models,” Jan. 30, 2024, arXiv: arXiv:2402.01761. doi: 10.48550/arXiv.2402.01761

    2024 doi arXiv

  6. [14]

    Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models,

    Y. Zhang et al., “Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models,” Sep. 24, 2023, arXiv: arXiv:2309.01219. doi: 10.48550/arXiv.2309.01219

    2023 doi arXiv

  7. [15]

    RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models,

    S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, “RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models,” Sep. 25, 2020, arXiv: arXiv:2009.11462. doi: 10.48550/arXiv.2009.11462

    2020 doi arXiv

  8. [16]

    The long but necessary road to responsible use of large language models in healthcare research,

    J. C. C. Kwong, S. C. Y. Wang, G. C. Nickel, G. E. Cacciamani, and J. C. Kvedar, “The long but necessary road to responsible use of large language models in healthcare research,” npj Digit. Med., vol. 7, no. 1, pp. 1–3, Jul. 2024, doi: 10.1038/s41746-024- 01180-y

    2024 doi

  9. [17]

    Communication-Efficient Learning of Deep Networks f rom Decentralized Data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-Efficient Learning of Deep Networks f rom Decentralized Data,” in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, PMLR, Apr. 2017, pp. 1273 –1282. Accesse...

    2017

  10. [18]

    Learning Differentially Private Recurrent Language Models,

    H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang, “Learning Differentially Private Recurrent Language Models,” Feb. 23, 2018, arXiv: arXiv:1710.06963. doi: 10.48550/arXiv.1710.06963. 37

    2018 doi arXiv

  11. [19]

    The Algorithmic Foundations of Differential P rivacy,

    C. Dwork and A. Roth, “The Algorithmic Foundations of Differential P rivacy,” TCS, vol. 9, no. 3–4, pp. 211–407, Aug. 2014, doi: 10.1561/0400000042

    2014 doi

  12. [20]

    A Survey on Bias and Fairness in Machine Learning,

    N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, “A Survey on Bias and Fairness in Machine Learning,” ACM Comput. Surv., vol. 54, no. 6, p. 115:1- 115:35, Jul. 2021, doi: 10.1145/3457607

    2021 doi

  13. [21]

    ‘Why Should I Trust You?’: Explaining the Predictions of Any Classifier,

    M. T. Ribeiro, S. Singh, and C. Guestrin, “‘Why Should I Trust You?’: Explaining the Predictions of Any Classifier,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, in KDD ’16. New York, NY, USA: Association for Computing ...

    2016

  14. [22]

    Model-Agnostic Interpretability of Machine Learning,

    M. T. Ribeiro, S. Singh, and C. Guestrin, “Model-Agnostic Interpretability of Machine Learning,” arXiv.org. Accessed: Jul. 07, 2024. [Online]. Available: https://arxiv.org/abs/1606.05386v1

    2024 arXiv

  15. [23]

    A Unified Approach to Interpreting Model Predictions,

    S. M. Lundberg and S. -I. Lee, “A Unified Approach to Interpreting Model Predictions,” in Advances in Neural Information Processing Systems , Curran Associates, Inc., 2017. Accessed: Jul. 07, 2024. [Online]. Available: https://proceedings.neurips.cc/paper/2017/hash/8a20a862197...

    2017

  16. [24]

    Retrieval -Augmented Generation for Knowledge -Intensive NLP Tasks,

    P. Lewis et al. , “Retrieval -Augmented Generation for Knowledge -Intensive NLP Tasks,” in Advances in Neural Information Processing Systems , Curran Associates, Inc., 2020, pp. 9459 –9474. Accessed: Jul. 07, 2024. [Online]. Available: https://proceedings.neurips.cc/paper/2020...

    2020

  17. [25]

    Rethinking Machine Unlearning for Large Language Models,

    S. Liu et al., “Rethinking Machine Unlearning for Large Language Models,” Feb. 13, 2024, arXiv: arXiv:2402.08787. doi: 10.48550/arXiv.2402.08787

    2024 doi arXiv

  18. [26]

    Llama Guard: LLM -based Input-Output Safeguard for Human -AI Conversations,

    H. Inan et al., “Llama Guard: LLM -based Input-Output Safeguard for Human -AI Conversations,” Dec. 07, 2023, arXiv: arXiv:2312.06674. doi: 10.48550/arXiv.2312.06674

    2023 doi arXiv

  19. [27]

    What In -Context Learning ‘Learns’ In - Context: Disentangling Task Recognition and Task Learning,

    J. Pan, T. Gao, H. Chen, and D. Chen, “What In -Context Learning ‘Learns’ In - Context: Disentangling Task Recognition and Task Learning,” May 16, 2023, arXiv: arXiv:2305.09731. Accessed: Nov. 17, 2023. [Online]. Available: http://arxiv.org/abs/2305.09731

    2023 arXiv

  20. [28]

    Few -shot Fine- tuning vs. In -context Learning: A Fair Comparison and Evaluation,

    M. Mosbach, T. Pimentel, S. Ravfogel, D. Klakow, and Y. Elazar, “Few -shot Fine- tuning vs. In -context Learning: A Fair Comparison and Evaluation,” May 30, 2023, arXiv: arXiv:2305.16938. doi: 10.48550/arXiv.2305.16938

    2023 doi arXiv

  21. [29]

    Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning,

    H. Liu et al., “Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning,” Advances in Neural Information Processing Systems , vol. 35, pp. 1950–1965, Dec. 2022

    1950

  22. [30]

    Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break- Even Performance,

    B. Pecher, I. Srba, and M. Bielikova, “Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break- Even Performance,” Apr. 26, 2024, arXiv: arXiv:2402.12819. doi: 10.48550/arXiv.2402.12819

    2024 doi

  23. [31]

    LongLoRA: Efficient Fine -tuning of Long -Context Large Language Models,

    Y. Chen et al., “LongLoRA: Efficient Fine -tuning of Long -Context Large Language Models,” Mar. 08, 2024, arXiv: arXiv:2309.12307. doi: 10.48550/arXiv.2309.12307

    2024 doi arXiv

  24. [32]

    Retrieval -Augmented Generation for Large Language Models: A Survey,

    Y. Gao et al. , “Retrieval -Augmented Generation for Large Language Models: A Survey,” Mar. 27, 2024, arXiv: arXiv:2312.10997. doi: 10.48550/arXiv.2312.10997

    2024 doi arXiv

  25. [33]

    Retrieval -Augmented Generation for AI -Generated Content: A Survey,

    P. Zhao et al. , “Retrieval -Augmented Generation for AI -Generated Content: A Survey,” Jun. 21, 2024, arXiv: arXiv:2402.19473. doi: 10.48550/arXiv.2402.19473

    2024 doi arXiv

  26. [34]

    Evaluating the Ideal Chunk Size for a RAG System using LlamaIndex — LlamaIndex, Data Framework for LLM Applications

    “Evaluating the Ideal Chunk Size for a RAG System using LlamaIndex — LlamaIndex, Data Framework for LLM Applications.” Accessed: Jul. 08, 2024. [Online]. Available: https://www.llamaindex.ai/blog/evaluating -the-ideal-chunk-size-for-a-rag- system-using-llamaindex-6207e5d3fec5

    2024

  27. [35]

    Recursively split by character | 🦜️🦜️ LangChain

    “Recursively split by character | 🦜️🦜️ LangChain.” Accessed: Jul. 08, 2024. [Online]. Available: https://python.langchain.com/v0.1/docs/modules/data_connection/document_transf ormers/recursive_text_splitter/ 38

    2024

  28. [36]

    Knowledge Graph Prompting for Multi -Document Question Answering,

    Y. Wang, N. Lipka, R. A. Rossi, A. Siu, R. Zhang, and T. Derr, “Knowledge Graph Prompting for Multi -Document Question Answering,” Dec. 25, 2023, arXiv: arXiv:2308.11730. doi: 10.48550/arXiv.2308.11730

    2023 doi arXiv

  29. [37]

    Distributed Representations of Words and Phrases and their Compositionality,

    T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed Representations of Words and Phrases and their Compositionality,” in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2013. Accessed: Jul. 07, 2024. [Onl ine]. Available: htt...

    2013

  30. [38]

    BERT: Pre -training of Deep Bidirectional Transformers for Language Understanding,

    J. Devlin, M. -W. Chang, K. Lee, and K. Toutanova, “BERT: Pre -training of Deep Bidirectional Transformers for Language Understanding,” May 2 4, 2019, arXiv: arXiv:1810.04805. doi: 10.48550/arXiv.1810.04805

    2019 doi arXiv

  31. [39]

    ChatQA: Surpassing GPT-4 on Conversational QA and RAG,

    Z. Liu et al., “ChatQA: Surpassing GPT-4 on Conversational QA and RAG,” May 22, 2024, arXiv: arXiv:2401.10225. doi: 10.48550/arXiv.2401.10225

    2024 doi arXiv

  32. [40]

    REALM: Retrieval - Augmented Language Model Pre -Training,

    K. Guu, K. Lee, Z. Tung, P. Pasupat , and M. -W. Chang, “REALM: Retrieval - Augmented Language Model Pre -Training,” Feb. 10, 2020, arXiv: arXiv:2002.08909. doi: 10.48550/arXiv.2002.08909

    2020 doi arXiv

  33. [41]

    REPLUG: Retrieval -Augmented Black-Box Language Models,

    W. Shi et al., “REPLUG: Retrieval -Augmented Black-Box Language Models,” May 24, 2023, arXiv: arXiv:2301.12652. doi: 10.48550/arXiv.2301.12652

    2023 doi arXiv

  34. [42]

    InstructRetro: Instruction Tuning post Retrieval -Augmented Pretraining,

    B. Wang et al. , “InstructRetro: Instruction Tuning post Retrieval -Augmented Pretraining,” May 29, 2024, arXiv: arXiv:2310.07713. doi: 10.48550/arXiv.2310.07713

    2024 doi arXiv

  35. [43]

    Efficient Estimation of Word Representations in Vector Space,

    T. Mikolov, K. Chen, G. Corrado, and J. D ean, “Efficient Estimation of Word Representations in Vector Space,” Sep. 06, 2013, arXiv: arXiv:1301.3781. doi: 10.48550/arXiv.1301.3781

    2013 doi arXiv

  36. [44]

    GloVe: Global Vectors for Word Representation,

    J. Pennington, R. Socher, and C. Manning, “GloVe: Global Vectors for Word Representation,” in Proceedings of th e 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), A. Moschitti, B. Pang, and W. Daelemans, Eds., Doha, Qatar: Association for Computatio...

    2014 doi

  37. [45]

    Sentence -BERT: Sentence Embeddings using Siamese BERT -Networks,

    N. Reimers and I. Gurevych, “Sentence -BERT: Sentence Embeddings using Siamese BERT -Networks,” Aug. 2019, [Online]. Available: http://arxiv.org/abs/1908.10084

    2019 arXiv

  38. [46]

    [2401.00368] Improving Text Embeddings with Large Language Models

    “[2401.00368] Improving Text Embeddings with Large Language Models.” Accessed: Jul. 09, 2024. [Online]. Available: https://arxiv.org/abs/2401.00368

    2024 arXiv

  39. [47]

    Mistral 7B,

    A. Q. Jiang et al., “Mistral 7B,” Oct. 10, 2023, arXiv: arXiv:2310.06825. Accessed: Oct. 25, 2023. [Online]. Available: http://arxiv.org/abs/2310.06825

    2023 arXiv

  40. [48]

    MTEB: Massive Text Embedding Benchmark,

    N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, “MTEB: Massive Text Embedding Benchmark,” Mar. 19, 2023, arXiv: arXiv:2210.07316. doi: 10.48550/arXiv.2210.07316

    2023 doi arXiv

  41. [49]

    LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders,

    P. BehnamGhader, V. Adlakha, M. Mosbach, D. Bahdanau, N. Chapados, and S. Reddy, “LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders,” Apr. 08, 2024, arXiv: arXiv:2404.05961. doi: 10.48550/arXiv.2404.05961

    2024 doi arXiv

  42. [50]

    SFR-Embedding-Mistral: Enhance Text Retrieval with Transfer Learning,

    “SFR-Embedding-Mistral: Enhance Text Retrieval with Transfer Learning,” Salesforce AI. Accessed: Jul. 09, 2024. [Online]. Available: https://blog.salesforceairesearch.com/sfr-embedded-mistral/

    2024

  43. [51]

    NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models,

    C. Lee et al., “NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models,” May 27, 2024, arXiv: arXiv:2405.17428. doi: 10.48550/arXiv.2405.17428

    2024 doi arXiv

  44. [52]

    ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT,

    O. Khattab and M. Zaharia, “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT,” Jun. 04, 2020, arXiv: arXiv:2004.12832. doi: 10.48550/arXiv.2004.12832

    2020 doi arXiv

  45. [53]

    ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction,

    K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia, “ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction,” Jul. 10, 2022, arXiv: arXiv:2112.01488. doi: 10.48550/arXiv.2112.01488. 39

    2022 doi arXiv

  46. [54]

    BERTScore: Evaluating Text Generation with BERT,

    T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “BERTScore: Evaluating Text Generation with BERT,” Feb. 24, 2020, arXiv: arXiv:1904.09675. doi: 10.48550/arXiv.1904.09675

    2020 doi arXiv

  47. [55]

    Benchmarking Large Language Models in Retrieval-Augmented Generation,

    J. Chen, H. Lin, X. Han, and L. Sun, “Benchmarking Large Language Models in Retrieval-Augmented Generation,” Dec. 20, 2023, arXiv: arXiv:2309.01431. Accessed: May 21, 2024. [Online]. Available: http://arxiv.org/abs/2309.01431

    2023 arXiv

  48. [56]

    MS MARCO: A Human Generated MAchine Reading COmprehension Dataset,

    P. Bajaj et al., “MS MARCO: A Human Generated MAchine Reading COmprehension Dataset,” Oct. 31, 2018, arXiv: arXiv:1611.09268. Accessed: May 21, 2024. [Online]. Available: http://arxiv.org/abs/1611.09268

    2018 arXiv

  49. [57]

    FActScore: Fine -grained Atomic Evaluation of Factual Precision in Long Form Text Generation,

    S. Min et al., “FActScore: Fine -grained Atomic Evaluation of Factual Precision in Long Form Text Generation,” Oct. 11, 2023, arXiv: arXiv:2305.14251. Accessed: May 21, 2024. [Online]. Available: http://arxiv.org/abs/2305.14251 40 41

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.