Pith. sign in

REVIEW 3 major objections 5 minor 17 references

Less Context, Same Performance: A RAG Framework for Resource-Efficient LLM-Based Clinical NLP

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Retrieval-augmented generation matches whole-text LLM classification of surgical complications while cutting token use by more than 90%.

desk verdict Useful head-to-head RAG vs whole-text comparison with credible cost savings, but the 'no sacrifice' conclusion overreaches: recall drops are real and equivalence is asserted, not tested. read the letter →

arxiv 2505.20320 v1 pith:QA6SGMFK submitted 2025-05-23 cs.CL cs.AI

Satya Narayana Cheetirala , Ganesh Raut , Dhavalkumar Patel , Fabio Sanatana , Robert Freeman , Matthew A Levin , Girish N. Nadkarni , Omar Dawkins
show 4 more authors
Reba Miller Randolph M. Steinhagen Eyal Klang Prem Timsina
This is my paper
classification cs.CLcs.AI
keywords retrieval-augmentedgenerationlargelanguagemodelsclinicalNLPsurgicalcomplicationclassificationFAISStokenefficiencylong-contextLLMstext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a retrieval-augmented generation (RAG) pipeline can classify post-operative complications as accurately as an LLM that reads the entire clinical note, and answers yes. Over 2,294 surgical patients, three LLMs (GPT4o, LLaMA, and Mistral) showed no statistically significant differences in AUROC, precision, recall, or F1 between RAG and whole-text ingestion, with DeLong test $p$-values all above $0.05$. The RAG arm used at most 4,000 retrieved words per patient instead of the complete 30-day note history, cutting token usage from roughly 172 million to 13.2 million tokens and API cost from about $430 to $33 for GPT4o, a reduction of more than 90%. If the finding holds, clinical NLP classifiers can be run at roughly a tenth of the inference cost without sacrificing average discriminative performance.

What carries the argument

The load-bearing mechanism is a vector-retrieval pipeline: each clinical note is split into 512-word chunks, converted to dense vector embeddings, and stored in a FAISS index; a targeted classification query retrieves the top $N$ chunks most semantically similar to the question, which are concatenated in their original order and kept under a 4,000-token limit before being passed to the LLM classifier. This machinery prunes irrelevant text before the model sees it, so the LLM's attention is concentrated on high-yield segments. It is what converts a long-document problem into a short-context classification problem, and it is the component whose token savings produce the reported cost reduction.

What would settle it

Run both pipelines on a cohort where every complication status is confirmed by blinded manual chart review rather than registry absence, and check whether RAG's parity with whole-text ingestion survives; if AUROC or F1 diverges on gold-standard labels, the paper's equivalence claim is an artifact of registry noise.

Watch

Extended reading notes

Core claim

The paper's central claim is that for identifying surgical complications in clinical text, most of a note is redundant: feeding only the top semantically relevant segments to an LLM preserves discriminative performance relative to feeding the whole document. In the RAG arm, notes from the prior 30 days are split into chunks of up to 512 words, embedded, indexed in FAISS, and the top chunks are concatenated in original order under a 4,000-token budget. On the same 2,294-patient cohort, GPT4o-RAG reached AUROC 0.67 and F1 0.61 versus 0.66 and 0.61 for GPT4o-Long Context; LLaMA reached 0.63 and 0.60 versus 0.63 and 0.61; Mistral reached 0.58 and 0.60 versus 0.57 and 0.61. The RAG method also improved per-patient inference speed for the open-source models (19% for Mistral, 23% for LLaMA). The authors conclude that RAG is a scalable, model-agnostic way to lower cost and latency in clinical NLP without a measurable accuracy trade-off.

Load-bearing premise

The negative cohort is defined as patients with no entry in the surgical complication registry, so any complication that was never flagged and adjudicated is silently counted as a true negative, and that labeling noise could differ systematically between long and short notes.

Editorial extensions

If this is right

  • Hospitals can run the same complication-classification task for roughly one-tenth of the API cost of whole-note ingestion, with no statistically significant change in AUROC, precision, recall, or F1.
  • The pipeline transfers across model families: parity held for a proprietary large model and two open-source models of different sizes, so the savings do not depend on one vendor's model.
  • The open-source models ran faster in RAG mode, so the approach can reduce both dollar cost and latency at scale.
  • Because the same FAISS index can be queried with different prompts, one indexed corpus could support multiple classification tasks without re-embedding the notes.
  • The authors propose extending the framework to ICD coding, radiology report classification, medication extraction, and real-time triage, where complication signals are likely concentrated in a few text regions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not ablate the retrieval step; an immediate testable extension is to compare semantic retrieval against random chunks of equal size, which would show whether the parity is caused by retrieval or simply by the LLM's robustness to shorter input.
  • Because the negative cohort is defined by absence from a registry that depends on resident flagging and manual adjudication, the comparison could be distorted if unrecorded complications cluster in long or short notes; a gold-standard chart-review cohort would test this.
  • The authors leave open whether the 4,000-token budget is task-dependent; for complications that require synthesizing evidence spread across many notes, the required budget may grow, and a scaling curve of performance versus token budget would make the trade-off explicit.
  • The equivalence of whole-text and RAG suggests that the LLM's decision is driven by local evidence rather than global narrative; if true, harder clinical questions that require cross-note reasoning may show a larger performance gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper compares two strategies for classifying post-operative complications from clinical notes: whole-text ingestion into large-context LLMs (GPT4o, LLaMA 8B, Mistral 12B) and a RAG pipeline that chunks notes, embeds them in FAISS, and retrieves a top-4,000-word (also described as 4,000-token) context. On a cohort of 2,294 patients (996 M&M-confirmed complications and 1,298 patients with no M&M registry entry), the authors report AUROC, precision, recall, F1, and PR-AUC for both strategies, along with a DeLong test for AUROC differences. They claim no statistically significant performance differences and a cost reduction of over 90%, concluding that RAG can match whole-text accuracy while cutting resource use. The paper includes a code repository and cost/time projections.

Significance. If the equivalence claim were established, the result would be practically valuable: hospitals could reduce API and compute costs by roughly an order of magnitude without measurable loss in discriminative performance. The study has notable strengths: it uses real clinical data across multiple sites within one health system, evaluates three different LLMs, reports a concrete cost analysis, and makes code available. However, the central claim is an equivalence claim, and the statistical evidence provided is incomplete: non-significant DeLong p-values for AUROC do not establish equivalence, and no tests or confidence intervals are reported for precision, recall, or F1. The observed recall drops for GPT4o (0.90 to 0.71) and LLaMA (0.86 to 0.73) are large in absolute terms and directly contradict the conclusion that there was 'no observed drop in accuracy or recall.' These issues are load-bearing because the paper's headline is that RAG preserves accuracy, not merely that AUROC is not significantly different.

major comments (3)
  1. The central claim that RAG matches whole-text performance on all reported metrics is not supported by the presented statistical analysis. The only significance test is the DeLong test applied to AUROC (Section IV-B, Table II). No p-values, confidence intervals, or non-inferiority tests are reported for precision, recall, or F1, yet the Abstract and Section VI assert that these metrics showed no statistically significant differences and that there was 'no observed drop in accuracy or recall.' The point estimates in Table I directly contradict that assertion for recall: GPT4o drops from 0.90 to 0.71 and LLaMA from 0.86 to 0.73 under RAG. Non-significance of an AUROC difference does not imply equivalence of recall, and the paper should either (a) pre-specify an equivalence/non-inferiority margin, report confidence intervals for all metrics, and conduct appropriate tests (e.g., bootstrap or DeLong-style tests for recall differences), or (b) substantially soften the conclusion to claim only comparable AUROC with a clear statement of the observed recall trade-off. As written, the headline conclusion is not established by the evidence provided.
  2. [Section III-A]
  3. [Section IV-A and Section III-B]
minor comments (5)
  1. [Abstract]
  2. [Section III-B]
  3. [Section III-C]
  4. [Section IV-A]
  5. [General]

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the RAG-vs-whole-text comparison is an independent empirical measurement against a fixed baseline; no fitted parameters or self-citation carry the central claim.

full rationale

This paper makes no derivation that reduces to its own inputs. The central result is an empirical comparison: the same 2,294 patient records are classified by a whole-text approach and by a RAG approach, and the measured AUROC, precision, recall, and F1 are reported. The RAG design constants (512-word chunks, a 4,000-token cap) are fixed hyperparameters chosen before evaluation; they are not fitted to the test set and are not renamed as predictions. The token-cost reduction is a direct consequence of the imposed context cap and is reported as an accounting calculation, not as an inferred property, so it is not circular in the sense of a fitted input being called a prediction. The references [16] and [17] include overlapping authors, but they are cited only for prompt engineering and cloud-infrastructure context; neither is load-bearing for the equivalence claim, and no uniqueness or external-support theorem is imported from them. The negative-cohort definition (no M&M registry entry) is an operational labeling rule; it may raise a correctness or bias concern, but it does not make the comparison logically circular because the whole-text and RAG arms use identical labels. The only limitation relevant to the paper's strength is statistical: only AUROC was tested with DeLong, while the abstract generalizes non-significance to precision, recall, and F1, and Table I shows large recall drops for GPT4o (0.90 to 0.71) and LLaMA (0.86 to 0.73). That is a gap between evidence and conclusion, not circularity. Accordingly, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. The central claim rests on a small set of hand-chosen retrieval constants, on the labeling quality of the M&M registry, and on a statistical inference that treats non-significance as equivalence.

free parameters (3)
  • RAG context cap = 4,000 words
    Hand-chosen limit on retrieved text; no sensitivity analysis, and the central comparison depends on this value.
  • Chunk size = 512 words
    Hand-chosen chunk size for embedding and retrieval; no sensitivity analysis.
  • Number of retrieved chunks = Top N set by the 4,000-word cap
    The number of chunks is an implicit consequence of the word cap and is not reported explicitly; it determines what the LLM sees.
assumptions (4)
  • domain assumption Top-N dense retrieval with FAISS captures the complication-relevant information in a clinical note.
    The whole RAG pipeline rests on the retrieval step selecting the segments that matter. Invoked in Section III-B2; if retrieval misses key evidence, the comparison is invalid.
  • domain assumption Patients with no M&M registry entry have no post-operative complication.
    Negative cohort definition in Section III-A. The registry depends on resident flagging, so undocumented complications are labeled negative, creating label noise.
  • ad hoc to paper A non-significant DeLong test p-value supports the claim that RAG matches whole-text performance.
    The paper concludes equivalence from non-significance (Section IV-B, Section V-A). This is a statistical logic assumption; non-significance does not establish equivalence without a pre-specified margin.
  • domain assumption The 30-day window of prior notes is sufficient to capture post-operative complications.
    Study design in Section III-B1. Complications documented outside the 30-day window or never documented would be missed by both approaches.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Less Context, Same Performance: A RAG Framework for Resource-Efficient LLM-Based Clinical NLP." pith.science (2026). https://pith.science/paper/QA6SGMFK

@misc{pith2026250520320,
  author       = {Pith},
  title        = {Pith review of: Less Context, Same Performance: A RAG Framework for Resource-Efficient LLM-Based Clinical NLP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QA6SGMFK}},
  note         = {Machine review of arXiv:2505.20320}
}
read the original abstract

Long text classification is challenging for Large Language Models (LLMs) due to token limits and high computational costs. This study explores whether a Retrieval Augmented Generation (RAG) approach using only the most relevant text segments can match the performance of processing entire clinical notes with large context LLMs. We begin by splitting clinical documents into smaller chunks, converting them into vector embeddings, and storing these in a FAISS index. We then retrieve the top 4,000 words most pertinent to the classification query and feed these consolidated segments into an LLM. We evaluated three LLMs (GPT4o, LLaMA, and Mistral) on a surgical complication identification task. Metrics such as AUC ROC, precision, recall, and F1 showed no statistically significant differences between the RAG based approach and whole-text processing (p > 0.05p > 0.05). These findings indicate that RAG can significantly reduce token usage without sacrificing classification accuracy, providing a scalable and cost effective solution for analyzing lengthy clinical documents.

Figures

Figures reproduced from arXiv: 2505.20320 by the authors.

Figure 1
Figure 1. Overview of the study cohort. We randomly selected 996 cases [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 4
Figure 4. Vector database creation using FAISS. Clinical notes are chunked, [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 2
Figure 2. Overall study design comparing a long-context approach vs. a RAG [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: RAG-based classification. The top-N chunks from the FAISS index are concatenated (under 4,000 tokens) and fed into an LLM classifier. b) Classification with RAG ( [PITH_FULL_IMAGE:figures/full_fig_p003_5.png]
Figure 6
Figure 6. Figure 6: RAG vs. Whole-Text Comparison for Three LLMs. The blue curves represent the RAG approach and the red curves represent the Whole-Text approach. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Total Cost vs. Number of Patients [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: Compute Time vs. Number of Patients. B. Implications for Clinical NLP 1) Token Efficiency and Computational Savings: By fo￾cusing on high-yield text segments, RAG conserves compu￾tational resources without degrading classification accuracy. Transformer-based models typ…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 11 canonical work pages

  1. [1]

    Big bird: Transformers for longer sequences,

    M. Zaheeret al., “Big bird: Transformers for longer sequences,” Advances in Neural Information Processing Systems, vol. 33, pp. 17283– 17297, 2020

    work page 2020

  2. [2]

    Longformer: The long- document transformer,

    I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long- document transformer,”arXiv preprint arXiv:2004.05150, 2020

    arXiv 2004

  3. [3]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,”Advances in Neural Information Processing Systems, vol. 35, pp. 16344–16359, 2022

    work page 2022

  4. [4]

    Large Language Models for Healthcare Text Classification: A Systematic Review,

    H. Sakai and S. S. Lam, “Large Language Models for Healthcare Text Classification: A Systematic Review,”arXiv preprint arXiv:2503.01159, 2025

    arXiv 2025

  5. [5]

    Almanac: Retrieval-Augmented Language Models for Clinical Medicine,

    C. Zakkaet al., “Almanac: Retrieval-Augmented Language Models for Clinical Medicine,”Res Sq, May 2023, doi: 10.21203/rs.3.rs- 2883198/v1

    doi:10.21203/rs.3.rs- 2023

  6. [6]

    Retrieval Augmented Generation Enabled Generative Pre-Trained Transformer 4 (GPT-4) Performance for Clinical Trial Screening,

    O. Unluet al., “Retrieval Augmented Generation Enabled Generative Pre-Trained Transformer 4 (GPT-4) Performance for Clinical Trial Screening,”medRxiv, Feb 2024, doi: 10.1101/2024.02.08.24302376

    work page doi:10.1101/2024.02.08.24302376 2024

  7. [7]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P. Lewiset al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,”Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020

    work page 2020

  8. [8]

    A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions,

    S. Gupta, R. Ranjan, and S. N. Singh, “A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions,”arXiv preprint arXiv:2410.12837, 2024

    arXiv 2024

Show all 17 references
  1. [9]

    Is Semantic Chunking Worth the Computa- tional Cost?,

    R. Qu, R. Tu, and F. Bao, “Is Semantic Chunking Worth the Computa- tional Cost?,”arXiv preprint arXiv:2410.13070, 2024

    2024 arXiv

  2. [10]

    A comparison of word embeddings for the biomedical natural language processing,

    Y . Wanget al., “A comparison of word embeddings for the biomedical natural language processing,”J Biomed Inform, vol. 87, pp. 12–20, Nov 2018

    2018

  3. [11]

    How to leverage large language models for automatic ICD coding,

    Y . Yoo and S. Kim, “How to leverage large language models for automatic ICD coding,”Computers in Biology and Medicine, vol. 189, p. 109971, 2025

    2025

  4. [12]

    Billion-scale similarity search with GPUs,

    J. Johnson, M. Douze, and H. J ´egou, “Billion-scale similarity search with GPUs,”IEEE Transactions on Big Data, vol. 7, no. 3, pp. 535– 547, 2019

    2019

  5. [13]

    Clinical Concept Embeddings Learned from Massive Sources of Multimodal Medical Data,

    A. L. Beamet al., “Clinical Concept Embeddings Learned from Massive Sources of Multimodal Medical Data,”Pac Symp Biocomput, vol. 25, pp. 295–306, 2020

    2020

  6. [14]

    Early Exit Strategies for Approximate k-NN Search in Dense Retrieval,

    F. Busolin, C. Lucchese, F. M. Nardini, S. Orlando, R. Perego, and S. Trani, “Early Exit Strategies for Approximate k-NN Search in Dense Retrieval,” inProceedings of the 33rd ACM International Conference on Information and Knowledge Management, 2024, pp. 3647–3652

    2024

  7. [15]

    Improving average ranking precision in user searches for biomedical research datasets,

    D. Teodoroet al., “Improving average ranking precision in user searches for biomedical research datasets,”Database (Oxford), 2017, doi: 10.1093/database/bax083

    2017 doi

  8. [16]

    Evaluating prompt engineering on GPT-3.5’s per- formance in USMLE-style medical calculations and clinical scenarios generated by GPT-4,

    D. Patelet al., “Evaluating prompt engineering on GPT-3.5’s per- formance in USMLE-style medical calculations and clinical scenarios generated by GPT-4,”Scientific Reports, vol. 14, p. 17341, 2024

    2024

  9. [17]

    Cloud Platforms for Developing Generative AI Solutions: A Scoping Review of Tools and Services,

    D. Patel, G. Raut, S. N. Cheetirala, G. N. Nadkarni, R. Freeman, B. S. Glicksberg, E. Klang,et al., “Cloud Platforms for Developing Generative AI Solutions: A Scoping Review of Tools and Services,”arXiv preprint arXiv:2412.06044, 2024. [Online]. doi: https://doi.org/10.48550/a...

    2024 doi arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.