The reviewed record of science sign in
Pith

arxiv: 2607.00008 · v1 · pith:EILWVUCY · submitted 2026-05-04 · cs.IR · cs.AI

SchemaRAG: Dynamic Large Schema Reduction for LLM-driven Structured Information Extraction

Sin Yu Bonnie Ho , Arlie Coles , Erik Larsson , Eric Marshall , Nathan Bodenstab , Paul Vozila This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-02 23:52 UTCgrok-4.3pith:EILWVUCYrecord.jsonopen to challenge →

classification cs.IR cs.AI
keywords schema pruninginformation extractionretrieval-augmented generationlarge language modelsstructured data extractiondynamic schema reductionhealthcaree-commerce
0
0 comments X

The pith

SchemaRAG dynamically prunes large schemas for LLM information extraction using metadata and few-shot examples.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents SchemaRAG as a retrieval-augmented framework that selects only relevant portions of a large target schema before prompting an LLM for structured extraction. This selection draws on schema metadata and any available few-shot examples to decide which fields matter for the current input text. The goal is to avoid prompt bloat that raises cost and latency, triggers lost-in-the-middle errors, or exceeds context windows. On healthcare and e-commerce datasets the method reports higher micro-F1 scores together with lower latency and token usage.

Core claim

SchemaRAG is a retrieval-augmented generation framework that dynamically prunes the output schema space for schema-conditioned information extraction tasks by leveraging schema metadata and few-shot examples when available. Evaluated on real-world healthcare and e-commerce datasets, it achieves up to an 8.8% increase in micro-F1, a 47% reduction in latency, and a 48% reduction in token costs.

What carries the argument

SchemaRAG, a retrieval-augmented generation framework that dynamically prunes the output schema space by retrieving relevant fields via schema metadata and few-shot examples.

If this is right

  • Schemas that exceed context limits become usable for extraction without manual splitting.
  • Token budgets for repeated extraction jobs drop substantially in production settings.
  • Latency-sensitive applications can run the same extraction pipeline at higher throughput.
  • Accuracy gains appear when irrelevant schema sections are removed from the prompt.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same retrieval step could be reused to adapt schemas that evolve over time rather than remaining fixed.
  • Combining SchemaRAG pruning with other prompt-compression methods might compound the observed cost reductions.
  • Domains with sparse metadata may require additional signals beyond the current retrieval sources to maintain pruning quality.

Load-bearing premise

Schema metadata and few-shot examples supply enough signal to prune the schema accurately without dropping fields that are actually needed for the extraction task.

What would settle it

A concrete test input where the pruning step omits a required field that the full schema would have captured, producing measurably lower recall on that field while the rest of the extraction remains unchanged.

Figures

Figures reproduced from arXiv: 2607.00008 by Arlie Coles, Eric Marshall, Erik Larsson, Nathan Bodenstab, Paul Vozila, Sin Yu Bonnie Ho.

Figure 1
Figure 1. Figure 1: Overview of SchemaRAG which extracts information by segmenting unstructured text and applying [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Schema reduction by SchemaRAG (with k = 5). The five closest embeddings to H(ti) are retrieved. Rows whose embeddings are found are added directly to the reduced schema, and examples whose embeddings are found have their annotated rows added to the reduced schema. The reduced schema is then converted to JSON Schema format for use in the extraction prompt, and the found examples are added as in-context lear… view at source ↗
Figure 3
Figure 3. Figure 3: Counts of unique rows that appear in each [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of example use method on SchemaRAG [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Effect of k on SchemaRAG F1 on Nursing [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
read the original abstract

Extracting structured data from unstructured text using large language models (LLMs) becomes challenging when target schemas are large and complex. In such cases, including the full schema in the prompt increases cost and latency, risks lost-in-the-middle performance degradation, and can exceed context length limits. We propose SchemaRAG, a retrieval-augmented generation (RAG) framework that dynamically prunes the output schema space for schema-conditioned information extraction tasks by leveraging schema metadata and few-shot examples when available. We evaluate SchemaRAG on real-world healthcare and e-commerce datasets. Our results show that SchemaRAG can achieve up to an 8.8% increase in micro-F1, a 47% reduction in latency, and a 48% reduction in token costs, demonstrating its practicality for large-schema extraction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes SchemaRAG, a RAG-based framework that dynamically prunes large output schemas for LLM-driven structured information extraction by retrieving relevant schema elements using metadata and few-shot examples. It is evaluated on real-world healthcare and e-commerce datasets and reports gains of up to 8.8% micro-F1, 47% lower latency, and 48% lower token cost compared to using the full schema.

Significance. If the pruning step reliably preserves all task-critical fields, the approach would address a practical bottleneck in large-schema extraction by reducing context length issues, cost, and latency while preserving or improving accuracy. The empirical results on domain-specific datasets indicate potential utility for production IE pipelines.

major comments (2)
  1. [Experiments] Experiments section: no field-level pruning metrics (e.g., recall of selected fields against an oracle of task-relevant fields, or precision of the RAG retriever) are reported. This is load-bearing for the central claim because the reported micro-F1, latency, and cost gains presuppose that the dynamic pruning never drops fields required by the extraction task.
  2. [§4] §4 (Evaluation): the performance tables and text supply no information on the exact baselines, dataset sizes, number of runs, statistical significance tests, or error bars for the 8.8% micro-F1, 47% latency, and 48% token-cost figures, preventing verification that the gains are attributable to SchemaRAG rather than experimental artifacts.
minor comments (2)
  1. [Method] Method section: the description of how schema metadata is encoded and how the few-shot examples are used for retrieval could be accompanied by pseudocode or a formal definition of the retrieval scoring function.
  2. Figure 2 or equivalent: the diagram of the SchemaRAG pipeline would benefit from explicit annotation of the pruning threshold or top-k parameter.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We respond to each major comment below, indicating where we will revise the manuscript to address the concerns raised.

read point-by-point responses

  1. Referee: [Experiments] Experiments section: no field-level pruning metrics (e.g., recall of selected fields against an oracle of task-relevant fields, or precision of the RAG retriever) are reported. This is load-bearing for the central claim because the reported micro-F1, latency, and cost gains presuppose that the dynamic pruning never drops fields required by the extraction task.

    Authors: We agree that field-level pruning metrics would provide direct evidence supporting the central claim. While the end-to-end micro-F1 gains indicate that task-critical fields are retained in practice, we will add recall and precision metrics for the RAG retriever against an oracle of relevant fields (derived from ground-truth annotations) to the revised Experiments section. revision: yes

  2. Referee: [§4] §4 (Evaluation): the performance tables and text supply no information on the exact baselines, dataset sizes, number of runs, statistical significance tests, or error bars for the 8.8% micro-F1, 47% latency, and 48% token-cost figures, preventing verification that the gains are attributable to SchemaRAG rather than experimental artifacts.

    Authors: We agree that these experimental details are necessary for reproducibility and verification. The revised Section 4 will explicitly report dataset sizes, the precise baselines compared, the number of runs, any statistical significance tests conducted, and error bars on the reported metrics. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical framework with no derivations or self-referential reductions

full rationale

The paper describes an empirical RAG-based pruning method evaluated on healthcare and e-commerce datasets, reporting measured gains in micro-F1, latency, and token cost. No equations, first-principles derivations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the provided text. All performance claims rest on external experimental outcomes rather than any step that reduces to the method's own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no technical details on free parameters, axioms, or invented entities can be extracted.

pith-pipeline@v0.9.1-grok · 5676 in / 993 out tokens · 20390 ms · 2026-07-02T23:52:10.655338+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 33 canonical work pages

  1. [1]

    2024 , url=

    Si-An Chen and Lesly Miculicich and Julian Martin Eisenschlos and Zifeng Wang and Zilong Wang and Yanfei Chen and Yasuhisa Fujii and Hsuan-Tien Lin and Chen-Yu Lee and Tomas Pfister , booktitle=. 2024 , url=

    work page 2024

  2. [2]

    Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval , pages=

    Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning , author=. Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval , pages=

    work page

  3. [3]

    InProceedings of the 30th International Conference on Intelligent User In- terfaces, IUI ’25, page 1398–1425, New York, NY , USA

    Chain-of-table: Evolving tables in the reasoning chain for table understanding , author=. arXiv preprint arXiv:2401.04398 , year=

    work page arXiv

  4. [4]

    Journal of Cleaner Production , volume =

    Yi Zou and Mengying Shi and Zhongjie Chen and Zhu Deng and Zongxiong Lei and Zihan Zeng and Shiming Yang and Hongxiang Tong and Lei Xiao and Wenwen Zhou , keywords =. Journal of Cleaner Production , volume =. 2025 , issn =. doi:https://doi.org/10.1016/j.jclepro.2024.144572 , url =

    work page doi:10.1016/j.jclepro.2024.144572 2025

  5. [5]

    Information Integration and Web Intelligence: 26th International Conference, IiWAS 2024, Bratislava, Slovak Republic, December 2–4, 2024, Proceedings, Part I , pages =

    Brinkmann, Alexander and Shraga, Roee and Bizer, Christian , title =. Information Integration and Web Intelligence: 26th International Conference, IiWAS 2024, Bratislava, Slovak Republic, December 2–4, 2024, Proceedings, Part I , pages =. 2024 , isbn =. doi:10.1007/978-3-031-78090-5_4 , abstract =

    work page doi:10.1007/978-3-031-78090-5_4 2024

  6. [6]

    and Ceder, Gerbrand and Persson, Kristin A

    Dagdelen, John and Dunn, Alexander and Lee, Sanghoon and Walker, Nicholas and Rosen, Andrew S. and Ceder, Gerbrand and Persson, Kristin A. and Jain, Anubhav , title=. Nature Communications , year=. doi:10.1038/s41467-024-45563-x , url=

    work page doi:10.1038/s41467-024-45563-x

  7. [7]

    Learning to Extract Structured Entities Using Language Models

    Wu, Haolun and Yuan, Ye and Mikaelyan, Liana and Meulemans, Alexander and Liu, Xue and Hensman, James and Mitra, Bhaskar. Learning to Extract Structured Entities Using Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.388

    work page doi:10.18653/v1/2024.emnlp-main.388 2024

  8. [8]

    Large language models are few-shot clinical information extractors

    Agrawal, Monica and Hegselmann, Stefan and Lang, Hunter and Kim, Yoon and Sontag, David. Large language models are few-shot clinical information extractors. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022. doi:10.18653/v1/2022.emnlp-main.130

    work page doi:10.18653/v1/2022.emnlp-main.130 2022

  9. [9]

    A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries

    Jiang, Min and Chen, Yukun and Liu, Mei and Rosenbloom, S Trent and Mani, Subramani and Denny, Joshua C and Xu, Hua. A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries. Journal of the American Medical Informatics Association

    work page

  10. [10]

    2023 , editor =

    Goel, Akshay and Gueta, Almog and Gilon, Omry and Liu, Chang and Erell, Sofia and Nguyen, Lan Huong and Hao, Xiaohong and Jaber, Bolous and Reddy, Shashir and Kartha, Rupesh and Steiner, Jean and Laish, Itay and Feder, Amir , booktitle =. 2023 , editor =

    work page 2023

  11. [11]

    Multimodal Joint Attribute Prediction and Value Extraction for E-commerce Product

    Zhu, Tiangang and Wang, Yue and Li, Haoran and Wu, Youzheng and He, Xiaodong and Zhou, Bowen. Multimodal Joint Attribute Prediction and Value Extraction for E-commerce Product. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. doi:10.18653/v1/2020.emnlp-main.166

    work page doi:10.18653/v1/2020.emnlp-main.166 2020

  12. [12]

    2025 , eprint=

    Extractive Schema Linking for Text-to-SQL , author=. 2025 , eprint=

    work page 2025

  13. [13]

    TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoning

    Sui, Yuan and Zou, Jiaru and Zhou, Mengyu and He, Xinyi and Du, Lun and Han, Shi and Zhang, Dongmei. TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoning. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024. doi:10.18653/v1/2024.findings-emnlp.603

    work page doi:10.18653/v1/2024.findings-emnlp.603 2024

  14. [14]

    An Inner Table Retriever for Robust Table Question Answering

    Lin, Weizhe and Blloshmi, Rexhina and Byrne, Bill and de Gispert, Adria and Iglesias, Gonzalo. An Inner Table Retriever for Robust Table Question Answering. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. doi:10.18653/v1/2023.acl-long.551

    work page doi:10.18653/v1/2023.acl-long.551 2023

  15. [15]

    and Lin, Kevin and Hewitt, John and Paranjape, Ashwin and Bevilacqua, Michele and Petroni, Fabio and Liang, Percy

    Liu, Nelson F. and Lin, Kevin and Hewitt, John and Paranjape, Ashwin and Bevilacqua, Michele and Petroni, Fabio and Liang, Percy. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics. 2024. doi:10.1162/tacl_a_00638

    work page doi:10.1162/tacl_a_00638 2024

  16. [16]

    2304.08085 , archivePrefix=

    Xiao Wang and Weikang Zhou and Can Zu and Han Xia and Tianze Chen and Yuansen Zhang and Rui Zheng and Junjie Ye and Qi Zhang and Tao Gui and Jihua Kang and Jingsheng Yang and Siyuan Li and Chunsai Du , year=. 2304.08085 , archivePrefix=

    work page arXiv

  17. [17]

    arXiv preprint arXiv:2302.10205 , year=

    Xiang Wei and Xingyu Cui and Ning Cheng and Xiaobin Wang and Xin Zhang and Shen Huang and Pengjun Xie and Jinan Xu and Yufeng Chen and Meishan Zhang and Yong Jiang and Wenjuan Han , year=. 2302.10205 , archivePrefix=

    work page arXiv

  18. [18]

    2310.12537 , archivePrefix=

    Alexander Brinkmann and Roee Shraga and Christian Bizer , year=. 2310.12537 , archivePrefix=

    work page arXiv

  19. [19]

    Advances in Neural Information Processing Systems , editor =

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K\". Advances in Neural Information Processing Systems , editor =

    work page

  20. [20]

    2024 , eprint=

    Retrieval-Augmented Generation-based Relation Extraction , author=. 2024 , eprint=

    work page 2024

  21. [21]

    IDNet: A Novel Dataset for Identity Document Analysis and Fraud Detection

    Kim, Edward and Shrestha, Manil and Foty, Richard and DeLay, Tom and Seyfert-Margolis, Vicki , booktitle =. 2024 , volume =. doi:10.1109/BigData62323.2024.10825160 , url =

    work page doi:10.1109/bigdata62323.2024.10825160 2024

  22. [22]

    Graph retrieval-augmented generation: A survey

    Boci Peng and Yun Zhu and Yongchao Liu and Xiaohe Bo and Haizhou Shi and Chuntao Hong and Yan Zhang and Siliang Tang , year=. 2408.08921 , archivePrefix=

    work page arXiv

  23. [23]

    CoRRabs/2501.08686(2025)

    Chuangtao Ma and Sriom Chakrabarti and Arijit Khan and Bálint Molnár , year=. 2501.08686 , archivePrefix=

    work page arXiv

  24. [24]

    An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaboration

    Li, Yihao and Zhang, Ru and Liu, Jianyi. An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaboration. Artificial Neural Networks and Machine Learning -- ICANN 2024. 2024

    work page 2024

  25. [25]

    Information Processing

    Yu Liu and Duantengchuan Li and Kaili Wang and Zhuoran Xiong and Fobo Shi and Jian Wang and Bing Li and Bo Hang , keywords =. Information Processing. 2024 , issn =. doi:https://doi.org/10.1016/j.ipm.2024.103809 , url =

    work page doi:10.1016/j.ipm.2024.103809 2024

  26. [26]

    Singhal, Karan and Azizi, Shekoofeh and Tu, Tao and Mahdavi, S. Sara and Wei, Jason and Chung, Hyung Won and Scales, Nathan and Tanwani, Ajay and Cole-Lewis, Heather and Pfohl, Stephen and Payne, Perry and Seneviratne, Martin and Gamble, Paul and Kelly, Chris and Babiker, Abubakr and Sch. Large language models encode clinical knowledge , journal=. 2023 , ...

    work page doi:10.1038/s41586-023-06291-2 2023

  27. [27]

    Breakthroughs in statistics: Methodology and distribution , pages=

    Individual comparisons by ranking methods , author=. Breakthroughs in statistics: Methodology and distribution , pages=. 1992 , publisher=

    work page 1992

  28. [28]

    Biometrika , volume=

    An analysis of variance test for normality (complete samples) , author=. Biometrika , volume=. 1965 , publisher=

    work page 1965

  29. [29]

    OpenAI , year=

    New and improved embedding model , author =. OpenAI , year=

    work page

  30. [30]

    GPT-4o: Large Multimodal Model , year =

    work page

  31. [31]

    2025 , howpublished =

    OpenAI , title =. 2025 , howpublished =

    work page 2025

  32. [32]

    Open. Prompt. OpenAI , year=

    work page

  33. [33]

    OpenAI , year=

    Open. OpenAI , year=

    work page