{"id":"216076fd-2edd-44d2-b49d-f48c441ff69d","arxiv_id":"2607.00008","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"SchemaRAG dynamically reduces large schemas via RAG for LLM information extraction, reporting up to 8.8% micro-F1 gain, 47% latency cut, and 48% token cost reduction on healthcare and e-commerce data.","lead":"SchemaRAG is a retrieval-augmented framework that dynamically prunes large output schemas for LLM-based structured information extraction from text. It targets cost, latency, and context issues in domains like healthcare and e-commerce by using schema metadata and few-shot examples.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Dynamic pruning accuracy is unverified; metadata/few-shot signal may fail to retain all task-critical fields","rationale":"The reader's weakest_assumption directly identifies the same retrieval-signal risk. Because the supplied abstract contains no pruning diagnostics, the concern remains load-bearing even after the instruction to treat the cached full text as available; the concrete test above would falsify or confirm it regardless of additional manuscript details.","tokens_in":1640,"tokens_out":337,"duration_ms":16905,"concrete_test":"On the two evaluation datasets, compute field-level pruning recall: for each test instance, obtain the set of schema fields actually used in the gold annotation, then measure what fraction of that set is retained by SchemaRAG's retrieved subset. If average recall < 0.95, re-run the end-to-end extraction with an oracle-pruned schema (perfect recall) and compare micro-F1 to the reported SchemaRAG numbers.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline performance gains (micro-F1, latency, cost) presuppose that the RAG retriever, using only schema metadata and few-shot examples, selects a pruned schema whose omitted fields are never required by the extraction task. If retrieval precision/recall is imperfect on the healthcare or e-commerce schemas, two failure modes arise: (1) dropped fields directly lower extraction F1, or (2) the reported gains are measured only on cases where pruning happened to be safe. The abstract supplies no pruning-stage metrics (e.g., field-level recall against oracle relevant fields) or ablation that isolates retrieval error from the downstream LLM extraction.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes SchemaRAG, a RAG-based framework that dynamically prunes large output schemas for LLM-driven structured information extraction by retrieving relevant schema elements using metadata and few-shot examples. It is evaluated on real-world healthcare and e-commerce datasets and reports gains of up to 8.8% micro-F1, 47% lower latency, and 48% lower token cost compared to using the full schema.","tokens_in":1752,"tokens_out":435,"duration_ms":17998,"significance":"If the pruning step reliably preserves all task-critical fields, the approach would address a practical bottleneck in large-schema extraction by reducing context length issues, cost, and latency while preserving or improving accuracy. The empirical results on domain-specific datasets indicate potential utility for production IE pipelines.","major_comments":[{"comment":"Experiments section: no field-level pruning metrics (e.g., recall of selected fields against an oracle of task-relevant fields, or precision of the RAG retriever) are reported. This is load-bearing for the central claim because the reported micro-F1, latency, and cost gains presuppose that the dynamic pruning never drops fields required by the extraction task.","section":"Experiments"},{"comment":"§4 (Evaluation): the performance tables and text supply no information on the exact baselines, dataset sizes, number of runs, statistical significance tests, or error bars for the 8.8% micro-F1, 47% latency, and 48% token-cost figures, preventing verification that the gains are attributable to SchemaRAG rather than experimental artifacts.","section":"§4"}],"minor_comments":[{"comment":"Method section: the description of how schema metadata is encoded and how the few-shot examples are used for retrieval could be accompanied by pseudocode or a formal definition of the retrieval scoring function.","section":"Method"},{"comment":"Figure 2 or equivalent: the diagram of the SchemaRAG pipeline would benefit from explicit annotation of the pruning threshold or top-k parameter.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive feedback. We respond to each major comment below, indicating where we will revise the manuscript to address the concerns raised.","responses":[{"response":"We agree that field-level pruning metrics would provide direct evidence supporting the central claim. While the end-to-end micro-F1 gains indicate that task-critical fields are retained in practice, we will add recall and precision metrics for the RAG retriever against an oracle of relevant fields (derived from ground-truth annotations) to the revised Experiments section.","revision_made":"yes","referee_comment":"[Experiments] Experiments section: no field-level pruning metrics (e.g., recall of selected fields against an oracle of task-relevant fields, or precision of the RAG retriever) are reported. This is load-bearing for the central claim because the reported micro-F1, latency, and cost gains presuppose that the dynamic pruning never drops fields required by the extraction task."},{"response":"We agree that these experimental details are necessary for reproducibility and verification. The revised Section 4 will explicitly report dataset sizes, the precise baselines compared, the number of runs, any statistical significance tests conducted, and error bars on the reported metrics.","revision_made":"yes","referee_comment":"[§4] §4 (Evaluation): the performance tables and text supply no information on the exact baselines, dataset sizes, number of runs, statistical significance tests, or error bars for the 8.8% micro-F1, 47% latency, and 48% token-cost figures, preventing verification that the gains are attributable to SchemaRAG rather than experimental artifacts."}],"tokens_in":1279,"tokens_out":366,"duration_ms":19916,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point on this paper is a RAG-based method that pulls relevant schema fields using metadata and few-shot examples, then feeds only the pruned schema to the LLM for structured extraction. It targets the real issues of token limits, latency, and lost-in-the-middle effects when schemas get big.\n\nThe work does a decent job showing the idea on actual healthcare and e-commerce data, with claims of up to 8.8% micro-F1 lift plus 47% lower latency and 48% fewer tokens. Reporting concrete numbers from real-world datasets is better than pure toy experiments, and the framing around dynamic reduction is a direct response to a common deployment headache.\n\nThe soft spot is that nothing in the abstract or available description checks whether the retriever actually keeps every field the task needs. If retrieval drops a critical field, downstream F1 should fall, yet there are no pruning-stage recall numbers, no oracle comparisons, and no ablation that separates retrieval error from the LLM step. The gains could be real only on easy cases. The evaluation is also limited to two domains with no mention of statistical tests or strong baselines, so the magnitude of the improvement is hard to judge.\n\nThis is for teams running schema-conditioned extraction at scale in IR or applied settings. The central claim is plausible and the problem is worth addressing, so the paper deserves a serious referee even if the pruning reliability needs more evidence in revision.","headline":"SchemaRAG uses RAG on metadata and examples to prune large schemas for LLM extraction and reports gains on two domains, but leaves pruning accuracy unmeasured.","tokens_in":2257,"tokens_out":364,"would_cite":false,"duration_ms":18754,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"SchemaRAG dynamically prunes large schemas for LLM information extraction using metadata and few-shot examples.","keywords":["schema pruning","information extraction","retrieval-augmented generation","large language models","structured data extraction","dynamic schema reduction","healthcare","e-commerce"],"falsifier":"A concrete test input where the pruning step omits a required field that the full schema would have captured, producing measurably lower recall on that field while the rest of the extraction remains unchanged.","tokens_in":2542,"feed_emoji":"","tokens_out":593,"duration_ms":21980,"temperature":0.7,"pith_summary":"The paper presents SchemaRAG as a retrieval-augmented framework that selects only relevant portions of a large target schema before prompting an LLM for structured extraction. This selection draws on schema metadata and any available few-shot examples to decide which fields matter for the current input text. The goal is to avoid prompt bloat that raises cost and latency, triggers lost-in-the-middle errors, or exceeds context windows. On healthcare and e-commerce datasets the method reports higher micro-F1 scores together with lower latency and token usage.","feed_headline":"Dynamic pruning lifts LLM extraction F1 while halving costs","feed_subtitle":"Retrieval from metadata and examples cuts latency 47% and tokens 48% on large-schema healthcare and e-commerce tasks.","key_machinery":"SchemaRAG, a retrieval-augmented generation framework that dynamically prunes the output schema space by retrieving relevant fields via schema metadata and few-shot examples.","core_discovery":"SchemaRAG is a retrieval-augmented generation framework that dynamically prunes the output schema space for schema-conditioned information extraction tasks by leveraging schema metadata and few-shot examples when available. Evaluated on real-world healthcare and e-commerce datasets, it achieves up to an 8.8% increase in micro-F1, a 47% reduction in latency, and a 48% reduction in token costs.","pith_inferences":["The same retrieval step could be reused to adapt schemas that evolve over time rather than remaining fixed.","Combining SchemaRAG pruning with other prompt-compression methods might compound the observed cost reductions.","Domains with sparse metadata may require additional signals beyond the current retrieval sources to maintain pruning quality."],"forward_implications":["Schemas that exceed context limits become usable for extraction without manual splitting.","Token budgets for repeated extraction jobs drop substantially in production settings.","Latency-sensitive applications can run the same extraction pipeline at higher throughput.","Accuracy gains appear when irrelevant schema sections are removed from the prompt."],"fun_headline_variants":["SchemaRAG prunes large schemas to increase micro-F1 8.8%","Reduces latency 47% with RAG on healthcare extraction tasks","Token costs drop 48% via metadata-based schema reduction","Dynamic pruning increases micro-F1 8.8% reducing costs 48%"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Schema metadata and few-shot examples supply enough signal to prune the schema accurately without dropping fields that are actually needed for the extraction task.","fun_headline_variants_meta":{"raw":{"variants":["SchemaRAG prunes large schemas to increase micro-F1 8.8%","Reduces latency 47% with RAG on healthcare extraction tasks","Token costs drop 48% via metadata-based schema reduction","Dynamic pruning increases micro-F1 8.8% reducing costs 48%"]},"model":"grok-4.3","cost_usd":0.009635,"raw_usage":{"total_tokens":4258,"prompt_tokens":593,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":96349500,"prompt_tokens_details":{"text_tokens":593,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3595,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":593,"tokens_out":70,"duration_ms":25115,"temperature":1.0,"reasoning_tokens":3595,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-02T23:52:10.655338+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A concrete test input where the pruning step omits a required field that the full schema would have captured, producing measurably lower recall on that field while the rest of the extraction remains unchanged.","supporting_citations":[],"review_version":1}