{"id":"0dde0b32-5a4c-42a0-9e63-9c34e707543b","arxiv_id":"2608.13384","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AnnoIndex reaches 0.87 average F1 on three document corpora by building an offline annotation index with induced schemas and a progressive SQL-based query engine that reuses extracted values.","lead":"AnnoIndex turns unstructured documents into a searchable structured index before answering queries, using a closed-loop schema induction step and a SQL-like execution engine that calls large language models only on the small remaining candidate set. In tests on legal, Wikipedia, and web datasets, it reached 0.87 average F1, beating retrieval-augmented and knowledge-graph baselines at lower LLM cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SchemaLoop's accept/reject gate uses 5-document samples and hand-set SR/FE thresholds; near threshold this is statistically noisy, and without seeds or sensitivity analysis the reported F1 and cost gains may not transfer.","rationale":"I read the paper's central claim as asserting that AnnoIndex is the current best accuracy and cost point for structured analytical queries over unstructured text. The table reports an average F1 of 0.87 for Performance mode and a clear cost advantage, so the claim is plausible if SchemaLoop genuinely produces extractable and discriminative schemas. The most load-bearing condition for that is the verification feedback loop: a bad schema decision propagates into missing index fields, larger EXTRACT candidate sets, and degraded F1 or cost. Section IV-B.d exposes that this decision is made from five randomly sampled documents per group with thresholds 0.6 and 0.3. At n=5, the estimation error for a binomial success rate is enormous, and the reported formulas even disagree with the described sampling procedure. No seeds, repeated runs, or threshold sweeps are reported, so the reader cannot tell whether the favorable Table I results are robust or seed-dependent. This is a concrete statistical weakness, not a stylistic objection, and it directly affects the central accuracy and cost claims. I also note an internal inconsistency: Section VI-C states both modes outperform or match every competitor on every dataset, but Table I shows Eco mode on WikiText at 0.88, below Lotus at 0.89. I did not make that the primary concern because the Performance-mode average and the cost comparisons remain as reported and a one-point gap without error bars is not decisive, but it should be corrected. The 5-document verification concern is addressable by a seed-variation and full-corpus verification experiment, so it supports a conditional verdict rather than rejection. Since the reader already assigned CONDITIONAL for related reproducibility reasons, my stress-test does not change the verdict.","tokens_in":17375,"tokens_out":9167,"duration_ms":101585,"concrete_test":"Run SchemaLoop end-to-end on WikiText with at least 20 different random seeds for choosing the 5 verification documents per group, keeping all other components fixed, and report the F1 and amortized token cost of the resulting indexes. Also recompute SR and FE on the full corpus for every accepted schema. If the F1 spread across seeds exceeds about two points, or if any accepted schema has full-corpus SR below 0.6, the 5-document gate is too noisy to support the reported gains; additionally vary the SR threshold from 0.4 to 0.8 and FE from 0.2 to 0.5 to test sensitivity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AnnoIndex is the best precision/cost point depends on SchemaLoop reliably inducing schemas that are both extractable and discriminative. Section IV-B.d accepts or rejects a candidate schema based on extraction success rate (SR) and filtering efficiency (FE) computed on a random sample of only 5 documents per group, with thresholds SR >= 0.6 and FE >= 0.3. For a true success rate near 0.6, a 5-document binomial sample has a Clopper-Pearson 95% interval of roughly [0.17, 0.93], so the accept/reject decision is dominated by sampling noise. If an unlucky sample accepts a schema whose true SR is below threshold, the materialized index will have empty values for queried attributes; the structured filter then silently drops valid documents, and the EXTRACT fallback is forced onto a much larger candidate set, raising cost and exposing LLMs to noisy contexts that degrade F1. Conversely, a usable schema can be rejected and over-refined, wasting induction effort and adding spurious fields. The paper reports no random seeds, no repeated verification runs, and no threshold sensitivity analysis, and the SR formula in Section IV-B.d is written over all |D_g| documents rather than the 5-doc sample actually used. The ablations remove whole modules but never vary the verification sample or thresholds, so the stability of this load-bearing gate is untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AnnoIndex, a system for precise analytical queries over unstructured documents. It consists of SchemaLoop, an offline closed-loop schema induction framework that derives three-layer hierarchical annotation schemas (dataset, table, document) and materializes an annotation index, and a Structured Query Engine that compiles natural-language queries into SQL-like execution plans, applies structured filtering against the index, and invokes an EXTRACT operator with progressively expensive extractors (regex, Mistral-7B, GPT-4o). Extracted values are written back into the index for reuse. The evaluation uses three datasets (LCR, WikiText, SWDE) with 500 queries each, reporting average F1 of 0.87 (Performance mode) and 0.83 (Economical mode) versus baselines such as QUEST (0.80) and a GPT-4o LLM baseline (0.73), while claiming substantially lower amortized LLM token costs. The paper argues that the offline-materialization and attribute-reuse design amortizes extraction costs and enables cheaper, precise analytic queries.","tokens_in":17674,"tokens_out":7609,"duration_ms":74970,"significance":"If the results hold up, AnnoIndex represents a useful step toward structured analytical querying over unstructured text, combining automatic schema induction with cost-aware execution. The strengths are the closed-loop verification of candidate schemas, the progressive EXTRACT strategy, and the persistence and reuse of extracted attributes, which together offer a plausible path to reducing LLM costs while preserving precision. However, the empirical evidence as presented is insufficient to support the strongest claims: Table I and Figure 7 report F1 without variance or significance tests; the schema verification gate is based on a statistically fragile five-document sample with hand-set thresholds; the efficiency comparison appears not to include baseline offline costs; and no code, query sets, or seeds are provided. These issues are load-bearing for the stated state-of-the-art conclusion and require additional experiments and reporting rather than a simple rewrite.","major_comments":[{"comment":"The verification step samples only 5 random documents per group, but the extraction success rate formula in Eq. (1) is written over the full group D_g, not the sample actually used. With n=5, a true success rate near the 0.6 threshold has a 95% Clopper-Pearson interval of roughly [0.17, 0.93], so accept/reject decisions are dominated by sampling noise. The paper reports no random seeds, no repeated verification runs, and no sensitivity analysis of the thresholds (SR≥0.6, FE≥0.3). Because schema quality underpins the index-based filtering and the claimed cost reductions, a systematic sensitivity study varying sample size and thresholds, with endpoint F1 and token cost as outputs, is needed to establish that the gate is robust.","section":"§IV-B.d, Eq. (1)"},{"comment":"Table I reports per-dataset F1 without standard deviations, confidence intervals, or significance tests. Several differences between AnnoIndex and the strongest baselines are small (e.g., WikiText 0.91 vs Lotus 0.89; SWDE 0.96 vs Lotus 0.95), and the claim that AnnoIndex \"consistently outperform[s] or match[es] all competing systems\" is not supported without evidence that these gaps exceed noise. Please report variance over repeated runs (e.g., multiple seeds or bootstrap resampling) and apply an appropriate paired significance test to the fixed query sets for both F1 and token costs.","section":"Table I, §VI-C"},{"comment":"The \"amortized LLM token consumption per query\" metric divides total offline plus online tokens by 500 queries. Although AnnoIndex's offline SchemaLoop and index population costs are included, it is not stated whether comparable offline costs for baselines (e.g., GraphRAG's graph construction, ZenDB's index/tree construction, Lotus's indexing, or QUEST's index build) are included. If these are excluded, the comparison is not apples-to-apples and the claim that AnnoIndex achieves the lowest amortized cost is unsupported. The paper should either include all systems' total (offline+online) costs or clearly justify any exclusion.","section":"§VI-D, Figure 4"},{"comment":"The abstract and contribution list state that SchemaLoop \"automatically\" creates hierarchical annotation schemas, but the dataset-level schema is explicitly predefined by humans in Section VI-B and Section IV-A says it is \"pre-defined by humans.\" This overstates the automation in the central contribution. Please either automate or generate this layer as well, or revise the claims to state precisely that table-level and document-level schemas are automatically induced given a manually specified dataset-level root schema.","section":"Abstract, §I Contributions, §IV-A, §VI-B"},{"comment":"Ground truth is initialized by GPT-4o extractions and then student-corrected, while Performance-mode EXTRACT operations also use GPT-4o. This creates a circularity-adjacent risk: the gold standard may be biased toward what GPT-4o can extract, favoring systems that use GPT-4o as their backend. Although students corrected and supplemented the initial output, the paper provides no inter-annotator agreement statistics and no measure of how much the gold standard differs from the raw GPT-4o output. Please report these quantities, and ideally include a gold standard constructed independently of GPT-4o (e.g., existing structured metadata or expert-built annotations) as a robustness check.","section":"§VI-A.b, §VI-B"},{"comment":"No code, datasets, query sets, or random seeds are made available, which severely limits the reproducibility of a systems paper. The 500-query sets per dataset, the sampling procedure for verification documents, and the seed/settings for LLM calls are essential to independently verify the F1 and cost numbers. The authors should release the implementation and data artifacts, or at a minimum provide detailed query sets and seed settings in an appendix.","section":"§VI (Experimental Setup)"}],"minor_comments":[{"comment":"The text says verification uses a random subset of 5 documents, but the SR and FE formulas are defined over all D_g. Please align the notation with the actual sample size used in the verification procedure.","section":"§IV-B.d, Eqs. (1)-(2)"},{"comment":"The term PR(S_row) appears without definition; presumably it is the filtering efficiency or precision of the document-level schema, but it is never introduced. Define the notation or replace it with the appropriate previously defined quantity.","section":"§IV-C, Eq. (3)"},{"comment":"There is a typo in the VectorDB+RAG description: \"We implement this into a RAG pipelin.\" should be a complete sentence. Also, Section I contains \"More worse,\" which should be \"What is worse\" or \"More importantly.\"","section":"§VI-A.c"},{"comment":"The system name is inconsistently spelled: \"AnnolIndex\" in Section IV-D and \"AnnoRe\" in Section VI-G should be corrected to \"AnnoIndex\" throughout.","section":"§IV-D, §VI-G"},{"comment":"The text states that Lotus has \"larger F1 variance\" on cross-table joins and aggregation, but no variance numbers are reported anywhere. Either provide quantitative variance data or soften this to a qualitative observation.","section":"§VI-C.a"},{"comment":"The token counts for the join query example (58.2K vs 44.5K) are reported without specifying the counting convention (input plus output tokens? prompt length only?). Please define the token accounting used for all efficiency figures.","section":"§VI-F"}],"recommendation":"major_revision","confidential_remarks":"The manuscript cites a large number of the first author's own prior works, including several arXiv preprints; this is not inherently problematic, but the editor may wish to check that the related-work discussion gives adequate credit to the broader community's systems. The main technical concerns are the statistically fragile five-document verification gate and the absence of significance testing, variance reporting, and reproducibility artifacts; these are fixable but require additional experiments rather than only text revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a worthwhile systems paper with a genuinely useful architecture, but the evaluation has a few holes that need patching before the headline numbers can be trusted.\n\nWhat's actually new: SchemaLoop's closed-loop schema induction with SLM-verified extractability and the virtual-field promotion mechanism aren't in QUEST, ZenDB, AutoSchemaKG, or SQUiD. The progressive cost-ordered execution with write-back of EXTRACT results is a nice practical idea that turns the index into a learning structure. The ablations are well designed and support the claim that both components matter.\n\nWhat it does well: the problem framing is clear, the three-layer schema hierarchy is a sensible way to exploit corpus structure, and the efficiency analysis (amortized tokens, EXTRACT invocations over batches) directly addresses the cost concerns that dominate this subfield. The writing is honest: the paper explicitly says the external feedback loop is future work and doesn't oversell the multi-loop engineering.\n\nWhere the soft spots are: no code or query sets are released, and Table I reports F1 without variance or significance tests. That's a real problem for a paper whose contribution is an architecture; a serious referee should demand artifacts. The dataset-level schema is pre-defined by humans while the abstract says 'automatically create hierarchical annotation schemas'; that's an overstatement that should be fixed. Ground truth starts from GPT-4o extractions before human correction, which reduces but doesn't eliminate model bias.\n\nThe stress-test concern about the verification gate is on point: SchemaLoop accepts or rejects candidate schemas based on SR and FE computed on a random sample of five documents per group, with hand-set thresholds (0.6 and 0.3). Near threshold, a five-document binomial sample is pure noise, and the paper reports no seeds or sensitivity analysis. Worse, the SR formula is written over |D_g|, not the 5-doc sample actually used—that's a discrepancy that needs explanation. I wouldn't call this fatal; the gate can be patched with a larger sample or a Bayesian approach, and the ablations already show the closed loop helps overall. But it's a load-bearing operational assumption that's untested, and the paper should include a sensitivity analysis varying sample size and thresholds.\n\nWho this is for: researchers and engineers building LLM-powered data systems, specifically the DB/IR community working on structured extraction over text. They'll get value from the architecture and the cost model. The paper deserves a serious referee, but only with a request for major revision: release artifacts, add error bars, fix the automatic claim, and test the verification gate.\n\nNet: I'd send it to peer review, but I'd read the revised version carefully before accepting.","headline":"Solid architecture and a genuinely useful cost model, but the evaluation needs artifacts and a sensitivity analysis before the headline F1 claims are trustworthy.","tokens_in":18191,"tokens_out":2770,"would_cite":true,"duration_ms":27536,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AnnoIndex claims that pre-building a hierarchical annotation index over documents and then executing SQL-like extraction plans yields precise analytical queries at higher F1 and lower LLM cost than prior retrieval-based systems.","keywords":["unstructured document analysis","annotation index","schema induction","structured query engine","LLM cost amortization","attribute reuse","EXTRACT operator","analytical queries"],"falsifier":"Run AnnoIndex on the WikiText corpus while withholding from the schema-verification sample every document containing a queried attribute that occurs in fewer than 5% of documents, then measure recall on the join and progressive-reasoning queries that reference that attribute; a large recall drop would show that the 5-document verification is the cause rather than the extraction model itself.","tokens_in":17175,"feed_emoji":"🗂️","tokens_out":5825,"duration_ms":59934,"temperature":0.7,"pith_summary":"AnnoIndex asks whether unstructured document collections can be queried as precisely as relational databases, and argues that they can if you first materialize an annotation index over the corpus instead of relying on vector similarity. The system's SchemaLoop automatically induces a three-layer schema (dataset, table, document), uses a lightweight extractor to populate field values once offline, and then answers queries with a SQL-like plan that filters on indexed fields first and applies extraction only to the dwindling remainder. The paper reports an average F1 of 0.87 across three real-world datasets, beating the strongest prior cost-optimized system by 7 points and a direct LLM prompt baseline by 14 points, while using fewer amortized tokens per query. If this holds, the practical consequence is that precise analytical workloads over text—range filters, joins, progressive reasoning—can be served without paying LLM cost on every document for every query.","feed_headline":"Offline index beats online LLM retrieval by 14 F1 points","feed_subtitle":"Turning text into a structured index first lets precise filters run without LLM calls; LLMs handle only the leftovers.","key_machinery":"The two load-bearing mechanisms are the Annotation Index, a materialized store of per-document attribute values governed by a three-layer schema, and the EXTRACT operator, a user-defined predicate in a SQL-like execution plan that can be backed by regex, a lightweight model, or an LLM. SchemaLoop produces the schema by a hypothesize-verify-refine loop: it groups documents by upper-layer annotations, generates candidate schemas with an LLM, verifies them by extracting from a random sample of 5 documents per group using a lightweight model, and keeps a schema only if extraction success is at least 0.6 and filtering efficiency at least 0.3, where filtering efficiency is one minus the maximum fraction of documents sharing the same value assignment. The query engine then evaluates predicates progressively: index filters first, then extraction in ascending cost, with short-circuiting and join-to-filter rewriting, and it persists newly extracted values back into the index for reuse.","core_discovery":"The central discovery is that the two obstacles named in the paper—fuzzy information location and unreliable algebraic reasoning—can be overcome by decoupling extraction from querying. AnnoIndex precomputes a hierarchical, queryable annotation index whose schemas are discovered automatically from the raw corpus, so that attribute-level filters such as birth year before 1985 become plain numerical comparisons that require no LLM call. During querying, the engine compiles the user question into a SQL-extended plan, filters using the index first, and then runs extraction in ascending order of cost—regex, then a lightweight model, then an LLM—calling the expensive model only on the minimal set of documents that survive the earlier filters. Extracted values are written back into the index, so repeated queries become cheaper. The paper claims this design yields state-of-the-art accuracy, with average F1 0.87, while consuming roughly 11 times fewer LLM tokens than the direct LLM baseline.","pith_inferences":["The authors leave implicit that the schema verification thresholds themselves could be learned from the query workload rather than fixed in advance; adapting acceptance criteria to observed query attributes would likely improve recall on rare-field questions.","If the cost-ascending extraction principle transfers beyond text, the same materialize-cheap-labels-first architecture could be applied to image or multimodal collections, provided a schema induction loop can define per-domain attributes.","The paper's amortized token comparison credits the offline build over a fixed 500-query evaluation; a deployment planner would need the break-even point—the workload size at which the one-time indexing cost is recovered—which the paper does not isolate as a single number."],"forward_implications":["Attribute extraction cost shifts from per-query to one-time build, so workloads with many repeated or related queries see marginal LLM cost approach zero.","Queries expressible as index filters, such as equality or range conditions, need no LLM at all, making high-throughput interactive analytics over text feasible.","Deep semantic predicates, such as determining whether a legal provision conflicts with superior law, remain answerable because the engine reserves LLM calls for the small candidate set that survives structured filtering.","Cross-table joins and multi-step reasoning become executable as SQL-like plans rather than opaque LLM reasoning, a capability the paper reports as a 0.86 F1 on three-way join queries.","Attribute reuse makes the system continuously cheaper: a virtual field promoted to the schema after repeated EXTRACT usage reduces the cost of future queries that reference it."],"supporting_citations":[{"why":"Supplies the strongest competing cost-optimized system, the instance-optimized ordering idea the query engine adapts, and the WikiText dataset construction.","marker":"[6]"},{"why":"Provides a declarative AI-powered analytics baseline whose fixed schema the paper uses to show the limitation on heterogeneous corpora.","marker":"[4]"},{"why":"Provides a hierarchical extraction baseline for templatized document collections that AnnoIndex compares against on precision and cost.","marker":"[5]"},{"why":"Provides a semantic-operator baseline that feeds full documents to an LLM, representing the high-cost accuracy comparison.","marker":"[37]"},{"why":"Provides the knowledge-graph-based baseline that the paper contrasts with algebraic query execution over a structured index.","marker":"[18]"},{"why":"Supplies the legal corpus used to test long-document progressive reasoning and conflict-of-law queries.","marker":"[33]"},{"why":"Supplies the web-extraction dataset used to test template-like noisy document pages.","marker":"[34]"},{"why":"Provides the direct LLM prompt baseline and the large language model used across the compared systems.","marker":"[39]"},{"why":"Supplies the lightweight extraction model used for offline annotation and economical-mode extraction.","marker":"[40]"}],"fun_headline_variants":["Precompute an index, then let LLMs handle only the leftovers","Structured index cuts LLM calls to a minimum, 11x fewer tokens","Filter first with a prebuilt annotation index, then ask the LLM","AnnoIndex: turn text into a queryable index, LLM only on leftovers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing operational premise is that a schema accepted on the evidence of a random sample of only 5 documents per group, with success-rate threshold 0.6 and filtering-efficiency threshold 0.3, will still extract the attributes users actually query across the full corpus; if the sample misses rare but queried attributes, the index filters out the right documents and the reported accuracy and cost gains do not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Precompute an index, then let LLMs handle only the leftovers","Structured index cuts LLM calls to a minimum, 11x fewer tokens","Filter first with a prebuilt annotation index, then ask the LLM","AnnoIndex: turn text into a queryable index, LLM only on leftovers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00078,"raw_usage":{"total_tokens":3480,"prompt_tokens":1013,"completion_tokens":2467,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":2384}},"tokens_in":629,"tokens_out":2467,"duration_ms":18181,"temperature":1.0,"reasoning_tokens":2384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:19:47.778967+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AnnoIndex on the WikiText corpus while withholding from the schema-verification sample every document containing a queried attribute that occurs in fewer than 5% of documents, then measure recall on the join and progressive-reasoning queries that reference that attribute; a large recall drop would show that the 5-document verification is the cause rather than the extraction model itself.","supporting_citations":[{"cited_title":"Quest: Query optimization in unstructured document analysis,","cited_arxiv_id":null,"evidence_quote":"Supplies the strongest competing cost-optimized system, the instance-optimized ordering idea the query engine adapts, and the WikiText dataset construction."},{"cited_title":"Palimpzest: Optimizing ai-powered analytics with declarative query processing,","cited_arxiv_id":null,"evidence_quote":"Provides a declarative AI-powered analytics baseline whose fixed schema the paper uses to show the limitation on heterogeneous corpora."},{"cited_title":"Querying templatized document collections with large language models,","cited_arxiv_id":null,"evidence_quote":"Provides a hierarchical extraction baseline for templatized document collections that AnnoIndex compares against on precision and cost."},{"cited_title":"Lexa: Towards automatic legal citation classification,","cited_arxiv_id":null,"evidence_quote":"Supplies the legal corpus used to test long-document progressive reasoning and conflict-of-law queries."}],"review_version":1}