{"id":"8554826c-1f36-4d27-8884-e1b107142e2b","arxiv_id":"2608.10396","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FormStruct-Bench is a hierarchical benchmark showing that current table-form recognition systems reach 83.85% at document level but under 18% on fine-grained structural scores.","lead":"This paper introduces FormStruct-Bench, a benchmark for testing how well computer systems recognize the structure of tables and forms in documents. It measures performance at document, page, schema, and component levels, and finds that current systems read the content well but fail to recover the underlying hierarchy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gap between 83.85% document-level and <18% fine-grained scores may be an artifact of metric asymmetry; the abstract does not demonstrate that the two scores are commensurable measures of the same underlying capability.","rationale":"The reader's weakest assumption concerns template representativeness. I partially agree but view the more load-bearing threat as metric commensurability: the claim is a comparative statement about two score families, and without definitions or an oracle baseline, the gap could be generated by scoring strictness. A template-distribution artifact would also produce a misleading gap, but the metric-calibration concern is more directly tied to the claim's wording 'pronounced gap between reading content and recovering hierarchy.' Since only the abstract was available, neither concern can be confirmed or refuted, so I do not change the UNVERDICTED verdict. The proposed check is one concrete way to adjudicate the metric-asymmetry threat.","tokens_in":716,"tokens_out":2416,"duration_ms":25460,"concrete_test":"Inspect the released evaluation code and compute the fine-grained structural score for an 'oracle-content, structure-scrambled' baseline: take ground-truth tokens and assign them to randomly permuted structural nodes, then score with the exact protocol. If this baseline also scores below 18%, the metric is floor-limited and the gap is an artifact; if it scores well above 18%, the gap reflects genuine structural failure. Also report human performance on the same fine-grained metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that current systems recover visible content but not the hierarchy/regional organization needed for reliable table-form understanding. The evidence is a gap between a document-level score of 83.85% and a fine-grained structural score below 18%. This inference is valid only if the two score families are commensurable, i.e., both are scalar summaries of the same failure space and neither is systematically saturated or floor-limited. The abstract gives no metric definitions, no matching tolerances, no partial-credit rules, and no human upper-bound baseline. If the fine-grained metric requires exact hierarchical region-and-label matches with zero partial credit, while the document-level metric uses loose token-level recall, then even a perfect structural system could score near 18% on the former. The absence of a 'structure oracle' baseline (ground-truth content, predicted structure) or human performance on the same metric means the reported gap cannot be attributed to model deficiency rather than to scoring strictness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"FormStruct-Bench is introduced as a hierarchical benchmark for table-form document structure recognition. The authors describe a pipeline that annotates 70 reusable templates and expands them into 7,000 verified instances, with 1,100 template-disjoint test instances receiving additional human review. The protocol uses five primary metrics and three structure-specific diagnostics across page, schema, and component levels, with slices over difficulty, structural constraints, and visual degradation. The empirical study covers 14 API-hosted or locally deployable systems and two SFT variants; the best document-level score is 83.85%, while the best fine-grained structural score is below 18%, which the authors interpret as a pronounced gap between content reading and hierarchy recovery.","tokens_in":1031,"tokens_out":3941,"duration_ms":37233,"significance":"If the benchmark's claims are substantiated, FormStruct-Bench would fill a genuine gap: existing benchmarks focus on holistic output or conventional grids and do not localize structural failures. The provenance-preserving Director-Artist-Verifier pipeline and template-disjoint test set are methodical choices that mitigate annotation leakage and template memorization. The diagnostic orientation toward page, schema, and component levels is a useful contribution for failure analysis. However, the significance is conditional: the headline gap is only meaningful if the metrics are commensurable and if the template space is representative. The abstract alone does not establish these conditions, so the empirical conclusion must be regarded as preliminary.","major_comments":[{"comment":"The central claim that current systems recover visible content but not hierarchy rests on the gap between the best document-level score of 83.85% and the best fine-grained structural score below 18%. The abstract does not report any metric definitions, matching tolerances, or partial-credit rules, so readers cannot tell whether the two scores are commensurable: a fine-grained metric requiring exact hierarchical matches could be floor-limited even for a perfect structural model, whereas a loose token-level document metric would naturally be high. The absence of a human upper bound or a structure-oracle baseline makes it impossible to attribute the gap to model deficiency rather than scoring strictness. Please provide full metric definitions and at least one baseline that anchors the fine-grained scale.","section":"Abstract, results paragraph"},{"comment":"The 70 reusable templates expanded into 7,000 instances are asserted to be 'verified' and the 1,100 test instances are 'template-disjoint,' but the abstract gives no evidence that this template space is representative of real-world table-form documents. If the templates are narrow or synthetic, the observed gap may be an artifact of the generator rather than a property of deployed systems. The authors should describe how templates were selected or sampled, compare their statistics with real document collections, and report diversity measures across templates.","section":"Abstract, data construction"},{"comment":"Results for 14 systems plus two SFT variants are reported only as best scores. Without per-system variance, confidence intervals, or significance tests, the robustness of the 83.85% versus <18% gap is unclear, particularly because different systems may be optimized for different metrics. The paper should report error bars or permutation-based significance for the gap, and ideally a scatter of scores across systems.","section":"Abstract, evaluation protocol"}],"minor_comments":[{"comment":"The abstract does not name the five primary metrics or the three diagnostics; including these names, even parenthetically, would help readers judge the protocol's scope.","section":"Abstract, evaluation protocol"},{"comment":"The terms 'Director-Artist-Verifier pipeline,' 'provenance-preserving,' and 'template-disjoint' are not defined; a one-sentence explanation in the abstract or a figure would improve accessibility.","section":"Abstract, data construction"},{"comment":"The abstract states that all 1,100 test instances received human review but does not report inter-annotator agreement; adding a Kappa or similar statistic would support the 'verified' claim.","section":"Abstract, data construction"}],"recommendation":"major_revision","confidential_remarks":"The review is based exclusively on the abstract because the full text was marked unavailable. The recommendation of major revision is driven by the absence of metric definitions and baselines in the abstract, which are necessary to assess the central claim; these may well exist in the full paper, but based on the available text the empirical conclusion is not yet fully supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Colleague],\n\nThe one thing to know about this paper: it's a benchmark proposal that could be genuinely useful if the full text backs up the abstract. The new bit is a hierarchical evaluation protocol—page, schema, component levels—with a template-disjoint test set and a provenance-preserving annotation pipeline. That's a real step beyond holistic-output benchmarks, and the fact that they tested 14 systems plus two SFT variants gives the gap claim some weight.\n\nBut the central claim—83.85% document-level vs. under 18% fine-grained—needs more than the abstract to be believed. The two scores may not be commensurable. If the fine-grained metric demands exact hierarchical matches with no partial credit and the document-level metric is loose token recall, the gap could be an artifact of scoring strictness rather than a genuine failure of models to recover hierarchy. The abstract gives no metric definitions, no matching tolerances, no human upper-bound baseline. Without those, the \"pronounced gap\" is a headline, not a result.\n\nThe template representativeness is the other soft spot. 70 templates expanded to 7,000 instances is fine, but we don't know how those templates were selected or whether they reflect real-world table-form distributions. A too-narrow template space can make the fine-grained scores artificially low.\n\nThat said, none of this is disqualifying. It's an abstract-only review, so we're judging a summary, not the paper. The design decisions they did mention—template-disjoint split, human review of the test set, progressive metrics—signal careful thought. The stress-test worry about metric asymmetry is a legitimate question, not a confirmed flaw. The authors may well have addressed it in the full text.\n\nBottom line: this deserves a serious referee. If the full paper ships the metric definitions, a structure-oracle or human baseline, and template construction details, it could be a useful resource. If not, the gap claim should be softened. I'd bring it to reading group once the full text is out, but I wouldn't cite it yet on the strength of the abstract.\n\nRecommendation: send it to review, but expect the referees to push on metric commensurability and template validity.","headline":"Benchmark with a promising hierarchical protocol, but the headline 83.85% vs <18% gap is unverifiable without metric definitions and a human upper bound.","tokens_in":1368,"tokens_out":1999,"would_cite":false,"duration_ms":19804,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FormStruct-Bench shows table-form systems read content but miss structure, with fine-grained scores below 18 percent.","keywords":["table-form document","structure recognition","hierarchical benchmark","document understanding","fine-grained evaluation","template expansion","diagnostic metrics"],"falsifier":"Take a random sample of real-world table-form documents, annotate them with the same page, schema, and component protocol, and evaluate the same systems; if fine-grained structural scores approach document-level scores on that sample, the benchmark's template space—not the systems' capability—is what produces the reported gap.","tokens_in":571,"feed_emoji":"📊","tokens_out":3636,"duration_ms":34742,"temperature":0.7,"pith_summary":"FormStruct-Bench is a benchmark for measuring whether table-form document systems recover not just the text but the multilevel structure—pages, schemas, and components—that organizes it. Using 70 annotated templates expanded into 7,000 verified instances, with a separate 1,100-instance human-reviewed test set, it evaluates 14 API-hosted and locally deployed systems plus two fine-tuned variants. The headline result is a large gap: the best document-level score is 83.85%, while the best fine-grained structural score is below 18%. The paper argues this shows current systems can read visible content but cannot reliably recover the hierarchy and regional layout needed for true table-form understanding, and that aggregate scores hide where such failures occur.","feed_headline":"Table-form AI scores 83.85% on content, under 18% on structure","feed_subtitle":"A new 7,000-instance benchmark traces recognition failures to hierarchy and layout, not content extraction.","key_machinery":"The central object is FormStruct-Bench itself, a hierarchical benchmark built from 70 reusable templates expanded through a provenance-preserving Director–Artist–Verifier pipeline into 7,000 verified instances; all 1,100 template-disjoint test instances receive human review. The evaluation protocol uses five primary metrics and three structure-specific diagnostics at page, schema, and component levels, with slices over difficulty, structural constraints, and visual degradation. This design lets an aggregate score be decomposed into specific structural failure modes, which is what exposes the content-versus-structure gap.","core_discovery":"The paper's central claim is that state-of-the-art table-form document systems, measured on FormStruct-Bench, consistently extract visible content while failing at structural organization. The best system earns 83.85% at the document level, yet none exceeds 18% on fine-grained structural scores spanning page, schema, and component levels. This document-versus-structure gap is the discovery: it is not a single weak system but a systematic property across 14 systems and two SFT variants. The paper therefore contends that structural recognition, not content extraction, is the unsolved bottleneck for converting table-form documents into machine-processable records.","pith_inferences":["The same diagnostic protocol could transfer to other structured documents such as invoices, receipts, and scientific tables to check whether the content-versus-structure gap generalizes.","Downstream applications like table question answering or database construction likely inherit the structural errors measured here, so fine-grained scores may predict end-task failures better than document-level scores.","Because the 70 templates are the source of all instances, testing on a broader corpus of real-world forms could narrow or widen the reported gap, clarifying whether it is inherent to current systems or amplified by template diversity."],"forward_implications":["If the gap is real, improving table-form understanding requires targeting hierarchy and regional organization, not just OCR or content extraction.","Benchmark scores can now be traced to failure modes, so a system's high document-level score and low structural score identify exactly which components need work.","The template-disjoint, human-reviewed test set gives a reusable testbed for measuring progress on structural recognition.","The 14-system evaluation provides a baseline that future systems must be compared against."],"supporting_citations":[],"fun_headline_variants":["AI reads tables but not their structure: 83.85% vs <18%","Table-form AI: 83.85% content, <18% structure on new benchmark","Systems ace table content, flunk structure: 83.85% vs <18%","New benchmark exposes table-form AI's structural blind spot","FormStruct-Bench: Content easy, structure hard for AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 70 templates, expanded into 7,000 instances, faithfully represent the variety of real-world table-form documents; if the templates are too narrow or artificial, the measured gap may be an artifact of the generator rather than a property of deployed systems.","fun_headline_variants_meta":{"raw":{"variants":["AI reads tables but not their structure: 83.85% vs <18%","Table-form AI: 83.85% content, <18% structure on new benchmark","Systems ace table content, flunk structure: 83.85% vs <18%","New benchmark exposes table-form AI's structural blind spot","FormStruct-Bench: Content easy, structure hard for AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000524,"raw_usage":{"total_tokens":2511,"prompt_tokens":900,"completion_tokens":1611,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":1510}},"tokens_in":516,"tokens_out":1611,"duration_ms":12174,"temperature":1.0,"reasoning_tokens":1510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:33:59.164637+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of real-world table-form documents, annotate them with the same page, schema, and component protocol, and evaluate the same systems; if fine-grained structural scores approach document-level scores on that sample, the benchmark's template space—not the systems' capability—is what produces the reported gap.","supporting_citations":[],"review_version":1}