Beyond String Matching: Semantic Evaluation of PDF Table Extraction

Janis Keuper; Pius Horn

REVIEW 3 major objections 4 minor 1 cited by

LLM judges of extracted PDF tables match human quality scores far better than TEDS or GriTS, ranking 21 parsers on synthetic arXiv tables.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 22:28 UTC pith:EUGZQ2GA

load-bearing objection Useful subfield benchmark and metric study if the human correlation holds; main risk is synthetic-only validation for real-PDF claims. the 3 major comments →

arxiv 2603.18652 v2 pith:EUGZQ2GA submitted 2026-03-19 cs.CV cs.AIcs.IR

Beyond String Matching: Semantic Evaluation of PDF Table Extraction

Pius Horn , Janis Keuper This is my paper

classification cs.CV cs.AIcs.IR

keywords PDF table extractionLLM-as-a-judgesemantic evaluationTEDSGriTSsynthetic benchmarksarXiv tablesparser ranking

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Reliably pulling tables out of PDFs matters for scientific data mining, but current scores treat tables as rigid trees or grids and miss when two tables mean the same thing. This paper builds a benchmark of synthetic PDFs whose tables come from arXiv and whose exact LaTeX source serves as ground truth, then scores extractions with an LLM acting as a semantic judge inside a matching pipeline that tolerates messy parser output. A human study of more than 1,500 quality judgments shows the LLM scores correlate with people at Pearson r=0.93, well above the 0.68–0.70 range of TEDS and GriTS. Running 21 contemporary parsers on 100 documents with 451 tables then yields a practical ranking and a reusable evaluation method.

Core claim

An LLM-as-a-judge semantic evaluation of PDF table extractions, placed inside a matching pipeline that handles inconsistent parser outputs, reaches Pearson r=0.93 with human quality judgments—substantially higher than Tree Edit Distance-based Similarity (TEDS, r=0.68) and Grid Table Similarity (GriTS, r=0.70)—and thereby supplies a more reliable metric for ranking parsers on a synthetic arXiv-derived benchmark.

What carries the argument

LLM-as-a-judge semantic scoring, integrated into a matching pipeline that accommodates structural and formatting inconsistencies between parser outputs and LaTeX ground truth; the judge supplies a continuous quality score that tracks human perception of table equivalence.

Load-bearing premise

That synthetically regenerated PDFs built from arXiv tables with perfect LaTeX ground truth are representative enough of real-world PDF table difficulty that the metric correlations and parser rankings transfer to production documents.

What would settle it

A held-out human study on real (non-synthetic) scientific PDFs in which the same LLM judge and the same parsers are scored against human quality labels; if the LLM–human correlation falls near TEDS/GriTS levels or the parser ranking reorders, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Parser selection for large-scale scientific table mining can be guided by the reported ranking of 21 systems rather than by TEDS or GriTS alone.
Future table-extraction papers can adopt the LLM-judge pipeline and the synthetic arXiv-derived corpus as a reproducible evaluation standard.
Semantic equivalence, not just structural edit distance, becomes the default notion of correctness for table extraction metrics.
Matching pipelines that tolerate layout and serialization differences become necessary scaffolding around any semantic judge.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same LLM-judge pattern could be reused for other semi-structured PDF elements (figures with captions, multi-column equations, nested lists) where structural metrics also diverge from human notions of correctness.
If the synthetic-to-real transfer holds only for clean academic LaTeX, commercial or scanned PDFs may still need a separate, harder calibration set.
High human correlation of the LLM judge suggests automated leaderboards for table extraction could be updated continuously without repeated large-scale human studies.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful subfield benchmark and metric study if the human correlation holds; main risk is synthetic-only validation for real-PDF claims.

read the letter

The one thing to know: this is a methods/benchmark paper that claims LLM-as-a-judge for PDF table extraction correlates with humans at r=0.93 versus ~0.7 for TEDS/GriTS, on >1500 judgments, plus a ranking of 21 parsers on 451 arXiv-sourced synthetic tables. That is the load-bearing result.

What is new and done well is the packaging. They build synthetic PDFs from real arXiv tables with LaTeX ground truth, add a matching pipeline for messy parser outputs, run a sizable human correlation study, and ship code/data links. LLM-as-judge is not novel as a pattern, and TEDS/GriTS are the standard baselines they position against, but the combination—arXiv tables, matching, large human study, 21-parser ranking—is a concrete empirical contribution for document AI. If the study design is clean, people will use the ranking and the metric. Credit for shipping the artifacts and for grounding the metric claim against external human judgments rather than against the LLM itself.

Soft spot, in proportion: we only have the abstract. The transfer premise is the real one. They assert the synthetic PDFs have “realistic complexity and diversity,” but there is no held-out real-PDF human study mentioned. If the 1500 judgments are all on the synthetic distribution, the high correlation may not travel to production PDFs with font quirks, multi-page spans, nested mess, and OCR-like noise—the exact setting where people need parser guidance. That is load-bearing for the practical claims. Secondary unknowns we cannot check: judge model identity, prompt, blinding, inter-annotator agreement, sampling of pairs, error bars, and whether any of the 21 parsers share an LLM family with the judge. Circularity risk looks low from the abstract structure, but residual family overlap is possible.

Who it is for: people who extract tables from scientific PDFs or who build/evaluate document parsers. A serious referee should see it; the numbers and the public ranking are important enough inside the subfield even if the synthetic-to-real gap forces revision. I would not desk-reject. I would want the full protocol and a real-PDF check before treating the ranking as production advice.

Recommendation: send to peer review. Ask specifically for human-study details and for evidence that the metric superiority and parser order hold outside the synthetic set.

Referee Report

3 major / 4 minor

Summary. The manuscript proposes a benchmarking framework for PDF table extraction that uses synthetically regenerated PDFs built from arXiv tables with precise LaTeX ground truth. Its central methodological claim is that an LLM-as-a-judge semantic metric, embedded in a matching pipeline that tolerates parser output inconsistencies, correlates substantially more strongly with human quality judgments (Pearson r=0.93) than standard structure-based metrics TEDS (r=0.68) and GriTS (r=0.70), based on a human validation study of over 1,500 judgments. The framework is then used to rank 21 contemporary PDF parsers on 100 synthetic documents containing 451 tables, with the stated goals of practical parser-selection guidance and a reproducible evaluation methodology. Code and data are released.

Significance. If the reported human-correlation advantage and the parser rankings transfer beyond the synthetic setting, the work would address a genuine gap: rule-based table metrics often penalize semantically equivalent extractions and mis-rank systems for scientific data mining. The combination of a large human validation study, multi-parser comparison, and public code/data is a clear strength and would give the community a usable, scalable alternative to TEDS/GriTS for semantic table evaluation. The result is therefore of practical interest to document AI and scientific knowledge-base construction, contingent on domain validity of the synthetic benchmark.

major comments (3)

Abstract: the claim that arXiv-sourced synthetic PDFs ensure 'realistic complexity and diversity' is load-bearing for both the metric superiority result and the parser-selection guidance, yet no held-out human study on real production PDFs is reported. Without quantitative comparison (layout quirks, multi-page spans, font/embedding artifacts, nested structures, OCR-like noise) or a real-PDF human correlation check, it remains unshown that r=0.93 vs ~0.7 and the 21-parser ranking transfer to the use case the paper advertises. A real-PDF validation subset or an explicit transfer experiment is needed.
Abstract (human validation, n>1500): the Pearson correlations are the central evidence for preferring LLM-as-a-judge over TEDS/GriTS, but the abstract does not specify judge model/family, prompt, blinding, inter-annotator agreement, pair sampling, score scale, or whether humans and the LLM saw identical matched pairs. These design choices determine whether r=0.93 is interpretable and whether residual circularity exists if the judge family overlaps LLM-based parsers among the 21 systems. Full study protocol and IAA must be reported and, if overlap exists, ablated.
Abstract (r=0.93 vs 0.68/0.70): the superiority claim needs uncertainty quantification—confidence intervals on the correlations, a test of difference between dependent correlations, and stratification by error type (structure vs content vs alignment). Without this, it is unclear whether the gap is robust across table difficulty and whether TEDS/GriTS fail systematically on the same cases the LLM judge handles.

minor comments (4)

Abstract: clarify how the matching pipeline defines table-level pairs when parsers split, merge, or drop tables; this affects both human labels and metric scores.
Abstract: state the number of human raters, rating scale, and aggregation method for the >1,500 judgments so readers can assess reliability before the full text.
Abstract: name which of the 21 parsers are rule-based, ML-based, or LLM-based so the ranking can be interpreted relative to the LLM judge.
Provide, in the full manuscript, error bars or significance tests on parser rankings across the 451 tables and document-level aggregation details.

Circularity Check

0 steps flagged

No circularity found: LLM-as-judge superiority is measured against external human judgments, not defined by construction from the metric or self-citation.

full rationale

Abstract-only review. The central claim is empirical: on a human validation study of >1,500 quality judgments, LLM-as-a-judge correlates with human scores at Pearson r=0.93 versus TEDS r=0.68 and GriTS r=0.70, then used to rank 21 parsers on 451 tables from synthetic arXiv-derived PDFs with LaTeX ground truth. That structure is not circular: the metric is scored against an external human reference, not against itself or a fitted parameter renamed as prediction. No equations, uniqueness theorems, or load-bearing self-citations appear in the abstract that would collapse the result into its inputs. Representativeness of synthetic PDFs for real-world PDFs is a transfer/validity concern (outside circularity scope), not a by-construction reduction. Residual risks (e.g., LLM family overlap between judge and parsers, or rater priming) are undisclosed and unproven from the given text; they do not constitute exhibited circular steps under the rules. Score 0 with empty steps is the honest finding.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

From the abstract alone, the claim rests on standard empirical ML evaluation practice plus two domain assumptions: synthetic LaTeX-regenerated PDFs stand in for real PDFs, and LLM judges plus a matching pipeline are valid proxies for human semantic table quality. No free parameters are fitted in the abstract narrative; no new physical entities are invented. The human study is the external anchor for the metric claim.

axioms (3)

domain assumption Synthetic PDFs regenerated from arXiv tables via LaTeX are realistic enough that parser rankings and metric correlations transfer to real-world PDFs.
Abstract presents this as the benchmarking design; validity of practical guidance depends on it.
domain assumption LLM-as-a-judge scores, after a structural matching pipeline, measure semantic table equivalence in a way that is stable across prompts, models, and table types.
Central methodological contribution; human correlation is offered as support but full protocol is not in the abstract.
standard math Pearson correlation with human quality judgments is an appropriate primary criterion for preferring one table-extraction metric over another.
Standard psychometrics/ML evaluation assumption; used to claim superiority of LLM judge over TEDS and GriTS.

pith-pipeline@v1.1.0-grok45 · 6134 in / 2634 out tokens · 30519 ms · 2026-07-13T22:28:48.729829+00:00 · methodology

0 comments

read the original abstract

Reliably extracting tables from PDFs is essential for large-scale scientific data mining and knowledge base construction, yet existing evaluation approaches rely on rule-based metrics that fail to capture semantic equivalence of table content. We present a benchmarking framework based on synthetically generated PDFs with precise LaTeX ground truth, using tables sourced from arXiv to ensure realistic complexity and diversity. As our central methodological contribution, we apply LLM-as-a-judge for semantic table evaluation, integrated into a matching pipeline that accommodates inconsistencies in parser outputs. Through a human validation study comprising over 1,500 quality judgments on extracted table pairs, we show that LLM-based evaluation achieves substantially higher correlation with human judgment (Pearson r=0.93) compared to currently used Tree Edit Distance-based Similarity (TEDS, r=0.68) and Grid Table Similarity (GriTS, r=0.70). Evaluating 21 contemporary PDF parsers across 100 synthetic documents containing 451 tables reveals significant performance disparities. Our results offer practical guidance for selecting parsers for tabular data extraction and establish a reproducible, scalable evaluation methodology for this critical task. Code and data: https://github.com/phorn1/pdf-parse-bench Metric study and human evaluation: https://github.com/phorn1/table-metric-study

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognition
cs.CL 2026-07 conditional novelty 5.0

Reference-free LLM judge scores failed to select better table-extraction outputs over eight regeneration iterations on FinTabNet and OmniDocBench; keeping the first output was safest.