Pith. sign in

REVIEW 2 major objections 6 minor 8 references

Tables as Graphs: A New Metric and Multilingual Benchmark

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A new multilingual table extraction benchmark (1,820 tables, 9 languages) and a graph-based metric (T-LAG) using optimal bipartite matching on directed adjacency edges to jointly score structure and content.

T0 review reviewed 2026-07-05 challenge →

load-bearing objection Real benchmark contribution, but the exclude-missing scoring mode changes mid-tier rankings and goes unreported in include-missing form. the 2 major comments →

arxiv 2606.07534 v1 pith:73KJMSUC submitted 2026-04-21 cs.IR cs.CL

PulseBench-Tab: A Multilingual Benchmark for Table Extraction with Graph-Based Evaluation

Ritvik Pandey , Sid Manchkanti , Mohammed Wazir Adain , Mohammed Hadi , Dushyanth Sekhar This is my paper
classification cs.IR cs.CL
keywords table extractionevaluation metricdirected graphbipartite matchingmultilingual benchmarkdocument intelligenceOCRHungarian algorithm
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces two things: a dataset and a metric. The dataset, PulseBench-Tab, is a collection of 1,820 human-annotated tables extracted from real-world documents in 9 languages across 4 writing systems (Latin, CJK, Arabic, Cyrillic), with nearly half containing merged or spanning cells. The metric, T-LAG (Table Logical Adjacency Graph), models a table as a directed graph where each edge represents a rightward or downward adjacency between two cells. To score a predicted table against ground truth, T-LAG computes a text-similarity weight for each candidate edge pair using a steeply decaying power function (exponent k=7, so that a cell 90% character-accurate contributes less than half credit), then uses the Hungarian algorithm to find the globally optimal one-to-one matching between predicted and ground-truth edges, and reports an F1 score over that matching. The authors evaluate 9 commercial and open-source table extraction systems and find a wide performance spread, with non-Latin scripts (especially Arabic and Korean) showing the largest cross-provider variance.

Core claim

The central claim is that table extraction quality is best evaluated by modeling tables as directed adjacency graphs and using optimal bipartite matching with aggressive text-similarity decay, rather than by tree edit distance (TEDS), greedy grid matching (GriTS), or sequence alignment. The paper demonstrates this by applying T-LAG to 9 systems across 1,820 multilingual tables, showing that rankings are stable across exponent values k in {7,8,9,11} and that structural hallucinations (fabricated rows, invented cells, shifted data) are pervasive even in top-ranked systems.

What carries the argument

T-LAG (Table Logical Adjacency Graph): a directed graph over cell adjacencies (RIGHT and BELOW edges), scored via Hungarian optimal bipartite matching with a power-decay text similarity kernel (exponent k=7).

Load-bearing premise

The choice of k=7 in the text-similarity decay function is justified by the fidelity requirements of enterprise document processing, but this is a domain-specific calibration that makes near-miss extraction score nearly as poorly as total failure. While rankings are stable at higher k values, the absolute scores and the characterization of approximate extraction as operationally equivalent to failure depend on this assumption.

What would settle it

If a different exponent value or a different edge-weighting scheme produced substantially different provider rankings, the claim that T-LAG provides a robust single-score evaluation would be undermined.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • T-LAG could be extended to multi-table layouts and multi-page spanning tables, which the current benchmark does not cover.
  • The graph-based edge-matching approach could be adapted to other structured-document tasks where 2D position carries meaning, such as form-field extraction or mathematical expression recognition.
  • The steep decay exponent k=7 reflects enterprise fidelity requirements; lower-stakes applications like summarization or exploratory analysis could use a lower k to reward partial credit, making the metric tunable across use cases.
  • The per-language breakdown methodology could be adopted by other benchmarks to expose script-specific weaknesses in OCR and structure recognition pipelines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The finding that provider rankings are invariant across k in {7,8,9,11} suggests the metric has effectively converged: once decay is steep enough, further strictness does not change relative ordering, only absolute scores. This means k=7 is not load-bearing for comparative conclusions, only for absolute score interpretation.
  • The large cross-provider variance on Arabic and Korean may reflect training-data scarcity rather than algorithmic limitations, since the same providers perform well on structurally similar Latin-script tables.
  • The exclude-missing scoring mode means some providers' scores are computed on self-selected subsets where they produced output, which could systematically favor providers with lower coverage but higher quality on attempted tables.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper introduces PulseBench-Tab, a multilingual benchmark of 1,820 human-annotated table images spanning 9 languages and 4 scripts, along with T-LAG, a graph-based evaluation metric that models tables as directed adjacency graphs and scores structural and content fidelity via optimal bipartite matching. The authors evaluate 9 commercial and open-source table extraction systems and report overall and per-language results. The dataset, scoring code, and provider outputs are publicly released.

Significance. The paper makes a solid contribution on two fronts. First, the multilingual dataset fills a genuine gap: existing benchmarks (PubTabNet, FinTabNet, RD-TableBench) are predominantly English or limited in script coverage, and the inclusion of Arabic, CJK, and Cyrillic tables with human-annotated structural markup is valuable. Second, the T-LAG metric is well-specified and addresses real limitations of TEDS (DOM-tree conflation of formatting with structure) and sequence-based metrics (loss of 2D adjacency). The use of optimal bipartite matching over directed edges is a principled choice over greedy alignment. The public release of dataset, code, and all provider outputs is a significant strength that enables reproducibility and independent verification. The sensitivity analysis for the k exponent (Table 5) is a commendable robustness check.

major comments (2)
  1. §5, Table 7: The exclude-missing scoring mode is load-bearing for the reported rankings but is not sensitivity-checked. Under exclude-missing, providers are scored only on successfully processed samples, which inflates absolute scores for low-coverage providers if failures correlate with table difficulty. The paper reports coverage alongside T-LAG but does not report include-missing scores (coverage × T-LAG) for comparison. This matters because it changes mid-tier rankings: Reducto (Agentic) at 0.795 × 78.8% = 0.627 would drop below Extend at 0.763 × 91.9% = 0.701, and Reducto at 0.718 × 80.4% = 0.577 would drop below AWS Textract at 0.603 × 98.5% = 0.594. The paper should report include-missing scores (or at minimum a coverage-weighted variant) alongside the exclude-missing scores so readers can assess how much of the mid-tier ordering is an artifact of differential coverage. The top排名（
  2. §1, §6.1: The first two authors are affiliated with Pulse AI, and the top-ranked system (Pulse Ultra 2, 93.5%) is a Pulse AI product. The benchmark, metric, and leading system all originate from the same organization. The public release of data, code, and outputs mitigates this concern substantially, and the exclude-missing bias actually compresses rather than inflates the leader's gap (Pulse Ultra 2 has 100% coverage). However, the paper does not explicitly disclose this relationship in the main text. A brief disclosure statement in §1 or §5 would strengthen the paper's credibility.
minor comments (6)
  1. §4.2.3, Eq. (3): The case for k=7 is justified by domain intuition ('enterprise document processing') rather than empirical calibration. While Table 5 shows ranking stability, the paper could strengthen its case by noting that k is a configurable parameter in the released code (mentioned only in §7.1).
  2. §3.2, Table 1: Korean has only 84 samples (4.6%). The paper acknowledges this in §7.1 but does not report confidence intervals or standard errors for any per-language scores in Table 8. Given the small sample sizes for some languages, error bars or intervals would help readers interpret cross-language differences.
  3. §4.2.2: The edge extraction suppresses edges within spanning cells (same cell ID). For tables with large merged regions, this could systematically reduce the number of edges and affect score comparability across table types. A brief discussion of how merged-cell density affects edge counts and thus T-LAG scores would be helpful.
  4. §7.2: The section header reads 'Limitations Future Work' — missing a colon or separator.
  5. §2: The reference to RD-TableBench [Reducto, 2024] cites a GitHub repository rather than a peer-reviewed publication. If a published version exists, it should be cited; if not, the citation format should make the repository nature explicit.
  6. §6.2, Table 8: Provider names are abbreviated inconsistently (e.g., 'Pulse U2', 'LlamaP.', 'Red. Ag.', 'Azure DI'). Standardizing column headers or using a legend would improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for a careful and constructive review. Both major comments are well-taken and will be addressed in the revised manuscript. The referee correctly identifies that (1) the exclude-missing scoring mode is load-bearing for mid-tier rankings and warrants a sensitivity check, and (2) the organizational relationship between the benchmark authors and the top-ranked system should be explicitly disclosed in the main text. We agree with both points and will revise accordingly.

read point-by-point responses

  1. Referee: §5, Table 7: The exclude-missing scoring mode is load-bearing for the reported rankings but is not sensitivity-checked. Under exclude-missing, providers are scored only on successfully processed samples, which inflates absolute scores for low-coverage providers if failures correlate with table difficulty. The paper should report include-missing scores (or at minimum a coverage-weighted variant) alongside the exclude-missing scores so readers can assess how much of the mid-tier ordering is an artifact of differential coverage.

    Authors: The referee is correct on all counts. The exclude-missing mode was adopted to avoid conflating detection failures with extraction quality, but we agree that reporting only this mode is insufficient when coverage varies substantially across providers (from 78.8% for Reducto Agentic to 100% for Pulse Ultra 2 and Unstructured). The referee's specific examples are accurate: under a coverage-weighted score (coverage × T-LAG), Reducto (Agentic) drops from 0.795 × 78.8% = 0.627 below Extend at 0.763 × 91.9% = 0.701, and standard Reducto drops from 0.718 × 80.4% = 0.577 below AWS Textract at 0.603 × 98.5% = 0.594. These are real reordering effects that readers should be able to see and assess. We will add an include-missing (coverage-weighted) column to Table 7 and discuss the mid-tier reordering in the results section. We note that the top and bottom of the ranking are unaffected—Pulse Ultra 2 retains its lead at 0.935 × 100% = 0.935, and Unstructured remains last at 0.360 × 100% = 0.360—but the mid-tier ordering is indeed sensitive to the scoring mode, and this should be transparent. We will also add a brief discussion of the assumption that failures correlate with table difficulty and note that include-missing scores represent a conservative lower bound under that assumption. revision: yes

  2. Referee: §1, §6.1: The first two authors are affiliated with Pulse AI, and the top-ranked system (Pulse Ultra 2, 93.5%) is a Pulse AI product. The benchmark, metric, and leading system all originate from the same organization. The paper does not explicitly disclose this relationship in the main text. A brief disclosure statement in §1 or §5 would strengthen the paper's credibility.

    Authors: We agree. The organizational relationship between the benchmark authors and the top-ranked system is not currently stated in the main text, and it should be. The affiliations on the title page show that the first two authors are at Pulse AI, but the connection to Pulse Ultra 2 as a Pulse AI product is not made explicit in §1 or §5. We will add a disclosure statement in §5 (Experimental Setup) noting that Pulse Ultra 2 is a product of Pulse AI, the employer of the first two authors, and that the benchmark dataset, T-LAG metric, and scoring code are publicly released to enable independent verification. We agree with the referee that the public release substantially mitigates the conflict-of-interest concern, and we note that the exclude-missing bias (as discussed above) actually compresses rather than inflates the leader's gap since Pulse Ultra 2 has 100% coverage. But the disclosure itself is appropriate and necessary regardless of these mitigating factors. revision: yes

Circularity Check

0 steps flagged

No circularity found: T-LAG is defined independently of evaluation results, and no self-citation chain is load-bearing.

full rationale

The paper defines T-LAG from first principles: tables are parsed into grid matrices, directed adjacency edges are extracted, edge similarity is computed via a Levenshtein-based kernel (Eq. 3), optimal matching is performed via the Hungarian algorithm (cited to Kuhn 1955, an external source), and an F1 score is computed (Eqs. 4-6). No step in this derivation chain reduces to its own inputs by construction. The k=7 exponent is a design choice justified by domain reasoning (enterprise fidelity requirements), not a parameter fitted to the benchmark data and then presented as a prediction. The sensitivity analysis (Table 5) confirms ranking invariance across k values, which is an empirical check, not a circular restatement. The conflict-of-interest concern (Pulse AI authors evaluating Pulse Ultra 2, which ranks first) is a legitimate methodological concern about bias, but it is not circularity: the metric definition does not reference any provider's output, and the dataset, scoring code, and provider outputs are all publicly released for external verification. The exclude-missing scoring mode is a methodological choice that could affect rankings, but it is openly stated and not hidden in a definition. No self-citation chain is load-bearing for the central claims. The derivation is self-contained.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The metric introduces one hand-selected parameter (k=7) justified by domain assumptions. The RTL DOM order axiom is standard but its application to skip column reversal is a modeling choice. The exclude-missing scoring is an ad-hoc protocol decision affecting all results.

free parameters (1)
  • k (Ψ exponent) = 7
    Chosen ad hoc based on domain assumptions about enterprise fidelity requirements (§4.3). Not fitted to data but selected by hand to enforce strict penalties.
axioms (3)
  • domain assumption HTML dir='rtl' is a visual rendering hint that does not change logical DOM order.
    Stated in §4.2.1 to justify not applying column reversal for RTL tables. This is a standard HTML specification behavior but a modeling choice for the metric.
  • domain assumption Near-miss extraction is operationally equivalent to failure for production use cases.
    Invoked in §4.3 to justify the aggressive decay of the Ψ function at k=7. This is an enterprise-specific assumption not universally true for all table extraction tasks.
  • ad hoc to paper Exclude-missing scoring mode is a valid way to evaluate extraction quality.
    Adopted in §5 to avoid penalizing providers for detection failures. This inflates scores for providers with low coverage and is a specific methodological choice.

reviewed 2026-07-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of PulseBench-Tab: A Multilingual Benchmark for Table Extraction with Graph-Based Evaluation." pith.science (2026). https://pith.science/paper/73KJMSUC

@misc{pith2026260607534,
  author       = {Pith},
  title        = {Pith review of: PulseBench-Tab: A Multilingual Benchmark for Table Extraction with Graph-Based Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/73KJMSUC}},
  note         = {Machine review of arXiv:2606.07534}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We introduce PulseBench-Tab, an open multilingual benchmark for evaluating table extraction from document images. The benchmark comprises 1,820 human-annotated tables spanning 9 languages and 4 scripts (Latin, CJK, Arabic, Cyrillic), drawn from 380 real-world source documents including financial filings, government reports, and regulatory disclosures. Tables range from 2 to 1,183 cells, with 48.1% containing merged or spanning cells. Alongside the dataset, we propose T-LAG (Table Logical Adjacency Graph), a novel evaluation metric that models tables as directed graphs over cell adjacencies and computes structural and content fidelity in a single score via optimal bipartite matching. We evaluate 9 commercial and open-source table extraction systems across the benchmark and report per-language breakdowns. The full dataset, scoring code, and all provider outputs are publicly available.

Figures

Figures reproduced from arXiv: 2606.07534 by Dushyanth Sekhar, Mohammed Hadi, Mohammed Wazir Adain, Ritvik Pandey, Sid Manchkanti.

Figure 1
Figure 1. Figure 1: Language distribution across PulseBench-Tab. English comprises 32.6% of samples; [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of table complexity. Tables range from 2 to 1,183 cells, with 48.1% [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Sensitivity of T-LAG scores to the Ψ exponent. All providers decline monotonically [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overall T-LAG scores across 9 providers. Pulse Ultra 2 (93.5%) leads by 11.9 per [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Per-language T-LAG scores across providers. Arabic and Korean exhibit the widest [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

8 extracted references · 8 canonical work pages · 1 internal anchor

  1. [1]

    Gartlehner, L

    G. Gartlehner, L. Kahwati, B. Nussbaumer-Streit, K. Crotty, R. Hilscher, S. Kugley, M. Viswanathan, I. Thomas, A. Konet, G. Booth, and R. Chew. From promise to practice: challenges and pitfalls in the evaluation of large language models for data extraction in evidence synthesis. BMJ Evidence-Based Medicine, 30(6):385--389, 2025

    work page 2025

  2. [2]

    H. W. Kuhn. The H ungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83--97, 1955

    work page 1955

  3. [3]

    Ouyang, Y

    L. Ouyang, Y. Qu, H. Zhou, J. Zhu, R. Zhang, Q. Lin, B. Wang, Z. Yuan, Z. Zhao, M. Jiang, and X. Zhao. Omnidocbench: Benchmarking diverse pdf document parsing with comprehensive annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025

    work page 2025

  4. [4]

    RD-TableBench : An open benchmark for PDF table extraction

    Reducto. RD-TableBench : An open benchmark for PDF table extraction. https://github.com/reductoai/rd-tablebench, 2024

    work page 2024

  5. [5]

    Smock, R

    B. Smock, R. Pesala, and R. Abraham. GriTS : Grid table similarity metric for table structure recognition. In International Conference on Document Analysis and Recognition (ICDAR), pages 535--549. Springer, 2023

    work page 2023

  6. [6]

    SCORE-Bench : An open document parsing benchmark

    Unstructured. SCORE-Bench : An open document parsing benchmark. https://huggingface.co/datasets/unstructuredio/SCORE-Bench, 2025

    work page 2025

  7. [7]

    Zheng, D

    X. Zheng, D. Burdick, L. Popa, X. Zhong, and N. X. R. Wang. Global table extractor ( GTE ): A framework for joint table identification and cell structure recognition using visual context. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 2021

    work page 2021

  8. [8]

    Image-based table recognition: data, model, and evaluation

    X. Zhong, E. ShafieiBavani, and A. Jimeno Yepes. Image-based table recognition: Data, model, and evaluation. arXiv preprint arXiv:1911.10683, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 1911

This paper was first reviewed by glm-5.2 on July 5, 2026.