Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

A judge LLM that can evaluate table outputs cannot reliably guide closed-loop regeneration: no tested score policy beat keeping the first output.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:24 UTC pith:QHYOIOPO

load-bearing objection The central negative result—reference-free judge scores don't drive selection in closed-loop table recognition—is solid and worth taking seriously; the secondary mechanism claim is softer but honestly hedged. the 3 major comments →

arxiv 2607.13347 v2 pith:QHYOIOPO submitted 2026-07-15 cs.CL cs.AIcs.LG

LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognition

Donghwan Kim This is my paper
classification cs.CL cs.AIcs.LG
keywords LLM-as-a-judgetable recognitionclosed-loop refinementTEDSselection utilityself-correctionreward hackingiterative optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether an LLM judge that can evaluate outputs can also guide their iterative improvement—an assumption behind many closed-loop pipelines. In a table-recognition setting where a deterministic structural metric (TEDS) audited every decision, the answer is no: across two benchmarks, judge scores tied frequently, rankings were not reproducible, and no tested selection or margin-guarded policy improved on simply keeping the first output. Better candidates existed, but judge policies recovered at most a small fraction of the available improvement. Severe breakage also occurred without judge feedback, indicting regeneration that fails to preserve the target structure; a structure-preserving instruction reduced the worst failures but never raised mean quality. If right, this means evaluation-style evidence alone is not enough to justify using judge scores as optimization signals, and iterative refinement needs a deterministic structural verification signal, not just LLM judgments.

Core claim

An eight-iteration closed loop regenerates HTML tables from images: a reference-free LLM judge scores each candidate and lists errors; the generator revises from that feedback; ground truth never enters; the deterministic tree-edit metric TEDS audits every decision. The judge signal is weak: scores tie on tens of percent of distinct candidate pairs, rankings are not reproducible, and no judge configuration, scoring format, or tie rule beats the trivial always-iter0 baseline on both datasets; margin-guarded acceptance does not help. Removing feedback leaves net degradation unchanged and severe losses persist, indicting target-preservation failure under unconstrained regeneration. A structure-

What carries the argument

The central object is a closed-loop regeneration pipeline whose every decision is audited by TEDS, a deterministic tree-edit-distance similarity between the generated HTML table and fixed ground truth. TEDS converts the otherwise fuzzy notion of table quality into a fixed benchmark objective, so the paper can quantify, per table and per iteration, how far each judge decision deviates from the target. The argument is carried by two crossed contrasts, both forking from the same stored initial output: a feedback-presence contrast (judge error list injected vs. generic instruction only) and a structure-preservation contrast (unconstrained regeneration vs. copy-preserving regeneration instructed

Load-bearing premise

The causal decomposition assumes that temperature-0 generation and scoring are near-deterministic; the paper concedes full determinism does not hold and that using more than one serving backend may account for part of the scoring nondeterminism, so if routing or sampling noise is substantial, attribution of changes to feedback weakens.

What would settle it

A judge configuration with near-perfect scoring reproducibility (e.g., exact agreement above 95% across repeated calls) and near-zero tie rates on TEDS-separated pairs, whose best-of-n or margin-guarded selection then beats keeping the first output on both benchmarks, would overturn the central claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Judge scores should not gate deployment in reference-free iterative extraction: keeping the first output was the safest baseline on both benchmarks.
  • Guarded margin rules are not a safety net; the margins that avoided degradation accepted almost no revisions, so they merely replicated the first-output policy.
  • A neutral average loop outcome can hide severe tail damage: 4–12% of tables suffered large losses, so mean metrics alone are insufficient for deciding whether to deploy a refine loop.
  • Regeneration itself, not the content of judge feedback, is the main source of damage; preventing structural changes during revision reduces the worst failures.
  • Iterative refinement needs a deterministic structural verification signal (e.g., checking row/column counts and merge structure) rather than judge scores alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial: If this pattern holds beyond the two benchmarks, evaluation-score alignment with humans should not be taken as evidence that the same scores can steer optimization; applications should first audit within-instance ranking reproducibility (tie rates, repeated-scoring agreement) before building a loop around a judge.
  • Editorial: The convention-conflict mechanism suggests a testable extension: the breakage rate should scale with the divergence between the target dataset's ground-truth HTML conventions and the markup conventions the judge prefers; measuring that divergence could predict which tables are at risk.
  • Editorial: The exploratory generator trend (larger oracle headroom associated with positive loop benefit) suggests a practical decision rule for practitioners: measure baseline error on a small labeled audit and enable closed-loop refinement only for generators/inputs with large headroom.
  • Editorial: A deterministic structural guard could be tested directly: add a checker that rejects any revision whose row/column count or merge structure differs from the previous candidate, and measure whether the loop's tail losses disappear.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies whether reference-free LLM-as-a-judge scores can serve as reliable optimization signals in closed-loop table recognition. Using a Gemini model as generator and judge, it runs eight-iteration regeneration loops on FinTabNet and OmniDocBench, with TEDS computed post hoc from ground truth as a deterministic audit metric. The paper reports three main findings: (1) judge scores are weak selection signals—frequent ties, low or negative rank correlations, non-reproducible rankings, negative or near-zero recovery relative to the first output, and no tested score policy improving over iter0 on both datasets; (2) severe degradation occurs even without judge feedback, with target-preservation failure under unconstrained regeneration as a proximate mechanism; (3) a structure-preserving instruction reduces the severe-loss rate in a frozen-iter0 paired contrast, significantly on FinTabNet and directionally on OmniDocBench. The paper concludes that evaluation-style evidence for LLM judges does not establish closed-loop optimization utility and that deterministic structural verification is needed.

Significance. If the central negative result holds, it is a valuable, well-scoped contribution: the task provides a deterministic external metric, and the authors run multiple independent diagnostics—tie rates, rank correlations, repeated-scoring repeatability, an independent TEDS implementation cross-check, a no-feedback control, an independent-candidate control, and artifact audits. These controls make the selection-failure result substantially more convincing than a single point estimate. The paper is also appropriately cautious in many places: it does not dispute LLM-as-a-judge as an evaluator, it discloses the development-set selection of the calibrated prompt, it reports the post hoc margin selection, and it includes a Limitations section that candidly documents non-replication. The main weakness is that the mechanistic claims about target-preservation failure and feedback content are built on across-run contrasts in a setting the paper itself shows to be temporally unstable. The selection-failure negative result is likely robust; the causal decomposition is not yet established at the same standard.

major comments (3)
  1. [§4.4 and Limitations] The proximate-mechanism claim rests on contrasts that are not protected against the paper's own documented run-to-run non-replication. The Limitations state that the original 2×2 unconstrained-condition mean (−0.0122) did not recur in later re-forking runs (+0.0000 and +0.0007), and that multi-backend serving is an unconfirmed possible cause. The full-n frozen-iter0 B/C severe-loss contrast (17/476 vs 4/476, two-sided exact McNemar p=0.0023) is paired by table, but the text does not report whether B and C were interleaved or randomized within the run, or whether backend allocation was pinned. If B and C were executed in separate time blocks, temporal drift could produce or inflate the severe-loss difference. The same concern applies to the no-feedback control in §4.1/Appendix B.6 (paired 99-table difference +0.0001), where the main loop and the no-feedback loop are separate runs. Please
  2. [§4.1 and Table 1; Limitations] The central selection-failure claim is presented as a general property of the loop, but the policy comparisons in Table 1 are computed on a single main-run candidate stream. The paper's own Limitations record that the unconstrained-condition mean did not recur in later re-forking runs (means +0.0000 and +0.0007), and the full-n B condition in §4.4 has mean −0.0030 on FinTabNet versus −0.0200 in the main run. Thus 'no tested judge score policy improved on the first output on both datasets' is, strictly, a claim about one realization. The tie-rate, rank-correlation, and repeatability diagnostics make the selection-failure result plausible, but they do not by themselves establish that the recovery numbers in Table 1 would survive replication on a re-forked run. Please report a replication of the selection policies on a re-forked candidate stream, or explicitly restrict the claim to the obse
  3. [§4.4 and Appendix J] The guarded-acceptance result is used to support the overall conclusion, but the OmniDocBench improvement at δ=20 is selected post hoc from a grid, as the Limitations state. The FinTabNet δ=20 result is negative (−0.0020) and the margin rule mostly rejects revisions. Given that the post hoc selection is not accounted for in the confidence interval, the sentence in §4.2 that 'the best margin (δ=20, selected post hoc from the grid) improved mean TEDS by +0.0116' should be explicitly labeled exploratory, and the abstract's summary of the guarded-policy finding should not rely on this value. A simple multiple-comparison acknowledgment (e.g., reporting all six margins, as Appendix J does) is already present, but the main text should not present the selected margin as a confirmed policy without a fixed-analysis qualification.
minor comments (5)
  1. [§4.1] 'Feedback content is not the source of net degradation' is too strong given the cross-run nature of the comparison; suggest rewording to 'not the source in this paired comparison' or adding the temporal-drift caveat where the sentence first appears.
  2. [Table 1] The 'Better'/'Worse' column compares against random-among-8, not against iter0. Since several policies are simultaneously 'better than random' and worse than iter0, the table would benefit from a footnote making this distinction explicit in each verdict cell.
  3. [§I.1] The statement 'Full determinism does not hold' appears only in an appendix footnote. Given that the paper's causal interpretation depends on near-determinism, this qualification should appear in the main-text experimental setup (Section 3.1) alongside the temperature-0 statement.
  4. [§3.1 / Appendix I.1] The multi-backend routing is mentioned in the Limitations and Appendix I.1 but not in Section 3.1. Consider moving a one-sentence mention into the main text so the reader is not surprised by the later caveat.
  5. [Appendix B.4] The blinded author validation reports Cohen's κ=0.648 and a relabeling of several REAL items to GT-inconsistent. The main-text Section 4.3 cites the LLM-assisted labels without noting this revision; a pointer to the sensitivity analysis near the main claim would improve transparency.

Circularity Check

0 steps flagged

No significant circularity: the central negative result is benchmarked against external deterministic TEDS, and the few fitted components are disclosed and non-load-bearing.

full rationale

The paper's central claim is empirical: judge-based selection and feedback do not improve over the first output when scored by TEDS. TEDS is computed deterministically from ground truth and is external to the loop; the paper also enforces a GT firewall so judge and generator never see GT. No equation derives the judge's weakness from a fitted parameter, and no load-bearing result is justified by self-citation. The only fitted elements are the calibrated_v2 prompt, selected against TEDS on 20 development tables, and the post hoc δ=20 margin rule. Both are explicitly disclosed as exploratory or non-confirmatory: the Limitations state 'The calibrated_v2 prompt was selected against TEDS using 20 development tables' and 'we do not treat positive signals by stratum as established claims,' and the δ=20 result is described as 'selected post hoc from a grid' and 'exploratory.' Moreover, calibrated_v2 does not support the main conclusion: its best-by-judge policy still has negative mean Δ on FinTabNet. The RQ4 mechanism claim uses paired frozen-iter0 contrasts with a pre-specified severe-loss endpoint and exact McNemar tests, not a fitted prediction. The acknowledged run-to-run non-replication of mean effects is a robustness/confound concern, not circularity. The paper does not import a uniqueness theorem, rename a known result, or defend its central premise by citing the author's prior work. The derivation chain is therefore self-contained and non-circular.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The experimental claims rely on TEDS as an external deterministic benchmark and on several background assumptions about generation determinism and GT conventions. No new entities are postulated. The main free parameters are disclosed design choices: dev-set prompt selection, the post hoc δ threshold, and the severe-loss threshold.

free parameters (3)
  • calibrated_v2 scoring prompt/rubric = selected on 20 development tables using TEDS
    The more granular scoring configuration was designed with development tables from FinTabNet; this is a form of parameter fitting to the evaluation metric before the main held-out evaluation.
  • Guard margin δ in guarded acceptance = 20 (best in post hoc grid over {0,20,40,60,80,100})
    The OmniDocBench improvement at δ=20 is explicitly described as exploratory and selected post hoc from a grid, so the claimed margin effect is fitted to the data.
  • Severe-loss threshold for primary frozen-iter0 contrast = ΔTEDS < -0.10
    The primary endpoint in Section 4.4 was fixed before inspecting the frozen-iter0 results, but the threshold was inherited from the main-run analysis rather than pre-registered independently.
axioms (4)
  • domain assumption TEDS against fixed GT is a valid deterministic measure of table-recognition quality.
    The entire audit treats TEDS as the benchmark objective, while the authors note in Limitations that TEDS depends on GT serialization conventions and that part of the judge-metric gap may be a convention conflict rather than a pure judge error.
  • domain assumption Generation at temperature 0 is near-deterministic enough to attribute iteration changes to injected judge feedback.
    Stated in Section 3.1 and Appendix I.1, but the authors admit full determinism does not hold and that two serving backends may contribute to scoring nondeterminism, weakening this premise.
  • domain assumption GT serialization conventions define correct structure, so departures such as converting section-label cells to merged headers count as structural damage.
    Section 4.3 classifies the judge's semantic HTML conventions as GT-inconsistent claims; the authors acknowledge in Limitations that some of these are convention conflicts rather than objectively wrong output.
  • domain assumption The two OpenRouter serving backends ('Google AI Studio' and 'Google') are functionally equivalent for this study.
    Appendix I.1 and the Limitations state that the model version string was identical across backends but that serving variability cannot be ruled out as a contributor to temperature-0 scoring nondeterminism.

pith-pipeline@v1.3.0-alltime-deepseek · 30107 in / 9788 out tokens · 103256 ms · 2026-08-02T05:24:51.809919+00:00 · methodology

0 comments
read the original abstract

LLM-as-a-judge is widely used to provide feedback and selection signals in closedloop regeneration, but this use remains insufficiently validated. We study it in table recognition, where deterministic TEDS evaluation provides a controlled testbed, using FinTabNet and OmniDocBench. Three findings emerge. First, judge signals were weak on both datasets: scores frequently tied, rankings were not reproducible, and no tested judge score policy, whether selecting among candidates or accepting revisions under a conservative score margin, improved on the first output on both datasets. Iteration produced better candidates, but the judge recovered them at most partially on one dataset and not at all on the other. Second, severe losses occurred even without specific judge feedback, supporting target-preservation failure under unconstrained regeneration as a proximate mechanism. Third, a structure-preserving instruction reduced the severe-loss rate, significantly on FinTabNet and directionally on OmniDocBench, but produced no improvement, and in an exploratory 2x2 analysis this protection was not stably observed when judge feedback was retained. These results do not dispute the value of LLMs as evaluators, but show that the tested reference-free judge signals were too weak and unstable to drive candidate selection in this setup, and that evaluation-style evidence alone was insufficient to establish closed-loop optimization utility. Iterative refinement requires, at minimum, a verification signal that deterministically detects structural change, rather than judge scores alone.

Figures

Figures reproduced from arXiv: 2607.13347 by Donghwan Kim.

Figure 1
Figure 1. Figure 1: Experimental framework. The generator produces HTML from a table image, and the judge returns a score and an error list based only on the image and the immediately preceding output. This feedback is injected into the next regeneration step (7 iterations). GT and TEDS computation are used only for evaluation after the loop terminates and are not exposed during either generation or scoring (right of the dash… view at source ↗
Figure 2
Figure 2. Figure 2: The same type of breakage observed on both datasets. Top (a), FinTabNet (TDG.2006); bottom (b), OmniDocBench (scihub). Each row, from left to right, shows the source image, the iter0 output, the iter1 output regenerated using judge feedback (changed cells marked in red), and the verbatim judge feedback injected into that regeneration (iteration 0 record, with score). In both examples, iter0 places a sectio… view at source ↗
Figure 3
Figure 3. Figure 3: Mean TEDS and S-TEDS by iteration on both datasets. Values are changes relative to iter0. Shading gives table-cluster bootstrap 95% CIs [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Severe-loss rate by iteration in the full-n frozen-iter0 B/C re-fork. The final rate on FinTabNet fell from 3.6% to 0.8% (C−B = −2.7 pp, two-sided exact McNemar p = 0.0023); OmniDocBench was directionally consistent, from 2.2% to 0.7% (p = 0.1250). claim recovery of improvements or an increase in mean performance. 5 Discussion Evaluation ability and optimization utility are different capabilities. Our resu… view at source ↗
Figure 5
Figure 5. Figure 5: Iteration curves for the no-feedback control. [PITH_FULL_IMAGE:figures/full_fig_p021_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Iteration curves for the 4 FinTabNet preservation conditions. [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Iteration curves for the 4 OmniDocBench preservation conditions. [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Oracle headroom versus the net effect of the loop (100-table means by generator, 6 points). The judge and protocol are fixed and only the generator is changed. The horizontal axis is oracle minus iter0; the vertical axis is final minus iter0. The dotted line marks a net effect of 0. Spearman correlation is 0.771 (n = 6) [PITH_FULL_IMAGE:figures/full_fig_p026_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Distribution of the table-level difference [PITH_FULL_IMAGE:figures/full_fig_p028_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles

    cs.CL 2026-07 conditional novelty 6.0

    Diversity metrics used to select LLM ensembles are largely capability proxies; after control, only a modest pairwise co-failure association with majority-vote gain remains.

Reference graph

Works this paper leans on

47 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    Surikuchi, Ece Takmaz, and Alberto Testoni

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fern \'a ndez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K. Surikuchi, Ece Takmaz, and Alberto Testoni. 2025. https...

    doi:10.18653/v1/2025.acl-short.20 2025

  2. [2]

    Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.474 Humans or LLM s as the judge? a study on judgement bias . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301--8327. Association for Computational Linguistics

    doi:10.18653/v1/2024.emnlp-main.474 2024

  3. [3]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    Pith/arXiv arXiv 2021

  4. [4]

    Leo Gao, John Schulman, and Jacob Hilton. 2023. https://proceedings.mlr.press/v202/gao23h.html Scaling laws for reward model overoptimization . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10835--10866. PMLR. ArXiv:2210.10760

    Pith/arXiv arXiv 2023

  5. [5]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. https://openreview.net/forum?id=WSrRF5Wy6v CRITIC : Large language models can self-correct with tool-interactive critiquing . In The Twelfth International Conference on Learning Representations

    2024

  6. [6]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Zhouchi Lin, Bowen Zhang, Lionel Ni, Wen Gao, Yuanzhuo Wang, and Jian Guo. 2026. https://doi.org/10.1016/j.xinn.2025.101253 A survey on LLM -as-a-judge . The Innovation, 7(6):101253. Open access; arXiv:2411.15594

    arXiv 2026

  7. [7]

    Rajarshi Haldar and Julia Hockenmaier. 2025. https://doi.org/10.18653/v1/2025.findings-emnlp.1361 Rating roulette: Self-inconsistency in LLM -as-a-judge frameworks . In Findings of the Association for Computational Linguistics: EMNLP 2025 , pages 24986--25004. Association for Computational Linguistics. ArXiv:2510.27106

    Pith/arXiv arXiv 2025

  8. [8]

    Pius Horn and Janis Keuper. 2026. https://arxiv.org/abs/2603.18652 Beyond string matching: Semantic evaluation of PDF table extraction . Preprint, arXiv:2603.18652

    Pith/arXiv arXiv 2026

  9. [9]

    Hui Huang, Xingyuan Bu, Hongli Zhou, Yingqi Qu, Jing Liu, Muyun Yang, Bing Xu, and Tiejun Zhao. 2025. https://doi.org/10.18653/v1/2025.findings-acl.306 An empirical study of LLM -as-a-judge for LLM evaluation: Fine-tuned judge model is not a general substitute for GPT -4 . In Findings of the Association for Computational Linguistics: ACL 2025 , pages 5880...

    Pith/arXiv arXiv 2025

  10. [10]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. https://openreview.net/forum?id=IkmD3fKBPQ Large Language Models Cannot Self-Correct Reasoning Yet . In The Twelfth International Conference on Learning Representations. ArXiv:2310.01798

    Pith/arXiv arXiv 2024

  11. [11]

    Dongwei Jiang, Alvin Zhang, Andrew Wang, Nicholas Andrews, and Daniel Khashabi. 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/4eb5cc598a6271528ed9b84bb9879e9b-Abstract-Conference.html Feedback friction: LLM s struggle to fully incorporate external feedback . In Advances in Neural Information Processing Systems, volume 38. ArXiv:2506.11930

    arXiv 2025

  12. [12]

    Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. https://doi.org/10.1162/tacl_a_00713 When can LLM s actually correct their own mistakes? a critical survey of self-correction of LLM s . Transactions of the Association for Computational Linguistics, 12:1417--1440. ArXiv:2406.01297

    Pith/arXiv arXiv 2024

  13. [13]

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. https://proceedings.iclr.cc/paper_files/paper/2024/hash/803485352e61e3ebf41221e4776c9fd4-Abstract-Conference.html Prometheus: Inducing fine-grained evaluation capability in language models . In The Twel...

    arXiv 2024

  14. [14]

    Eddie Landesberg. 2026. https://arxiv.org/abs/2603.12520 When LLM judge scores look good but best-of-n decisions fail . Preprint, arXiv:2603.12520

    arXiv 2026

  15. [15]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://proceedings.iclr.cc/paper_files/paper/2024/hash/aca97732e30bcf1303bc22ac3924fd16-Abstract-Conference.html Let's verify step by step . In The Twelfth International Conference on Learning Representation...

    Pith/arXiv arXiv 2024

  16. [16]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G-Eval : NLG evaluation using GPT-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511--2522. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.153 2023

  17. [17]

    Jack Lu, Ryan Teehan, Jinran Jin, and Mengye Ren. 2025. https://arxiv.org/abs/2512.02304 When does verification pay off? a closer look at LLM s as solution verifiers . Preprint, arXiv:2512.02304. Accepted at the ICLR 2026 AI with Recursive Self-Improvement Workshop

    Pith/arXiv arXiv 2025

  18. [18]

    Maksym Lysak, Ahmed Nassar, Nikolaos Livathinos, Christoph Auer, and Peter Staar. 2023. https://doi.org/10.1007/978-3-031-41679-8_3 Optimized table tokenization for table structure recognition . In Document Analysis and Recognition -- ICDAR 2023 , volume 14188 of Lecture Notes in Computer Science, pages 37--50. Springer Nature Switzerland. ArXiv:2305.03393

    Pith/arXiv arXiv 2023

  19. [19]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/91edff07232fb1b55a505a9e9f6c0ff3-Abs...

    Pith/arXiv arXiv 2023

  20. [20]

    Nikita Mehrotra, Aayush Kumar, Sumit Gulwani, Arjun Radhakrishna, and Ashish Tiwari. 2026. https://doi.org/10.18653/v1/2026.acl-industry.138 TEN : Table explicitization, neurosymbolically . In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 2050--2086. Association for Computational ...

    Pith/arXiv arXiv 2026

  21. [21]

    Deepak Nathani, David Wang, Liangming Pan, and William Yang Wang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.407 MAF : Multi-aspect feedback for improving reasoning in large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6591--6616. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.407 2023

  22. [22]

    Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongying Tu, and Conghui He. 2025. https://doi.org/10.1109/CVPR52734.2025.02313 OmniDocBench : Benchmarking diverse PDF document parsing with comprehe...

    arXiv 2025

  23. [23]

    Bowman, and Shi Feng

    Jane Pan, He He, Samuel R. Bowman, and Shi Feng. 2024 a . https://arxiv.org/abs/2407.04549 Spontaneous reward hacking in iterative self-refinement . Preprint, arXiv:2407.04549

    Pith/arXiv arXiv 2024

  24. [24]

    Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2024 b . https://doi.org/10.1162/tacl_a_00660 Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies . Transactions of the Association for Computational Linguistics, 12:484--506

    doi:10.1162/tacl_a_00660 2024

  25. [25]

    Kelly Buchanan, Mayee F

    Jon Saad-Falcon, E. Kelly Buchanan, Mayee F. Chen, Tzu-Heng Huang, Brendan McLaughlin, Tanvir Bhathal, Shang Zhu, Ben Athiwaratkun, Frederic Sala, Scott Linderman, Azalia Mirhoseini, and Christopher R \'e . 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/715f48023c26ad210321a4c0da67f27c-Abstract-Conference.html Weaver : Shrinking the gene...

    arXiv 2025

  26. [26]

    Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.779 PICARD : Parsing incrementally for constrained auto-regressive decoding from language models . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895--9901. Association for Computational Linguistics

    doi:10.18653/v1/2021.emnlp-main.779 2021

  27. [27]

    Farseen Shaikh. 2026. https://doi.org/10.18653/v1/2026.surgellm-1.19 Can LLM s self-correct table reasoning errors? In Proceedings of the First Workshop on Structured Understanding, Retrieval, and Generation in the LLM Era , pages 298--312. Association for Computational Linguistics

    work page doi:10.18653/v1/2026.surgellm-1.19 2026

  28. [28]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html Reflexion: Language agents with verbal reinforcement learning . In Advances in Neural Information Processing Systems, volume 36, pages 8634--8652

    2023

  29. [29]

    Brandon Smock, Rohith Pesala, and Robin Abraham. 2022. https://doi.org/10.1109/CVPR52688.2022.00459 PubTables-1M : Towards comprehensive table extraction from unstructured documents . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4634--4642

    arXiv 2022

  30. [30]

    Brandon Smock, Rohith Pesala, and Robin Abraham. 2023 a . https://doi.org/10.1007/978-3-031-41734-4_23 Aligning benchmark datasets for table structure recognition . In Document Analysis and Recognition -- ICDAR 2023 , volume 14191 of Lecture Notes in Computer Science, pages 371--386. Springer. ArXiv:2303.00716

    Pith/arXiv arXiv 2023

  31. [31]

    Brandon Smock, Rohith Pesala, and Robin Abraham. 2023 b . https://doi.org/10.1007/978-3-031-41734-4_33 GriTS : Grid table similarity metric for table structure recognition . In Document Analysis and Recognition -- ICDAR 2023 , volume 14191 of Lecture Notes in Computer Science, pages 535--549. Springer. ArXiv:2203.12555

    Pith/arXiv arXiv 2023

  32. [32]

    Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. 2025. https://openreview.net/forum?id=HiUZtgAPoH SynCode : LLM generation with grammar augmentation . Transactions on Machine Learning Research. ArXiv:2403.01632

    Pith/arXiv arXiv 2025

  33. [33]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. https://doi.org/10.18653/v1/2024.acl-long.511 Large language models are not fair evaluators . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9440...

    doi:10.18653/v1/2024.acl-long.511 2024

  34. [34]

    Bosi Wen, Yilin Niu, Cunxiang Wang, Xiaoying Ling, Ying Zhang, Pei Ke, Hongning Wang, and Minlie Huang. 2026. https://doi.org/10.18653/v1/2026.acl-long.1092 IF-RewardBench : Benchmarking judge models for instruction-following evaluation . In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    work page doi:10.18653/v1/2026.acl-long.1092 2026

  35. [35]

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason E Weston, and Sainbayar Sukhbaatar. 2025. https://doi.org/10.18653/v1/2025.emnlp-main.583 Meta-rewarding language models: Self-improving alignment with LLM -as-a-meta-judge . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages ...

    doi:10.18653/v1/2025.emnlp-main.583 2025

  36. [36]

    Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang. 2024. https://doi.org/10.18653/v1/2024.acl-long.826 Pride and prejudice: LLM amplifies self-bias in self-refinement . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15474--15492. Association for Co...

    doi:10.18653/v1/2024.acl-long.826 2024

  37. [37]

    Yusuke Yamauchi, Taro Yano, and Masafumi Oyamada. 2026. https://doi.org/10.18653/v1/2026.gem-main.19 An empirical study of LLM -as-a-judge: How design choices impact evaluation reliability . In Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics, pages 167--176. Association for Computational Linguistics

    work page doi:10.18653/v1/2026.gem-main.19 2026

  38. [38]

    Peiying Yu, Guoxin Chen, and Jingjing Wang. 2025. https://doi.org/10.18653/v1/2025.acl-long.853 Table-critic: A multi-agent framework for collaborative criticism and refinement in table reasoning . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17432--17451. Association for Computa...

    doi:10.18653/v1/2025.acl-long.853 2025

  39. [39]

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. https://proceedings.mlr.press/v235/yuan24d.html Self-rewarding language models . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 57905--57923. PMLR

    2024

  40. [40]

    Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. https://proceedings.iclr.cc/paper_files/paper/2024/hash/afc8b034823271816d14f7c1aefe1dff-Abstract-Conference.html Evaluating large language models at evaluating instruction following . In The Twelfth International Conference on Learning Representations. Introduces LLMBar; ar...

    Pith/arXiv arXiv 2024

  41. [41]

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Seyed Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2025. https://iclr.cc/virtual/2025/poster/30506 Generative verifiers: Reward modeling as next-token prediction . In The Thirteenth International Conference on Learning Representations. ArXiv:2408.15240

    Pith/arXiv arXiv 2025

  42. [42]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html Judging LLM -as-a-judge with MT -bench an...

    Pith/arXiv arXiv 2023

  43. [43]

    Xinyi Zheng, Douglas Burdick, Lucian Popa, Xu Zhong, and Nancy Xin Ru Wang. 2021. https://doi.org/10.1109/WACV48630.2021.00074 Global table extractor ( GTE ): A framework for joint table identification and cell structure recognition using visual context . In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 697--706

    arXiv 2021

  44. [44]

    Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. 2020. https://doi.org/10.1007/978-3-030-58589-1_34 Image-Based Table Recognition: Data, Model, and Evaluation . In Computer Vision -- ECCV 2020 , pages 564--580. Springer International Publishing. Introduces TEDS ; arXiv:1911.10683

    Pith/arXiv arXiv 2020

  45. [45]

    Chenyu Zhou. 2026. https://arxiv.org/abs/2607.05904 More convincing, not more correct: Self-play reward hacking of reference-free LLM judges . Preprint, arXiv:2607.05904

    Pith/arXiv arXiv 2026

  46. [46]

    Yitong Zhou, Mingyue Cheng, Qingyang Mao, Jiahao Wang, Feiyang Xu, and Xin Li. 2025. https://doi.org/10.24963/ijcai.2025/279 Enhancing table recognition with vision LLM s: A benchmark and neighbor-guided toolchain reasoner . In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, pages 2503--2511. International Joint...

    Pith/arXiv arXiv 2025

  47. [47]

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/7f8f73134e253845a8f82983219a8452-Abstract-Conference.html JudgeLM : Fine-tuned large language models are scalable judges . In The Thirteenth International Conference on Learning Representations. ArXiv:2310.17631

    Pith/arXiv arXiv 2025