arXiv2Table: Toward Realistic Benchmarking and Evaluation for LLM-Based Literature-Review Table Generation

Benjamin Van Durme; Daniel Khashabi; Jiefu Ou; Weiqi Wang; Yangqiu Song

arxiv: 2504.10284 · v5 · submitted 2025-04-14 · 💻 cs.CL

arXiv2Table: Toward Realistic Benchmarking and Evaluation for LLM-Based Literature-Review Table Generation

Weiqi Wang , Jiefu Ou , Yangqiu Song , Benjamin Van Durme , Daniel Khashabi This is my paper

Pith reviewed 2026-05-22 20:06 UTC · model grok-4.3

classification 💻 cs.CL

keywords evaluationarxiv2tablegenerationgoldtablesuserdemandshuman

0 comments

The pith

arXiv2Table supplies a large benchmark, realistic user-query simulation, distractor papers, and a decomposed utility evaluation to test LLM generation of literature-review tables.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Literature review tables help researchers compare many papers at once. Earlier AI systems for making these tables were often tested in easy settings where they already knew the exact columns or answers they should produce. This work changes the test by creating user requests that do not reveal the desired table structure or content. It also adds papers that are related but off-topic, checked by people, to mimic noisy search results. A new scoring method looks at whether the table covers the needed topics, gets individual facts right, and shows correct relationships between papers. The authors release a dataset of 1,957 tables and an iterative process where the AI refines both which papers to include and what columns to use over several rounds. Experiments indicate the new method beats basic approaches, yet overall performance stays low, showing the task remains difficult.

Core claim

The arXiv2Table benchmark, schema-agnostic user demands, human-verified distractors, and utilization-oriented evaluation (schema coverage, unary fidelity, pairwise consistency plus two-way QA) together enable more realistic assessment of LLM-based literature-review table generation, with the iterative batch method showing consistent gains over baselines.

Load-bearing premise

The simulated well-specified yet schema-agnostic user demands and the human-verified semantically related distractor papers are assumed to faithfully represent real-world information needs and retrieval noise encountered when users request literature review tables.

read the original abstract

Literature review tables are essential for summarizing and comparing collections of scientific papers. In this paper, we study the automatic generation of such tables from a pool of papers to satisfy a user's information need. Building on recent work (Newman et al., 2024), we move beyond oracle settings by (i) simulating well-specified yet schema-agnostic user demands that avoid leaking gold column names or values, (ii) explicitly modeling retrieval noise via semantically related but out-of-scope distractor papers verified by human annotators, and (iii) introducing a lightweight, annotation-free, utilization-oriented evaluation that decomposes utility into schema coverage, unary cell fidelity, and pairwise relational consistency, while measuring paper selection through a two-way QA procedure (gold to system and system to gold) with recall, precision, and F1. To support reproducible evaluation, we introduce arXiv2Table, a benchmark of 1,957 tables referencing 7,158 papers, with human-verified distractors and rewritten, schema-agnostic user demands. We also develop an iterative, batch-based generation method that co-refines paper filtering and schema over multiple rounds. We validate the evaluation protocol with human audits and cross-evaluator checks. Extensive experiments show that our method consistently improves over strong baselines, while absolute scores remain modest, underscoring the task's difficulty. Our data and code is available at https://github.com/JHU-CLSP/arXiv2Table.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This adds a public benchmark and decomposed metrics for LLM paper-table generation with some realism upgrades, but the simulations of user needs and distractors lack external validation against real queries.

read the letter

This paper's main contribution is a benchmark that tries to make LLM-based literature table generation evaluation more realistic by simulating user queries without schema leaks and adding human-verified distractors. It also provides a decomposed evaluation metric and an iterative generation method that shows gains. What stands out is the release of arXiv2Table with 1957 tables and 7158 papers, along with the protocol for schema-agnostic demands and distractor verification. The iterative batch method refines paper selection and schema over rounds, and experiments indicate it outperforms baselines consistently. Human audits back up the evaluation setup, and the modest absolute performance is a fair note on the task's difficulty. The public data and code are a plus for reproducibility. The weaker part is the claim to greater realism. The simulated queries and distractors are internally consistent and human-checked, but there's no evidence they match real researcher queries or retrieval noise from actual systems. Without that external check, the more realistic framing is more aspirational than demonstrated. The metric definitions seem solid on paper, but full implementation details would help confirm no circularity. This work is aimed at researchers developing or testing LLMs for scientific summarization tasks. Anyone building benchmarks in this area would find the dataset and evaluation approach worth looking at. I would send it to peer review. The dataset and protocol are concrete enough to merit referee feedback, even with the open question on how well the simulations generalize.

Referee Report

1 major / 2 minor

Summary. The paper introduces the arXiv2Table benchmark consisting of 1,957 tables referencing 7,158 papers for evaluating LLM-based literature-review table generation. It moves beyond oracle settings by simulating well-specified yet schema-agnostic user demands (rewritten to avoid leaking gold column names/values), explicitly modeling retrieval noise with human-verified semantically related distractor papers, and proposing a lightweight utilization-oriented evaluation that decomposes utility into schema coverage, unary cell fidelity, pairwise relational consistency, and two-way QA (gold-to-system and system-to-gold) with recall/precision/F1. An iterative batch-based generation method is developed that co-refines paper filtering and schema over multiple rounds; experiments show consistent gains over baselines, with human audits validating the protocol, though absolute scores remain modest.

Significance. If the construction holds, the work supplies an open, reproducible benchmark (data and code released) that enables more realistic assessment of practical table-generation utility than prior oracle settings. The human-verified distractors, annotation-free evaluation, and cross-evaluator checks are concrete strengths that could help standardize evaluation in this area while underscoring remaining task difficulty.

major comments (1)

[Abstract / Benchmark Construction] Abstract and benchmark-construction description: the claim that the benchmark enables 'more realistic assessment' relative to oracle settings rests on the assumption that the simulated schema-agnostic demands and human-verified distractors faithfully approximate real researcher information needs and retrieval noise. No comparison to actual query logs, user studies, or observed search-engine outputs is reported, leaving external validity untested and load-bearing for the central contribution.

minor comments (2)

[Evaluation Protocol] The two-way QA procedure is described at a high level; explicit formulas or pseudocode for how system-to-gold precision is computed when system tables contain extra columns would aid reproducibility.
[Abstract] Table counts and paper counts are given, but the abstract does not state how many distinct schemas or domains are represented; adding this statistic would clarify coverage.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the careful reading and constructive feedback. We respond to the single major comment below and indicate planned revisions.

read point-by-point responses

Referee: [Abstract / Benchmark Construction] Abstract and benchmark-construction description: the claim that the benchmark enables 'more realistic assessment' relative to oracle settings rests on the assumption that the simulated schema-agnostic demands and human-verified distractors faithfully approximate real researcher information needs and retrieval noise. No comparison to actual query logs, user studies, or observed search-engine outputs is reported, leaving external validity untested and load-bearing for the central contribution.

Authors: We agree that the absence of direct comparisons to real query logs or user studies leaves the external validity of our simulation as an assumption rather than an empirically validated claim. Our design decisions—rewriting user demands to be schema-agnostic and avoid leaking gold column names or values, together with human-verified semantically related distractors—were chosen specifically to move beyond oracle settings that presuppose perfect retrieval and known schemas. These choices are documented in Sections 3.2 and 3.3 and are supported by the human audits reported in Section 5.1. Nevertheless, we recognize that stronger external validation would require access to proprietary search logs or new user studies, both of which were outside the scope of the present work. We will therefore (i) qualify the phrasing in the abstract and introduction to describe the benchmark as “a step toward more realistic assessment” and (ii) add a dedicated paragraph in the Limitations section that explicitly discusses the modeling assumptions, the rationale for the chosen simulation, and the value of future log-based or user-study validation. revision: partial

Circularity Check

0 steps flagged

No circularity: new benchmark and evaluation protocol are independently constructed

full rationale

The paper introduces arXiv2Table (1,957 tables, 7,158 papers) and a utilization-oriented evaluation (schema coverage, unary fidelity, pairwise consistency, two-way QA) by simulating schema-agnostic demands and adding human-verified distractors, then testing an iterative batch method. These steps rely on explicit data creation and human annotation rather than any equation, fitted parameter, or self-citation chain that reduces the reported gains to quantities already present in the inputs. The single external citation to Newman et al. (2024) merely identifies the oracle baseline being extended and carries no load-bearing uniqueness claim. No self-definitional, fitted-input, or renaming patterns appear.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work rests on standard NLP benchmarking assumptions about the validity of human annotation for distractors and the representativeness of simulated queries; no free parameters or invented entities are introduced in the abstract.

axioms (1)

domain assumption Human annotators can reliably identify semantically related but out-of-scope distractor papers.
The abstract states that distractors are 'verified by human annotators' without further qualification.

pith-pipeline@v0.9.0 · 5806 in / 1294 out tokens · 70120 ms · 2026-05-22T20:06:38.140084+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator
cs.DL 2025-07 unverdicted novelty 4.0

The paper proposes a four-role framework for LLMs in scientific innovation and reviews methods, benchmarks, and limitations across Assistant, Collaborator, Scientist, and Evaluator roles.