NEXT-EVAL: Next Evaluation of Traditional and LLM Web Data Record Extraction

Namhee Kim; Soyeon Kim; Yeonwoo Jeong

REVIEW 4 major objections 4 minor 1 cited by

NEXT-EVAL: Next Evaluation of Traditional and LLM Web Data Record Extraction

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Feeding an LLM a flat map of XPath addresses to text extracts web data records with an F1 score of 0.9567, far above HTML-based input formats.

desk verdict Useful evaluation framework, but the Flat JSON superiority claim is largely a scoring artifact — a reworked metric is needed before the numbers can be trusted. read the letter →

arxiv 2505.17125 v1 pith:T66XBGRL submitted 2025-05-21 cs.DB cs.AIcs.IR

Soyeon Kim , Namhee Kim , Yeonwoo Jeong This is my paper · ORCID

classification cs.DBcs.AIcs.IR

keywords webdatarecordextractionLLMevaluationXPathannotationsFlatJSONinputrepresentationhallucinationratebenchmarkframeworkDOMstructure

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish a reproducible way to compare traditional and LLM-based web data record extraction, and to show that input representation decides how well an LLM can segment a page into repeated records. The central empirical claim is that a Flat JSON format, where every absolute XPath is a key and the text content is the value, lets an off-the-shelf LLM reach an F1 score of 0.9567 with a hallucination rate of only 0.0305, while the same model on Slimmed HTML reaches only 0.1014 F1. The paper also contributes an evaluation framework that builds datasets from arbitrary MHTML snapshots, uses human-refined XPath annotations as ground truth, and scores predictions with structure-aware metrics that give partial credit. If correct, this means that simple, unambiguous localization of text fields is more important for LLM extraction than preserving the full DOM hierarchy.

What carries the argument

The load-bearing mechanism is the Flat JSON text map: a serialization that pairs every non-empty text node's absolute XPath with its textual content, for example "/html/body/ul/li[1]/span": "Sample Product". This representation removes the DOM hierarchy while preserving exact positional identity, so the LLM is effectively asked to select existing XPath keys rather than reconstruct structure from markup. The scoring framework then compares predicted record sets to ground-truth XPath sets using Jaccard overlap $Overlap(P_i, G_j) = |P_i \cap G_j| / |P_i \cup G_j|$, an optimal one-to-one matching via the Hungarian algorithm, and the resulting precision, recall, and F1. The hallucination rate is a per-URL binary flag for whether any predicted record is empty, which the paper uses to distinguish text hallucination (prevented by the XPath constraint) from positional hallucination.

What would settle it

Take a random sample of the 164 pages and have independent annotators label data records from scratch in XPath form, then measure agreement with the paper's ground truth; if agreement is low, the reported precision, recall, and F1 numbers are not stable. Separately, count hallucinated fields inside non-empty records, not just empty records; if that count is clearly above zero for Flat JSON, the claim of minimal hallucination needs qualification.

Watch

Extended reading notes

Core claim

The paper claims that the input format is the dominant factor in LLM-based web data record extraction, and that a flat key-value serialization of the DOM outperforms both a hierarchy-preserving JSON and slimmed HTML. Concretely, the LLM with Flat JSON input achieves precision 0.9939, recall 0.9392, and F1 0.9567, whereas Hierarchical JSON gives F1 0.4048 and Slimmed HTML gives F1 0.1014; the traditional MDR baseline sits at F1 0.0830. The authors attribute the Flat JSON advantage to unambiguous field localization: each key is an absolute XPath, so the model must output paths that exist in the cleaned DOM, which suppresses hallucinated elements. The paper further claims that a hallucination rate defined as the fraction of URLs where the model outputs at least one empty record drops to 0.0305 with Flat JSON, versus 0.9146 with Slimmed HTML and 0.5976 with Hierarchical JSON.

Load-bearing premise

The ground-truth XPath annotations are complete and correct for every page, even though they were produced by an LLM and refined by humans with no reported check for inter-annotator agreement or label quality.

Editorial extensions

If this is right

Flat JSON should become the default input serialization for LLM-based web record extraction when accuracy is the priority, since it outperforms hierarchy-preserving and HTML formats by a wide margin.
The framework lets researchers build evaluation datasets from any MHTML snapshot without redistributing crawled pages, which sidesteps licensing and robots.txt restrictions that limit previous benchmarks.
XPath-based supervision makes evaluation deterministic and DOM-grounded, so model outputs can be verified as real elements rather than plausible-sounding text.
There is a token-cost tradeoff: Flat JSON has the lowest error but the highest average token count (116,698 tokens) versus Hierarchical JSON (34,107) and Slimmed HTML (86,084), so cost-sensitive deployments may prefer a cheaper format despite worse accuracy.
The synthetic dataset created by transforming DOM structures and modifying content enables public benchmarking even for pages whose original text cannot be redistributed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported advantages could shift if the ground-truth labels are noisy: the annotations are LLM-proposed and human-refined, but no inter-annotator reliability is reported, so an independent re-labeling of a sample of pages would test whether the Flat JSON edge is real or an artifact of label noise.
The hallucination rate only flags records that are entirely empty, so a model that invents extra fields inside an otherwise valid record would not be penalized; the claimed 3 percent may understate positional hallucination.
The Flat JSON effect may partly reflect the model's pretraining on JSON key-value structures rather than genuine DOM reasoning; testing open-weight LLMs that have seen less structured serialization would separate these explanations.
The paper does not break down results by record layout type, so it is unclear whether Flat JSON's advantage persists for nested or non-contiguous records, where the lost hierarchy might be needed for correct grouping.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing you should know: this is a useful evaluation artefact with a compromised headline result. The framework — generating datasets from MHTML snapshots, annotating with XPaths, scoring with structure-aware overlap — is genuinely reusable, and the comparison of Slimmed HTML, Hierarchical JSON, and Flat JSON as LLM inputs is new. The synthetic dataset (164 pages, 12k+ records) is a real contribution, assuming they actually release it.

What the paper does well: it formalizes data records as sets of XPath expressions, prevents text hallucination by requiring predicted paths to have text present in the cleaned DOM, and uses Hungarian matching to give partial credit. That is a sensible baseline for the community.

The soft spots are serious, though. The F1 gap between Flat JSON (0.9567) and Hierarchical JSON (0.4048) is largely a scoring artifact. The metric compares sets of XPath strings, not DOM nodes. A DOM node has many valid XPath spellings (positional vs. attribute-based, indexed vs. unindexed siblings). Flat JSON supplies the exact absolute-positional XPath strings used as ground truth as its keys, so the model copies them verbatim. Slimmed HTML and Hierarchical JSON force the model to re-serialize those paths from markup or nested objects; any equivalent but differently spelled path is scored as a miss. So the reported gap conflates record grouping quality with XPath serialization difficulty. The circularity concern is real.

The hallucination metric also understates hallucination: it only flags empty records, so a predicted record with a valid-looking but wrong XPath (whose text exists somewhere on the page) is not penalized. The paper says it allows only assessment of positional hallucination, but the metric doesn't actually measure that — it measures whether a record is empty.

The paper claims multiple seeds and averaging in the checklist, but no error bars or per-seed numbers appear anywhere. Without them, a 0.9567 vs 0.4048 gap is hard to trust, especially with a temperature-1.0 API. Also, ground-truth labels were human-refined LLM suggestions, but no inter-annotator reliability is reported; if labels are incomplete or wrong, every number shifts.

Who gets value: anyone building or comparing web extraction systems will want the framework and dataset, once released. But the headline "Flat JSON is best" should not be cited without re-running the evaluation with canonicalized node identity and a proper hallucination metric.

Recommendation: send it to peer review. The framework deserves referee time. The evaluation protocol needs real revision before acceptance: canonicalize predicted and ground-truth XPaths to DOM node identity, fix the hallucination metric, and report variance. With those changes, the claim about Flat JSON might survive — or might not. Right now, it is not established.

Referee Report

4 major / 4 minor

Summary. The paper introduces NEXT-EVAL, a framework for evaluating web data record extraction methods on MHTML snapshots. It formalizes data records as sets of XPath expressions, constructs ground-truth labels through LLM-proposed and human-refined XPath annotations, and scores predictions with a Jaccard-based partial-credit metric. The framework compares traditional methods such as MDR with LLM-based extraction using three input representations: Slimmed HTML, Hierarchical JSON, and Flat JSON. The headline empirical claim, stated in the abstract and in Section 5.3, is that Flat JSON input enables Gemini-2.5-pro-preview to achieve an F1 score of 0.9567 with a hallucination rate of 0.0305, substantially outperforming Slimmed HTML (F1 0.1014) and Hierarchical JSON (F1 0.4048). The paper also contributes a dataset of 164 crawled web pages with 12,278 annotated records and a synthetic dataset.

Significance. If the central claim is established, the framework would be a useful step toward standardized benchmarking of web record extraction, addressing a real gap in the literature. The paper contributes a clean formal definition of records as XPath sets, a reproducible dataset-construction pipeline, a public synthetic dataset, and a structure-aware scoring procedure with partial credit. These are genuine strengths. However, the current evaluation does not yet support the claim that Flat JSON yields superior extraction quality: the scoring metric compares raw XPath strings, and Flat JSON supplies exactly those strings as input keys. The paper also defines hallucination in a way that ignores fabricated but non-empty XPath predictions. The framework is promising, but the headline comparison needs to be re-run under a scorer that is invariant to XPath serialization and that penalizes invalid paths.

major comments (4)

[Section 4.4, Eq. (1); Fig. 1(d); Table 2] The central F1 comparison is computed over raw XPath strings without canonicalizing predicted and ground-truth paths to a common DOM node identity. Eq. (1) defines overlap as the Jaccard similarity of XPath string sets. A DOM node admits many equivalent XPath spellings, and different input formats give the model very different access to those spellings. Flat JSON (Fig. 1d) presents the exact absolute XPath strings used as ground-truth labels as input keys, so the model can copy them verbatim; Slimmed HTML and Hierarchical JSON force the model to regenerate the same strings from markup or nested fragments, so any equivalent but differently spelled path is scored as a miss. The F1 gap of 0.9567 vs. 0.4048 therefore conflates record grouping quality with XPath-serialization difficulty. The claim that Flat JSON produces 'superior extraction accuracy' is not yet established as a claim about extraction quality. Please re-score all predictions by resolving predicted and ground-truth XPaths to DOM node IDs before computing the Jaccard overlap, and report the resulting F1 values.
[Section 4.4, Hallucination Rate; Section 4.3] The hallucination metric as defined only flags pages where the model predicts at least one empty record (a record containing no XPaths). It does not penalize non-empty records whose XPaths do not correspond to real DOM nodes. The paper asserts in Section 4.3 that 'every model output paths are drawn from the cleaned DOM' and that 'the model cannot hallucinate elements that do not exist,' but no procedure is described for validating each predicted path against the DOM or for counting invalid paths as hallucinations. Depending on the implementation, such fabricated paths are either silently discarded (which can inflate precision) or counted as negatively matching (which would reduce precision), but in neither case are they reflected in the reported Hallucination Rate. Please define a hallucination metric that counts predicted XPaths that do not resolve to a node in the cleaned DOM, and report that rate separately for each input format.
[Section 5.3, Table 2; NeurIPS Checklist item 7] Table 2 reports single precision, recall, F1, and hallucination-rate numbers for each method with no error bars, confidence intervals, or significance tests. This contradicts the checklist claim that 'we use multiple seeds for the zero-shot LLM and average the results to ensure statistical reliability.' Since Gemini-2.5-pro-preview is evaluated at temperature 1.0, run-to-run variance is expected, and the lack of any variance information makes it impossible to assess whether the large gaps in Table 2 are stable. Please report the number of runs, the mean and standard deviation (or equivalent) for each metric, and, where appropriate, a paired significance test across input formats.
[Section 4.2] Ground-truth records are constructed by having LLMs propose candidate repetitive blocks and human annotators 'review and refine' them, but the paper reports no inter-annotator reliability, no label audit, and no details about the number of annotators or the refinement protocol. Every precision, recall, and F1 value in Table 2 is computed against these labels, so label errors could shift the reported ranking, including the Flat JSON advantage. Please report the annotation protocol in detail, provide agreement statistics on a held-out subset of pages (e.g., pairwise overlap between independent annotators), and describe how ambiguous or nested record boundaries were resolved.

minor comments (4)

[Abstract and Section 4.4] The abstract says the scoring 'specifically prevents text hallucination and allowing only for the assessment of positional hallucination,' but the Hallucination Rate defined in Section 4.4 only counts empty records and says nothing about invalid XPaths; this wording should be aligned with the actual metric.
[Table 2] The row for MDR lists 'Full / Slimmed HTML' as the input type; it is unclear whether the reported numbers come from Full HTML, Slimmed HTML, both, or a union, and this should be specified explicitly.
[Section 5, Checklist items 4 and 5] The paper and checklist state that crawling code and dataset construction instructions are open access, but no URL or repository is given in the main text; please provide concrete links in the final version.
[Section 4.2 and Fig. 1] The paper should state the exact XPath serialization convention used for ground-truth labels (e.g., positional indexes as in Fig. 1d versus attribute predicates), since the scoring and the Flat JSON input both depend on this convention.

Circularity Check

1 steps flagged · score 6.0 of 10

Flat JSON's reported F1 advantage is partly built into the evaluation: the input supplies the exact XPath strings that Eq. (1) scores against ground-truth labels.

self definitional [Section 4.3 (Flat JSON input), Section 4.4 Eq. (1), Section 5.3 Table 2]
"Second, we construct a Flat JSON (Figure 1d), where each key is an absolute XPath and the value is the corresponding textual content. Overlap(Pi, Gj) = |Pi ∩ Gj| / |Pi ∪ Gj| (1)"

Ground-truth labels are sets of XPath expressions (Section 4.2), and Eq. (1) scores predictions by exact Jaccard intersection of raw XPath strings. Flat JSON's keys are exactly those XPath strings, so by construction a model that copies keys into its predicted records emits strings from the same vocabulary as the labels; every correct membership is counted as an exact string match. In Slimmed HTML and Hierarchical JSON, the model must regenerate XPath expressions from markup or nested keys, and a correct node addressed with an equivalent but differently spelled XPath is scored as a miss because no canonicalization to DOM node identity is described before Eq. (1).

full rationale

The load-bearing circular step is the Flat JSON condition itself: its input representation and the ground-truth/scoring representation are the same XPath strings, and Eq. (1) compares strings without canonicalizing node identity. This makes the headline 'Flat JSON enables superior extraction accuracy' substantially self-fulfilling, because the hard subproblem of deriving canonical XPath addresses from raw HTML is eliminated in that condition while retained in the others. The hallucination-rate definition (only empty records count) further rewards the format that supplies the complete key set, though I treat that as a supporting weakness rather than a separate circular step. The paper does not rely on self-citations or imported uniqueness theorems, and the MDR baseline, token statistics, and dataset description are externally checkable. The limitations section acknowledges token-budget and attribute-slimming issues but does not acknowledge the coordinate-system confound. Because the LLM must still discover the record partition, the result is not wholly forced; however, the reported comparison is partially circular, so a score of 6 is appropriate.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; the framework uses fixed design choices such as Jaccard similarity, Hungarian matching, and a binary hallucination flag rather than values fitted to data. No new physical or conceptual entities are introduced. The main unstated support comes from the assumed correctness of the human-refined labels and the adequacy of the Jaccard/Hungarian metric for the record extraction task.

assumptions (3)

domain assumption Human-refined XPath annotations are assumed to be correct and complete ground truth for the data records on each page.
Section 4.2 describes LLM-generated candidate blocks that human annotators refine, but no inter-annotator agreement or quality metric is reported. Label errors directly bias all reported F1 scores and the Flat JSON advantage.
domain assumption Jaccard overlap with Hungarian one-to-one matching is a fair record-level similarity measure.
Section 4.4 defines the scoring. It assumes one-to-one matches and treats set overlap as proportional credit, which may favor over-segmented or under-segmented predictions and does not penalize all forms of hallucination.
ad hoc to paper A hallucination event is defined as predicting at least one empty record per URL.
Section 4.4. This binary page-level definition ignores invalid XPath predictions that do not correspond to any DOM node, so the reported hallucination rates understate real fabrication of nonexistent record fields.

how reviews work

0 comments

Cite this review

Pith. "Pith review of NEXT-EVAL: Next Evaluation of Traditional and LLM Web Data Record Extraction." pith.science (2026). https://pith.science/paper/T66XBGRL

@misc{pith2026250517125,
  author       = {Pith},
  title        = {Pith review of: NEXT-EVAL: Next Evaluation of Traditional and LLM Web Data Record Extraction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T66XBGRL}},
  note         = {Machine review of arXiv:2505.17125}
}

read the original abstract

Effective evaluation of web data record extraction methods is crucial, yet hampered by static, domain-specific benchmarks and opaque scoring practices. This makes fair comparison between traditional algorithmic techniques, which rely on structural heuristics, and Large Language Model (LLM)-based approaches, offering zero-shot extraction across diverse layouts, particularly challenging. To overcome these limitations, we introduce a concrete evaluation framework. Our framework systematically generates evaluation datasets from arbitrary MHTML snapshots, annotates XPath-based supervision labels, and employs structure-aware metrics for consistent scoring, specifically preventing text hallucination and allowing only for the assessment of positional hallucination. It also incorporates preprocessing strategies to optimize input for LLMs while preserving DOM semantics: HTML slimming, Hierarchical JSON, and Flat JSON. Additionally, we created a publicly available synthetic dataset by transforming DOM structures and modifying content. We benchmark deterministic heuristic algorithms and off-the-shelf LLMs across these multiple input formats. Our benchmarking shows that Flat JSON input enables LLMs to achieve superior extraction accuracy (F1 score of 0.9567) and minimal hallucination compared to other input formats like Slimmed HTML and Hierarchical JSON. We establish a standardized foundation for rigorous benchmarking, paving the way for the next principled advancements in web data record extraction.

Figures

Figures reproduced from arXiv: 2505.17125 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers
cs.CY 2026-07 conditional novelty 6.0 of 10

Under controlled tests, several major AI assistants (Gemini, Grok, DeepSeek, Qwen) accessed robots.txt-disallowed web pages without ever requesting the robots.txt file, while Claude and Mistral generally complied.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · cited by 1 Pith paper

[1]

These sections accurately cover the paper’s goal of providing a standard way to evaluate and compare web record extraction systems

Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and introduction clearly state our main contributions: a repro- ducible framework for creating web data extraction evaluation datasets from MHTMLs, a consistent scoring system for tradi...

work page
[2]

Limitations

Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Yes, the paper dedicates a section in the conclusion to discuss limitations and future work. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those a...

work page
[3]

We do, however, clearly define our concepts

Theory assumptions and proofs 11 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: Our paper focuses on an evaluation framework and empirical results, not new theoretical proofs. We do, however, clearly define our concepts. Guidelines: • The answer NA m...

work page
[4]

Guidelines: • The answer NA means that the paper does not include experiments

Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Yes, we provide the necess...

work page
[5]

Guidelines: • The answer NA means that paper does not include experiments requiring code

Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Yes, we offer open access to our crawling code and the methods for building evaluation datasets from MHTMLs...

work page
[6]

A list of websites is in the appendix, and prompting strategies are described

Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Yes, we detail our experimental setup, including the LLM (Gemini-2.5-pro- preview), dataset sources, MHTML fo...

work page
[7]

Guidelines: 13 • The answer NA means that the paper does not include experiments

Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: Yes, we use multiple seeds for the zero-shot LLM and average the results to ensure statistical reliability. Guidelines: 13 • The a...

work page
[8]

Guidelines: • The answer NA means that the paper does not include experiments

Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Yes, we specify that experiments used the Gemini-2.5-pro-preview LLM (zero-shot) and report average tok...

work page

Show all 16 references

[9]

It doesn’t involve human experiments or sensitive data generation that would pose ethical issues beyond responsible web data and LLM use

Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: Yes, our research on a framework for evaluating web data extraction aligns with...
[10]

Guidelines: • The answer NA means that there is no societal impact of the work performed

Broader impacts 14 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Yes, the paper addresses both positive and negative societal impacts of this work. Guidelines: • The a...
[11]

The dataset is built with a provided framework on public web pages (MHTMLs), not a high-risk asset needing special safeguards beyond ethical data practices

Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our paper doesn...
[12]

Web sources (public sites, listed in appendix), the MDR baseline, and the Gemini-2.5-pro-preview LLM (zero-shot) are cited

Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] 15 Justification: Yes, we credi...
[13]

Guidelines: • The answer NA means that the paper does not release new assets

New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Yes, new assets (the dataset framework, annotation methodology, and MHTML dataset characteristics) are well-document...
[14]

Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...
[15]

Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

Institutional review board (IRB) approvals or equivalent for research with human subjects 16 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approv...
[16]

Answer: [Yes] Justification: We used an LLM only for writing and editing, not for the core methodology

Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

These sections accurately cover the paper’s goal of providing a standard way to evaluate and compare web record extraction systems

Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and introduction clearly state our main contributions: a repro- ducible framework for creating web data extraction evaluation datasets from MHTMLs, a consistent scoring system for tradi...

work page

[2] [2]

Limitations

Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Yes, the paper dedicates a section in the conclusion to discuss limitations and future work. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those a...

work page

[3] [3]

We do, however, clearly define our concepts

Theory assumptions and proofs 11 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: Our paper focuses on an evaluation framework and empirical results, not new theoretical proofs. We do, however, clearly define our concepts. Guidelines: • The answer NA m...

work page

[4] [4]

Guidelines: • The answer NA means that the paper does not include experiments

Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Yes, we provide the necess...

work page

[5] [5]

Guidelines: • The answer NA means that paper does not include experiments requiring code

Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Yes, we offer open access to our crawling code and the methods for building evaluation datasets from MHTMLs...

work page

[6] [6]

A list of websites is in the appendix, and prompting strategies are described

Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Yes, we detail our experimental setup, including the LLM (Gemini-2.5-pro- preview), dataset sources, MHTML fo...

work page

[7] [7]

Guidelines: 13 • The answer NA means that the paper does not include experiments

Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: Yes, we use multiple seeds for the zero-shot LLM and average the results to ensure statistical reliability. Guidelines: 13 • The a...

work page

[8] [8]

Guidelines: • The answer NA means that the paper does not include experiments

Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Yes, we specify that experiments used the Gemini-2.5-pro-preview LLM (zero-shot) and report average tok...

work page