{"id":"4d0bbffc-0a59-47d7-b7e4-476f660462d0","arxiv_id":"2506.14927","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MDBench is a synthetically generated, knowledge-guided benchmark for multi-document QA on which frontier LLMs achieve only about 60% exact match.","lead":"This paper introduces MDBench, a synthetic benchmark for evaluating how well language models reason across multiple documents. It offers a low-cost, controllable way to generate fresh reasoning tests that may resist data contamination.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline difficulty numbers rest on a GPT-4o judge that also generated and validated the examples; without independent rescoring, the ~81% accuracy / ~62% exact-match claim is not established.","rationale":"The reader's weakest assumption is that GPT-4o can reliably validate examples it generated itself, and the paper's own 87% human-validity rate supports that concern. I agree with this, but I see the same circularity as broader and more direct to the paper's central quantitative claim: GPT-4o is not only the generator and validator of the benchmark but also the judge that extracts and scores every model prediction. The paper's headline accuracy and exact-match results therefore depend on a single model family for both ground-truth construction and answer evaluation. This is a correctness risk, not an ad hominem critique: the authors did include human validation and public artifacts, which are genuinely useful, but the published numbers do not include independent scoring. The ablation on document ordering and the table-versus-document comparison are informative, yet they do not settle whether the difficulty numbers would survive human scoring. The concrete test of blind human rescoring on a stratified sample would directly determine whether the GPT-4o judge biases the leaderboard. Given the existing 13% invalidity in the auto-validated subset and the absence of scoring-quality evidence, the reader's CONDITIONAL verdict remains appropriate; I would not change it to accept or reject based on this pass. My agreement with the reader is partial because I emphasize a closely related but distinct weakness: the same model also constitutes the evaluation metric, so even the results on the human-validated subset are not yet independently verified.","tokens_in":31386,"tokens_out":4477,"duration_ms":48686,"concrete_test":"Take a stratified sample of roughly 100 model outputs from Table 3 across GPT-4o, GPT-o1, Claude, Gemini, and LLaMA and across prompt variants. Have two independent human annotators score each prediction against the gold answer using the paper's 0-10 rubric and a deterministic exact/containment check. Compare human scores with the GPT-4o judge scores using agreement metrics such as quadratic-weighted kappa and a 3-point accuracy shift threshold. If human-judge agreement is low, or if rescoring changes GPT-4o/o1 scores by more than 3-5 points relative to other models, the reported leaderboard and headline 81%/62% figures are not stable. As a secondary check, recompute the leaderboard on the 300 human-validated examples only; if the rankings change materially, the invalid auto-validated subset is driving the main conclusions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is quantitative: MDBench is hard, with the best models at roughly 81% accuracy and 62% exact-match performance. Both links in the measurement chain are GPT-4o. First, ground truth is created by GPT-4o, which edits the table, writes the question and answer, and then filters examples through an oracle self-consistency check in which the same model is given the construction history. The human audit of auto-validated examples finds only 87% validity, implying roughly 91 of the 700 machine-validated examples may have incorrect or unanswerable gold answers. Second, the reported scores are produced by GPT-4o: it extracts the final answer from each model response and then assigns a 0-10 correctness score by comparing that prediction to the gold answer. The paper does not report the threshold used for exact-match, nor does it provide any human validation of the scoring decisions. Because the same model that decides what is correct also decides which predicted answers match, the headline numbers may partially reflect stylistic compatibility with GPT-4o rather than reasoning ability. Invalid gold answers can further suppress scores on otherwise-correct outputs, making the benchmark appear harder than it is. The claims that MDBench poses a strong challenge and that GPT-4o/o1 lead the ranking are load-bearing on this unvalidated judge.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MDBench, a 1,000-example synthetic multi-document question-answering benchmark. Examples are generated from TabFact tables by using GPT-4o to apply knowledge-augmentation edits, inject cross-row dependencies, create questions and gold answers, and then map each row to a natural-language document. A further GPT-4o-based oracle self-consistency check is used to filter examples, and 300 examples are additionally human-validated. The authors evaluate several open and proprietary LLMs under zero-shot, one-shot, and chain-of-thought prompting, reporting that the best models reach roughly 60% exact-match and 82% accuracy on the document setting, with lower performance than on the intermediate tabular versions. The central claim is that MDBench poses a strong, contamination-resistant challenge for current LLMs and that the knowledge-guided pipeline enables controllable, targeted analysis of multi-document reasoning.","tokens_in":31620,"tokens_out":5440,"duration_ms":55941,"significance":"If the quality concerns are resolved, MDBench would be a worthwhile contribution. The synthetic knowledge-guided generation pipeline is cost-efficient and adaptable, the benchmark and code are publicly released, and the comparison between tabular and document reasoning plus the document-ordering ablations are informative. The design goal of producing unseen, counterfactual-tinged examples is sensible for contamination resistance. However, the quantitative claims about model difficulty rest on ground-truth labels and answer scoring that are only partially human-validated, and the paper's own reported 87% human-validity rate for machine-validated examples directly limits confidence in the headline numbers.","major_comments":[{"comment":"The released benchmark contains 700 examples whose gold answers were never human-checked, and the paper's own human audit found only 87% validity for auto-validated examples. On the reported numbers, roughly 91 of those 700 machine-validated examples could have incorrect or unanswerable gold answers. Because Table 3 and Figure 3 compute the headline difficulty claims over the full 1,000-example set, the central claim that frontier LLMs plateau around 60% exact-match is not yet established. Please report scores separately for the 300 human-validated examples and the 700 machine-validated examples, or fully human-validate the benchmark before release.","section":"§3.2 and Step 4 of §3.1"},{"comment":"The scoring procedure uses GPT-4o both to extract a predicted answer and to assign a 0-10 consistency score, but the paper never defines the exact-match threshold or reports any human agreement on scoring decisions. Since the same model family generated the gold answers, the reported exact-match and accuracy numbers could reflect GPT-4o's stylistic preferences rather than reasoning ability. Please specify how exact-match is derived from the 0-10 score, report agreement between the GPT-4o scorer and human scoring on a sample, and consider model-independent answer extraction and scoring for at least the headline results.","section":"§4 and Appendix C"},{"comment":"The characteristic-level difficulty bins are produced by GPT-4o pairwise rankings with no human validation, and the hard soft-reasoning bin contains only 45 examples. The claim of an inverse relationship between soft reasoning and explicit characteristics is therefore not supported by the evidence. Please either obtain human-validated difficulty labels for the characteristic analysis or explicitly frame the soft-reasoning result as an exploratory observation requiring larger samples.","section":"§5, Figure 4, and Table 6"}],"minor_comments":[{"comment":"The relationship between the Appendix D two-prompt validity check with a 0-5 score and the Step 4 three-variation oracle self-consistency check is unclear; please clarify whether these are sequential components of the same filter or alternative implementations, and specify how the 32% retention rate is computed.","section":"§3.1 vs. Appendix D"},{"comment":"The document-ordering ablation reports accuracy differences without confidence intervals, significance tests, or per-condition sample sizes; the phrase 'implying the existence of temporal and cross-document dependencies' is stronger than the evidence supports.","section":"Table 5"},{"comment":"The benchmark name is inconsistent ('MDBench' vs. 'MDBENCH' in the abstract and 'DMDBench' in the Appendix D heading); please standardize the capitalization and the appendix title.","section":"Throughout"},{"comment":"The statement that automated validation 'yields an overall example validity rate of 87%' is important enough that it should be stated prominently, rather than appearing only in the validation paragraph; the current phrasing in the abstract and introduction implies higher quality than the reported human audit supports.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The central difficulty claim is interesting but the benchmark's credibility depends on ground-truth validity and scoring fairness. The 87% human-validity rate for machine-validated examples is the most important fact in the paper and should be foregrounded in any revision. I would be willing to revisit after the authors either human-validate the full benchmark or restrict the headline results to the human-validated subset and provide scorer validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: MDBench is worth a look, but the headline difficulty numbers are not independently established. The same model — GPT-4o — generates the examples, validates them with an oracle self-consistency check, and scores all the models under test.\n\nWhat's actually new is the generation pipeline. Starting from tabular seed knowledge (TabFact), the authors use GPT-4o to apply structured edits that inject cross-row reasoning dependencies — multi-hop, numeric, temporal, soft reasoning, aggregation — then generate one document per row. That's a real idea: it gives you controllable, targeted construction of multi-document QA, and it's much more scalable than hand-annotation. The paper also ships 1,000 examples, code, and prompts. The document-vs-table comparison is well designed and shows a consistent, nontrivial surface-form drop. The ordering/delimiter ablation is a nice sanity check that the examples actually require cross-document tracking.\n\nThe soft spots are in the measurement chain. The oracle check gives GPT-4o the original table, edit plan, and generated table, and asks it to reproduce the answer. Human validation of auto-validated examples found only 87% validity. So out of the 700 machine-validated examples, roughly nine in a hundred may have wrong or unanswerable gold answers. That inflates difficulty and adds noise to every model score. To the authors' credit, they report this number instead of hiding it. Second, the reported EM and accuracy come from GPT-4o parsing each model's answer and assigning a 0-10 correctness score. The threshold for exact match isn't reported. Since GPT-4o also wrote the gold answers, the score may reflect stylistic compatibility as much as reasoning. The ranking of GPT-4o/o1 vs Claude/Gemini is within a couple of points, and with a single run and no confidence intervals, those differences aren't meaningful.\n\nThese are fixable issues, not fatal ones. The benchmark has value as a synthetic, contamination-resistant evaluation tool, and the human-validated 300-example subset is presumably cleaner. But the paper's central claim — that MDBench is genuinely hard for frontier models — is only as strong as the least reliable link, and that link is the GPT-4o judge.\n\nWho benefits: researchers working on LLM evaluation, synthetic data, and multi-document reasoning. I'd bring it to a reading group and I'd cite the pipeline. But I'd want the authors to release the human-validated subset separately, rescore with at least one independent judge, and report the exact-match threshold and run-to-run variance. With those changes this becomes a solid benchmark paper; without them, it's a promising prototype.","headline":"MDBench has a genuinely useful generation pipeline, but the headline difficulty numbers rest on a GPT-4o judge that validates and scores its own creations, so treat them as preliminary.","tokens_in":32156,"tokens_out":3409,"would_cite":true,"duration_ms":31264,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MDBench is a synthetic benchmark that turns structured tabular knowledge into multi-document QA examples, and it shows that even the best LLMs reach only around 62% exact-match and 81% accuracy on this task.","keywords":["multi-document reasoning","synthetic benchmark","knowledge-guided generation","table-to-text generation","LLM evaluation","chain-of-thought prompting","data contamination","question answering"],"falsifier":"Re-run the human validation protocol on a random sample of the 700 auto-validated examples; if the validity rate falls well below 87%, or if annotators can demonstrate a valid alternative answer for a sizable share, then the reported exact-match ceiling and model ranking are not reliable measurements of multi-document reasoning.","tokens_in":31167,"feed_emoji":"📊","tokens_out":11225,"duration_ms":97593,"temperature":0.7,"pith_summary":"Multi-document reasoning—answering questions that require synthesizing several texts—has few rigorous benchmarks because long inputs are expensive to annotate and public examples can be memorized by models during training. MDBench attacks both problems at once: it starts from condensed structured tabular knowledge, uses a strong language model to edit the table so that rows carry deliberate cross-document dependencies, and then generates a natural-language document for each row, producing a multi-document question-answer example. The authors release 1,000 examples and evaluate models from several families, finding that even the strongest models reach only about 81% accuracy and 62% exact-match performance. They also find that the same questions are easier when posed over the table than over the documents, and that chain-of-thought prompting helps strong models more than weak ones. If the benchmark is sound, it offers a scalable, controllable, and relatively contamination-resistant way to test multi-document reasoning.","feed_headline":"Synthetic benchmark trips top LLMs at 62% exact match","feed_subtitle":"Editing tabular data into document sets yields question sets that best models exactly match only about 62% of the time.","key_machinery":"The carrying mechanism is the table as a compressed knowledge substrate: each row will become one document, so cross-row edits in the table become cross-document dependencies in the final benchmark. Knowledge augmentation prompts demonstrate five reasoning skills (multi-hop, numeric, temporal, soft reasoning, knowledge aggregation) in simple and hard variants and provide edit plans; a generator then maps each row to fluent natural-language text. A final oracle self-consistency check runs each question under three construction-knowledge contexts—original table, edited table, document set—and rejects examples unless all three agree, which is what makes automated scaling possible.","core_discovery":"The paper's central claim is that synthetic generation guided by structured knowledge can produce a valid, challenging, and targeted multi-document reasoning benchmark. Concretely, editing seed tables to inject cross-row dependencies—multiple rows for one entity that must be added, obfuscated values defined relative to earlier rows, dates that impose temporal order—and then rendering each row as a document yields question-answer pairs whose answers cannot be found in any single document. The released 1,000-example MDBench tops out at roughly 60% exact match (best 62.1% zero-shot chain-of-thought) and roughly 81–82% accuracy, and every evaluated model drops when moving from the table version to the document version, showing that surface form itself is part of the difficulty. The generation method also supports targeted analysis: difficulty can be binned by reasoning characteristic, and ablations show models rely on document ordering and delimiters.","pith_inferences":["If the pipeline is reused with fresh seed domains and skill definitions, it can generate new benchmark versions on demand, which would make training-data contamination harder while keeping difficulty calibrated.","The sensitivity to document delimiters and ordering suggests current models may be exploiting layout cues rather than truly integrating information; a controlled study varying only those cues could quantify how much of their success is genuine reasoning.","The inverse difficulty trend for soft reasoning may be an artifact of the LLM judge's difficulty rankings rather than a property of the benchmark; constructing pairs that vary only soft-reasoning content would settle whether this is real."],"forward_implications":["Current state-of-the-art models cluster near 60% exact match on MDBench, so the benchmark leaves clear headroom for measuring future progress in multi-document reasoning.","Because the table version of the same questions is easier, MDBench can separate raw reasoning difficulty from the extra difficulty imposed by natural-language surface form.","Chain-of-thought prompting helps larger models more than smaller ones, indicating that prompting alone is not a substitute for model capacity on cross-document reasoning.","Ablations that shuffle documents or remove delimiters lower accuracy, confirming that MDBench examples require tracking cross-document dependencies rather than reading each document in isolation."],"supporting_citations":[{"why":"Supplies the structured seed tables whose rows become the documents in every MDBench example.","marker":"Chen et al., 2020"},{"why":"The model family used for knowledge augmentation, document generation, and oracle scoring; the pipeline is built on its outputs.","marker":"OpenAI et al., 2024"},{"why":"Defines chain-of-thought prompting, the main prompting intervention whose effect on model size is a headline finding.","marker":"Wei et al., 2022"},{"why":"Provides the LLaMA-3 open-weight models whose comparatively weak results ground the claim that capacity, not prompting, is the bottleneck.","marker":"Dubey et al., 2024"},{"why":"Documents benchmark contamination, the problem that motivates generating new counterfactual text instead of scraping public data.","marker":"Xu et al., 2024"},{"why":"Earlier synthetic structure-to-text benchmark that the MDBench generation method builds on and distinguishes itself from.","marker":"Sprague et al., 2023"}],"fun_headline_variants":["Synthetic benchmark exposes multi-document reasoning gap","From tables to text: LLMs drop on multi-doc QA","MDBench: LLMs only 62% exact on multi-doc reasoning","Knowledge-guided synthetic docs create hard QA sets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's quality rests on the automated consistency check, which uses the same language model that built the examples to decide whether the answers are correct; human re-checking of machine-validated examples found only 87% valid.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic benchmark exposes multi-document reasoning gap","From tables to text: LLMs drop on multi-doc QA","MDBench: LLMs only 62% exact on multi-doc reasoning","Knowledge-guided synthetic docs create hard QA sets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000419,"raw_usage":{"total_tokens":2183,"prompt_tokens":999,"completion_tokens":1184,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":1115}},"tokens_in":615,"tokens_out":1184,"duration_ms":11913,"temperature":1.0,"reasoning_tokens":1115,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:09:00.487094+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the human validation protocol on a random sample of the 700 auto-validated examples; if the validity rate falls well below 87%, or if annotators can demonstrate a valid alternative answer for a sizable share, then the reported exact-match ceiling and model ranking are not reliable measurements of multi-document reasoning.","supporting_citations":[],"review_version":1}