Pith. sign in

REVIEW 3 major objections 4 minor 34 references

MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MDBench is a synthetic benchmark that turns structured tabular knowledge into multi-document QA examples, and it shows that even the best LLMs reach only around 62% exact-match and 81% accuracy on this task.

desk verdict MDBench has a genuinely useful generation pipeline, but the headline difficulty numbers rest on a GPT-4o judge that validates and scores its own creations, so treat them as preliminary. read the letter →

arxiv 2506.14927 v1 pith:LON2GVRI submitted 2025-06-17 cs.CL cs.AI

Joseph J. Peper , Wenzhao Qiu , Ali Payani , Lu Wang This is my paper
classification cs.CLcs.AI
keywords multi-documentreasoningsyntheticbenchmarkknowledge-guidedgenerationtable-to-textLLMevaluationchain-of-thoughtpromptingdatacontaminationquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-document reasoning—answering questions that require synthesizing several texts—has few rigorous benchmarks because long inputs are expensive to annotate and public examples can be memorized by models during training. MDBench attacks both problems at once: it starts from condensed structured tabular knowledge, uses a strong language model to edit the table so that rows carry deliberate cross-document dependencies, and then generates a natural-language document for each row, producing a multi-document question-answer example. The authors release 1,000 examples and evaluate models from several families, finding that even the strongest models reach only about 81% accuracy and 62% exact-match performance. They also find that the same questions are easier when posed over the table than over the documents, and that chain-of-thought prompting helps strong models more than weak ones. If the benchmark is sound, it offers a scalable, controllable, and relatively contamination-resistant way to test multi-document reasoning.

What carries the argument

The carrying mechanism is the table as a compressed knowledge substrate: each row will become one document, so cross-row edits in the table become cross-document dependencies in the final benchmark. Knowledge augmentation prompts demonstrate five reasoning skills (multi-hop, numeric, temporal, soft reasoning, knowledge aggregation) in simple and hard variants and provide edit plans; a generator then maps each row to fluent natural-language text. A final oracle self-consistency check runs each question under three construction-knowledge contexts—original table, edited table, document set—and rejects examples unless all three agree, which is what makes automated scaling possible.

What would settle it

Re-run the human validation protocol on a random sample of the 700 auto-validated examples; if the validity rate falls well below 87%, or if annotators can demonstrate a valid alternative answer for a sizable share, then the reported exact-match ceiling and model ranking are not reliable measurements of multi-document reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that synthetic generation guided by structured knowledge can produce a valid, challenging, and targeted multi-document reasoning benchmark. Concretely, editing seed tables to inject cross-row dependencies—multiple rows for one entity that must be added, obfuscated values defined relative to earlier rows, dates that impose temporal order—and then rendering each row as a document yields question-answer pairs whose answers cannot be found in any single document. The released 1,000-example MDBench tops out at roughly 60% exact match (best 62.1% zero-shot chain-of-thought) and roughly 81–82% accuracy, and every evaluated model drops when moving from the table version to the document version, showing that surface form itself is part of the difficulty. The generation method also supports targeted analysis: difficulty can be binned by reasoning characteristic, and ablations show models rely on document ordering and delimiters.

Load-bearing premise

The benchmark's quality rests on the automated consistency check, which uses the same language model that built the examples to decide whether the answers are correct; human re-checking of machine-validated examples found only 87% valid.

Editorial extensions

If this is right

  • Current state-of-the-art models cluster near 60% exact match on MDBench, so the benchmark leaves clear headroom for measuring future progress in multi-document reasoning.
  • Because the table version of the same questions is easier, MDBench can separate raw reasoning difficulty from the extra difficulty imposed by natural-language surface form.
  • Chain-of-thought prompting helps larger models more than smaller ones, indicating that prompting alone is not a substitute for model capacity on cross-document reasoning.
  • Ablations that shuffle documents or remove delimiters lower accuracy, confirming that MDBench examples require tracking cross-document dependencies rather than reading each document in isolation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the pipeline is reused with fresh seed domains and skill definitions, it can generate new benchmark versions on demand, which would make training-data contamination harder while keeping difficulty calibrated.
  • The sensitivity to document delimiters and ordering suggests current models may be exploiting layout cues rather than truly integrating information; a controlled study varying only those cues could quantify how much of their success is genuine reasoning.
  • The inverse difficulty trend for soft reasoning may be an artifact of the LLM judge's difficulty rankings rather than a property of the benchmark; constructing pairs that vary only soft-reasoning content would settle whether this is real.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces MDBench, a 1,000-example synthetic multi-document question-answering benchmark. Examples are generated from TabFact tables by using GPT-4o to apply knowledge-augmentation edits, inject cross-row dependencies, create questions and gold answers, and then map each row to a natural-language document. A further GPT-4o-based oracle self-consistency check is used to filter examples, and 300 examples are additionally human-validated. The authors evaluate several open and proprietary LLMs under zero-shot, one-shot, and chain-of-thought prompting, reporting that the best models reach roughly 60% exact-match and 82% accuracy on the document setting, with lower performance than on the intermediate tabular versions. The central claim is that MDBench poses a strong, contamination-resistant challenge for current LLMs and that the knowledge-guided pipeline enables controllable, targeted analysis of multi-document reasoning.

Significance. If the quality concerns are resolved, MDBench would be a worthwhile contribution. The synthetic knowledge-guided generation pipeline is cost-efficient and adaptable, the benchmark and code are publicly released, and the comparison between tabular and document reasoning plus the document-ordering ablations are informative. The design goal of producing unseen, counterfactual-tinged examples is sensible for contamination resistance. However, the quantitative claims about model difficulty rest on ground-truth labels and answer scoring that are only partially human-validated, and the paper's own reported 87% human-validity rate for machine-validated examples directly limits confidence in the headline numbers.

major comments (3)
  1. [§3.2 and Step 4 of §3.1] The released benchmark contains 700 examples whose gold answers were never human-checked, and the paper's own human audit found only 87% validity for auto-validated examples. On the reported numbers, roughly 91 of those 700 machine-validated examples could have incorrect or unanswerable gold answers. Because Table 3 and Figure 3 compute the headline difficulty claims over the full 1,000-example set, the central claim that frontier LLMs plateau around 60% exact-match is not yet established. Please report scores separately for the 300 human-validated examples and the 700 machine-validated examples, or fully human-validate the benchmark before release.
  2. [§4 and Appendix C] The scoring procedure uses GPT-4o both to extract a predicted answer and to assign a 0-10 consistency score, but the paper never defines the exact-match threshold or reports any human agreement on scoring decisions. Since the same model family generated the gold answers, the reported exact-match and accuracy numbers could reflect GPT-4o's stylistic preferences rather than reasoning ability. Please specify how exact-match is derived from the 0-10 score, report agreement between the GPT-4o scorer and human scoring on a sample, and consider model-independent answer extraction and scoring for at least the headline results.
  3. [§5, Figure 4, and Table 6] The characteristic-level difficulty bins are produced by GPT-4o pairwise rankings with no human validation, and the hard soft-reasoning bin contains only 45 examples. The claim of an inverse relationship between soft reasoning and explicit characteristics is therefore not supported by the evidence. Please either obtain human-validated difficulty labels for the characteristic analysis or explicitly frame the soft-reasoning result as an exploratory observation requiring larger samples.
minor comments (4)
  1. [§3.1 vs. Appendix D] The relationship between the Appendix D two-prompt validity check with a 0-5 score and the Step 4 three-variation oracle self-consistency check is unclear; please clarify whether these are sequential components of the same filter or alternative implementations, and specify how the 32% retention rate is computed.
  2. [Table 5] The document-ordering ablation reports accuracy differences without confidence intervals, significance tests, or per-condition sample sizes; the phrase 'implying the existence of temporal and cross-document dependencies' is stronger than the evidence supports.
  3. [Throughout] The benchmark name is inconsistent ('MDBench' vs. 'MDBENCH' in the abstract and 'DMDBench' in the Appendix D heading); please standardize the capitalization and the appendix title.
  4. [§3.2] The statement that automated validation 'yields an overall example validity rate of 87%' is important enough that it should be stated prominently, rather than appearing only in the validation paragraph; the current phrasing in the abstract and introduction implies higher quality than the reported human audit supports.

Circularity Check

1 steps flagged · score 3.0 of 10

GPT-4o generates, validates, and scores the same examples, making 'correct' partially self-defined; human validation covers only 300 of 1,000 examples.

  1. self definitional [Sec. 3.1 (Step 4, Automated Quality Validation) and Sec. 4 (Experimental Setup)]
    "Specifically, for each QA example, we prompt GPT-4o under three variations of oracle knowledge (original table and edit plan) and context (generated table and generated document set)... An example is rejected if any of the three variations yield an answer which does not match the generated ground-truth. ... To evaluate on the QA task, we use GPT-4o as a reference-based scorer, first parsing the final answer from each output, then comparing the similarity of the predicted answer with the ground-truth answer."

    Correctness is defined by GPT-4o in both construction and measurement. GPT-4o generates the augmented table, the question, and the ground-truth answer; the oracle filter keeps only examples where GPT-4o, prompted with the construction history, reproduces that same answer; and the final scorer is again GPT-4o comparing model predictions with these self-generated ground truths. For the 700 machine-validated examples there is no human anchor, so 'valid' and 'correct' are, by construction, 'consistent with GPT-4o's own generated answer'. Model scores therefore measure agreement with GPT-4o's stylistic and reasoning conventions as much as independent correctness.

full rationale

Most of the paper is an empirical benchmark study, not a derivation, so there is no fitted-parameter or self-citation chain that forces the conclusions. The central difficulty result is an external measurement of model performance on generated documents. However, one load-bearing step is partially circular: the same model (GPT-4o) generates the ground-truth answers, validates them through an oracle self-consistency filter, and then scores every model prediction against those answers. For the 700 machine-validated examples, this makes 'correct' equivalent to 'GPT-4o-consistent' by construction, which can inflate scores for models whose output style matches GPT-4o and deflate scores for equally correct answers phrased differently. The paper's honest reporting of the 87% human validity rate and the 300 human-validated examples provides an independent anchor but does not cover the full benchmark. Related-work self-citations (Peper et al. 2023; Xiong et al. 2024) are not load-bearing. On balance: partial circularity in the validation/scoring loop, but the benchmark construction and model comparison retain substantial independent content.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity depends on a chain of domain assumptions: TabFact tables are correct, GPT-4o faithfully translates edited tables into documents, and GPT-4o's oracle check reliably identifies inconsistent examples. The paper provides limited evidence for each assumption, and the 87% human-agreement rate on auto-validated examples highlights the fragility of the last one. No new physical or mathematical entities are introduced.

free parameters (2)
  • Table filter thresholds (row and column counts) = 5-17 rows, 3-9 columns
    The authors chose to include tables with 5 to 17 rows and 3 to 9 columns to balance content richness and manageability; this choice shapes the difficulty and length distribution of the benchmark but is not fitted to any target.
  • Validity cutoff for oracle check = 5 (on a 0-5 scale)
    The validity prompt asks GPT-4o to score each example 0-5 and discards any example with a score below 5; this strict cutoff is arbitrary and affects which examples are retained.
assumptions (4)
  • domain assumption TabFact tables provide reliable, curated seed knowledge.
    The pipeline sources structured knowledge from TabFact without independently verifying the tables' correctness; wrong seeds could propagate into generated documents.
  • domain assumption GPT-4o can faithfully execute edit plans and generate documents that preserve the augmented table's information.
    The whole pipeline rests on the generator being able to map table rows to documents without losing or adding information needed for the QA answer.
  • domain assumption The oracle self-consistency check performed by GPT-4o correctly identifies inconsistent examples.
    The paper treats GPT-4o's agreement across three oracle/context variations as the ground-truth filter for the 700 auto-validated examples; the 87% human-agreement rate shows this assumption is only partially validated.
  • domain assumption The five reasoning skills (multi-hop, numeric, temporal, soft reasoning, knowledge aggregation) are the relevant dimensions of multi-document reasoning.
    The benchmark's targeted analysis claims depend on these categories being meaningful and complete.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance." pith.science (2026). https://pith.science/paper/LON2GVRI

@misc{pith2026250614927,
  author       = {Pith},
  title        = {Pith review of: MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LON2GVRI}},
  note         = {Machine review of arXiv:2506.14927}
}
read the original abstract

Natural language processing evaluation has made significant progress, largely driven by the proliferation of powerful large language mod-els (LLMs). New evaluation benchmarks are of increasing priority as the reasoning capabilities of LLMs are expanding at a rapid pace. In particular, while multi-document (MD) reasoning is an area of extreme relevance given LLM capabilities in handling longer-context inputs, few benchmarks exist to rigorously examine model behavior in this setting. Moreover, the multi-document setting is historically challenging for benchmark creation due to the expensive cost of annotating long inputs. In this work, we introduce MDBench, a new dataset for evaluating LLMs on the task of multi-document reasoning. Notably, MDBench is created through a novel synthetic generation process, allowing us to controllably and efficiently generate challenging document sets and the corresponding question-answer (QA) examples. Our novel technique operates on condensed structured seed knowledge, modifying it through LLM-assisted edits to induce MD-specific reasoning challenges. We then convert this structured knowledge into a natural text surface form, generating a document set and corresponding QA example. We analyze the behavior of popular LLMs and prompting techniques, finding that MDBENCH poses significant challenges for all methods, even with relatively short document sets. We also see our knowledge-guided generation technique (1) allows us to readily perform targeted analysis of MD-specific reasoning capabilities and (2) can be adapted quickly to account for new challenges and future modeling improvements.

Figures

Figures reproduced from arXiv: 2506.14927 by the authors.

Figure 1
Figure 1. MDBench generation pipeline overview. We 1) source structured knowledge, then use in-context multi-document reasoning demonstrations to 2) intentionally modify the existing knowledge with challenging dependencies. We then 3) map this seed knowledge into document form to produce the multi-document QA example. Finally, we 4) employ an added validity check to filter inconsistent examples that yield erratic outputs in a… view at source ↗
Figure 2
Figure 2. Example skill description on “Multi-hop Reasoning”. During knowledge augmentations, we demonstrate the multi-document skills relevant to the document sets. 3.1 Pipeline Overview Our benchmark generation pipeline begins with structured knowledge sourced from tabular data, which serves as the seed for the augmentation pro￾cess. This structured knowledge is systematically enriched and refined through a strong LLM to in… view at source ↗
Figure 3
Figure 3. Overall exact-match performance of models [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Characteristic-level performance breakdown. We partition the examples by difficulty for each characteristic [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: MDBench human validation guidelines [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Demonstration of table edit plan used during the knowledge augmentation component of the [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: Knowledge Aggregation Skill Description [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: Multi-hop Reasoning Skill Description [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: Numeric Reasoning Skill Description [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: Soft Reasoning Skill Description [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: Temporal Reasoning Skill Description [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]
Figure 12
Figure 12. Figure 12: MDBench Document QA Example – Part 1 [PITH_FULL_IMAGE:figures/full_fig_p029_12.png]
Figure 13
Figure 13. Figure 13: MDBench Document QA Example – Part 2 [PITH_FULL_IMAGE:figures/full_fig_p030_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 9 canonical work pages

  1. [1]

    Anthropic. 2024. Meet the claude 3 family. https://www.anthropic.com/news/claude-3-family. Accessed: 2024-09-06

    work page 2024

  2. [2]

    Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan, and Ido Dagan. 2021. Cdlm: Cross-document language modeling. arXiv preprint arXiv:2101.00406

    work page Pith review arXiv 2021

  3. [3]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

    2024

  4. [4]

    Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. https://arxiv.org/abs/1909.02164 Tabfact: A large-scale dataset for table-based fact verification . Preprint, arXiv:1909.02164

    arXiv 2020

  5. [5]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany...

    arXiv 2024

  6. [6]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.580 Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps . In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609--6625, Barcelona, Spain (Online). International Committee ...

    doi:10.18653/v1/2020.coling-main.580 2020

  7. [7]

    Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ramachandran. 2023. https://arxiv.org/abs/2306.07934 Boardgameqa: A dataset for natural language reasoning with contradictory information . Preprint, arXiv:2306.07934

    work page Pith review arXiv 2023

  8. [8]

    Yoonsang Lee, Xi Ye, and Eunsol Choi. 2024. https://arxiv.org/abs/2404.12447 Ambigdocs: Reasoning across documents on different entities under the same name . Preprint, arXiv:2404.12447

    work page Pith review arXiv 2024

Show all 34 references
  1. [9]

    Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and Zhaoxiang Zhang. 2023 a . https://arxiv.org/abs/2305.19308 Sheetcopilot: Bringing software productivity to the next level through large language models . Preprint, arXiv:2305.19308

    2023 arXiv

  2. [10]

    Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.647 Synthetic data generation with large language models for text classification: Potential and limitations . In Proceedings of the 2023 Conference on Empirical Methods in ...

    2023 doi

  3. [11]

    Gili Lior, Avi Caciularu, Arie Cattan, Shahar Levy, Ori Shapira, and Gabriel Stanovsky. 2024. https://arxiv.org/abs/2406.16086 Seam: A stochastic benchmark for multi-document tasks . Preprint, arXiv:2406.16086

    2024 arXiv

  4. [12]

    Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. 2024. https://arxiv.org/abs/2404.07503 Best practices and lessons learned on synthetic data . Preprint, arXiv:2404.07503

    2024 arXiv

  5. [13]

    Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. https://arxiv.org/abs/2406.15126 On llms-driven synthetic data generation, curation, and evaluation: A survey . Preprint, arXiv:2406.15126

    2024 arXiv

  6. [14]

    Weizheng Lu, Jiaming Zhang, Jing Zhang, and Yueguo Chen. 2024. https://arxiv.org/abs/2402.05121 Large language model for table processing: A survey . Preprint, arXiv:2402.05121

    2024 arXiv

  7. [15]

    Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt. 2024. https://arxiv.org/abs/2204.09140 Multi-hop question answering . Preprint, arXiv:2204.09140

    2024 arXiv

  8. [16]

    Md Mahadi Hasan Nahid and Davood Rafiei. 2024. https://arxiv.org/abs/2404.10150 Tabsqlify: Enhancing reasoning capabilities of llms through table decomposition . Preprint, arXiv:2404.10150

    2024 arXiv

  9. [17]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  10. [18]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  11. [19]

    Peper, Wenzhao Qiu, and Lu Wang

    Joseph J. Peper, Wenzhao Qiu, and Lu Wang. 2023. https://arxiv.org/abs/2311.09836 Pelms: Pre-training for effective low-shot multi-document summarization . Preprint, arXiv:2311.09836

    2023 arXiv

  12. [20]

    Julian Schnitzler, Xanh Ho, Jiahao Huang, Florian Boudin, Saku Sugawara, and Akiko Aizawa. 2024. https://arxiv.org/abs/2406.13397 Morehopqa: More than multi-hop reasoning . Preprint, arXiv:2406.13397

    2024 arXiv

  13. [21]

    Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. 2023. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. arXiv preprint arXiv:2310.16049

    2023 arXiv

  14. [22]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lil...

    2024 arXiv

  15. [23]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, A...

    2024 arXiv

  16. [24]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. https://arxiv.org/abs/2108.00573 Musique: Multihop questions via single-hop question composition . Preprint, arXiv:2108.00573

    2022 arXiv

  17. [25]

    Xiaojun Wan. 2007. https://doi.org/10.1145/1277741.1277949 Timedtextrank: adding the temporal dimension to multi-document summarization . In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '07, page ...

    2007

  18. [26]

    Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2024. https://arxiv.org/abs/2401.04398 Chain-of-table: Evolving tables in the reasoning chain for table...

    2024 arXiv

  19. [27]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  20. [28]

    Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2021. Primera: Pyramid-based masked sentence pre-training for multi-document summarization. arXiv preprint arXiv:2110.08499

    2021 arXiv

  21. [29]

    Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. 2024. https://arxiv.org/abs/2401.06853 Large language models can learn temporal reasoning . Preprint, arXiv:2401.06853

    2024 arXiv

  22. [30]

    Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. 2024. https://arxiv.org/abs/2406.04244 Benchmark data contamination of large language models: A survey . Preprint, arXiv:2406.04244

    2024 arXiv

  23. [31]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA : A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing ( EMNLP )

    2018

  24. [32]

    Andrew Zhu, Alyssa Hwang, Liam Dugan, and Chris Callison-Burch. 2024. https://arxiv.org/abs/2402.14116 Fanoutqa: A multi-hop, multi-document question answering benchmark for large language models . Preprint, arXiv:2402.14116

    2024 arXiv

  25. [33]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  26. [34]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.