Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SCIVER is the first benchmark for multimodal scientific claim verification; the strongest model tested, o4-mini, hits 77.7% test accuracy while human experts reach 93.8%.

desk verdict A genuinely new and carefully built multimodal claim-verification benchmark whose headline human-vs-model gap rests on a 40-claim sample, and which still deserves a serious referee. read the letter →

arxiv 2506.15569 v1 pith:HHYIE2QW submitted 2025-06-18 cs.CL

Chengye Wang , Yifei Shen , Zexi Kuang , Arman Cohan , Yilun Zhao This is my paper
classification cs.CL
keywords scientificclaimverificationmultimodalreasoningbenchmarkevaluationfoundationmodelsentailmentclassificationevidenceannotationretrieval-augmentedgenerationliteraturecomprehension
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SCIVER is proposed as the first benchmark built to evaluate foundation models on multimodal scientific claim verification: checking a claim against the full context of a scientific paper, including paragraphs, tables, and charts. It contains 3,000 expert-annotated examples drawn from 1,113 computer science papers, organized into four subsets that isolate direct, parallel, sequential, and analytical reasoning. Every example comes with expert-identified supporting evidence, which lets evaluation say not only whether a model got the answer right but whether it used the right evidence. Evaluation of 21 multimodal models shows the best system, o4-mini at 77.7% test accuracy, still trails expert annotators at 93.8%, with the gap largest on analytical reasoning. The paper reads this gap as evidence that integrating multiple modalities and multi-step reasoning remains an open problem for current foundation models.

What carries the argument

The load-bearing object is the benchmark itself: 3,000 claims over 1,113 computer science papers, each with a curated multimodal context consisting of text paragraphs, tables rendered as screenshots, and charts, plus a binary entailment label and expert-annotated supporting evidence. The design encodes four reasoning types, direct, parallel, sequential, and analytical, as separate subsets, and the evidence annotations are what turn a simple accuracy number into a diagnostic: they let the authors attribute errors to retrieval failure, visual misinterpretation, multi-step reasoning failure, text over-reliance, or domain misconceptions. The task formulation feeds the model the paper context and asks it to decide whether the claim is entailed or refuted, with tables given as screenshots to preserve their structure.

What would settle it

Recruit expert annotators to independently label a large random sample, say 500 examples, from SciVer's test set and compare their accuracy with the reported 93.8%; if the larger-sample expert accuracy is materially below that figure, the headline human-versus-model gap is not established. A complementary check is whether refuted claims sampled from real papers, rather than created by perturbing entailed claims, change model and expert error rates.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that multimodal scientific claim verification is a distinct task that current foundation models cannot yet perform at expert level. The benchmark's four subsets operationalize the reasoning demands: direct extraction from a single source, parallel integration of several sources, sequential inference chains across modalities, and analytical application of domain knowledge. With 3,000 expert-annotated examples and per-example supporting-evidence annotations, SCIVER lets an evaluation separate failures of retrieval, interpretation, and reasoning. The empirical headline is that o4-mini, the strongest model tested, reaches 77.7% accuracy on the test set against 93.8% for human experts, and that even the best proprietary models fall from near-human performance on direct reasoning to roughly 70% on analytical reasoning.

Load-bearing premise

The paper's central comparison takes the 93.8% human accuracy as the reference point, but that number comes from 40 claims; if a larger sample of experts scored lower on the full test set, the claimed gap would be smaller.

Editorial extensions

If this is right

  • If SCIVER is accepted as a reference task, scientific claim verification should no longer be treated as text-only or table-only; claims that depend on charts and tables jointly are in scope.
  • The 93.8% versus 77.7% accuracy gap sets a concrete performance target for multimodal foundation models, with analytical reasoning identified as the hardest subset.
  • The evidence-requirement analysis implies that model performance drops as the number of supporting evidence pieces grows, so multi-hop cross-modal integration is a key bottleneck.
  • The retrieval-augmented generation experiments imply that supplying better-retrieved evidence, especially via stronger retrievers plus an LLM evidence filter, can improve verification accuracy, but even oracle retrieval leaves a residual gap.
  • The error taxonomy implies that the largest failure of open-source models is not answering but finding the relevant evidence inside a long multimodal context.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the human baseline of 93.8% rests on only 40 expert-judged claims, so the reported 16-point gap may be an overestimate; a larger expert sample could place the true gap elsewhere.
  • Editorial inference: because refuted claims are created by perturbing entailed claims, the benchmark measures detection of locally contradicted true statements, not the full variety of false or misleading statements that occur naturally in scientific writing; the two may pose different challenges.
  • Editorial inference: the retrieval results suggest a testable extension in which models are given oracle evidence, or are trained to call retrievers explicitly, to isolate whether the bottleneck is selecting information or reasoning over it.
  • Editorial inference: the four reasoning subtypes suggest building a diagnostic at the subtype level, for example reporting separate accuracies on direct versus sequential claims, to match model capabilities to specific scientific-reading workflows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. SciVer introduces a new benchmark for multimodal scientific claim verification: 3,000 expert-annotated examples drawn from 1,113 computer science papers, organized into four reasoning subsets (direct, parallel, sequential, analytical), with expert-annotated supporting evidence for each example. The paper evaluates 21 open-source and proprietary multimodal foundation models and reports that the best model (o4-mini, 77.7% test accuracy) still lags a human-expert estimate of 93.8%. It also presents error analyses and a retrieval-augmented generation study with BM25, Contriever, and OpenAI embeddings.

Significance. If the benchmark's quality holds, this is a useful and timely resource: existing claim-verification benchmarks are text-only or single-table/single-chart, while SciVer targets the full multimodal context of scientific papers. The annotation protocol is a genuine strength: 18 graduate-student experts with peer-reviewed publications in assigned subfields, a two-hour training session, dual annotation with adjudication, 94% inter-annotator agreement on entailment labels, and a five-aspect validation pass that revised 232 examples. The paper also releases data and code, and the evaluation spans 21 models with fine-grained evidence annotations that enable error analysis. The main weaknesses are statistical: the human-expert baseline rests on only 40 claims, the model evaluations are single runs without error bars, and the refuted-claim construction is entirely perturbation-based. These issues affect the headline 'substantial performance gap' claim but do not undermine the benchmark's underlying data value.

major comments (4)
  1. [§3.6, Table 3] The human-expert accuracy of 93.8% is estimated from 10 randomly sampled claims per subset, 40 claims total, judged by two experts. With 80 binary judgments, the approximate 95% confidence interval is at least ±5 points, and it is wider if the 40 claims are treated as clusters. A true expert accuracy of 88% would shrink the reported 77.7%-vs-93.8% gap from 16.1 to about 10 points, and at 84% the gap would be about 7 points. The section also does not state whether the two experts saw the same curated multimodal context that was given to the models (average 583 words, 0.55 tables, and 0.94 charts per Table 2) or the full paper. If the latter, the human number is an upper bound rather than a directly comparable ceiling. Because this estimate is the sole anchor for the paper's central 'substantial performance gap' finding, the authors should expand the human evaluation to a protocol-matched sample, report confidence intervals, and specify exactly what evidence the experts saw.
  2. [§3.3] All refuted claims are generated by perturbing entailed claims: the annotators modify an initially entailed claim by introducing factual errors that contradict the supporting evidence. This construction means the negative class is always a minimal edit of a true claim, with the original evidence present and the contradiction local. Real-world false or misleading claims in scientific literature can take other forms, such as unsupported generalizations, swapped methods, or hallucinated numbers with no corresponding evidence. The paper should report the distribution of perturbation types, check for lexical or positional artifacts that allow trivial discrimination, and either add a subset of naturally occurring refuted claims or explicitly narrow the claim about real-world validity.
  3. [§4.1, Table 3] Model accuracies are reported from single runs without variance, confidence intervals, or significance tests. Several adjacent models differ by less than one point (e.g., GPT-4.1 at 73.2 vs. Gemini-2.0-Flash at 73.3; Phi-4-Multimodal at 51.0 vs. LLaVA-OneVision at 50.3), so the ranking, the 'open-source lag' conclusion, and the comparison with the human baseline are not statistically grounded. The authors should provide bootstrap confidence intervals or multiple-seed runs for at least the top-performing models and treat differences within noise as ties.
  4. [Between Abstract and Section 1] The manuscript contains an extraneous, unreferenced passage about 'KnowAda' and 'DOCCI' with two caption-like paragraphs and the note '(...The remaining one table, two charts, and 64 paragraphs are not presented due to space constraints….)' inserted between the abstract and Section 1. This text is unrelated to SciVer and indicates that the submitted file is malformed or includes content from another paper. It must be removed and the complete, correct manuscript verified; the note about omitted content also contradicts the paper's claim that full multimodal context is provided to models.
minor comments (5)
  1. [§3.1] In the task formulation, 'the model is is tasked' contains a duplicated 'is' that should be corrected.
  2. [§4.4 / Figure 5] Two different tables are both numbered Table 4: one in the RAG analysis and one inside the Figure 5 caption. The numbering should be made consistent throughout.
  3. [Table 5] The caption says annotator biographies are hidden to protect identity confidentiality, but the 'Author?' column reveals that annotators 4-6 are authors. This should be stated explicitly in the main text as a potential conflict of interest, or the column should be removed.
  4. [Limitations] The Limitations section acknowledges the CS-only scope and the exclusion of equations and experimental images, but it does not acknowledge the small size of the human-expert baseline or the perturbation-based construction of refuted claims; both should be discussed given their role in the main claims.
  5. [Appendix C] The error-analysis examples are helpful, but several are drawn from papers about 'KnowAda' and 'HiGen' that do not appear in the main data statistics; the relationship between these examples and the four subsets should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SciVer is a self-contained benchmark paper whose central results rest on external expert annotation and external model evaluations.

full rationale

The paper introduces a new benchmark (3,000 expert-annotated examples over 1,113 papers) and evaluates 21 foundation models. I walked the derivation chain looking for the six enumerated circularity patterns. No fitted parameter is later renamed as a prediction; no derived quantity reduces to an input by definition; no uniqueness theorem or prior-work assertion is used to force the benchmark design; and no known result is repackaged as a new contribution. The only author-overlapping citations (Zhao et al. 2024 FinDVer; Li et al. 2024b M3SciQA) appear in related-work positioning and in incidental design choices (e.g., table screenshots following Zheng et al. 2024 and Deng et al. 2024), and none of them is load-bearing for the measured accuracies. The human-expert reference of 93.8% (Section 3.6) is an external measurement computed from 40 claims, not an output of the evaluation pipeline; the small sample and the unspecified evidence-access protocol are legitimate robustness concerns, but they are not circularity. Likewise, generating refuted claims by perturbing entailed claims (Section 3.3) is a data-construction choice that may affect how representative the negative class is, but it does not make the evaluation circular. The central results are externally measured model accuracies compared against a gold label set produced by expert annotation with 94.0% inter-annotator agreement and third-expert adjudication. The paper does not derive its benchmark's validity from its own conclusions, nor does it import any contested premise from prior work by the same authors. For these reasons, no circular step is present and the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central evidence in this paper is an expert-annotated dataset, not a derivation. The benchmark's validity rests on three domain assumptions: annotator expertise, the representativeness of perturbed refuted claims, and the adequacy of a 40-claim human baseline. No free parameters or invented entities are introduced.

assumptions (4)
  • domain assumption Annotators with at least two peer-reviewed publications in the assigned subfield produce accurate entailment labels and evidence spans.
    Section 3.2 sets this recruitment bar; if expertise is miscalibrated, gold labels could carry systematic errors.
  • domain assumption Refuted claims are adequately represented by introducing factual errors into entailed claims.
    Section 3.3 constructs all refuted examples by perturbation, which may miss naturalistic false claims that are not simple edits of true ones.
  • domain assumption A 40-claim sample (10 per subset) suffices to estimate human-expert accuracy for the whole benchmark.
    Section 3.6 uses this sample to report 93.8% human accuracy; the resulting confidence interval is wide and could change the stated gap.
  • domain assumption Each claim can be verified using only the curated context (three randomly sampled elements plus surrounding paragraphs) without additional unannotated context.
    Section 3.3 claim annotation protocol restricts claims to the sampled multimodal elements; this shapes the task into a closed-set verification problem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification." pith.science (2026). https://pith.science/paper/HHYIE2QW

@misc{pith2026250615569,
  author       = {Pith},
  title        = {Pith review of: SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HHYIE2QW}},
  note         = {Machine review of arXiv:2506.15569}
}
read the original abstract

We introduce SciVer, the first benchmark specifically designed to evaluate the ability of foundation models to verify claims within a multimodal scientific context. SciVer consists of 3,000 expert-annotated examples over 1,113 scientific papers, covering four subsets, each representing a common reasoning type in multimodal scientific claim verification. To enable fine-grained evaluation, each example includes expert-annotated supporting evidence. We assess the performance of 21 state-of-the-art multimodal foundation models, including o4-mini, Gemini-2.5-Flash, Llama-3.2-Vision, and Qwen2.5-VL. Our experiment reveals a substantial performance gap between these models and human experts on SciVer. Through an in-depth analysis of retrieval-augmented generation (RAG), and human-conducted error evaluations, we identify critical limitations in current open-source models, offering key insights to advance models' comprehension and reasoning in multimodal scientific literature tasks.

Figures

Figures reproduced from arXiv: 2506.15569 by the authors.

Figure 1
Figure 1. An illustration of the four subsets in the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An overview of the SCIVER benchmark construction pipeline. only interpret the provided data but also apply rel￾evant scientific principles and methodological un￾derstanding to arrive at valid conclusions. Appendix C presents detailed examples of each sub￾set. These subsets enable fine-grained evaluation across different reasoning paradigms commonly encountered in scientific literature comprehension. 3.2 Preliminary … view at source ↗
Figure 3
Figure 3. The Chain-of-Thought prompt used. ria, validators are responsible for making necessary revisions. In practice, 232 initially annotated exam￾ples required revisions before being finalized. 3.6 Data Statistics and Analysis [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Comparison of model performance on the validation set, with claims requiring varying amounts of annotated supporting evidence. Each piece of evidence is defined as a single table, chart, or paragraph (§3.4). SCIVER presents substantial challenges for cur￾rent models. W…
Figure 3
Figure 3. Figure 3: Dense captioning descriptiveness precision-recall results for LLaVA-7B fine-tuned with DOCCI captions, adapted using different methods. …(abbreviated) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]
Figure 5
Figure 5. Figure 5: Illustration of two error types: Visual Element Misinterpretation (left) and Failure in Multi-step Reasoning [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The prompt for evidence filtering in §4.4. • Heavy reliance on text modality (12%), where models focus primarily on textual input, failing to properly integrate crucial information from tables and charts. • Domain-specific misconceptions (10%), where models misapply do…
Figure 7
Figure 7. Figure 7: Illustration of Failure to Retrieve Relevant Information with the example from the Analytical Reasoning subset [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 4
Figure 4. Figure 4: Exact Match values for TriviaQA, NQ, and WebQ datasets categorized by question type, based on the optimal [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 8
Figure 8. Figure 8: Illustration of Visual element misinterpretation with the example from the Direct Reasoning subset [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 10
Figure 10. Figure 10: Playing performance with different numbers of [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 9
Figure 9. Figure 9: Illustration of Heavy Reliance on Text Modality with the example from the Analytical Reasoning subset [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Illustration of Domain-Specific Misconceptions with the example from the Analytical Reasoning subset [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 5
Figure 5. Figure 5: The Physical Failure Rates (top) and Cumulative Percentage of Collisions (bottom). This [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 11
Figure 11. Figure 11: Illustration of Other Observation Error with the example from the Parallel Reasoning subset [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research

    cs.CL 2025-07 conditional novelty 7.0 of 10

    A benchmark of 1,500 expert-annotated ablation study designs from 807 NLP papers shows frontier LLMs underperform human experts and that LLM-as-a-judge evaluations correlate weakly with human judgments.

  2. Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers

    cs.CL 2025-07 conditional novelty 7.0 of 10

    MISS-QA, a new benchmark for information-seeking QA over schematic diagrams, shows the best open-source multimodal model at 61.6% accuracy versus 89.0% for human experts.

Reference graph

Works this paper leans on

45 extracted references · 5 canonical work pages · cited by 2 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall,...

    arXiv 2024

  4. [4]

    Mubashara Akhtar, Nikesh Subedi, Vivek Gupta, Sahar Tahmasebi, Oana Cocarascu, and Elena Simperl. 2024. https://doi.org/10.18653/v1/2024.findings-acl.828 C hart C heck: Explainable fact-checking over real-world chart images . In Findings of the Association for Computational Linguistics: ACL 2024, pages 13921--13937, Bangkok, Thailand. Association for Comp...

    doi:10.18653/v1/2024.findings-acl.828 2024

  5. [5]

    Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. https://doi.org/10.18653/v1/2021.fever-1.1 The fact extraction and VER ification over unstructured and structured information ( FEVEROUS ) shared task . In Proceedings of the Fourth Workshop on Fact Extrac...

    work page doi:10.18653/v1/2021.fever-1.1 2021

  6. [6]

    Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D'arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Graham Neubig, Dan Weld, Doug Downey, Wen tau Yih, Pang Wei Koh, and Hannaneh Hajishirzi. 2024. http://...

    arXiv 2024

  7. [7]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966

    arXiv 2023

  8. [8]

    Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. https://openreview.net/forum?id=rkeJRhNYDH Tabfact: A large-scale dataset for table-based fact verification . In International Conference on Learning Representations

    work page 2020

Show all 45 references
  1. [9]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. 2024 a . Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271

    2024 arXiv

  2. [10]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024 b . How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821

    2024 arXiv

  3. [11]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv ...

    2023 arXiv

  4. [12]

    Smith, and Matt Gardner

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. https://doi.org/10.18653/v1/2021.naacl-main.365 A dataset of information-seeking questions and answers anchored in research papers . In Proceedings of the 2021 Conference of the North Amer...

    2021 doi

  5. [13]

    Naihao Deng, Zhenjie Sun, Ruiqi He, Aman Sikka, Yulong Chen, Lin Ma, Yue Zhang, and Rada Mihalcea. 2024. https://doi.org/10.18653/v1/2024.findings-acl.23 Tables as texts or images: Evaluating the table reasoning ability of LLM s and MLLM s . In Findings of the Association for ...

    2024 doi

  6. [14]

    e , James Thorne, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych

    Max Glockner, Ieva Stali \= u nait \. e , James Thorne, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2024. https://doi.org/10.1162/tacl_a_00629 A mbi FC : Fact-checking ambiguous claims with evidence . Transactions of the Association for Computational Linguistics, 12:1--18

    2024 doi

  7. [15]

    Google. 2024. http://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    2024 arXiv

  8. [16]

    Google. 2025. https://deepmind.google/models/gemini/flash/ Gemini 2.5 flash

    2025

  9. [17]

    Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. https://doi.org/10.18653/v1/2020.acl-main.210 INFOTABS : Inference on tables as semi-structured data . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309--23...

    2020 doi

  10. [18]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. https://doi.org/10.48550/ARXIV.2112.09118 Unsupervised dense information retrieval with contrastive learning

    2021 doi arXiv

  11. [19]

    Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.470 W i CE : Real-world entailment for claims in W ikipedia . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7561...

    2023 doi

  12. [20]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  13. [21]

    Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-In Lee, and Moontae Lee. 2023. https://proceedings.mlr.press/v202/lee23n.html QASA : Advanced question answering on scientific articles . In Proceedings of the 40th International Conference on Machine Le...

    2023

  14. [22]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 a . http://arxiv.org/abs/2408.03326 Llava-onevision: Easy visual task transfer

    2024 arXiv

  15. [23]

    Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, and Arman Cohan. 2024 b . https://doi.org/10.18653/v1/2024.findings-emnlp.904 M 3 S ci QA : A multi-modal multi-document scientific QA benchmark for evaluating foundation models . In Findings of the Association for ...

    2024 doi

  16. [24]

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. 2024 c . https://doi.org/10.18653/v1/2024.acl-long.775 Multimodal A r X iv: A dataset for improving scientific comprehension of large vision-language models . In Proceedings of the 62nd Annual ...

    2024 doi

  17. [25]

    Wilson, Woosang Lim, and William Yang Wang

    Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, Linda Ruth Petzold, Stephen D. Wilson, Woosang Lim, and William Yang Wang. 2024 d . http://arxiv.org/abs/2407.04903 Mmsci: A dataset for graduate-le...

    2024 arXiv

  18. [26]

    Xinyuan Lu, Liangming Pan, Qian Liu, Preslav Nakov, and Min-Yen Kan. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.483 SCITAB : A challenging benchmark for compositional reasoning and claim verification on scientific tables . In Proceedings of the 2023 Conference on Empiri...

    2023 doi

  19. [27]

    Meta. 2024. http://arxiv.org/abs/2407.21783 The llama 3 herd of models

    2024 arXiv

  20. [28]

    Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, Xiyang Dai, Ruchao ...

    2025 arXiv

  21. [29]

    Microsoft. 2024. http://arxiv.org/abs/2404.14219 Phi-3 technical report: A highly capable language model locally on your phone

    2024 arXiv

  22. [30]

    Mistral AI . 2025. Mistral-small-3.1-24b-instruct-2503. https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503. Apache 2.0 License

    2025

  23. [31]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Hello gpt-4o

    2024

  24. [32]

    OpenAI. 2025 a . https://openai.com/index/o3-o4-mini-system-card-addendum-operator-o3/ Addendum to openai o3 and o4-mini system card: Openai o3 operator

    2025

  25. [33]

    OpenAI. 2025 b . https://openai.com/index/gpt-4-1/ Introducing gpt-4.1 in the api

    2025

  26. [34]

    Aman Rangapur, Haoran Wang, Ling Jian, and Kai Shu. 2024. http://arxiv.org/abs/2309.08793 Fin-fact: A benchmark dataset for multimodal financial fact checking and explanation generation

    2024 arXiv

  27. [35]

    Skarlinski, Sam Cox, Jon M

    Michael D. Skarlinski, Sam Cox, Jon M. Laurent, James D. Braza, Michaela Hinks, Michael J. Hammerling, Manvitha Ponnapati, Samuel G. Rodriques, and Andrew D. White. 2024. http://arxiv.org/abs/2409.13740 Language agents achieve superhuman synthesis of scientific knowledge

    2024 arXiv

  28. [36]

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. https://doi.org/10.18653/v1/N18-1074 FEVER : a large-scale dataset for fact extraction and VER ification . In Proceedings of the 2018 Conference of the North A merican Chapter of the Associatio...

    2018 doi

  29. [37]

    Andreas Vlachos and Sebastian Riedel. 2014. https://doi.org/10.3115/v1/W14-2508 Fact checking: Task definition and dataset construction . In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science , pages 18--22, Baltimore, MD, USA. Assoc...

    2014 doi

  30. [38]

    David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.609 Fact or fiction: Verifying scientific claims . In Proceedings of the 2020 Conference on Empirical Methods in Na...

    2020 doi

  31. [39]

    David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.347 S ci F act-open: Towards open-domain scientific claim verification . In Findings of the Association for Computational ...

    2022 doi

  32. [40]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024 a . Qwen2-vl: Enhancing vision-language m...

    2024 arXiv

  33. [41]

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. 2024 b . https://doi.org/10.48550/arXiv.2406.18521 Charxiv: Charting gaps in realistic chart understand...

    2024 doi arXiv

  34. [42]

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. 2024. http://arxiv.org/abs/2409.02813 Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark

    2024 arXiv

  35. [43]

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. 2025. https://openreview.net/forum?id=2jTdHYuguF MMMU -pro: A more robust multi-discipline multimodal understanding benchmark

    2025

  36. [44]

    Yilun Zhao, Yitao Long, Tintin Jiang, Chengye Wang, Weiyuan Chen, Hongjun Liu, Xiangru Tang, Yiming Zhang, Chen Zhao, and Arman Cohan. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.818 F in DV er: Explainable claim verification over long and hybrid-content financial docume...

    2024 doi

  37. [45]

    Mingyu Zheng, Xinwei Feng, Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang, and Weiping Wang. 2024. https://doi.org/10.18653/v1/2024.acl-long.493 Multimodal table understanding . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volum...

    2024 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.