Pith. sign in

REVIEW 10 cited by

SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.09413 v3 pith:2NWP3M5O submitted 2024-07-12 cs.CL cs.AIcs.CV

Shraman Pramanick , Rama Chellappa , Subhashini Venugopalan This is my paper · ORCID
classification cs.CLcs.AIcs.CV
keywords scientificdatasetresearcharticlesmultimodalspiqaabilityanswering
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Seeking answers to questions within long scientific research articles is a crucial area of study that aids readers in quickly addressing their inquiries. However, existing question-answering (QA) datasets based on scientific papers are limited in scale and focus solely on textual content. We introduce SPIQA (Scientific Paper Image Question Answering), the first large-scale QA dataset specifically designed to interpret complex figures and tables within the context of scientific research articles across various domains of computer science. Leveraging the breadth of expertise and ability of multimodal large language models (MLLMs) to understand figures, we employ automatic and manual curation to create the dataset. We craft an information-seeking task on interleaved images and text that involves multiple images covering plots, charts, tables, schematic diagrams, and result visualizations. SPIQA comprises 270K questions divided into training, validation, and three different evaluation splits. Through extensive experiments with 12 prominent foundational models, we evaluate the ability of current multimodal systems to comprehend the nuanced aspects of research articles. Additionally, we propose a Chain-of-Thought (CoT) evaluation strategy with in-context retrieval that allows fine-grained, step-by-step assessment and improves model performance. We further explore the upper bounds of performance enhancement with additional textual information, highlighting its promising potential for future research and the dataset's impact on revolutionizing how we interact with scientific literature.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MuSciClaims: Multimodal Scientific Claim Verification

    cs.CL 2025-06 conditional novelty 7.0 of 10

    MuSciClaims provides 1,515 claim-figure pairs across physics, chemistry, and biology and shows that even the best vision-language models only reach 0.72 F1 on claim verification.

  2. DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections

    cs.CL 2025-08 conditional novelty 6.0 of 10

    DocHop-QA introduces an 11,379-instance multimodal, multi-document, multi-hop QA benchmark from PubMed, and shows current models achieve only a best BLEU of 31.8.

  3. MMAFFBen: A Multilingual and Multimodal Affective Analysis Benchmark for Evaluating LLMs and VLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    MMAFFBen is an open-source multilingual and multimodal benchmark for evaluating sentiment and emotion understanding of LLMs and VLMs.

  4. AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning

    cs.CL 2025-12 reject novelty 5.0 of 10

    AutoTool's two-phase SFT/RL plus ranking training lets 8B LLM agents beat larger fixed-tool agents across math, search, code, and vision benchmarks, though unseen-tool gains are asserted, not isolated.

  5. Evaluating Multimodal Large Language Models on Educational Textbook Question Answering

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Adding retrieved textbook context drops LLaMA 3.2-Vision's zero-shot diagram-question accuracy on CK12-QA from 74% to 26%, while fine-tuning reverses the picture.

  6. Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A new 1,430-item multimodal benchmark shows that leading multimodal LLMs rarely notice small visual traps needed for commonsense safety reasoning, with top scores near 0.46 on a scale whose maximum is about 0.97.

  7. LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancements

    cs.CL 2025-05 conditional novelty 5.0 of 10

    HSE-Bench is a new 1,020-question LLM benchmark for HSE compliance reasoning, and the paper claims LLMs rely on semantic matching rather than structured legal reasoning.

  8. Enhancing Scientific Visual Question Answering through Multimodal Reasoning and Ensemble Modeling

    cs.CV 2025-07 conditional novelty 4.0 of 10

    On the SciVQA 2025 benchmark, an InternVL3 model with optimized prompts and chain-of-thought instructions reaches ROUGE-1 and ROUGE-L F1 of 0.740, and a figure-type-aware ensemble ranks 5th.

  9. E-FreeM2: Efficient Training-Free Multi-Scale and Cross-Modal News Verification via MLLMs

    cs.MM 2025-06 conditional novelty 4.0 of 10

    A training-free pipeline using image and text retrieval plus two-stage Gemini and GPT-4o mini reasoning reaches 90.0% accuracy on NewsCLIPpings out-of-context detection, but code, prompts, and error bars are missing.

  10. Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models

    cs.CL 2025-08 unverdicted novelty 3.0 of 10

    The submission cannot be reviewed as a coherent paper: its abstract and full text are two different papers, so the abstract's claims have no supporting body.

Pith tools