Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Current multimodal LLMs cannot reliably draw accurate scientific figures from text alone, even the best one tested.

desk verdict A solid, reusable human-scored benchmark for scientific text-to-image generation; the headline result (even GPT-4o below 4/5 correctness) holds, but the per-dimension claims need error bars and more data. read the letter →

arxiv 2412.02368 v1 pith:5ECPK2PJ submitted 2024-12-03 cs.AI cs.CLcs.CV

Leixin Zhang , Steffen Eger , Yinjie Cheng , Weihe Zhai , Jonas Belouadi , Christoph Leiter , Simone Paolo Ponzetto , Fahimeh Moafian
show 1 more author
Zhixue Zhao
This is my paper · ORCID
classification cs.AIcs.CLcs.CV
keywords ScImagescientifictext-to-imagegenerationmultimodallargelanguagemodelsbenchmarkhumanevaluationspatialunderstandingnumericattributebinding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ScImage, a benchmark for testing how well multimodal large language models generate scientific figures from text. It argues that current models, even the best tested (GPT-4o), cannot reliably produce accurate scientific diagrams: GPT-4o averages below 4 out of 5 on correctness, relevance, and scientific style, and every other model scores far lower. The benchmark decomposes scientific image understanding into spatial, numeric, and attribute comprehension, and finds different failure modes for code-based versus direct image models. The authors provide the prompts, the human-annotated scores for roughly 3,000 generated images, and an analysis across object categories and input languages, positioning ScImage as a reusable reference for future work. If the finding holds, automated scientific figure production from text alone is not yet trustworthy enough for publication without human verification.

What carries the argument

ScImage is a template-and-dictionary benchmark: 101 query templates are instantiated with elements sampled from a hand-built dictionary of scientific objects, attributes, spatial relations, and numeric values, yielding 404 prompts that each explicitly require one or more of three comprehension dimensions (attribute binding, numeric understanding, spatial understanding). The evaluation machinery is a panel of 11 scientists scoring generated images on correctness, relevance, and scientific style (1 to 5, with 0 for compile errors), with pairwise annotation and a calibration session; it is this human scoring that carries the claims, since automatic metrics correlate at most 0.26 (Kendall) with it.

What would settle it

Take a fresh sample of about 200 prompts drawn from real scientific figure captions (for example, from the DaTikZ corpus), score them with the same rubric, and check whether GPT-4o's mean correctness stays below 4 and whether the model ranking by object category is reproduced. If the mean rises above 4.5 or the rankings flip, the benchmark's conclusions are specific to ScImage rather than to scientific text-to-image generation as a whole.

Watch

Extended reading notes

Core claim

The paper's central claim is that on ScImage, a new benchmark of 404 text prompts covering spatial, numeric, and attribute comprehension of scientific objects, current multimodal LLMs cannot reliably generate accurate scientific figures. In both text-code-image mode (Python or TikZ) and direct image mode, even the best model, GPT-4o, scores below 4 out of 5 on correctness, relevance, and scientific style; other models score mostly between 1.5 and 2.2. The benchmark and its roughly 3,000 human-annotated images are offered as a reusable yardstick for measuring progress in scientific text-to-image generation.

Load-bearing premise

The 404 prompts assembled from 101 templates, and the three-dimensional decomposition they encode, are representative of scientific image generation; if not, the model rankings and per-category findings do not generalize beyond the test set.

Editorial extensions

If this is right

  • GPT-4o in text-code-image mode achieves the best scores but still averages below 4 on correctness, relevance, and scientific style, meaning every output has at least some mistakes on average.
  • Code-based generation yields more scientifically styled images than direct image generation; direct models such as DALL·E and Stable Diffusion tend to add irrelevant details and score worst on scientificness.
  • Spatial understanding is the weakest dimension for code-generating models, while numeric understanding is the weakest for direct image generators.
  • Graph theory representations (nodes and edges) are the hardest object category across models, with average correctness below 1.7.
  • Automatic metrics (best PickScore Kendall correlation 0.26) correlate only weakly with human judgments, so the human-annotated scores remain the reference for this task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the below-4 finding holds, automated scientific-figure pipelines (for example, LLM-to-code-to-image) need a verification stage, such as a separate visual checker, before outputs are used in papers.
  • The complementary failure modes (spatial hardest for code models, numeric hardest for image models) suggest that an ensemble or a router combining code-based layout with direct image generation could outperform any single approach.
  • A testable extension is to run newer models through the same 404 prompts and track whether the gap closes; the 20-prompt multilingual subset could also be expanded to test whether the observed language effects are stable.
  • The three comprehension dimensions could serve as a diagnostic for targeted fine-tuning, for example training code models on spatial relations and image models on counting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ScImage, a benchmark for evaluating multimodal large language models on scientific text-to-image generation. The benchmark consists of 101 prompt templates and 404 English generation queries constructed from a dictionary of scientific objects, attributes, spatial relations, and numeric requests, organized around three understanding dimensions (attribute, numeric, spatial) and their combinations. The authors evaluate seven model-output configurations (GPT-4o, Llama 3.1 8B, AutomaTikZ, DALL·E, Stable Diffusion, with code-based and direct-image modes) using 11 human annotators who score correctness, relevance, and scientific style on 1--5 scales, plus a multilingual extension into German, Chinese, and Farsi. The central finding is that GPT-4o is the best model but still scores below 4 on all three criteria, indicating systematic mistakes, and that all models struggle with combined understanding dimensions. The paper also reports per-dimension and per-object-category analyses, and shows that existing automatic metrics correlate only weakly with human judgments.

Significance. If the benchmark is validated, this is a useful contribution: it is one of the first structured evaluations of scientific text-to-image generation, with a carefully designed human protocol that includes calibration, double annotation, detailed guidelines, and inter-annotator agreement reporting. The authors' decision to release the benchmark and the human scores is a concrete strength, as is the demonstration that standard automatic metrics achieve at most Kendall correlation 0.26 with human judgments. The headline result -- that current MLLMs cannot reliably produce accurate scientific diagrams, with GPT-4o remaining below 4 on correctness -- is plausible and is supported by the reported means and the large gaps between GPT-4o and other models. However, the paper's more fine-grained claims about which understanding dimension is hardest for which model type, and about performance by object category, rest on small and uneven samples with no uncertainty quantification; these claims need strengthening before the benchmark can serve as a reusable reference.

major comments (4)
  1. [Section 5, Table 4] The per-dimension conclusions, such as "spatial understanding appears to be the most challenging across all textual models" and the corresponding claims in Section 6, are based on point averages over 40--80 prompts per category with no confidence intervals, error bars, or significance tests. Since Section 3.2 generates only four queries per template, a single hard template can dominate a category mean. For example, the "Annotation" template "The English text (the name of the object) is {preposition} the {object}" is labeled Spatial and may be unusually difficult for code-based models. Please report per-template clustered means, bootstrap confidence intervals, and pairwise significance tests before asserting dimension-level rankings as findings.
  2. [Section 3.2, Table 5] The object-category analysis has severely imbalanced and small sample sizes: table n=4, annotations n=9, matrices n=8, and graph theory n=20. Statements such as "graph theory representation poses great challenges for models" and the discussion of Llama's near-zero table-generation score are therefore fragile; a category mean can be driven by one or two templates. Appendix H acknowledges that sample sizes are "comparatively small," but the main text does not apply this caution consistently. Please either report per-template scores and confidence intervals for these categories, collapse small categories, or explicitly mark these comparisons as exploratory in the abstract and conclusion.
  3. [Section 3.2, Table 6] The multilingual evaluation compares 20 translated prompts per language across seven model configurations, and claims such as "English does not always lead to best results" and "Farsi is worst on average" are reported without any inferential statistics. Given the small number of prompts and the known sensitivity of LLM outputs to prompt wording, these cross-language ranking differences could easily be noise. Please add significance tests or confidence intervals, and consider mixed-effects models that treat prompts and annotators as random factors.
  4. [Section 3.2] The paper presents ScImage as a "comprehensive benchmark" and generalizes findings beyond the specific 404 prompts, but it provides no external validation that the 101 templates and the four sampled dictionary entries per template are representative of the distribution of real scientific figure-generation requests. The construction is internally motivated by a survey of existing benchmarks and DaTikZ, yet there is no coverage analysis against held-out scientific captions or other external sources. Please add a coverage or representativeness analysis, or temper the "comprehensive" claim in the abstract and Section 6.
minor comments (5)
  1. [Section 3.1] The first sentence of Section 3.1 reads "he ScImage evaluates" and should be "The ScImage evaluates."
  2. [Appendix E.2] The heading "Relevence" should be "Relevance," and the paper alternates between "scientificness" and "scientific style" for the same criterion; please harmonize the terminology.
  3. [Tables 10--12] The star ratings in the failure and gold-standard example tables are difficult to read, especially when averaged across annotators; consider showing the numeric scores alongside or instead of stars.
  4. [Section 3.3] The assignment of 0 for compile errors is applied to all three criteria, but this is only stated explicitly in the correctness guideline (Table 13) and in the context of correctness in Section 4; please clarify in Section 3.3 that the 0 penalty applies uniformly to relevance and scientific style as well.
  5. [Table 5] The table caption and the surrounding text should note that some categories contain very few instances; at minimum, add the sample sizes to the caption and refer readers to the limitations discussion in Appendix H.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark results are grounded in external human evaluations, and the self-citations to DaTikZ and AutomaTikZ are not load-bearing.

full rationale

ScImage is an empirical benchmark rather than a derivation, so the self-definitional and fitted-input patterns do not apply. The object vocabulary is sourced from DaTikZ by overlapping authors, and AutomaTikZ is included as a baseline, but the central claims are not justified by these citations: the top-line finding (GPT-4o below 4 on correctness) comes from independent human ratings of generated images, and the paper's own system (AutomaTikZ) is among the worst performers, which goes against a self-serving interpretation. The per-dimension and per-category analyses rest on the authors' template construction, but that is ordinary benchmark design, not an equation that reduces a prediction to an input; the paper also flags the small sample sizes in Appendix H. No step in the paper equates an output with an input by construction, and no load-bearing self-citation chain forces the conclusions. The main risk is external generalizability of the 404 hand-built prompts, which is a validity concern, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical benchmark, so there are no fitted physical parameters or new entities. The central claim rests on design assumptions about task representativeness, prompt fairness, and the validity of human scoring, all of which are only partially defended.

assumptions (4)
  • domain assumption Scientific image generation can be represented by three understanding dimensions (spatial, numeric, attribute) and their combinations.
    The benchmark is built around these dimensions (Section 3.1); if real scientific figures require other competencies, the benchmark's coverage is incomplete.
  • ad hoc to paper The 101 templates and sampled dictionary entries are representative of scientific visualization requests.
    Objects are manually extracted from the authors' DaTikZ corpus and each template is instantiated only four times (Section 3.2); no external coverage validation is provided.
  • ad hoc to paper Single fixed auxiliary prompts give a fair comparison across all models.
    Section 3.1 and Limitations H; the authors acknowledge prompt sensitivity, so the fixed-prompt policy is a load-bearing, partially tensioned choice.
  • domain assumption Human ratings on correctness, relevance, and scientificness are valid ground truth.
    Section 3.3; inter-annotator weighted kappas of 0.41 to 0.66 are moderate, so this assumption is only partially supported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?." pith.science (2026). https://pith.science/paper/5ECPK2PJ

@misc{pith2026241202368,
  author       = {Pith},
  title        = {Pith review of: ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ECPK2PJ}},
  note         = {Machine review of arXiv:2412.02368}
}
read the original abstract

Multimodal large language models (LLMs) have demonstrated impressive capabilities in generating high-quality images from textual instructions. However, their performance in generating scientific images--a critical application for accelerating scientific progress--remains underexplored. In this work, we address this gap by introducing ScImage, a benchmark designed to evaluate the multimodal capabilities of LLMs in generating scientific images from textual descriptions. ScImage assesses three key dimensions of understanding: spatial, numeric, and attribute comprehension, as well as their combinations, focusing on the relationships between scientific objects (e.g., squares, circles). We evaluate five models, GPT-4o, Llama, AutomaTikZ, Dall-E, and StableDiffusion, using two modes of output generation: code-based outputs (Python, TikZ) and direct raster image generation. Additionally, we examine four different input languages: English, German, Farsi, and Chinese. Our evaluation, conducted with 11 scientists across three criteria (correctness, relevance, and scientific accuracy), reveals that while GPT-4o produces outputs of decent quality for simpler prompts involving individual dimensions such as spatial, numeric, or attribute understanding in isolation, all models face challenges in this task, especially for more complex prompts.

Figures

Figures reproduced from arXiv: 2412.02368 by the authors.

Figure 1
Figure 1. Illustration of scientific text-to-image generation. The text shown below is the generation query. Images on [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the three understanding dimensions. The first row shows the individual dimensions of Attribute, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of text-code-image and text-image: [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Generation performance of models on different object types. The same scale is used for three radar bars, with [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Incorrect output from models arguably due to a lack of world knowledge [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: TikZ code snippets of GPT-4o and Llama models for generating a cone and ellipsoid. [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Python code snippets of GPT-4o and Llama models for generating a cone and ellipsoid. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SridBench: Benchmark of Scientific Research Illustration Drawing of Image Generation Model

    cs.CV 2025-05 conditional novelty 5.0 of 10

    SridBench provides a large multi-discipline benchmark for scientific illustration generation and shows current image generation models, especially GPT-4o-image, remain far below human expert quality.

Reference graph

Works this paper leans on

60 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    Automatikz: Text-guided synthesis of scientific vector graphics with tikz

    Jonas Belouadi, Anne Lauscher, and Steffen Eger. Automatikz: Text-guided synthesis of scientific vector graphics with tikz. In The Twelfth International Conference on Learning Representations, 2024 a . URL https://openreview.net/forum?id=v3K5TVP8kZ

    work page 2024

  2. [2]

    Detikzify: Synthesizing graphics programs for scientific figures and sketches with tikz, 2024 b

    Jonas Belouadi, Simone Paolo Ponzetto, and Steffen Eger. Detikzify: Synthesizing graphics programs for scientific figures and sketches with tikz, 2024 b . URL https://arxiv.org/abs/2405.15306

    arXiv 2024

  3. [3]

    Scene text visual question answering

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Mar c al Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 4291--4301, 2019

    work page 2019

  4. [4]

    Elicit: Language models as research tools

    Jungwon Byun and Andreas Stuhlmüller. Elicit: Language models as research tools. https://doi.org/https://doi.org/10.1787/174aee8f-en, 2023. URL https://www.oecd-ilibrary.org/content/component/174aee8f-en

    work page doi:10.1787/174aee8f-en 2023

  5. [5]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015

    arXiv 2015

  6. [6]

    Transformers go for the LOL s: Generating (humourous) titles from scientific abstracts end-to-end

    Yanran Chen and Steffen Eger. Transformers go for the LOL s: Generating (humourous) titles from scientific abstracts end-to-end. In Daniel Deutsch, Rotem Dror, Steffen Eger, Yang Gao, Christoph Leiter, Juri Opitz, and Andreas R \"u ckl \'e (eds.), Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, pp.\ 62--84, Bali, Indonesia, No...

    work page doi:10.18653/v1/2023.eval4nlp-1.6 2023

  7. [7]

    Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models

    Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 3043--3054, 2023

    work page 2023

  8. [8]

    Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation

    Jaemin Cho, Yushi Hu, Jason Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. In ICLR, 2024

    2024

Show all 60 references
  1. [9]

    Exams-v: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models

    Rocktim Jyoti Das, Simeon Emilov Hristov, Haonan Li, Dimitar Iliyanov Dimitrov, Ivan Koychev, and Preslav Nakov. Exams-v: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models. CoRR, 2024

    2024

  2. [10]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 248--255, 2009. doi:10.1109/CVPR.2009.5206848

    2009

  3. [11]

    The mnist database of handwritten digit images for machine learning research [best of the web]

    Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29 0 (6): 0 141--142, 2012. doi:10.1109/MSP.2012.2211477

    2012

  4. [12]

    Scaling rectified flow transformers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transf...

    2024 arXiv

  5. [13]

    CLIPS core: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPS core: A reference-free evaluation metric for image captioning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical M...

    2021 doi

  6. [14]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36: 0 78723--78747, 2023

    2023

  7. [15]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation, 2023

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation, 2023. URL https://arxiv.org/abs/2305.01569

    2023 arXiv

  8. [16]

    Learning multiple layers of features from tiny images

    A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009

    2009

  9. [17]

    West, and Bill Howe

    Po-Shen Lee, Jevin D. West, and Bill Howe. Viziometrics: Analyzing visual information in the scientific literature. IEEE Transactions on Big Data, 4: 0 117--129, 2016. URL https://api.semanticscholar.org/CorpusID:3665638

    2016

  10. [18]

    Holistic evaluation of text-to-image models

    Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  11. [19]

    Prexme! large scale prompt exploration of open source llms for machine translation and summarization evaluation

    Christoph Leiter and Steffen Eger. Prexme! large scale prompt exploration of open source llms for machine translation and summarization evaluation. ArXiv, abs/2406.18528, 2024. URL https://api.semanticscholar.org/CorpusID:270737974

    2024 arXiv

  12. [20]

    The E val4 NLP 2023 shared task on prompting large language models as explainable metrics

    Christoph Leiter, Juri Opitz, Daniel Deutsch, Yang Gao, Rotem Dror, and Steffen Eger. The E val4 NLP 2023 shared task on prompting large language models as explainable metrics. In Daniel Deutsch, Rotem Dror, Steffen Eger, Yang Gao, Christoph Leiter, Juri Opitz, and Andreas R \...

    2023 doi

  13. [21]

    Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. arXiv preprint arXiv:2403.00231, 2024 a

    2024 arXiv

  14. [22]

    Mmsci: A dataset for graduate-level multi-discipline multimodal scientific understanding

    Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, et al. Mmsci: A dataset for graduate-level multi-discipline multimodal scientific understanding. arXiv preprint arXiv:2407.04903, 2024 b

    2024 arXiv

  15. [23]

    Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, Diyi Yang, Christopher Potts, Christopher D Manning, and James Y. Zou. Mapping the increasing use of LLM s in scientific papers. In First Conference on...

    2024

  16. [24]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Procee...

    2014

  17. [25]

    Figurefirst: A layout-first approach for scientific figures

    Theodore Lindsay, Peter Weir, and Floris van Breugel. Figurefirst: A layout-first approach for scientific figures. In Python in Science Conference, pp.\ 57--63, 01 2017. doi:10.25080/shinma-7f4c6e7-009

    2017 doi

  18. [26]

    The ai scientist: Towards fully automated open-ended scientific discovery

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024 a

    2024 arXiv

  19. [27]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun ...

    2022

  20. [28]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning...

    2024

  21. [29]

    SimPO : Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO : Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  22. [30]

    State of What Art? A Call for Multi-Prompt LLM Evaluation

    Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of What Art? A Call for Multi-Prompt LLM Evaluation . Transactions of the Association for Computational Linguistics, 12: 0 933--949, 08 2024. ISSN 2307-387X. doi:10.1162/tacl_a_00681....

    2024 doi

  23. [31]

    Evaluating large language models for structured science summarization in the open research knowledge graph

    Vladyslav Nechakhin, Jennifer D’Souza, and Steffen Eger. Evaluating large language models for structured science summarization in the open research knowledge graph. Information, 15 0 (6), 2024. ISSN 2078-2489. doi:10.3390/info15060328. URL https://www.mdpi.com/2078-2489/15/6/328

    2024 doi

  24. [32]

    Introducing openai o1 (preview)

    OpenAI. Introducing openai o1 (preview). https://openai.com/index/introducing-openai-o1-preview/, oct 2024. Accessed: 2024-10-26

    2024

  25. [33]

    Zero-shot text-to-image generation, 2021

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021. URL https://arxiv.org/abs/2102.12092

    2021 arXiv

  26. [34]

    Scifibench: Benchmarking large multimodal models for scientific figure interpretation

    Jonathan Roberts, Kai Han, Neil Houlsby, and Samuel Albanie. Scifibench: Benchmarking large multimodal models for scientific figure interpretation. arXiv preprint arXiv:2405.08807, 2024

    2024 arXiv

  27. [35]

    Ocr-vqgan: Taming text-within-image generation

    Juan A Rodriguez, David Vazquez, Issam Laradji, Marco Pedersoli, and Pau Rodriguez. Ocr-vqgan: Taming text-within-image generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.\ 3689--3698, 2023

    2023

  28. [36]

    Who evaluates the evaluations? objectively scoring text-to-image prompt coherence metrics with t2iscorescore (ts2), 2024

    Michael Saxon, Fatima Jahara, Mahsa Khoshnoodi, Yujie Lu, Aditya Sharma, and William Yang Wang. Who evaluates the evaluations? objectively scoring text-to-image prompt coherence metrics with t2iscorescore (ts2), 2024. URL https://arxiv.org/abs/2404.04251

    2024 arXiv

  29. [37]

    Kanell, Peter Xu, Omar Khattab, and Monica S

    Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu, Omar Khattab, and Monica S. Lam. Assisting in writing wikipedia-like articles from scratch with large language models, 2024. URL https://arxiv.org/abs/2402.14207

    2024 arXiv

  30. [38]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computation...

    2018 doi

  31. [39]

    Chartmimic: Evaluating lmm's cross-modal reasoning capability via chart-to-code generation

    Chufan Shi, Cheng Yang, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran Xu, Xinyu Zhu, Siheng Li, Yuxiang Zhang, et al. Chartmimic: Evaluating lmm's cross-modal reasoning capability via chart-to-code generation. arXiv preprint arXiv:2406.09961, 2024

    2024 arXiv

  32. [40]

    Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers, 2024

    Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers, 2024. URL https://arxiv.org/abs/2409.04109

    2024 arXiv

  33. [41]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019

    2019

  34. [42]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  35. [43]

    Winoground: Probing vision and language models for visio-linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

    2022

  36. [44]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401.06209, 2024

    2024 arXiv

  37. [45]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023 arXiv

  38. [46]

    Plots made quickly: An efficient approach for generating visualizations from natural language queries

    Henrik Voigt, Kai Lawonn, and Sina Zarrie . Plots made quickly: An efficient approach for generating visualizations from natural language queries. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the...

    2024

  39. [47]

    Contextual: Evaluating context-sensitive text-rich visual reasoning in large multimodal models

    Rohan Wadhawan, Hritik Bansal, Kai-Wei Chang, and Nanyun Peng. Contextual: Evaluating context-sensitive text-rich visual reasoning in large multimodal models. In Forty-first International Conference on Machine Learning, 2024

    2024

  40. [48]

    Scibench: Evaluating college-level scientific problem-solving abilities of large language models

    Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. In Forty-first International Conference on...

    2024

  41. [49]

    Charxiv: Charting gaps in realistic chart understanding in multimodal llms

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. arXiv preprint arXiv:2406.18521, 2024 b

    2024 arXiv

  42. [50]

    seaborn: statistical data visualization

    Michael Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6: 0 3021, 04 2021. doi:10.21105/joss.03021

    2021 doi

  43. [51]

    Evaluating and analyzing relationship hallucinations in large vision-language models

    Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, and Rongrong Ji. Evaluating and analyzing relationship hallucinations in large vision-language models. In Forty-first International Conference on Machine Learning, 2024

    2024

  44. [52]

    An automatic graph generation method for scholarly papers based on table structure analysis

    Ryoya Yamada, Manabu Ohta, and Atsuhiro Takasu. An automatic graph generation method for scholarly papers based on table structure analysis. In Richard Chbeir, Hiroshi Ishikawa, Kazutoshi Sumiya, Kenji Hatano, and Mario Koeppen (eds.), Proceedings of the 10th International Con...

    2018

  45. [53]

    Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. In Forty-first International Conference on...

    2024

  46. [54]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In Forty-first International Conference on Machine Learning, 2024

    2024

  47. [55]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502, 2023

    2023 arXiv

  48. [56]

    Vgbench: Evaluating large language models on vector graphics understanding and generation

    Bocheng Zou, Mu Cai, Jianrui Zhang, and Yong Jae Lee. Vgbench: Evaluating large language models on vector graphics understanding and generation. arXiv preprint arXiv:2407.10972, 2024

    2024 arXiv

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  50. [58]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  51. [59]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  52. [61]

    !1A Qa

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.