Pith. sign in

REVIEW 4 major objections 5 minor 59 references

Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Professional editors overwhelmingly prefer GPT-4V's scientific figure captions to author-written ones, even on papers published after the model's training cutoff.

desk verdict Solid challenge report with a well-run editor study; the GPT-4V preference is real for these three editors, but the paper generalizes it a bit beyond its evidence. read the letter →

arxiv 2501.19353 v3 pith:T5W3J5KI submitted 2025-01-31 cs.CL cs.AIcs.CV

Ting-Yao E. Hsu , Yi-Li Hsu , Shaurya Rohatgi , Chieh-Yang Huang , Ho Yin Sam Ng , Ryan Rossi , Sungchul Kim , Tong Yu
show 3 more authors
Lun-Wei Ku C. Lee Giles Ting-Hao K. Huang
This is my paper · ORCID
classification cs.CLcs.AIcs.CV
keywords scientificfigurecaptioninglargemultimodalmodelsGPT-4VSciCapChallengehumanevaluationprofessionaleditorstextsummarizationautomaticmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large multimodal models have solved the task of writing captions for scientific figures, using the first SciCap Challenge of 2023 as a natural experiment. The key claim is that professional editors overwhelmingly preferred captions written by GPT-4V over captions from every other model, and over the original captions written by the papers' authors. That preference held under three different caption-length constraints and on a fresh set of figures published after the model's training cutoff, suggesting it is not just memorization. The paper's answer is a qualified no: caption generation has reached a milestone but is not solved, because caption quality itself, evaluation methods, and personalization for different readers remain open problems. A sympathetic reader should care because the result suggests practical AI caption-writing assistance is within reach, while also exposing that standard automatic metrics disagree with what expert humans value.

What carries the argument

The load-bearing mechanism is the human ranking protocol: for each sampled figure, six captions (one author-written, five machine-generated, including the two challenge winners) were ranked from best to worst by three professional editors using a drag-and-drop interface, with rankings then aggregated. The paper complements this with a 'Quality Subset' of the hidden test set, curated by using GPT-4 to score caption quality and then manual verification, so that reference-based evaluation is not skewed by low-quality author captions. The central access point that carries the argument is the figure-mentioning paragraph: GPT-4V was given both the figure image and the paragraphs that refer to it, and the image-plus-paragraph setting consistently beat image-only and text-only settings.

What would settle it

Recruit a substantially larger and broader panel (for example, two dozen editors and researchers from biology, medicine, and engineering) and have them rank the same six caption sources without knowing the source; if GPT-4V does not win a clear majority in that ranking, or if a comprehension test shows readers understand figures better with author captions, the paper's central claim would be falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery is that in a controlled human evaluation, three professional editors with academic editing experience ranked GPT-4V-generated captions higher than all competing outputs, including author-written captions, across all three length settings (unconstrained, a 25-word cap, and a cap matching the author caption's length). This held when the experiment was repeated on papers published after GPT-4V's data cutoff. The editors' comments indicate that GPT-4V's advantage comes primarily from providing sufficient details and highlighting the figure's takeaway message. The paper argues that automatic metrics such as BLEU and ROUGE do not align with this human judgment, with GPT-4V scoring poorly under those metrics despite winning the human evaluation.

Load-bearing premise

The load-bearing premise is that three professional editors constitute a valid and representative measure of caption quality for the broad range of scientists, a generalization the paper itself flags as uncertain.

Editorial extensions

If this is right

  • If editors' preferences reflect what makes captions useful, then GPT-4V-class large multimodal models can already serve as practical assistants for scientific figure caption writing, especially where author captions are weak.
  • Since automatic metrics ranked GPT-4V far below the challenge winners while human editors ranked it first, any caption-generation leaderboard that relies on BLEU or ROUGE should be interpreted cautiously.
  • The result implies that the bottleneck is no longer pure generation quality but evaluation, factual reliability, and customization to individual readers' needs.
  • The persistence of the result on post-cutoff papers indicates that the advantage is not primarily due to memorization of the test captions.
  • Because the best-performing model is proprietary and closed, the field cannot fully inspect why it works, which limits reproducibility.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the editors' preference is driven by the 'detail plus takeaway' style, then one testable extension is to fine-tune an open-weight large multimodal model on GPT-4V captions and check whether professional editors transfer their preference to the open model, which would decouple the capability from the proprietary API.
  • The paper compares editors with Ph.D. and undergraduate students and finds disagreements; a natural extension would be a task-based study measuring whether readers actually answer questions about a figure better with GPT-4V captions than with author captions, rather than relying on preference rankings.
  • The authors' own caution that the dataset excludes fields like biology and medicine suggests the result should be re-run on biomedical figures, where caption conventions differ, before generalizing beyond the paper's scope.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports on the first SciCap Challenge (2023), compares the challenge-winning systems with GPT-4V and open LMMs, and asks whether advanced large multimodal models have solved scientific figure caption generation. Using automatic metrics and a human evaluation by three professional editors, the study finds that GPT-4V (image+paragraph input) is overwhelmingly preferred over all other model outputs and over author-written captions, across three length conditions and on a post-cutoff unseen dataset. The paper also documents dataset expansion, challenge procedures, inter-annotator agreement, and a secondary comparison with prior reader studies.

Significance. If the findings hold, the paper provides a valuable snapshot of the state of scientific figure captioning: it documents the first community challenge, releases a much larger SciCap version, and offers a carefully controlled human evaluation with inter-annotator agreement (Kendall's Tau 0.53-0.71, Kappa@1 0.53-0.81). The replication on post-2024 arXiv papers is a strong positive feature, as is the transparent reporting of the evaluation protocol. The main risk is external validity: the headline result rests on three Upwork-recruited editors, and the paper's own Section 5 shows that other user groups may disagree. The work is nevertheless a useful benchmark contribution for the community.

major comments (4)
  1. [Section 4.2, Table 2] The claim that "The Leaderboard Winner, NJUST, consistently scored the highest in every category" is directly contradicted by the physics row of Table 2, where Pegasus achieves higher ROUGE-2-Normalized (2.045 vs. 1.986), higher ROUGE-1-Normalized, and higher BLEU-4 than NJUST. This also undermines the immediately following statement that "Model score rankings were consistent across categories." The text must be corrected to note the exception, or the claim should be qualified to specific metrics/domains.
  2. [Section 4.3, Condition 3 (Figure 4C)] The length-control condition fails: the paper states that in Condition 3 the human-written captions average 39.68 words while GPT-4V's captions average 44.59 words, despite the prompt instructing GPT-4V to generate captions "no longer than the corresponding human-written captions." Thus this condition does not actually control for length, and the conclusion that GPT-4V's superiority persists under strict length constraints is not supported by Condition 3. The authors should either enforce the constraint (e.g., post-hoc truncation or stricter prompting) or soften the claim to rest on Condition 2 only.
  3. [Section 4.4, Footnote 5] Study 3, which is presented as evidence of generalizability to post-cutoff papers, uses gpt-4-0125-preview rather than the gpt-4-vision-preview used in the main study. The paper does not explain whether gpt-4-0125-preview accepts image inputs; if it is a text-only model, then the Figure 5 human evaluation and Tables 4-5 are not evaluating a multimodal model, and the conclusion that "GPT-4V's captions were consistently favored" on unseen data is not supported. Please clarify the exact model version, its vision capability, and the API settings used.
  4. [Sections 4.3, 5, and 8] The abstract's central claim, that "professional editors overwhelmingly preferred" GPT-4V captions, is a generalization from three Upwork-recruited editors. While the paper acknowledges the small panel in Section 8, the headline claim is still phrased generically, and Section 5 shows that another user group (undergraduates) can disagree with editors' quality judgments. The claim should be explicitly restricted to the three study editors, or the paper should provide a larger and more representative editor sample, before asserting a general preference among professional editors. The current evidence supports a statement about the specific judges, not the broader population implied by the title and abstract.
minor comments (5)
  1. [Section 4.3, Figure 4] The paper says GPT-4V was "overwhelmingly preferred" but does not report the concrete proportion of figures where GPT-4V was ranked first or in the top two; providing these numbers would make the strength of the preference quantitative and more informative.
  2. [Section 3.1, Quality Subset] The manual evaluation step that selected 460 captions from the 769 GPT-4-passing captions is not described; please add details on who performed it, the criteria, and inter-annotator agreement, if any.
  3. [Section 5] The comparison with Hsu et al. (2023) is cross-study and indirect, as the paper acknowledges in Section 8; in Section 5 the conclusions are stated rather strongly, and it would be helpful to label these as exploratory or suggestive.
  4. [Section 4.2] The text does not define the ROUGE normalization factor; please state explicitly how the normalized scores are computed (e.g., ROUGE score divided by caption length) or point to the exact formula.
  5. [General] The paper alternates between "S CICAP" and "SCICAP" in several places; please standardize the spelling throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical human-evaluation result, not a derivation fitted to its own inputs.

full rationale

The paper's key claim, that professional editors overwhelmingly preferred GPT-4V captions over all other models and over author-written captions, is an empirical observation from a human ranking study (Section 4.3), replicated on an unseen post-cutoff dataset in Study 3 (Section 4.4). There is no fitted parameter that is later renamed as a prediction, no quantity defined in terms of the outcome it is said to predict, and no uniqueness theorem imported from the authors' prior work to force a choice. The authors do cite their own prior work (Hsu et al., 2021; Huang et al., 2023; Hsu et al., 2023) for dataset provenance, evaluation conventions, and the GPT-4-based quality scoring used to construct the Quality Subset, but these citations are lineage and methodology, not load-bearing for the central finding. The three-editor panel is small, which the paper itself acknowledges in Section 8, and Section 5 shows that other reader groups do not always align with editors; these are external-validity limitations, not circularity. The automatic evaluation scores even run counter to the human-preference finding, illustrating that the result is not an artifact of the evaluation setup. The manuscript is therefore self-contained with respect to its main empirical claim, and no circular step can be exhibited.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

This is an empirical benchmark paper, so the ledger captures methodological choices rather than theoretical postulates. The central claim depends on the validity of the human evaluation design, not on fitted parameters or invented entities.

free parameters (4)
  • ROUGE normalization factor = Computed on hidden test set
    Used to produce normalized ROUGE scores in automatic evaluation; a standard length-adjustment method, not a fitted constant for the main human-eval claim.
  • GPT-4 quality threshold for Quality Subset = 5 (on 1-6 scale)
    Hand-picked cutoff used to curate the Quality Subset from GPT-4 scores; affects the Quality Winner track but not the main GPT-4V versus author finding.
  • 25-word length constraint = 25 words
    Chosen to align GPT-4V caption length with the cs-domain average (25.52 words); a design choice for a robustness condition.
  • Sample size per evaluation condition = 100 figures
    Chosen by the authors for each of three conditions; affects statistical precision but not the direction of results.
assumptions (4)
  • domain assumption Professional editors' rankings are a valid measure of caption quality for the intended audience.
    The paper's central claim rests on the assumption that the three editors' preferences reflect what makes captions useful to readers; the paper acknowledges this assumption is limited by the small, specialized judge pool (Section 8).
  • domain assumption The 100 figures sampled per condition are representative of the hidden test set.
    Random sampling from the hidden test set is described, but the paper does not report the exact seed or sample IDs, so representativeness is assumed.
  • domain assumption The post-cutoff dataset (papers published January to March 2024) is free from GPT-4V data contamination.
    The paper assumes that figures from papers published after the model's cutoff date have not been seen during pretraining; this is a reasonable empirical assumption but cannot be fully verified for a proprietary model.
  • domain assumption GPT-4V and gpt-4-0125-preview produce comparable caption quality for the purpose of generalizing Study 2 to Study 3.
    The main human eval uses gpt-4-vision-preview, while the generalization study uses gpt-4-0125-preview; the paper does not establish that these models behave equivalently for caption generation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023." pith.science (2026). https://pith.science/paper/T5W3J5KI

@misc{pith2026250119353,
  author       = {Pith},
  title        = {Pith review of: Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T5W3J5KI}},
  note         = {Machine review of arXiv:2501.19353}
}
read the original abstract

Since the SciCap datasets launch in 2021, the research community has made significant progress in generating captions for scientific figures in scholarly articles. In 2023, the first SciCap Challenge took place, inviting global teams to use an expanded SciCap dataset to develop models for captioning diverse figure types across various academic fields. At the same time, text generation models advanced quickly, with many powerful pre-trained large multimodal models (LMMs) emerging that showed impressive capabilities in various vision-and-language tasks. This paper presents an overview of the first SciCap Challenge and details the performance of various models on its data, capturing a snapshot of the fields state. We found that professional editors overwhelmingly preferred figure captions generated by GPT-4V over those from all other models and even the original captions written by authors. Following this key finding, we conducted detailed analyses to answer this question: Have advanced LMMs solved the task of generating captions for scientific figures?

Figures

Figures reproduced from arXiv: 2501.19353 by the authors.

Figure 1
Figure 1. In S [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. ROUGE-2 normalized scores of each model across eight arXiv domains, highlighting similar [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. ROUGE-2 scores by model across five figure types, showing similar trends. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Rankings of generated captions by all models in Study 2 across three evaluation conditions [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Rankings of captions generated by different models on an unseen dataset of arXiv papers [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Ph.D. students’ ranking results from Hsu et al. (2023). Note that lower ranks mean better performance [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Undergraduate students’ ratings on help [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Two examples where experts favored GPT-4V’s captions for providing sufficient details and [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: The drag-and-drop interface used by professional editors to rank captions for a figure. [Figure [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 27 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Simone Balloccu, Patr \' cia Schmidtov \'a , Mateusz Lango, and Ondrej Dusek. 2024. https://aclanthology.org/2024.eacl-long.5 Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLM s . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    work page 2024

  4. [4]

    John Bransford. 1979. Human cognition: Learning, understanding, and remembering. (No Title)

    work page 1979

  5. [5]

    Vitaly Bulgakov. 2016. Iterative aggregation method for solving principal component analysis problems. arXiv preprint arXiv:1602.08800

    work page Pith review arXiv 2016

  6. [6]

    Dian Chao, Xin Song, Shupeng Zhong, Boyuan Wang, Xiangyu Wu, Chen Zhu, and Yang Yang. 2023. The solution for the iccv 2023 1st scientific figure captioning challenge. arXiv preprint

    work page 2023

  7. [7]

    Charles Chen, Ruiyi Zhang, Eunyee Koh, Sungchul Kim, Scott Cohen, and Ryan Rossi. 2020. Figure captioning with relation maps for reasoning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1537--1545

    work page 2020

  8. [8]

    Christopher Clark and Santosh Divvala. 2016. Pdffigures 2.0: Mining figures from research papers. In Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries, pages 143--152

    work page 2016

Show all 59 references
  1. [9]

    Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. https://doi.org/10.18653/v1/P19-1483 Handling divergent reference texts when evaluating table-to-text generation . In Proceedings of the 57th Annual Meeting of the Association ...

    2019 doi

  2. [10]

    Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, et al. 2020. Pp-ocr: A practical ultra lightweight ocr system. arXiv preprint arXiv:2009.09941

    2020 arXiv

  3. [11]

    Hal Finkel, David Poliakoff, Jean-Sylvain Camier, and David F Richards. 2019. Clangjit: Enhancing c++ with just-in-time compilation. In 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), pages 82--95. IEEE

    2019

  4. [12]

    Andrew Gelman, Cristian Pasarica, and Rahul Dodhia. 2002. Let's practice what we preach: turning tables into graphs. The American Statistician, 56(2):121--130

    2002

  5. [13]

    Dimitra Gkatzia, Helen Hastie, and Oliver Lemon. 2014. https://doi.org/10.3115/v1/E14-4041 Finding middle ground? multi-objective natural language generation from time-series data . In Proceedings of the 14th Conference of the E uropean Chapter of the Association for Computati...

    2014 doi

  6. [14]

    James Hartley. 2003. Single authors are not alone: Colleagues often help. Journal of Scholarly Publishing, 34(2):108--113

    2003

  7. [15]

    Mary Hegarty and Marcel-Adam Just. 1993. Constructing mental models of machines from text and diagrams. Journal of memory and language, 32(6):717--742

    1993

  8. [16]

    Sameera Horawalavithana, Sai Munikoti, Ian Stewart, and Henry Kvinge. 2023. Scitune: Aligning large language models with scientific multimodal instructions. arXiv preprint arXiv:2307.01139

    2023 arXiv

  9. [17]

    Ting-Yao Hsu, C Lee Giles, and Ting-Hao Huang. 2021. Scicap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3258--3264

    2021

  10. [18]

    Ting-Yao Hsu, Chieh-Yang Huang, Shih-Hong Huang, Ryan Rossi, Sungchul Kim, Tong Yu, C Lee Giles, and Ting-Hao K. Huang. 2024. Scicapenter: Supporting caption composition for scientific figures with machine-generated captions and ratings. In Extended Abstracts of the 2024 CHI C...

    2024

  11. [19]

    Ting-Yao Hsu, Chieh-Yang Huang, Ryan Rossi, Sungchul Kim, C Giles, and Ting-Hao Huang. 2023. Gpt-4 as an effective zero-shot evaluator for scientific figure captions. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5464--5474

    2023

  12. [20]

    Chieh-Yang Huang, Ting-Yao Hsu, Ryan Rossi, Ani Nenkova, Sungchul Kim, Gromit Yeuk-Yin Chan, Eunyee Koh, Clyde Lee Giles, and Ting-Hao 'Kenneth' Huang. 2023. http://arxiv.org/abs/2302.12324 Summaries as captions: Generating figure captions for scientific documents with automat...

    2023 arXiv

  13. [21]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  14. [22]

    Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. Chart-to-text: A large-scale benchmark for chart summarization. arXiv preprint arXiv:2203.06486

    2022 arXiv

  15. [23]

    Zeba Karishma, Shaurya Rohatgi, Kavya Shrinivas Puranik, Jian Wu, and C Lee Giles. 2023. Acl-fig: A dataset for scientific figure classification. arXiv preprint arXiv:2301.12293

    2023 arXiv

  16. [24]

    Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, and Noah A. Smith. 2022. https://doi.org/10.18653/v1/2022.naacl-main.254 Transparent human evaluation for image captioning . In Proceedings of the 2022 Conference of the North American ...

    2022 doi

  17. [25]

    Jaeyoung Kim, Jongho Lee, Hong-Jun Choi, Ting-Yao Hsu, Chieh-Yang Huang, Sungchul Kim, Ryan Rossi, Tong Yu, Clyde Lee Giles, Ting-Hao'Kenneth' Huang, et al. 2025. Multi-llm collaborative caption generation in scientific documents. arXiv preprint arXiv:2501.02552

    2025 arXiv

  18. [26]

    Graham, F.Q

    Rodney Michael Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Go...

    2023 arXiv

  19. [27]

    Andrew Large, Jamshid Beheshti, Alain Breuleux, and Andre Renaud. 1995. Multimedia and comprehension: The relationship among text, animation, and captions. Journal of the American society for information science, 46(5):340--347

    1995

  20. [28]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. http://arxiv.org/abs/2301.12597 Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    2023 arXiv

  21. [29]

    Shengzhi Li and Nima Tajbakhsh. 2023. Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs. arXiv preprint arXiv:2308.03349

    2023 arXiv

  22. [30]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  23. [31]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning

    2023

  24. [32]

    Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. https://doi.org/10.18653/v1/2022.acl-long.207 BRIO : Bringing order to abstractive summarization . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2022 doi

  25. [33]

    Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. 2023. http://arxiv.org/abs/2305.14761 Unichart: A universal vision-language pretrained model for chart comprehension and reasoning

    2023 arXiv

  26. [34]

    Ho Yin Sam Ng, Ting-Yao Hsu, Jiyoo Min, Sungchul Kim, Ryan A Rossi, Tong Yu, Hyunggu Jung, and Ting-Hao Kenneth Huang. 2025. Understanding how paper writers use ai-generated captions in figure caption writing. In 2nd AI4Research Workshop: Towards a Knowledge-grounded Scientifi...

    2025

  27. [35]

    Gwen C Nugent. 1983. Deaf students' learning from captioned instruction: The relationship between the visual and caption display. The Journal of Special Education, 17(2):227--234

    1983

  28. [36]

    Jason Obeid and Enamul Hoque. 2020. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. arXiv preprint arXiv:2010.09142

    2020 arXiv

  29. [37]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  30. [38]

    Siemenn, Saisamrit Surbehera, Zad Chin, Keith Tyser, Gregory Hunter, Arvind Raghavan, Yann Hicke, Bryan A

    Vitali Petsiuk, Alexander E. Siemenn, Saisamrit Surbehera, Zad Chin, Keith Tyser, Gregory Hunter, Arvind Raghavan, Yann Hicke, Bryan A. Plummer, Ori Kerret, Tonio Buonassisi, Kate Saenko, Armando Solar-Lezama, and Iddo Drori. 2022. http://arxiv.org/abs/2211.12112 Human evaluat...

    2022 arXiv

  31. [39]

    Jason Phang, Yao Zhao, and Peter J. Liu. 2022. http://arxiv.org/abs/2208.04347 Investigating efficiently extending transformers for long input summarization

    2022 arXiv

  32. [40]

    Xin Qian, Eunyee Koh, Fan Du, Sungchul Kim, Joel Chan, Ryan A Rossi, Sana Malik, and Tak Yeon Lee. 2021. Generating accurate caption units for figure captioning. In Proceedings of the Web Conference 2021, pages 2792--2804

    2021

  33. [41]

    Raian Rahman, Rizvi Hasan, Abdullah Al Farhad, Md Tahmid Rahman Laskar, Md Hamjajul Ashmafee, and Abu Raihan Mostofa Kamal. 2023. Chartsumm: A comprehensive benchmark for automatic chart summarization of long and short summaries. arXiv preprint arXiv:2304.13620

    2023 arXiv

  34. [42]

    google research. 2022. Python rouge implementation. https://github.com/google-research/google-research/tree/master/rouge

    2022

  35. [43]

    Shaurya Rohatgi, Yanxia Qin, Benjamin Aw, Niranjana Unnithan, and Min-Yen Kan. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.640 The ACL OCL corpus: Advancing open science in computational linguistics . In Proceedings of the 2023 Conference on Empirical Methods in Natural ...

    2023 doi

  36. [44]

    Noah Siegel, Zachary Horvitz, Roie Levin, Santosh Divvala, and Ali Farhadi. 2016. Figureseer: Parsing result-figures in research papers. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part VII 14, pages 6...

    2016

  37. [45]

    Avneet Singh. 2017. Gravitational wave transient signal emission via ekman pumping in neutron stars during post-glitch relaxation phase. Physical Review D, 95(2):024022

    2017

  38. [46]

    Simeng Sun, Ori Shapira, Ido Dagan, and Ani Nenkova. 2019. https://doi.org/10.18653/v1/W19-2303 How to compare summarizers without target length? pitfalls, solutions and re-examination of the neural summarization literature . In Proceedings of the Workshop on Methods for Optim...

    2019 doi

  39. [47]

    Tim Tarsi, Heike Adel, Jan Hendrik Metzen, Dan Zhang, Matteo Finco, and Annemarie Friedrich. 2024. Sciol and mulms-img: Introducing a large-scale multimodal scientific dataset and models for image-text tasks in the scientific domain. In Proceedings of the IEEE/CVF Winter Confe...

    2024

  40. [48]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  41. [49]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  42. [50]

    Satoshi Tsutsui and David Crandall. 2017. A Data Driven Approach for Compound Figure Separation Using Convolutional Neural Networks . In The IAPR International Conference on Document Analysis and Recognition (ICDAR)

    2017

  43. [51]

    Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, et al. 2024. Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning. arXiv preprint arXiv:2402.12185

    2024 arXiv

  44. [52]

    Deshraj Yadav, Rishabh Jain, Harsh Agrawal, Prithvijit Chattopadhyay, Taranjeet Singh, Akash Jain, Shiv Baran Singh, Stefan Lee, and Dhruv Batra. 2019. http://arxiv.org/abs/1902.03570 Evalai: Towards better evaluation systems for ai agents

    2019 arXiv

  45. [53]

    Zhishen Yang, Raj Dabre, Hideki Tanaka, and Naoaki Okazaki. 2023. Scicap+: A knowledge augmented dataset to study the challenges of scientific figure captioning. arXiv preprint arXiv:2306.03491

    2023 arXiv

  46. [54]

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257

    2023 arXiv

  47. [55]

    Jun Yu, Xiaohua Qi, Biao Zhang, and Qiang Ling. 2023. A solution to the 1st scientific figure captioning (scicap) challenge. arXiv preprint

    2023

  48. [56]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2023. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502

    2023 arXiv

  49. [57]

    Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328--11339. PMLR

    2020

  50. [58]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  51. [59]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.