Pith. sign in

REVIEW 3 major objections 6 minor 49 references

Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multimodal large language models can match events to images, but TempVS shows they largely fail to reason about the order of those events in time.

desk verdict TempVS is solid benchmark infrastructure, but only MT1 actually tests temporal reasoning; MT2 and MT3 reduce to grounding plus reading off order supplied by one modality. read the letter →

arxiv 2506.10415 v1 pith:7EUIV3CI submitted 2025-06-12 cs.CL cs.CV

Yingjin Song , Yupei Du , Denis Paperno , Albert Gatt This is my paper · ORCID
classification cs.CLcs.CV
keywords temporalreasoningmultimodallargelanguagemodelsimagesequenceunderstandingeventrelationinferencesentenceorderingvisualgroundingbenchmarkevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TempVS asks whether multimodal large language models truly understand the order of events in image sequences, rather than just recognizing what is in each image. The paper builds 2,085 visual-story sequences and 15,192 multiple-choice questions across three tasks—event-relation inference, sentence ordering, and image ordering—each paired with a basic event-grounding test. Across 38 models, the authors find that models can often ground a text event to the correct image yet fail when the question requires ordering events in time, with the best models still far below human accuracy on relation inference and image ordering. If the paper is right, current multimodal models lack a distinct capability for multi-event temporal reasoning that is not captured by single-image benchmarks, and simple step-by-step prompting will not fix it.

What carries the argument

The carrying mechanism is the paired test design of TempVS: every main test (relation inference, sentence ordering, image ordering) has a matching grounding test that asks which image corresponds to a described event, so the benchmark can separate 'can the model see the event?' from 'can the model order the events?'. The items are generated from four visual-story datasets using statement templates that express the same event pair in iconic and non-iconic word orders (before/then vs. after/earlier), and a filtering pipeline removes sequences with stative verbs, near-duplicate captions or images (BERTScore and CLIP thresholds), ambiguous image-event pairs, and any question a text-only LLM can answer without images. This machinery is what lets the paper attribute failures to temporal reasoning rather than to perception, language bias, or world knowledge.

What would settle it

Permute the image order while keeping the same images and re-run MT1: any model that still answers near its original accuracy is using something other than the visual event order, which would undercut the claim that TempVS measures temporal understanding.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that state-of-the-art multimodal models do not genuinely capture the order of events in image sequences. Across 38 MLLMs on 15,192 questions, grounding tests show models can often find the image that matches a described event, but the paired main tests—deciding whether a before/after statement is true, reordering shuffled sentences, and reordering shuffled images—drop sharply, with image ordering near random for most models and the best system reaching only 53.8%. Human annotators score 77.9–89.3% on the same tasks. The paper reads this as a dissociation: matching events to images and reasoning about their temporal order are different capabilities, and current models have the first without the second.

Load-bearing premise

The whole measurement rests on the source datasets' annotated image order being true and on the filtering pipeline removing every shortcut, so if either is wrong the reported grounding-versus-reasoning gap is inflated.

Editorial extensions

If this is right

  • Applications that rely on MLLMs to read multi-image narratives—comics, photo albums, storyboards, surveillance logs—should not assume temporal comprehension even when single-event recognition looks strong.
  • The grounding-test design gives a reusable template: evaluating any multi-event capability should pair a main task with a prerequisite perception task to attribute failures correctly.
  • Chain-of-thought prompting improves sentence and image ordering substantially but leaves event-relation inference nearly unchanged, so reasoning-time compute alone is not a sufficient fix.
  • Scale and preference-optimization post-training consistently raise TempVS scores, pointing to training-objective work rather than architecture alone as the path forward.
  • The benchmark's public data and code allow direct comparison of future models against these 38 results.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the grounding-ordering dissociation predicts that fine-tuning on pairwise order discrimination will improve TempVS-style tasks more than additional grounding data will.
  • The iconicity effect—better accuracy when word order mirrors image order—suggests models may rely on surface-order heuristics, and a TempVS variant built with non-iconic temporal constructions would expose this directly.
  • The authors' acknowledged data-leakage uncertainty for closed models implies the numerical gap between GPT-4o and open models should be treated as an upper bound until tested on fresh unpublished image sequences.
  • Extending TempVS to video clips or open-ended generation could show whether motion cues or generative planning give models temporal structure that static frame ordering does not.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces TempVS, a benchmark for multi-event temporal grounding and reasoning in image sequences, built from four visual-story datasets (FlintstonesSV, PororoSV, VIST, VWP). It contains three main tests—event relation inference (MT1), sentence ordering (MT2), and image ordering (MT3)—each accompanied by grounding tests (GT). The authors evaluate 38 multimodal LLMs of varying sizes and report that models perform far below humans, especially on event relation inference and image ordering, while often being able to ground individual events to images. They also analyze the effect of temporal expressions, event distance, and chain-of-thought prompting. The data and evaluation code are released.

Significance. If the central claim is correct, TempVS is a useful diagnostic for a genuine weakness of current MLLMs: the ability to ground individual events without understanding their temporal order. The paper's strengths include broad model coverage (38 models), the grounding-test control, filtering against text-only shortcuts with three unimodal LLMs, a human study with reasonably high inter-annotator agreement, and public release of data and code. However, the task-validity issue for MT2 and MT3, described in the major comments, means that the paper currently overstates the breadth of the measured temporal-reasoning deficit. MT1 alone is a valid temporal-reasoning test, and the resource may still be valuable after the claims are re-scoped or the ordering tasks are redesigned.

major comments (3)
  1. [Section 3.1, Appendix C] MT2 and MT3 do not require inferring temporal order from the image sequence, because the target order is already supplied by one input modality. In MT2, the images are presented in an ordered, labeled sequence; after a model grounds each sentence to an image, it can read off the sentence order directly from the image indices. In MT3, the event/caption text is listed in chronological order (see the prompt example in Appendix C: 'The events are: Some people came... The girls enjoyed... We played...'); after matching each text item to an image, the model outputs the images in text order. Thus both tasks primarily measure cross-modal grounding and sequence copying, not temporal reasoning. The headline claim in the abstract and Section 4.2 that models 'struggle with image ordering' is therefore not evidence of a temporal-reasoning deficit; it is evidence of grounding difficulty in shuffled-image conditions. I recommend re-scoping the paper's temporal-reasoning claims to MT1, or redesigning MT2/MT3 so that neither modality provides the target order (for example, by shuffling both sentences and images and requiring the model to align them by event content).
  2. [Section 3.2, Appendix A] The benchmark's ground-truth order is inherited from four source datasets without verification that the source image order is the intended chronological event order. This is especially concerning for VIST, which is built from Flickr albums with user-written stories, and VWP, which aligns movie synopses with scene sequences; in both cases, captions may reflect narrative or album organization rather than strict temporal progression. The manual quality check described in Section 3.2 removes ambiguous images and mismatched text, but it does not validate that the source order is chronologically correct. If some source sequences are not truly ordered, MT1 accuracy is artificially deflated and the human-model gap is overstated. Please add a validation study (e.g., human judgments of source order on a sample) or report per-source results to show that the main findings are consistent across datasets.
  3. [Section 4.2, Table 19] Table 19 shows that with images only, the best models perform near chance on the image ordering task (e.g., InternVL2.5-78B-MPO at 27.09% vs. 20% random). The paper interprets this as evidence that the benchmark requires joint text-image understanding, but it is also direct evidence that models are not extracting temporal order from visual content in the main MT3 setting; the text is acting as a scaffold that supplies the order. This reinforces the concern raised in the first major comment and should be discussed explicitly when interpreting the MT3 results.
minor comments (6)
  1. [Table 1] The template line 'Neg: Ei. Ealier, Ej' contains a typo ('Ealier' should be 'Earlier'); there are also capitalization inconsistencies in the templates (e.g., 'Then' vs. 'then').
  2. [Appendix D.1] The sentence 'The usage of a combined image was alsp motivated by practical constraints' contains a typo ('alsp' should be 'also').
  3. [Table 3] The column header 'GTs' is not defined in the table caption; please define it as the strict grounding test (GT strict) and use consistent terminology throughout.
  4. [Tables 14-16] Many MT|GTstrict entries in the appendix are computed over very small denominators (e.g., 1 or 0 sequences), which can produce misleading values such as 100% or 0% based on a single instance. Please consider suppressing such entries or reporting the underlying sample size.
  5. [Section 4.2] The paper does not report confidence intervals or significance tests for model comparisons, yet several claims rest on differences of only 1-2 percentage points (e.g., two-event vs. three-event relation inference for the strongest models). Adding such statistics would strengthen the analysis.
  6. [Appendix B] The human evaluation reports only aggregate accuracy and Fleiss' kappa. Reporting agreement and accuracy per source dataset or per statement subtype would help assess where human performance is robust.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: TempVS is an externally grounded benchmark; no fitted prediction, load-bearing self-citation, or definitional equivalence found.

full rationale

TempVS is a benchmark-construction and evaluation paper, not a fitted derivation. Its ground truth is inherited from four external visual-story datasets (FlintstonesSV, PororoSV, VIST, VWP), where image order defines event chronology, plus human annotator responses on a 280-sequence sample; no target value is defined by the models evaluated or by the authors' earlier results. GPT-4 event extraction, CLIP/BERTScore filtering, and text-only LLM filtering are data-cleaning tools that remove shortcuts, but they do not define the answers. The empirical accuracy numbers are measured, not predicted from fitted parameters; no parameter is fit to a subset and then renamed as a prediction. The benchmark comparisons cited (MileBench, Mementos, MuirBench, etc.) are prior external work by other groups, and there is no load-bearing self-citation, uniqueness theorem, or ansatz imported from the authors' own papers. The skeptical observation that MT2 and MT3 can be solved by grounding plus one-modality order information is a construct-validity or correctness concern, not a circularity of the derivation chain by the criteria here; moreover, MT1 independently requires cross-modal temporal verification between text and image order. The paper's own Limitations section identifies data-contamination uncertainty for closed models, which is an honest disclosure rather than a circular step. Therefore the analysis finds no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The benchmark's validity rests on five main assumptions: source dataset ordering, filtering effectiveness, human ground truth, input format equivalence, and contamination status. The filtering thresholds are hand-tuned parameters. No new physical or conceptual entities are postulated; TempVS is a dataset, not a theoretical construct.

free parameters (3)
  • BERTScore text-similarity thresholds = FlintstonesSV P/R<0.98, F1<0.96; PororoSV <0.96/<0.96/<0.95; VWP <0.98/<0.98/<0.97; VIST <0.92/<0.92/<0.90
    Chosen by manual inspection and empirical tuning per dataset; they determine which sequences are removed and therefore affect benchmark difficulty (Appendix A.2).
  • CLIP image-similarity thresholds = FlintstonesSV <0.94; PororoSV <0.90; VWP <0.95; VIST <0.88
    Hand-tuned thresholds to remove similar images; they influence event distinctness and task difficulty (Appendix A.2).
  • Detectron2 PERSON detection threshold = at least 60% of images must contain a person
    A hand-chosen threshold to retain character-centered sequences; it shapes the domain composition of the benchmark (Section 3.2).
assumptions (5)
  • domain assumption Source visual story datasets provide reliable ground-truth temporal order for image sequences and captions.
    TempVS builds all tasks on FlintstonesSV, PororoSV, VIST, and VWP assuming image order equals event order; no independent verification of this ordering is provided (Section 3.2).
  • domain assumption The filtering pipeline removes all shortcuts and ambiguities so that correct answers require multimodal temporal reasoning.
    Relies on stative-verb removal, manually tuned BERTScore/CLIP thresholds, GPT-4 event extraction, and three text-only LLMs to filter text-answerable questions; these heuristics are not proven to eliminate all unimodal or associative shortcuts (Section 3.2, Appendix A).
  • domain assumption Human annotations on TempVS are reliable ground truth.
    Human evaluation on 280 sequences shows agreement (kappa above 0.68), but only a subset is validated and annotator errors are not modeled (Table 4, Appendix B).
  • domain assumption Merging multiple images into a single input preserves the temporal reasoning ability being measured.
    The authors found little difference between merged and separate inputs for 6 models, but the main evaluation uses merged images; this format could still affect temporal reasoning in other models (Section 4.1, Appendix D.1).
  • domain assumption Evaluated open-source models were not trained on TempVS, and closed models may have contamination.
    The authors state no evidence of training on TempVS but acknowledge data leakage for GPT-4o is unclear (Limitations); contamination would inflate performance, not the reported weakness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?." pith.science (2026). https://pith.science/paper/7EUIV3CI

@misc{pith2026250610415,
  author       = {Pith},
  title        = {Pith review of: Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7EUIV3CI}},
  note         = {Machine review of arXiv:2506.10415}
}
read the original abstract

This paper introduces the TempVS benchmark, which focuses on temporal grounding and reasoning capabilities of Multimodal Large Language Models (MLLMs) in image sequences. TempVS consists of three main tests (i.e., event relation inference, sentence ordering and image ordering), each accompanied with a basic grounding test. TempVS requires MLLMs to rely on both visual and linguistic modalities to understand the temporal order of events. We evaluate 38 state-of-the-art MLLMs, demonstrating that models struggle to solve TempVS, with a substantial performance gap compared to human capabilities. We also provide fine-grained insights that suggest promising directions for future research. Our TempVS benchmark data and code are available at https://github.com/yjsong22/TempVS.

Figures

Figures reproduced from arXiv: 2506.10415 by the authors.

Figure 1
Figure 1. Illustrative examples from the main tests of TempVS benchmark. Additional examples are provided in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustrative TempSV benchmark results of selected models with different number of parameters. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Accuracy of the two-event relation inference [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Qualitative cases of four representative [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: An example of task instruction and example [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Example question interface used by the human [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 24 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  5. [5]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  6. [6]

    Emanuele Bugliarello, Laurent Sartran, Aishwarya Agrawal, Lisa Anne Hendricks, and Aida Nematzadeh. 2023. https://doi.org/10.18653/v1/2023.acl-long.87 Measuring progress in fine-grained vision-and-language understanding . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1559--1582, T...

    work page doi:10.18653/v1/2023.acl-long.87 2023

  7. [7]

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. Janus-pro: Unified multimodal understanding and generation with data and model scaling

    work page 2025

  8. [8]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185--24198

    2024

Show all 49 references
  1. [9]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/9a6a435e75419a836fe47ab6793623e6-Abstract-Conference.html Instructblip: Towards gener...

    2023

  2. [10]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. 2023. https://doi.org/10.48550/ARXIV.2306.13394 MME: A comprehensive evaluation benchmark for multimodal large language mo...

    2023 doi arXiv

  3. [11]

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148--166. Springer

    2024

  4. [12]

    Team Gemini, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  5. [13]

    Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. 2018. Imagine this! scripts to compositions to videos. In Proceedings of the European conference on computer vision (ECCV), pages 598--613

    2018

  6. [14]

    Xudong Hong, Asad Sayeed, Khushboo Mehra, Vera Demberg, and Bernt Schiele. 2023. Visual writing prompts: Character-grounded story generation with curated image sequences. Transactions of the Association for Computational Linguistics, 11:565--581

    2023

  7. [15]

    Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. 2016. Visual storytelling. In Proceedings of the 2016 conference of the North American chapter of the association...

    2016

  8. [16]

    Yupan Huang, Zaiqiao Meng, Fangyu Liu, Yixuan Su, Nigel Collier, and Yutong Lu. 2024. https://openreview.net/forum?id=oq5EF8parZ Sparkles: Unlocking chats across multiple images for multimodal instruction-following models

    2024

  9. [17]

    Ku, Qian Liu, and Wenhu Chen

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max W.F. Ku, Qian Liu, and Wenhu Chen. 2024. https://openreview.net/forum?id=skLtdUVaJa Mantis: Interleaved multi-image instruction tuning . Transactions on Machine Learning Research, 2024

    2024

  10. [18]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159--174

    1977

  11. [19]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 a . https://arxiv.org/abs/2408.03326 Llava-onevision: Easy visual task transfer . Preprint, arXiv:2408.03326

    2024 arXiv

  12. [20]

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024 b . Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299--13308

    2024

  13. [21]

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024 c . https://arxiv.org/abs/2407.07895 Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models . Preprint, arXiv:2407.07895

    2024 arXiv

  14. [22]

    Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, et al. 2024 d . A survey on benchmarks of multimodal large language models. arXiv preprint arXiv:2408.08632

    2024 arXiv

  15. [23]

    Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. 2024 e . https://openreview.net/forum?id=BXY6fe7q31 Fine-tuning multimodal LLM s to follow zero-shot demonstrative instructions . In The Twelf...

    2024

  16. [24]

    Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao. 2019. Storygan: A sequential conditional gan for story visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, page...

    2019

  17. [25]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 a . Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  18. [26]

    Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, et al. 2024 b . Mibench: Evaluating multimodal large language models over multiple images. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan...

    2024

  19. [27]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2024 c . Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233. Springer

    2024

  20. [28]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Le...

    2024

  21. [29]

    Fanqing Meng, Chuanhao Li, Jin Wang, Quanfeng Lu, Hao Tian, Tianshuo Yang, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. 2025. https://openreview.net/forum?id=WsgEWL8i0K MMIU : Multimodal multi-image understanding for evaluating large vi...

    2025

  22. [30]

    OpenAI. 2022. https://openai.com/blog/chatgpt Introducing chatgpt

    2022

  23. [31]

    QwenTeam. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  24. [32]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  25. [33]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. https://openreview.net/forum?id=RIu5lyNXjT Quantifying language models' sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting . In The Twelfth International...

    2024

  26. [34]

    Carlota Smith. 2003. Modes of Discourse . Cambridge University Press, Cambridge

    2003

  27. [35]

    Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. 2024. https://openreview.net/forum?id=Uhwze2LEwq Milebench: Benchmarking MLLM s in long context . In First Conference on Language Modeling

    2024

  28. [36]

    Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418--6428

    2019

  29. [37]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  30. [38]

    Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and Muhao Chen. 20...

    2025

  31. [39]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024 a . Qwen2-vl: Enhancing vision-language m...

    2024 arXiv

  32. [40]

    Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Fuxiao Liu, Gedas Bertasius, Mohit Bansal, Huaxiu Yao, and Furong Huang. 2024 b . https://doi.org/10.18653/V1/2024.ACL-LONG.25 Mementos: A comprehensive benchmark for multimodal ...

    2024 doi

  33. [41]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  34. [42]

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang,...

    2024 arXiv

  35. [43]

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, jiayi lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. 2024. https://open...

    2024

  36. [44]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conferenc...

    2024

  37. [45]

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024 a . https://doi.org/10.48550/ARXIV.2406.16852 Long context transfer from language to vision . CoRR, abs/2406.16852

    2024 doi arXiv

  38. [46]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  39. [47]

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. 2024 b . https://llava-vl.github.io/blog/2024-04-30-llava-next-video/ Llava-next: A strong zero-shot video understanding model

    2024

  40. [48]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. https://openreview.net/forum?id=shr9PXz7T0 Large language models are not robust multiple choice selectors . In The Twelfth International Conference on Learning Representations

    2024

  41. [49]

    Zwaan and Gabriel A

    Rolf A. Zwaan and Gabriel A. Radvansky. 1998. http://www.ncbi.nlm.nih.gov/pubmed/9522683 Situation models in language comprehension and memory. Psychological Bulletin, 123(2):162--85

    1998

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.