Pith. sign in

REVIEW 2 major objections 2 minor 3 cited by

Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding

T0 review · 2 major / 2 minor · reviewed 2026-05-15 · grok-4.3

Challenge this review Download PDF Read on arXiv ↗

Pith's one-line read Multimodal models underperform on elementary symbol recognition yet perform better on complex reasoning tasks that use those symbols.

desk verdict The paper flags a real practical issue with MLLM symbol handling but the inversion claim rests on thin evidence and possible task confounds. read the letter →

arxiv 2603.18472 v2 submitted 2026-03-19 cs.AI cs.CV

Yinghui Li , Jiayi Kuang , Peng Xing , Daixian Liu , Yongheng Zhang , Junnan Dong , Shu-Yu Guo , Yangning Li
show 6 more authors
Qingyu Zhou Wenhao Jiang Hai-Tao Zheng Ying Shen Liang Lin Philip S. Yu
This is my paper · ORCID
classification cs.AIcs.CV
keywords multimodallargelanguagemodelssymbolrecognitioncognitivemismatchvisualgroundingdiscretesymbolsreasoninginversionbenchmarkevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes that leading MLLMs display a consistent cognitive mismatch across a multi-domain benchmark covering language, culture, mathematics, physics and chemistry. Models are weaker at basic perception and recognition of discrete symbols such as handwritten characters, formula graphs, circuit diagrams and chemical structures, yet appear more capable on higher-level combination, reasoning and critical thinking tasks. A sympathetic reader would care because the pattern suggests current systems rely on linguistic priors, template retrieval or procedural shortcuts instead of building robust visual grounding for symbolic content. This inversion implies that apparent reasoning competence may rest on weak perceptual foundations, making symbolic understanding a central remaining bottleneck for multimodal systems.

What carries the argument

The three-level cognitive benchmark that separates perception and recognition from combination and reasoning and from association and critical thinking, exposing the performance inversion on discrete symbols.

What would settle it

A controlled test in which linguistic context and priors are removed from the input while keeping the same symbol images, measuring whether recognition accuracy rises to match or exceed the reported reasoning performance.

Watch

Extended reading notes

Core claim

Across leading MLLMs, we observe a consistent cognitive mismatch. Models frequently underperform on elementary symbol recognition while appearing relatively competent on more complex reasoning tasks. This recognition-reasoning inversion indicates that current systems often compensate with linguistic priors, template retrieval or procedural reasoning instead of robust visual grounding. The pattern is especially clear for sparse, low-redundancy symbols such as handwritten characters, formula graphs, circuit diagrams and chemical structures.

Load-bearing premise

The benchmark's division into cognitive levels isolates genuine visual grounding failures rather than reflecting artifacts of task design, prompting or overlap with training data.

Editorial extensions

If this is right

  • Symbolic understanding remains a major bottleneck for multimodal intelligence.
  • Current systems compensate for weak visual grounding by drawing on language-based shortcuts.
  • Training and evaluation schemes should prioritize grounded perception in discrete semantic spaces.
  • The mismatch is clearest for low-redundancy symbols such as handwritten characters and chemical structures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the inversion persists, targeted improvements to visual encoders for discrete symbols could raise overall capability without further language scaling.
  • The same mismatch may appear in other sparse visual domains such as technical diagrams or musical notation.
  • Future work could test whether fine-tuning solely on recognition tasks closes the gap on higher-level reasoning benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces a multi-domain benchmark for discrete symbol understanding in MLLMs spanning language, culture, mathematics, physics, and chemistry, organized into three cognitive levels (perception/recognition, combination/reasoning, and association/critical thinking). It reports a consistent recognition-reasoning inversion: models underperform on elementary symbol recognition (especially sparse symbols like handwritten characters, circuit diagrams, and chemical structures) but appear more competent on complex reasoning tasks, attributing this to reliance on linguistic priors and template retrieval rather than robust visual grounding.

Significance. If the inversion is shown to arise from visual grounding failures rather than benchmark artifacts, the result would identify a substantive limitation in current MLLMs for grounded symbolic reasoning in technical domains, motivating targeted training and evaluation protocols that prioritize perceptual grounding over linguistic compensation.

major comments (2)
  1. [Methods/Benchmark] Benchmark construction (Methods section): No quantitative controls are described for symbol frequency, visual redundancy, or differential textual context across the three cognitive levels. Without these, the reported performance inversion cannot be isolated from task-design artifacts, directly undermining the central claim that the mismatch reflects absent visual grounding.
  2. [Results] Results and evaluation (Results section): The abstract and results provide no model names/versions, raw accuracy numbers, error breakdowns, or statistical tests comparing recognition versus reasoning performance. This absence prevents assessment of whether the inversion is consistent, significant, or generalizable across the claimed domains.
minor comments (2)
  1. [Abstract] The abstract would benefit from a single sentence summarizing the number of models tested and the magnitude of the observed gaps to allow readers to gauge the effect size immediately.
  2. [Benchmark Description] Notation for the three cognitive levels is introduced without a clear table or diagram mapping example items from each domain to each level; this reduces readability when interpreting the inversion pattern.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback, which highlights important areas for strengthening the methodological transparency and empirical reporting in our work. We address each major comment below and commit to revisions that enhance the rigor of the benchmark and results presentation.

read point-by-point responses

  1. Referee: [Methods/Benchmark] Benchmark construction (Methods section): No quantitative controls are described for symbol frequency, visual redundancy, or differential textual context across the three cognitive levels. Without these, the reported performance inversion cannot be isolated from task-design artifacts, directly undermining the central claim that the mismatch reflects absent visual grounding.

    Authors: We agree that the absence of explicit quantitative controls leaves room for potential task-design confounds. In the revised manuscript, we will add a dedicated subsection under Methods that reports: (1) symbol frequency and sparsity metrics (e.g., average stroke density and pixel entropy per symbol class), (2) visual redundancy measures (e.g., symmetry indices and intra-class visual similarity via perceptual hashing), and (3) textual context standardization (uniform minimal-prompt templates across levels). These additions will allow readers to better assess whether the observed inversion is attributable to visual grounding limitations rather than benchmark artifacts. revision: yes

  2. Referee: [Results] Results and evaluation (Results section): The abstract and results provide no model names/versions, raw accuracy numbers, error breakdowns, or statistical tests comparing recognition versus reasoning performance. This absence prevents assessment of whether the inversion is consistent, significant, or generalizable across the claimed domains.

    Authors: We acknowledge that the current Results section lacks the granular reporting needed for full reproducibility and statistical evaluation. The revised manuscript will expand the Results section to include: specific model names and versions (e.g., GPT-4o, Claude-3-Opus, LLaVA-1.6, etc.), full raw accuracy tables disaggregated by domain and cognitive level, categorized error breakdowns (perceptual misrecognition rates versus reasoning failures), and statistical tests (paired Wilcoxon signed-rank tests with effect sizes) comparing recognition versus reasoning performance within and across domains. These details will substantiate the consistency and generalizability of the inversion. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical benchmark observation with no derivations or self-referential reductions

full rationale

The paper introduces a multi-domain benchmark organized into three cognitive levels and reports observed performance patterns across MLLMs, noting underperformance on symbol recognition relative to reasoning tasks. No equations, fitted parameters, ansatzes, or derivation steps are present in the provided text. The central claim rests on direct model evaluations rather than any chain that reduces to its own inputs by construction. Self-citations are not invoked as load-bearing premises, and the analysis remains self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The mismatch interpretation rests on the untested premise that benchmark levels cleanly separate visual perception from linguistic reasoning.

assumptions (1)
  • domain assumption The three defined cognitive levels validly distinguish perception failures from reasoning capabilities without confounding factors.
    Invoked to label the performance gap as a cognitive mismatch rather than a methodological artifact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding." pith.science (2026). https://pith.science/paper/2603.18472

@misc{pith2026260318472,
  author       = {Pith},
  title        = {Pith review of: Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2603.18472}},
  note         = {Machine review of arXiv:2603.18472}
}
read the original abstract

Multimodal large language models (MLLMs) perform strongly on natural images, yet their ability to understand discrete visual symbols remains unclear. We present a multi-domain benchmark spanning language, culture, mathematics, physics and chemistry, organized into three cognitive levels: perception and recognition, combination and reasoning, and association and critical thinking. Across leading MLLMs, we observe a consistent cognitive mismatch. Models frequently underperform on elementary symbol recognition while appearing relatively competent on more complex reasoning tasks. This recognition-reasoning inversion indicates that current systems often compensate with linguistic priors, template retrieval or procedural reasoning instead of robust visual grounding. The pattern is especially clear for sparse, low-redundancy symbols such as handwritten characters, formula graphs, circuit diagrams and chemical structures. These results show that symbolic understanding remains a major bottleneck for multimodal intelligence and motivate training and evaluation schemes that prioritize grounded perception in discrete semantic spaces.

Figures

Figures reproduced from arXiv: 2603.18472 by the authors.

Figure 1
Figure 1. Continuous and discrete semantic spaces differ fundamentally in how visual information is organized. Discrete symbol understanding requires precise recognition of individual symbolic units and their structured relations. Specialized methods have also begun to address symbolic domains such as documents, music, art, geometry, chemistry and other structured scientific inputs28–40. However, these approaches remain fragm… view at source ↗
Figure 2
Figure 2. Performance and representative cases for language-symbol understanding across three cognitive levels. Models are evaluated on faked-character detection, misspelled-character detection and visual-semantic correction. 3/46 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance and representative cases for cultural-symbol understanding. The tasks progress from lexical grounding of emojis to idiomatic composition and culturally mediated interpretation. 5/46 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Performance and representative cases for mathematical-symbol understanding, covering function graphs and geometric figures across recognition, reasoning, and verification tasks. 7/46 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Performance and representative cases for physical-symbol understanding across mechanics and electromagnetism. Sparse physical symbols remain difficult for most models, especially in visually grounded mapping tasks. 8/46 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Performance and representative cases for chemical-symbol understanding. The benchmark examines molecular structure recognition, reaction reasoning and higher-level correction or prediction. 9/46 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Overview of the benchmark task design framework and illustrative examples. (a) The instantiation of the three-level symbolic understanding hierarchy across five distinct domains: Language, Cultural, Mathematical, Physical, and Chemical symbols. (b) Representative examp…
Figure 8
Figure 8. Figure 8: Cross-domain summary of benchmark results. (a) Radar charts illustrate the fine-grained performance of models across General, Language, Culture, Math, Physics and Chemistry domains. (b) Global performance aggregated by difficulty, with accuracy averaged across five sym…
Figure 9
Figure 9. Figure 9: Human performance on the benchmark compared with the best-performing model Gemini. tasks than on basic symbol recognition. This disparity confirms that current models do not adhere to human-like visual-cognitive logic. Instead, they appear to rely on learned linguistic…
Figure 10
Figure 10. Figure 10: Case studies from the comprehensive analysis, highlighting model failures in recognizing and integrating symbols. B.7.3 Reasoning ability sometimes exists as a compensatory mechanism A model’s language reasoning ability can, to some extent, compensate for its deficien…
Figure 11
Figure 11. Figure 11: Overall performances of different models across the five symbolic domains. 33/46 [PITH_FULL_IMAGE:figures/full_fig_p033_11.png]
Figure 12
Figure 12. Figure 12: Overview of the benchmark task design, including language symbols, cultural symbols, math symbols, chemistry symbols and physical symbols. tasks. In stark contrast, these same top-performing models often exhibited a significant performance decline in domains less repr…
Figure 13
Figure 13. Figure 13: The data construction pipeline of our benchmark. At the second level (Combination and Reasoning), the tasks move beyond static structures to symbolic reasoning under chemical laws. The model is required to classify reaction types and balance chemical equations by enfo…
Figure 14
Figure 14. Figure 14: Dataset distribution. (a) Overall data distribution across five domains. (b) Detailed task distribution specifically for Mathematical symbols. levels [PITH_FULL_IMAGE:figures/full_fig_p040_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TopoAgent: A Self-Evolving Topological Agent for Multimodal Scientific Reasoning

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A DAG-based agent framework using context isolation and runtime atomic fission reports average accuracy gains over linear agents on multimodal math, physics, and chemistry benchmarks.

  2. Latent Visual Cache for Video Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A recurrent latent visual cache in the decoder, trained with contrastive key-frame alignment and a vision-grounded GRPO reward, improves video reasoning accuracy while shortening answers.

  3. From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI

    cs.AI 2026-06 conditional novelty 4.0 of 10

    Autonomous AI becomes dependable when tool use is embedded in persistent workspaces with reusable skills, shifting evaluation from answers to task closure.

Reference graph

Works this paper leans on

126 extracted references · 126 canonical work pages · cited by 3 Pith papers

  1. [1]

    InFindings of the Association for Computational Linguistics: ACL 2024, 13590–13618 (2024)

    Caffagni, D.et al.The revolution of multimodal large language models: A survey. InFindings of the Association for Computational Linguistics: ACL 2024, 13590–13618 (2024)

    work page 2024

  2. [2]

    Data Eng.(2025)

    Song, S.et al.How to bridge the gap between modalities: Survey on multimodal large language model.IEEE Transactions on Knowl. Data Eng.(2025)

    work page 2025

  3. [3]

    InFindings of the Association for Computational Linguistics ACL 2024, 12401–12430 (2024)

    Zhang, D.et al.Mm-llms: Recent advances in multimodal large language models. InFindings of the Association for Computational Linguistics ACL 2024, 12401–12430 (2024)

    work page 2024

  4. [4]

    & Tang, Y

    Xu, J., Sun, Q., Han, Q.-L. & Tang, Y . When embodied ai meets industry 5.0: Human-centered smart manufacturing. IEEE/CAA J. Autom. Sinica12, 485–501 (2025). 5.Turgunbaev, R.et al.From perception to action with integrated vla systems.Tech. Sci. Integr. Res.1, 11–17 (2025)

    work page 2025

  5. [5]

    InProceedings of the Computer Vision and Pattern Recognition Conference, 10644–10655 (2025)

    Szot, A.et al.From multimodal llms to generalist embodied agents: Methods and lessons. InProceedings of the Computer Vision and Pattern Recognition Conference, 10644–10655 (2025)

    work page 2025

  6. [6]

    Du, C.et al.Human-like object concept representations emerge naturally in multimodal large language models.Nat. Mach. Intell.1–16 (2025)

    work page 2025

  7. [7]

    InPro- ceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024): Tutorial Summaries, 1–8 (2024)

    Fei, H.et al.From multimodal llm to human-level ai: Modality, instruction, reasoning, efficiency and beyond. InPro- ceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024): Tutorial Summaries, 1–8 (2024)

    work page 2024

  8. [8]

    InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 6613–6629 (2025)

    Shen, H.et al.Zoomeye: Enhancing multimodal llms with human-like zooming capabilities through tree-based image exploration. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 6613–6629 (2025)

    work page 2025

Show all 126 references
  1. [9]

    Perlovsky, L. I. Symbols: Integrated cognition and language. InSemiotics and intelligent systems development, 121–151 (IGI Global Scientific Publishing, 2007)

    2007

  2. [10]

    Taniguchi, T.et al.Symbol emergence in cognitive developmental systems: a survey.IEEE transactions on Cogn. Dev. Syst.11, 494–516 (2018)

    2018

  3. [11]

    Symbol and structure: a comprehensive framework for language evolution.Stud

    Bickerton, D. Symbol and structure: a comprehensive framework for language evolution.Stud. Evol. Lang.3, 77–93 (2003)

    2003

  4. [12]

    & Nóbrega, V

    Miyagawa, S., Lesure, C. & Nóbrega, V . A. Cross-modality information transfer: A hypothesis about the relationship among prehistoric cave paintings, symbolic thinking, and the emergence of language.Front. Psychol.9, 115 (2018). 14.Rapoport, A. The role of symbols in human beh...

    2018

  5. [13]

    Ahn, T., Janssen, M. A. & Ostrom, E. Signals, symbols, and human cooperation. InThe origins and nature of sociality, 122–139 (Routledge, 2017)

    2017

  6. [14]

    Abdulgalil, H. D. & Basir, O. A. Next-generation image captioning: A survey of methodologies and emerging challenges from transformers to multimodal large language models.Nat. Lang. Process. J.100159 (2025)

    2025

  7. [15]

    Surv.57, 1–36 (2025)

    Kuang, J.et al.Natural language understanding and inference with mllm in visual question answering: A survey.ACM Comput. Surv.57, 1–36 (2025)

    2025

  8. [16]

    Xiao, L., Yang, X., Lan, X., Wang, Y . & Xu, C. Towards visual grounding: A survey.IEEE Transactions on Pattern Analysis Mach. Intell.1–20, DOI: 10.1109/TPAMI.2025.3630635 (2025). 12/46

    2025 doi

  9. [17]

    Liu, Y .et al.Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, 216–233 (Springer, 2024)

    2024

  10. [18]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13299–13308 (2024)

    Li, B.et al.Seed-bench: Benchmarking multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13299–13308 (2024)

    2024

  11. [19]

    InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2025)

    Fu, C.et al.MME: A comprehensive evaluation benchmark for multimodal large language models. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2025)

    2025

  12. [20]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9556–9567 (2024)

    Yue, X.et al.Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9556–9567 (2024)

    2024

  13. [21]

    In Proceedings of the Computer Vision and Pattern Recognition Conference, 1587–1606 (2025)

    Li, Z.et al.A survey of state of the art large vision language models: Benchmark evaluations and challenges. In Proceedings of the Computer Vision and Pattern Recognition Conference, 1587–1606 (2025)

    2025

  14. [22]

    Intell.(2024)

    Xu, P.et al.Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models.IEEE Transactions on Pattern Analysis Mach. Intell.(2024)

    2024

  15. [23]

    InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2025)

    Fu, L.et al.OCRBench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2025)

    2025

  16. [24]

    InThe Twelfth International Conference on Learning Representations(2024)

    Lu, P.et al.Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InThe Twelfth International Conference on Learning Representations(2024)

    2024

  17. [25]

    Zhang, R.et al.Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InEuropean Conference on Computer Vision, 169–186 (Springer, 2024)

    2024

  18. [26]

    Liu, Y .et al.Textmonkey: An ocr-free large multimodal model for understanding document.arXiv preprint arXiv:2403.04473(2024)

    2024

  19. [27]

    Yu, Y .-Q.et al.Texthawk: Exploring efficient fine-grained perception of multimodal large language models.arXiv preprint arXiv:2404.09204(2024)

    2024

  20. [28]

    S.et al.Docvlm: Make your vlm an efficient reader

    Nacson, M. S.et al.Docvlm: Make your vlm an efficient reader. InProceedings of the Computer Vision and Pattern Recognition Conference, 29005–29015 (2025)

    2025

  21. [29]

    In Chiruzzo, L., Ritter, A

    Tang, M.et al.NOTA: Multimodal music notation understanding for visual large language model. In Chiruzzo, L., Ritter, A. & Wang, L. (eds.)Findings of the Association for Computational Linguistics: NAACL 2025, 7160–7173, DOI: 10.18653/v1/2025.findings-naacl.399 (Association for...

    2025 doi

  22. [30]

    & Chen, C

    Jiang, R. & Chen, C. W. Multimodal llms can reason about aesthetics in zero-shot. InProceedings of the 33rd ACM International Conference on Multimedia, 6634–6643 (2025)

    2025

  23. [31]

    & Castellano, G

    Fanelli, N., Vessio, G. & Castellano, G. Artseek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval.arXiv preprint arXiv:2507.21917(2025)

    2025

  24. [32]

    InThe Thirteenth International Conference on Learning Representations(2025)

    Gao, J.et al.G-LLaV A: Solving geometric problem with multi-modal large language model. InThe Thirteenth International Conference on Learning Representations(2025)

    2025

  25. [33]

    In Al-Onaizan, Y ., Bansal, M

    Shi, W.et al.Math-LLaV A: Bootstrapping mathematical reasoning for multimodal large language models. In Al-Onaizan, Y ., Bansal, M. & Chen, Y .-N. (eds.)Findings of the Association for Computational Linguistics: EMNLP 2024, 4663–4680, DOI: 10.18653/v1/2024.findings-emnlp.268 (...

    2024 doi

  26. [34]

    In Proceedings of the 33rd ACM International Conference on Multimedia, 5394–5403 (2025)

    Pan, Y .et al.Enhancing the geometric problem-solving ability of multimodal llms via symbolic-neural integration. In Proceedings of the 33rd ACM International Conference on Multimedia, 5394–5403 (2025)

    2025

  27. [35]

    38.Tan, Q.et al.Chemmllm: Chemical multimodal large language model.arXiv preprint arXiv:2505.16326(2025)

    Shi, W.et al.Multimodal mathematical reasoning with diverse solving perspective.arXiv preprint arXiv:2507.02804 (2025). 38.Tan, Q.et al.Chemmllm: Chemical multimodal large language model.arXiv preprint arXiv:2505.16326(2025)

    2025

  28. [36]

    InProceedings of the AAAI Conference on Artificial Intelligence, vol

    Li, J.et al.Chemvlm: Exploring the power of multimodal large language models in chemistry area. InProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, 415–423 (2025). 40.Zhao, Z.et al.Chemdfm-x: towards large multimodal model for chemistry.Sci. China Inf. Sc...

    2025

  29. [37]

    Hiippala, T.et al.Ai2d-rst: a multimodal corpus of 1000 primary school science diagrams.Lang. Resour. Eval.55, 661–688 (2021)

    2021

  30. [38]

    In Ku, L.-W., Martins, A

    Qin, Y .et al.InFoBench: Evaluating instruction following ability in large language models. In Ku, L.-W., Martins, A. & Srikumar, V . (eds.)Findings of the Association for Computational Linguistics: ACL 2024, DOI: 10.18653/v1/2024. findings-acl.772 (Association for Computation...

    2024 doi

  31. [39]

    InFindings of the Association for Computational Linguistics: ACL 2025, 7748–7763 (2025)

    Tang, J.et al.Mtvqa: Benchmarking multilingual text-centric visual question answering. InFindings of the Association for Computational Linguistics: ACL 2025, 7748–7763 (2025)

    2025

  32. [40]

    Neural Inf

    Wang, Y .et al.Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Adv. Neural Inf. Process. Syst.37, 95266–95290 (2024)

    2024

  33. [41]

    InEuropean Conference on Computer Vision, 148–166 (Springer, 2024)

    Fu, X.et al.Blink: Multimodal large language models can see but not perceive. InEuropean Conference on Computer Vision, 148–166 (Springer, 2024)

    2024

  34. [42]

    In Salakhutdinov, R.et al.(eds.)Proceedings of the 41st International Conference on Machine Learning, vol

    Ying, K.et al.MMT-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask AGI. In Salakhutdinov, R.et al.(eds.)Proceedings of the 41st International Conference on Machine Learning, vol. 235 ofProceedings of Machine Learning Res...

    2024

  35. [43]

    InThe Thirteenth International Conference on Learning Representations(2025)

    Meng, F.et al.MMIU: Multimodal multi-image understanding for evaluating large vision-language models. InThe Thirteenth International Conference on Learning Representations(2025)

    2025

  36. [44]

    Jiang, D.et al.Mantis: Interleaved multi-image instruction tuning.Transactions on Mach. Learn. Res.(2024). Featured Certification, Outstanding Certification

    2024

  37. [45]

    InThe Thirteenth International Conference on Learning Representations(2025)

    Chen, J.et al.MEGA-bench: Scaling multimodal evaluation to over 500 real-world tasks. InThe Thirteenth International Conference on Learning Representations(2025)

    2025

  38. [46]

    InThe Thirteenth International Conference on Learning Representations(2025)

    Wang, F.et al.Muirbench: A comprehensive benchmark for robust multi-image understanding. InThe Thirteenth International Conference on Learning Representations(2025)

    2025

  39. [47]

    InWorkshop on Reasoning and Planning for Large Language Models(2025)

    Wang, S.et al.MFC-bench: Benchmarking multimodal fact-checking with large vision-language models. InWorkshop on Reasoning and Planning for Large Language Models(2025)

    2025

  40. [48]

    & Berg, T

    Kazemzadeh, S., Ordonez, V ., Matten, M. & Berg, T. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 787–798 (2014)

    2014

  41. [49]

    InProceedings of the IEEE conference on computer vision and pattern recognition, 11–20 (2016)

    Mao, J.et al.Generation and comprehension of unambiguous object descriptions. InProceedings of the IEEE conference on computer vision and pattern recognition, 11–20 (2016)

    2016

  42. [50]

    Neural Inf

    Li, C.et al.Elevater: A benchmark and toolkit for evaluating language-augmented visual models.Adv. Neural Inf. Process. Syst.35, 9287–9301 (2022)

    2022

  43. [51]

    In Bouamor, H., Pino, J

    Li, Y .et al.Evaluating object hallucination in large vision-language models. In Bouamor, H., Pino, J. & Bali, K. (eds.)Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 292–305, DOI: 10.18653/v1/2023.emnlp-main.20 (Association for Computa...

    2023 doi

  44. [52]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14375–14385 (2024)

    Guan, T.et al.Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14375–14385 (2024)

    2024

  45. [53]

    & Evans, O

    Lin, S., Hilton, J. & Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. InProceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), 3214–3252 (2022)

    2022

  46. [54]

    & Wen, J.-R

    Li, J., Cheng, X., Zhao, X., Nie, J.-Y . & Wen, J.-R. Halueval: A large-scale hallucination evaluation benchmark for large language models. InThe 2023 Conference on Empirical Methods in Natural Language Processing(2023)

    2023

  47. [55]

    InThe Thirty-eighth Annual Conference on Neural Information Processing Systems(2024)

    Zeng, Z.et al.MR-ben: A meta-reasoning benchmark for evaluating system-2 thinking in LLMs. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems(2024)

    2024

  48. [56]

    InFirst Conference on Language Modeling(2024)

    Rein, D.et al.GPQA: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling(2024)

    2024

  49. [57]

    In Forty-second International Conference on Machine Learning(2025)

    Hao, Y .et al.Can MLLMs reason in multimodality? EMMA: An enhanced multimodal reasoning benchmark. In Forty-second International Conference on Machine Learning(2025)

    2025

  50. [58]

    & Scialom, T

    Mialon, G., Fourrier, C., Wolf, T., LeCun, Y . & Scialom, T. GAIA: a benchmark for general AI assistants. InThe Twelfth International Conference on Learning Representations(2024). 14/46

    2024

  51. [59]

    F., Alon, U., Neubig, G

    Xu, F. F., Alon, U., Neubig, G. & Hellendoorn, V . J. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, MAPS 2022, 1–10, DOI: 10.1145/3520312.3534862 (Association for Computing Machin...

    2022 doi

  52. [60]

    Uniyal, M., Singh, M., Verbruggen, G., Gulwani, S. & Le, V . One-to-many testing for code generation from (just) natural language. In Al-Onaizan, Y ., Bansal, M. & Chen, Y .-N. (eds.)Findings of the Association for Computational Linguistics: EMNLP 2024, 15397–15402, DOI: 10.18...

    2024 doi

  53. [61]

    S., Wang, Y

    Liu, J., Xia, C. S., Wang, Y . & Zhang, L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Adv. Neural Inf. Process. Syst.36, 21558–21572 (2023)

    2023

  54. [62]

    InThe Thirteenth International Conference on Learning Representations(2025)

    Jain, N.et al.Livecodebench: Holistic and contamination free evaluation of large language models for code. InThe Thirteenth International Conference on Learning Representations(2025)

    2025

  55. [63]

    & Levy, O

    Shaham, U., Ivgi, M., Efrat, A., Berant, J. & Levy, O. ZeroSCROLLS: A zero-shot benchmark for long text understanding. InThe 2023 Conference on Empirical Methods in Natural Language Processing(2023)

    2023

  56. [64]

    & Arik, S

    Jin, B., Yoon, J., Han, J. & Arik, S. O. Long-context LLMs meet RAG: Overcoming challenges for long inputs in RAG. InThe Thirteenth International Conference on Learning Representations(2025)

    2025

  57. [65]

    ∞Bench: Extending long context evaluation beyond 100K tokens

    Zhang, X.et al. ∞Bench: Extending long context evaluation beyond 100K tokens. In Ku, L.-W., Martins, A. & Srikumar, V . (eds.)Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 15262–15277, DOI: 10.18653/v1/2024.acl...

    2024 doi

  58. [66]

    In Chiruzzo, L., Ritter, A

    Wang, H.et al.Multimodal needle in a haystack: Benchmarking long-context capability of multimodal large language models. In Chiruzzo, L., Ritter, A. & Wang, L. (eds.)Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational ...

    2025 doi

  59. [67]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22195–22206 (2024)

    Li, K.et al.Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22195–22206 (2024)

    2024

  60. [68]

    Neural Inf

    Fang, X.et al.Mmbench-video: A long-form multi-shot benchmark for holistic video understanding.Adv. Neural Inf. Process. Syst.37, 89098–89124 (2024)

    2024

  61. [69]

    InProceedings of the Computer Vision and Pattern Recognition Conference, 13691–13701 (2025)

    Zhou, J.et al.Mlvu: Benchmarking multi-task long video understanding. InProceedings of the Computer Vision and Pattern Recognition Conference, 13691–13701 (2025)

    2025

  62. [70]

    Wu, H., Li, D., Chen, B. & Li, J. Longvideobench: A benchmark for long-context interleaved video-language understand- ing.Adv. Neural Inf. Process. Syst.37, 28828–28857 (2024)

    2024

  63. [71]

    InProceedings of the IEEE/CVF International Conference on Computer Vision, 22958–22967 (2025)

    Wang, W.et al.Lvbench: An extreme long video understanding benchmark. InProceedings of the IEEE/CVF International Conference on Computer Vision, 22958–22967 (2025)

    2025

  64. [72]

    & Srikumar, V

    Liu, Y .et al.TempCompass: Do video LLMs really understand videos? In Ku, L.-W., Martins, A. & Srikumar, V . (eds.) Findings of the Association for Computational Linguistics: ACL 2024, 8731–8772, DOI: 10.18653/v1/2024.findings-acl. 517 (Association for Computational Linguistic...

    2024 doi

  65. [73]

    In Proceedings of the Computer Vision and Pattern Recognition Conference, 24108–24118 (2025)

    Fu, C.et al.Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, 24108–24118 (2025)

    2025

  66. [74]

    InICLR 2024 Workshop on Large Language Model (LLM) Agents(2024)

    Cheng, K.et al.Seeclick: Harnessing GUI grounding for advanced visual GUI agents. InICLR 2024 Workshop on Large Language Model (LLM) Agents(2024)

    2024

  67. [75]

    Qiu, Z.et al.Can large language models understand symbolic graphics programs? InThe Thirteenth International Conference on Learning Representations(2025)

    2025

  68. [76]

    InThe Thirteenth International Conference on Learning Representations(2025)

    Rawles, C.et al.Androidworld: A dynamic benchmarking environment for autonomous agents. InThe Thirteenth International Conference on Learning Representations(2025)

    2025

  69. [77]

    InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2024)

    Li, W.et al.On the effects of data scale on UI control agents. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2024)

    2024

  70. [78]

    InWorkshop on Reasoning and Planning for Large Language Models(2025)

    Li, K.et al.Screenspot-pro: GUI grounding for professional high-resolution computer use. InWorkshop on Reasoning and Planning for Large Language Models(2025). 15/46

    2025

  71. [79]

    InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2025)

    Xie, T.et al.Scaling computer-use grounding via user interface decomposition and synthesis. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2025). 86.De Saussure, F. The linguistic sign.Semiot. An introductory antholo...

    2025

  72. [80]

    Valdez, J. L. C.et al.Semiotics and artificial intelligence (ai): An analysis of symbolic communication in the age of technology. InFuture of Information and Communication Conference, 481–494 (Springer, 2024)

    2024

  73. [81]

    Ai, popular culture, semiotics

    Danesi, M. Ai, popular culture, semiotics. InAI-Generated Popular Culture: A Semiotic Perspective, 1–22 (Springer, 2024). 90.Hatt, M. & Klonk, C. Semiotics. InArt history, 200–222 (Manchester University Press, 2025)

    2024

  74. [82]

    What is semiotics? InUnveiling Semiotic Codes of Fake News and Misinformation: Contemporary Theories and Practices for Media Professionals, 5–9 (Springer, 2024)

    Iskanderova, T. What is semiotics? InUnveiling Semiotic Codes of Fake News and Misinformation: Contemporary Theories and Practices for Media Professionals, 5–9 (Springer, 2024)

    2024

  75. [83]

    2509.09731

    Yu, H.et al.Benchmarking vision-language models on chinese ancient documents: From ocr to knowledge reasoning (2025). 2509.09731

    2025

  76. [84]

    In Ku, L.-W., Martins, A

    Li, Y .et al.Towards real-world writing assistance: A Chinese character checking benchmark with faked and misspelled characters. In Ku, L.-W., Martins, A. & Srikumar, V . (eds.)Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: L...

    2024 doi

  77. [85]

    & Zhai, G

    Chen, Z., tingzhu chen, Zhang, W. & Zhai, G. OBI-bench: Can LMMs aid in study of ancient script on oracle bones? In The Thirteenth International Conference on Learning Representations(2025)

    2025

  78. [86]

    & Herrera, P

    Fuentes-Ferrer, R., Duque-Domingo, J. & Herrera, P. J. Recognition of egyptian hieroglyphic texts through focused generic segmentation and cross-validation voting.Appl. Soft Comput.171, 112793 (2025)

    2025

  79. [87]

    & Ruan, Q

    Liu, X., Han, X., Chen, S., Dai, W. & Ruan, Q. Ancient yi script handwriting sample repository.Sci. Data11, 1183 (2024). 97.Liu, C.et al.Improving sentiment analysis accuracy with emoji embedding.J. Saf. Sci. Resil.2, 246–252 (2021)

    2024

  80. [88]

    & Luo, J

    Zheng, Y ., Lyu, H. & Luo, J. Irony in emojis: A comparative study of human and llm interpretation.arXiv preprint arXiv:2501.11241(2025)

    2025

  81. [89]

    Kuang, J.et al.Express what you see: Can multimodal llms decode visual ciphers with intuitive semiosis comprehension? InFindings of the Association for Computational Linguistics: ACL 2025, 12743–12774 (2025)

    2025

  82. [90]

    In Al-Onaizan, Y ., Bansal, M

    Nayak, S.et al.Benchmarking vision language models for cultural understanding. In Al-Onaizan, Y ., Bansal, M. & Chen, Y .-N. (eds.)Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 5769–5790, DOI: 10.18653/v1/2024.emnlp-main.329 (Associati...

    2024 doi

  83. [91]

    InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 1–18 (2025)

    Zheng, C.et al.Artmentor: Ai-assisted evaluation of artworks to explore multimodal large language models capabilities. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, 1–18 (2025)

    2025

  84. [92]

    In Christodoulopoulos, C., Chakraborty, T., Rose, C

    Mundada, G.et al.WildScore: Benchmarking MLLMs in-the-wild symbolic music reasoning. In Christodoulopoulos, C., Chakraborty, T., Rose, C. & Peng, V . (eds.)Proceedings of the 2025 Conference on Empirical Methods in Natural Lan- guage Processing, 16858–16874, DOI: 10.18653/v1/2...

    2025 doi

  85. [93]

    Neural Inf

    Wang, K.et al.Measuring multimodal mathematical reasoning with math-vision dataset.Adv. Neural Inf. Process. Syst. 37, 95095–95169 (2024)

    2024

  86. [94]

    InThe Thirteenth International Conference on Learning Representations(2025)

    Zou, C.et al.Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. InThe Thirteenth International Conference on Learning Representations(2025)

    2025

  87. [95]

    Nath, O., Bathina, H., Khan, M. S. U. R. & Khapra, M. M. Can vision-language models evaluate handwritten math? In Che, W., Nabende, J., Shutova, E. & Pilehvar, M. T. (eds.)Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

    2025 doi

  88. [96]

    Shi, Y .et al.Amsbench: A comprehensive benchmark for evaluating mllm capabilities in ams circuits.arXiv preprint arXiv:2505.24138(2025)

    2025

  89. [97]

    Shen, H.et al.Phyx: Does your model have the" wits" for physical reasoning?arXiv preprint arXiv:2505.15929(2025). 16/46

    2025

  90. [98]

    In Christodoulopoulos, C., Chakraborty, T., Rose, C

    Dai, S.et al.PhysicsArena: The first multimodal physics reasoning benchmark exploring variable, process, and solution dimensions. In Christodoulopoulos, C., Chakraborty, T., Rose, C. & Peng, V . (eds.)Findings of the Association for Computational Linguistics: EMNLP 2025, 17290...

    2025 doi

  91. [99]

    Wang, L.et al.Physunibench: An undergraduate-level physics reasoning benchmark for multimodal models.arXiv preprint arXiv:2506.17667(2025)

    2025

  92. [100]

    InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2025)

    Xiang, K.et al.Seephys: Does seeing help thinking? – benchmarking vision-based physics reasoning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2025)

    2025

  93. [101]

    Cheminformatics13, 97 (2021)

    Zhai, Z.et al.Chemtables: a dataset for semantic classification on tables in chemical patents.J. Cheminformatics13, 97 (2021)

    2021

  94. [102]

    Huang, Y .et al.Chemeval: a comprehensive multi-level chemical evaluation for large language models.arXiv preprint arXiv:2409.13989(2024)

    2024

  95. [103]

    Neural Inf

    Guo, K.et al.Can llms solve molecule puzzles? a multimodal benchmark for molecular structure elucidation.Adv. Neural Inf. Process. Syst.37, 134721–134746 (2024)

    2024

  96. [104]

    Yin, S.et al.A survey on multimodal large language models.Natl. Sci. Rev.11, nwae403, DOI: 10.1093/nsr/nwae403 (2024). https://academic.oup.com/nsr/article-pdf/11/12/nwae403/61201557/nwae403.pdf

    2024 doi

  97. [105]

    Chu, W.et al.Scaling llama 3 training with efficient parallelism strategies. InProceedings of the 52nd Annual International Symposium on Computer Architecture, ISCA ’25, 1703–1716, DOI: 10.1145/3695053.3731410 (Association for Computing Machinery, New York, NY , USA, 2025)

    2025 doi

  98. [106]

    InEuropean Conference on Computer Vision, 304–323 (Springer, 2024)

    McKinzie, B.et al.Mm1: methods, analysis and insights from multimodal llm pre-training. InEuropean Conference on Computer Vision, 304–323 (Springer, 2024)

    2024

  99. [107]

    Neural Inf

    Wu, J.et al.Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks.Adv. Neural Inf. Process. Syst.37, 69925–69975 (2024)

    2024

  100. [108]

    InInternational conference on machine learning, 8748–8763 (PmLR, 2021)

    Radford, A.et al.Learning transferable visual models from natural language supervision. InInternational conference on machine learning, 8748–8763 (PmLR, 2021). 119.Jia, C.et al.Scaling up visual and vision-language representation learning with noisy text supervision. InInterna...

    2021

  101. [109]

    & Hoi, S

    Li, J., Li, D., Savarese, S. & Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, 19730–19742 (PMLR, 2023)

    2023

  102. [110]

    & Lee, Y

    Liu, H., Li, C., Wu, Q. & Lee, Y . J. Visual instruction tuning.Adv. neural information processing systems36, 34892–34916 (2023)

    2023

  103. [111]

    In Al-Onaizan, Y ., Bansal, M

    Lin, B.et al.Video-LLaV A: Learning united visual representation by alignment before projection. In Al-Onaizan, Y ., Bansal, M. & Chen, Y .-N. (eds.)Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 5971–5984, DOI: 10.18653/v1/2024.emnlp-m...

    2024 doi

  104. [112]

    & Lee, Y

    Liu, H., Li, C., Li, Y . & Lee, Y . J. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 26296–26306 (2024). 124.Li, B.et al.LLaV A-onevision: Easy visual task transfer.Transactions on Mach. ...

    2024 arXiv

  105. [113]

    5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265(2025)

    Wang, W.et al.Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265(2025)

    2025 arXiv

  106. [114]

    InThe Thirty-ninth Annual Conference on Neural Information Processing Systems(2025)

    Yang, L.et al.MMaDA: Multimodal large diffusion language models. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems(2025)

    2025

  107. [115]

    InThe Thirty-ninth Annual Conference on Neural Information Processing Systems(2025)

    Fu, C.et al.VITA-1.5: Towards GPT-4o level real-time vision and speech interaction. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems(2025)

    2025

  108. [116]

    InThe Thirteenth International Conference on Learning Representations(2025)

    Zhu, E.et al.MAPS: Advancing multi-modal reasoning in expert-level physical science. InThe Thirteenth International Conference on Learning Representations(2025). 17/46 131.Wiesner, F., Wessling, M. & Baek, S. Towards a physics foundation model.arXiv preprint arXiv:2509.13805(2025)

    2025

  109. [117]

    In Goldberg, Y ., Kozareva, Z

    Edwards, C.et al.Translation between molecules and natural language. In Goldberg, Y ., Kozareva, Z. & Zhang, Y . (eds.)Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 375–413, DOI: 10.18653/v1/2022.emnlp-main.26 (Association for Computat...

    2022 doi

  110. [118]

    134.Zhang, D.et al.Chemllm: A chemical large language model.arXiv preprint arXiv:2402.06852(2024)

    Peng, S.et al.Multimath: Bridging visual and mathematical reasoning for large language models.arXiv preprint arXiv:2409.00147(2024). 134.Zhang, D.et al.Chemllm: A chemical large language model.arXiv preprint arXiv:2402.06852(2024)

    2024

  111. [119]

    & Liu, C.-L

    Ding, L., Zhao, M., Yin, F., Zeng, S. & Liu, C.-L. A large-scale database for chemical structure recognition and preliminary evaluation. In2022 26th International Conference on Pattern Recognition (ICPR), 1464–1470 (IEEE, 2022)

    2022

  112. [120]

    InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3828–3850 (2024)

    He, C.et al.Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3828–3850 (2024)

    2024

  113. [121]

    Zhang, X.et al.Evaluating the performance of large language models on gaokao benchmark.arXiv preprint arXiv:2305.12474(2023)

    2023

  114. [122]

    139.Grattafiori, A.et al.The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

    Wu, Z.et al.Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302(2024). 139.Grattafiori, A.et al.The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

    2024 arXiv

  115. [123]

    141.Hurst, A.et al.Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)

    Zhu, J.et al.Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479(2025). 141.Hurst, A.et al.Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024). 142.Anthropic. Claude 4. https://www.anthropic.com/ne...

    2025 arXiv

  116. [124]

    Introducing o3 and o4-mini

    OpenAI. Introducing o3 and o4-mini. https://openai.com/zh-Hans-CN/index/introducing-o3-and-o4-mini/ (2025). Accessed: 2025-10-13

    2025

  117. [125]

    Comanici, G.et al.Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)

    2025 arXiv

  118. [126]

    recognition-reasoning inversion

    Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311–318 (2002). 18/46 Contents Supplementary 21 A Extended Introduction 2...

    2002

Pith tools

Reviewed May 15, 2026 · model on record in the stance chip above.