Pith. sign in

REVIEW 3 major objections 3 minor 114 references

CoMemo: LVLMs Need Image Context with Image Memory

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vision-language models lose images in the middle of long contexts, and CoMemo fixes that by giving every image two routes into the model: an autoregressive context path and a gated cross-attention memory path.

desk verdict Solid architectural study with believable controlled gains on long-context and multi-image, but the abstract oversells OCR and Eq. (2) looks misprinted. read the letter →

arxiv 2506.06279 v1 pith:KLPB7XL3 submitted 2025-06-06 cs.CV

Shi Liu , Weijie Su , Xizhou Zhu , Wenhai Wang , Jifeng Dai This is my paper
classification cs.CV
keywords largevision-languagemodelsdual-patharchitecturecross-attentionmemoryRoPE-DHRlong-contextmultimodalunderstandinglostinthemiddledynamichighresolutionpositionalencoding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoMemo is a vision-language architecture built on the claim that today's image-aware LLMs discard visual information for two structural reasons: causal self-attention concentrates on the start and the most recent tokens, so middle content fades as context grows, and rotary position embeddings turn dynamic high-resolution image tokens into long, spatially flat sequences that decay with distance. To counter this, CoMemo feeds the same image tokens down two paths at once—a normal autoregressive context path and a cross-attention memory path—and adds a position scheme, RoPE-DHR, that anchors high-resolution tile patches to a compact image thumbnail's position grid. The paper reports that, with identical 2B-scale backbones, training data, and compute, CoMemo beats both a LLaVA-style autoregressive model and a Flamingo-style cross-attention model on seven benchmark families, with 17.2%, 7.0%, and 5.6% relative gains on captioning, long-generation, and long-context tasks. The point is that image neglect can be fixed architecturally rather than by adding more data or scale.

What carries the argument

The machinery is a dual-stream image processor plus a compressed position map. The context path concatenates image tokens with text and processes them autoregressively; the memory path, inserted every fourth transformer block as a mixin layer, runs gated cross-attention where text tokens query the image tokens with RoPE-based bidirectional positional awareness, so each decoding step can pull image information independent of context length. RoPE-DHR assigns the image thumbnail a normal sequential position grid, then maps every high-resolution tile patch to the position ID of the thumbnail patch it overlaps, replacing one-dimensional token distances with a 2D-aware, shorter position sequence. The learnable attention and feed-forward gates, applied as tanh-gated residuals, are what keep the two paths balanced.

What would settle it

Read the released CoMemo code and compute the RoPE position IDs for all tile patches of one high-resolution image; if the ratios in Eq. (2) are applied exactly as printed, patches in the lower-right tile will map outside the thumbnail position grid, which is a directly checkable mismatch. A second check is to rerun the Table 7 ablations with RoPE-DHR active but with tile position IDs shuffled; if 2D layout is really load-bearing, performance on multi-image and OCR tasks should drop sharply.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that the two dominant LVLM designs fail in complementary ways: the autoregressive path grounds tokens well but inherits causal attention's bimodal focus, while the cross-attention path retrieves images on demand but underuses the LLM's decoding capacity. CoMemo's dual-path design lets both mechanisms run on the same image features, with a learnable gate controlling how much the memory path contributes; three-stage training—alignment first, then freezing the gate, then full fine-tuning—prevents the model from over-relying on the memory branch. With RoPE-DHR, the position ID of every high-resolution tile patch is inherited from its corresponding thumbnail patch, so visual sequences stay short in position space and keep their 2D layout. The paper claims this combination yields consistent wins in controlled comparisons and in ablations, and that the memory path does not increase decoding-time key-value cache size.

Load-bearing premise

The load-bearing premise is that the coordinate mapping in Eq. (2) correctly places each high-resolution tile patch onto its thumbnail anchor, preserving 2D layout while compressing position IDs; if the implementation follows the printed formula literally, the width and height ratios are reversed and the spatial structure would collapse instead of being preserved.

Editorial extensions

If this is right

  • With the same InternLM-1.8B backbone and InternViT-300M encoder, CoMemo outperforms both controlled baselines on captioning, long-generation, multi-image, and long-context sets (Tables 2 and 3).
  • The memory path alone already raises overall performance in the ablation (Variant 3), and adding RoPE-DHR on top gives the largest gain (Variant 5), so the two components are complementary rather than redundant.
  • RoPE-DHR's compressed position space improves needle-in-the-haystack retrieval for both image and text needles, including cases where the stored image is unrelated to the target text needle.
  • Because cross-attention only touches the current decoding token against cached visual states, the memory path adds no growing key-value cache during generation, keeping inference latency close to the plain autoregressive model.
  • At 8B scale the same architecture keeps its advantage over the LVLM-S baseline, and it remains ahead when the fine-tuning dataset is swapped, so the result does not depend on one data mixture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If RoPE-DHR is the main driver of the long-context needle gains, then the same thumbnail-anchored position mapping could be applied to any dynamic-high-resolution autoregressive LVLM without the mixin layers, giving a cheaper route to the same long-context benefit.
  • The paper's ablation split (memory path lifts captioning, RoPE-DHR lifts long-context) suggests a testable specialization: the memory path should matter more for generation-heavy and multi-image tasks, while position compression should matter more for retrieval-style long-context tasks; running CoMemo with only one component on each task family would confirm this division.
  • A further consequence for multimodal architecture design is that pathway reliance can be controlled by when the cross-attention gate is frozen, so the three-stage schedule may transfer to other dual-path or retrieval-augmented LVLMs as a regularization recipe.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper identifies two failure modes of LVLMs—'lost in the middle' caused by causal attention and remote decay/dimensional collapse caused by RoPE on dynamic high-resolution image tokens—and proposes CoMemo, a dual-path architecture that adds gated cross-attention 'memory' layers to a standard autoregressive LLaVA-style 'context' path, together with RoPE-DHR positional encoding that maps high-resolution tile patch positions to thumbnail anchor positions. The authors reimplement LVLM-X and LVLM-S baselines under identical backbone, data, and training settings at 2B scale and report gains on captioning, long generation, multi-image, and long-context benchmarks, with mixed results on OCR-related benchmarks; ablations at 2B and 8B scales and on an alternative SFT dataset are also reported.

Significance. If the controlled comparisons are taken at face value, the paper makes a useful architectural point: a dual-path design can combine the captioning strength of cross-attention and the reasoning strength of autoregressive alignment, while RoPE-DHR improves long-context and long-generation behavior. The main strengths are the controlled experimental protocol (same InternLM-1.8B backbone, same InternViT encoder, same pretraining/fine-tuning data for all three architectures), the explicit attention/gradient analysis motivating the design, and the component ablations including compression-ratio and dataset-switch checks. The work would be more significant if the claimed seven-category superiority were stated accurately and if the coordinate-mapping formula and training-stage description were corrected; as it stands, the evidence supports a narrower claim of architectural gains on long-context/generation tasks with an OCR trade-off.

major comments (3)
  1. [Abstract; Tables 2–4] The unqualified claim that CoMemo 'demonstrates superior performance compared to conventional LVLM architectures' across seven benchmark categories is not supported by the controlled comparison. In Table 4, CoMemo is below LVLM-S on every OCR-related benchmark: AI2D 74.2 vs 74.3, ChartQA 73.6 vs 75.6, and TextVQA 72.6 vs 74.2. The paper itself concedes in §4.5 that text/OCR tasks favor more granular image representations, so the contribution should be stated as gains on captioning, long-generation, long-context, multi-image, and general VQA, with a documented OCR trade-off. This is not a request for new experiments; it is a request to make the central claim match the reported numbers.
  2. [§3.1, Eq. (2)] The printed coordinate mapping cannot be correct as written. The x-coordinate multiplies x_tile by W_tile/W_orig, while the y-coordinate multiplies y_tile by H_orig/H_thumb; neither expression maps a tile patch coordinate to a thumbnail patch index in a dimensionally consistent way. If the implementation uses the printed formula, RoPE-DHR would assign tile patches to wrong thumbnail locations and would not preserve 2D structure; if the implementation uses a corrected formula, the paper must state it. Since RoPE-DHR is one of the two central contributions, please correct Eq. (2) (and make Figure 6 consistent with the corrected formula) and confirm that the experiments use the corrected mapping.
  3. [§3.3; Appendix B] The three-stage training strategy described in §3.3 includes a gate-freeze stage that is claimed to prevent over-reliance on the cross-attention path, yet Appendix B states that the 2B model, which is the model in all main controlled comparisons (Tables 2–4), used only Pretraining Phase 1 and treated Phase 2 as optional. Thus the main empirical evidence for CoMemo appears to be produced without the gate-freeze mechanism that is presented as part of the method. Please clarify whether Phase 2 was used for any main-table result and, if not, state that the main results correspond to a two-stage variant.
minor comments (3)
  1. [Throughout] There are several typographical errors that should be corrected in revision: 'tow' for 'two' (§2.3 and §3.2), 'thumnail' for 'thumbnail' (Figure 6), 'contruct' for 'construct' (Figure 2), 'Expereiment' for 'Experiment' (Appendix C), and 'Cation' for 'Caption' (§4.7).
  2. [References; Table 4] The paper cites Zhong et al. 2023 as the source of MMVP, but that reference is 'MMVP: Motion-matrix-based video prediction'; the MMVP benchmark used in Table 4 appears to be 'Multimodal Visual Patterns' from Tong et al. (CVPR 2024). Please correct the citation and the benchmark description.
  3. [§1] The stated relative improvements of 17.2%, 7.0%, and 5.6% on Caption, Long-Generation, and Long-Context are not directly recoverable from the per-benchmark numbers in Tables 2 and 3; please specify the task averages or normalization used to compute these percentages.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: benchmark results are genuine empirical evaluations, not consequences of the method's construction.

full rationale

CoMemo's load-bearing claims are empirical comparisons run under controlled data, backbone, and training settings. The design motivation in Section 2.3 (DHR-B allocation and pretraining step choice) is supported by balancing experiments, but those choices do not encode benchmark answers, and the ablations in Section 4.7 test components independently. RoPE-DHR is motivated by standard RoPE formulas in Appendix A, yet its benefit is established by Table 7 ablations rather than by the formula itself. No load-bearing result is justified by a self-citation chain; the citations to attention sinks, lost-in-the-middle, and cross-attention LVLMs are external prior art. The abstract's unqualified 'superior performance' claim is weakened by the paper's own Table 4, where CoMemo trails LVLM-S on AI2D, ChartQA, and TextVQA, and the paper concedes in Sections 4.5 and 4.7 that OCR favors more granular representations; however, that is an evidence-quality issue, not circularity. The apparent inversion in Eq. (2) is a correctness concern, not a circular one, and is not counted here. I find no step where a 'prediction' or 'first-principles result' is equivalent to its inputs by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim depends on hand-chosen training hyperparameters and on empirical assumptions about the causes of LVLM underperformance. No fitted constants are introduced to manufacture the benchmark results, so the circularity burden is low.

free parameters (3)
  • Pretraining phase lengths = 2000 / 2000 / 9000 steps
    Chosen by hand; the paper reports that 2000/2000 balances paths but does not report sensitivity to these values.
  • Mixin layer frequency = 1:4 (4 mixin layers in a 16-layer LLM)
    The interleaving ratio is a design choice; no ablation on this ratio is reported.
  • Gate freeze point = After phase 1 (2000 steps)
    The freeze point is chosen based on observed over-reliance on the memory path; no criterion is given.
assumptions (5)
  • standard math RoPE exhibits remote decay with relative distance
    Inherited from sinusoidal position encodings; cited to Su (2021) and used in Section 2.2 to motivate RoPE-DHR.
  • domain assumption Causal self-attention causes a bimodal attention distribution and 'lost in the middle'
    Based on the empirical analysis in Section 2.1 (Figure 3); accepted as the cause of image neglect without causal verification.
  • domain assumption DHR token expansion (7x) aggravates remote decay and image neglect
    Based on Figure 4 in Section 2.2; assumed to be a dominant factor.
  • ad hoc to paper Mapping tile patch coordinates via Eq. (2) preserves 2D structure
    The formula uses W_tile/W_orig and H_tile/H_orig, which appears inverted for the stated mapping, so the correctness of this geometric correspondence is assumed.
  • standard math Abel summation / triangle inequality
    Used in Appendix A to bound the RoPE inner product.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoMemo: LVLMs Need Image Context with Image Memory." pith.science (2026). https://pith.science/paper/KLPB7XL3

@misc{pith2026250606279,
  author       = {Pith},
  title        = {Pith review of: CoMemo: LVLMs Need Image Context with Image Memory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KLPB7XL3}},
  note         = {Machine review of arXiv:2506.06279}
}
read the original abstract

Recent advancements in Large Vision-Language Models built upon Large Language Models have established aligning visual features with LLM representations as the dominant paradigm. However, inherited LLM architectural designs introduce suboptimal characteristics for multimodal processing. First, LVLMs exhibit a bimodal distribution in attention allocation, leading to the progressive neglect of middle visual content as context expands. Second, conventional positional encoding schemes fail to preserve vital 2D structural relationships when processing dynamic high-resolution images. To address these limitations, we propose CoMemo - a dual-path architecture that combines a Context image path with an image Memory path for visual processing, effectively alleviating visual information neglect. Additionally, we introduce RoPE-DHR, a novel positional encoding mechanism that employs thumbnail-based positional aggregation to maintain 2D spatial awareness while mitigating remote decay in extended sequences. Evaluations across seven benchmarks,including long-context comprehension, multi-image reasoning, and visual question answering, demonstrate CoMemo's superior performance compared to conventional LVLM architectures. Project page is available at https://lalbj.github.io/projects/CoMemo/.

Figures

Figures reproduced from arXiv: 2506.06279 by the authors.

Figure 1
Figure 1. Evaluation results of three architectures with same train￾ing data and model size (2B). Please refer to Tables 2 to 4 for details. Tokenizer Input Sequence LLM Layer 0 Layer 1 Layer n Img. Encoder … (a) LVLM-S. Tokenizer Input Sequence Mixin LLM Layer 0 Layer n Img. Encoder … …MixinLayer (b) LVLM-X. Tokenizer Input Sequence Mixin LLM Layer 0 Layer n Img. Encoder . MixinLayer … Memory … Path Context Path (c) Ours [P… view at source ↗
Figure 3
Figure 3. Average gradients and attention weights assigned [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 5
Figure 5. Balancing experiments. Experiment settings are [PITH_FULL_IMAGE:figures/full_fig_p003_5.png] view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: The computation process of Rope-DHR. The colors are assigned based on a mapping of position IDs in RoPE. Input Sequence Mixin LLM Layer 0 Encoder Dynamic High Resolution Pixel Shuffle Projector Image Tokens User Message Tokenizer Mixin Layer Layer n .. [PITH_FULL_IMA…
Figure 8
Figure 8. Figure 8: Heatmap of results for the NIAH evaluation on [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

114 extracted references · 32 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Nocaps: Novel object captioning at scale

    Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., and Anderson, P. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 8948--8957, 2019

    2019

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 0 23716--23736, 2022

    2022

  4. [4]

    Mathqa: Towards interpretable math word problem solving with operation-based formalisms

    Amini, A., Gabriel, S., Lin, P., Koncel-Kedziorski, R., Choi, Y., and Hajishirzi, H. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019

    arXiv 1905

  5. [5]

    A., Datla, V

    Ben Abacha, A., Hasan, S. A., Datla, V. V., Demner-Fushman, D., and M \"u ller, H. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. In Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes. 9-12 September 2019, 2019

    2019

  6. [6]

    F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C., and Karatzas, D

    Biten, A. F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C., and Karatzas, D. Scene text visual question answering. pp.\ 4291--4301, 2019

    2019

  7. [7]

    Coyo-700m: Image-text pair dataset

    Byeon, M., Park, B., Kim, H., Lee, S., Baek, W., and Kim, S. Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/coyo-dataset, 2022

    2022

  8. [8]

    and Xiao, J

    Cao, J. and Xiao, J. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In COLING, pp.\ 1511--1520, 2022

    2022

Show all 114 references
  1. [9]

    Textocr-gpt4v

    Carter, J. Textocr-gpt4v. https://huggingface.co/datasets/jimmycarter/textocr-gpt4v, 2024

    2024

  2. [10]

    Mapqa: A dataset for question answering on choropleth maps

    Chang, S., Palzer, D., Li, J., Fosler-Lussier, E., and Xiao, N. Mapqa: A dataset for question answering on choropleth maps. arXiv preprint arXiv:2211.08545, 2022

    2022 arXiv

  3. [11]

    H., Chen, S., Zhang, R., Chen, J., Wu, X., Zhang, Z., Chen, Z., Li, J., Wan, X., and Wang, B

    Chen, G. H., Chen, S., Zhang, R., Chen, J., Wu, X., Zhang, Z., Chen, Z., Li, J., Wan, X., and Wang, B. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684, 2024 a

    2024 arXiv

  4. [12]

    Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression

    Chen, J., Li, T., Qin, J., Lu, P., Lin, L., Chen, C., and Liang, X. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression. arXiv preprint arXiv:2212.02746, 2022

    2022 arXiv

  5. [13]

    Shikra: Unleashing multimodal llm's referential dialogue magic

    Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and Zhao, R. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023

    2023 arXiv

  6. [14]

    Evlm: An efficient vision-language model for visual understanding

    Chen, K., Shen, D., Zhong, H., Zhong, H., Xia, K., Xu, D., Yuan, W., Hu, Y., Wen, B., Zhang, T., et al. Evlm: An efficient vision-language model for visual understanding. arXiv preprint arXiv:2407.14177, 2024 b

    2024 arXiv

  7. [15]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024 c

    2024 arXiv

  8. [16]

    Complicated table structure recognition

    Chi, Z., Huang, H., Xu, H.-D., Yu, H., Yin, W., and Mao, X.-L. Complicated table structure recognition. arXiv preprint arXiv:1908.04729, 2019

    1908 arXiv

  9. [17]

    K., Liu, Y., Sun, Y., Ng, C

    Chng, C. K., Liu, Y., Sun, Y., Ng, C. C., Luo, C., Ni, Z., Fang, C., Zhang, S., Han, J., Ding, E., et al. Icdar2019 robust reading challenge on arbitrary-shaped text-rrc-art. In ICDAR, pp.\ 1571--1576. IEEE, 2019

    2019

  10. [18]

    and Gardner, M

    Clark, C. and Gardner, M. Simple and effective multi-paragraph reading comprehension. arXiv preprint arXiv:1710.10723, 2017

    2017 arXiv

  11. [19]

    Nvlm: Open frontier-class multimodal llms

    Dai, W., Lee, N., Wang, B., Yang, Z., Liu, Z., Barker, J., Rintamaki, T., Shoeybi, M., Catanzaro, B., and Ping, W. Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409.11402, 2024

    2024 arXiv

  12. [20]

    Deep visual template-free form parsing

    Davis, B., Morse, B., Cohen, S., Price, B., and Tensmeyer, C. Deep visual template-free form parsing. In ICDAR, pp.\ 134--141. IEEE, 2019

    2019

  13. [21]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  14. [22]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  15. [23]

    A., Ma, W.-C., and Krishna, R

    Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N. A., Ma, W.-C., and Krishna, R. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pp.\ 148--166. Springer, 2025

    2025

  16. [24]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. pp.\ 6904--6913, 2017

    2017

  17. [25]

    Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark

    Gu, J., Meng, X., Lu, G., Hou, L., Minzhe, N., Liang, X., Yao, L., Huang, R., Zhang, W., Jiang, X., et al. Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark. 35: 0 26418--26431, 2022

    2022

  18. [26]

    Eaten: Entity-aware attention for single shot visual text extraction

    Guo, H., Qin, X., Liu, J., Han, J., Liu, J., and Ding, E. Eaten: Entity-aware attention for single shot visual text extraction. In ICDAR, pp.\ 254--259. IEEE, 2019

    2019

  19. [27]

    Icpr2018 contest on robust reading for multi-type web images

    He, M., Liu, Y., Yang, Z., Zhang, S., Luo, C., Gao, F., Zheng, Q., Wang, Y., Zhang, X., and Jin, L. Icpr2018 contest on robust reading for multi-type web images. pp.\ 7--12. IEEE, 2018

    2018

  20. [28]

    Pathvqa: 30000+ questions for medical visual question answering

    He, X., Zhang, Y., Mou, L., Xing, E., and Xie, P. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286, 2020

    2003 arXiv

  21. [29]

    Hiippala, T., Alikhani, M., Haverinen, J., Kalliokoski, T., Logacheva, E., Orekhova, S., Tuomainen, A., Stone, M., and Bateman, J. A. Ai2d-rst: A multimodal corpus of 1000 primary school science diagrams. Language Resources and Evaluation, 55: 0 661--688, 2021

    2021

  22. [30]

    Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment

    Hosu, V., Lin, H., Sziranyi, T., and Saupe, D. Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment. 29: 0 4041--4056, 2020

    2020

  23. [31]

    mplug-docowl 1.5: Unified structure learning for ocr-free document understanding

    Hu, A., Xu, H., Ye, J., Yan, M., Zhang, L., Zhang, B., Li, C., Zhang, J., Jin, Q., Huang, F., et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403.12895, 2024

    2024 arXiv

  24. [32]

    Medical-diff-vqa: a large-scale medical dataset for difference visual question answering on chest x-ray images, 2023

    Hu, X., Gu, L., An, Q., Zhang, M., Liu, L., Kobayashi, K., Harada, T., Summers, R., and Zhu, Y. Medical-diff-vqa: a large-scale medical dataset for difference visual question answering on chest x-ray images, 2023

    2023

  25. [33]

    Movienet: A holistic dataset for movie understanding

    Huang, Q., Xiong, Y., Rao, A., Wang, J., and Lin, D. Movienet: A holistic dataset for movie understanding. pp.\ 709--727. Springer, 2020

    2020

  26. [34]

    Icdar2019 competition on scanned receipt ocr and information extraction

    Huang, Z., Chen, K., He, J., Bai, X., Karatzas, D., Lu, S., and Jawahar, C. Icdar2019 competition on scanned receipt ocr and information extraction. In ICDAR, pp.\ 1516--1520. IEEE, 2019

    2019

  27. [35]

    Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. pp.\ 6700--6709, 2019

    2019

  28. [36]

    Mantis: Interleaved multi-image instruction tuning

    Jiang, D., He, X., Zeng, H., Wei, C., Ku, M., Liu, Q., and Chen, W. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024

    2024 arXiv

  29. [37]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. pp.\ 2901--2910, 2017

    2017

  30. [38]

    Dvqa: Understanding data visualizations via question answering

    Kafle, K., Price, B., Cohen, S., and Kanan, C. Dvqa: Understanding data visualizations via question answering. pp.\ 5648--5656, 2018

    2018

  31. [39]

    E., Michalski, V., Atkinson, A., K \'a d \'a r, \'A ., Trischler, A., and Bengio, Y

    Kahou, S. E., Michalski, V., Atkinson, A., K \'a d \'a r, \'A ., Trischler, A., and Bengio, Y. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017

    2017 arXiv

  32. [40]

    Kantharaj, S., Leong, R. T. K., Lin, X., Masry, A., Thakkar, M., Hoque, E., and Joty, S. Chart-to-text: A large-scale benchmark for chart summarization. arXiv preprint arXiv:2203.06486, 2022

    2022 arXiv

  33. [41]

    Referitgame: Referring to objects in photographs of natural scenes

    Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. Referitgame: Referring to objects in photographs of natural scenes. pp.\ 787--798, 2014

    2014

  34. [42]

    A diagram is worth a dozen images

    Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth a dozen images. pp.\ 235--251, 2016

    2016

  35. [43]

    Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension

    Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., and Hajishirzi, H. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. pp.\ 4999--5007, 2017

    2017

  36. [44]

    Visual information extraction in the wild: practical dataset and end-to-end solution

    Kuang, J., Hua, W., Liang, D., Yang, M., Jiang, D., Ren, B., and Bai, X. Visual information extraction in the wild: practical dataset and end-to-end solution. In ICDAR, pp.\ 36--53. Springer, 2023

    2023

  37. [45]

    Laion-gpt4v dataset

    LAION. Laion-gpt4v dataset. https://huggingface.co/datasets/laion/gpt4v-dataset, 2023

    2023

  38. [46]

    J., Gayen, S., Ben Abacha, A., and Demner-Fushman, D

    Lau, J. J., Gayen, S., Ben Abacha, A., and Demner-Fushman, D. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5 0 (1): 0 1--10, 2018

    2018

  39. [47]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    Lauren c on, H., Tronchon, L., Cord, M., and Sanh, V. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    2024 arXiv

  40. [48]

    G., and Lov \'o n Melgarejo, J

    Lerner, P., Ferret, O., Guinaudeau, C., Le Borgne, H., Besan c on, R., Moreno, J. G., and Lov \'o n Melgarejo, J. Viquae, a dataset for knowledge-based visual question answering about named entities. In SIGIR, pp.\ 3108--3120, 2022

    2022

  41. [49]

    Chemvlm: Exploring the power of multimodal large language models in chemistry area

    Li, J., Zhang, D., Wang, X., Hao, Z., Lei, J., Tan, Q., Zhou, C., Liu, W., Yang, Y., Xiong, X., et al. Chemvlm: Exploring the power of multimodal large language models in chemistry area. arXiv preprint arXiv:2408.07246, 2024

    2024

  42. [50]

    Li, Z., Wang, X., Stengel-Eskin, E., Kortylewski, A., Ma, W., Van Durme, B., and Yuille, A. L. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. pp.\ 14963--14973, 2023

    2023

  43. [51]

    Vila: On pre-training for visual language models

    Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 26689--26699, 2024

    2024

  44. [52]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp....

    2014

  45. [53]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. pp.\ 740--755. Springer, 2014 b

    2014

  46. [54]

    Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering

    Liu, B., Zhan, L.-M., Xu, L., Ma, L., Yang, Y., and Wu, X.-M. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In ISBI, pp.\ 1650--1654. IEEE, 2021

    2021

  47. [55]

    Casia online and offline chinese handwriting databases

    Liu, C.-L., Yin, F., Wang, D.-H., and Wang, Q.-F. Casia online and offline chinese handwriting databases. In ICDAR, pp.\ 37--41. IEEE, 2011

    2011

  48. [56]

    Visual spatial reasoning

    Liu, F., Emerson, G., and Collier, N. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11: 0 635--651, 2023 a

    2023

  49. [57]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. Mitigating hallucination in large multi-modal models via robust instruction tuning. 2023 b

    2023

  50. [58]

    Mmc: Advancing multimodal chart understanding with large-scale instruction tuning

    Liu, F., Wang, X., Yao, W., Chen, J., Song, K., Cho, S., Yacoob, Y., and Yu, D. Mmc: Advancing multimodal chart understanding with large-scale instruction tuning. arXiv preprint arXiv:2311.10774, 2023 c

    2023 arXiv

  51. [59]

    Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024 a . URL https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  52. [60]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems, 36, 2024 b

    2024

  53. [61]

    F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P

    Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024 c

    2024

  54. [62]

    Paying more attention to image: A training-free method for alleviating hallucination in lvlms

    Liu, S., Zheng, K., and Chen, W. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. In European Conference on Computer Vision, pp.\ 125--140. Springer, 2025 a

    2025

  55. [63]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp.\ 216--233

    Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp.\ 216--233. Springer, 2025 b

    2025

  56. [65]

    Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms

    Liu, Z., Chu, T., Zang, Y., Wei, X., Dong, X., Zhang, P., Liang, Z., Xiong, Y., Qiao, Y., Lin, D., et al. Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms. arXiv preprint arXiv:2406.11833, 2024 e

    2024 arXiv

  57. [66]

    Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning

    Lu, P., Gong, R., Jiang, S., Qiu, L., Huang, S., Liang, X., and Zhu, S.-C. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.04165, 2021

    2021 arXiv

  58. [67]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. 35: 0 2507--2521, 2022 a

    2022

  59. [68]

    N., Zhu, S.-C., Rajpurohit, T., Clark, P., and Kalyan, A

    Lu, P., Qiu, L., Chang, K.-W., Wu, Y. N., Zhu, S.-C., Rajpurohit, T., Clark, P., and Kalyan, A. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209.14610, 2022 b

    2022 arXiv

  60. [69]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  61. [70]

    Deepart: Learning joint representations of visual arts

    Mao, H., Cheung, M., and She, J. Deepart: Learning joint representations of visual arts. pp.\ 1183--1191, 2017

    2017

  62. [71]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok-vqa: A visual question answering benchmark requiring external knowledge. pp.\ 3195--3204, 2019

    2019

  63. [72]

    and Bunke, H

    Marti, U.-V. and Bunke, H. The iam-database: an english sentence database for offline handwriting recognition. International journal on document analysis and recognition, 5: 0 39--46, 2002

    2002

  64. [73]

    L., Tan, J

    Masry, A., Do, X. L., Tan, J. Q., Joty, S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL, pp.\ 2263--2279, 2022 a

    2022

  65. [74]

    X., Tan, J

    Masry, A., Long, D. X., Tan, J. Q., Joty, S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022 b

    2022 arXiv

  66. [75]

    L., Hoque, E., and Joty, S

    Masry, A., Kavehzadeh, P., Do, X. L., Hoque, E., and Joty, S. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. arXiv preprint arXiv:2305.14761, 2023

    2023 arXiv

  67. [76]

    Infographicvqa

    Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., and Jawahar, C. Infographicvqa. pp.\ 1697--1706, 2022

    2022

  68. [77]

    M., and Kumar, P

    Methani, N., Ganguly, P., Khapra, M. M., and Kumar, P. Plotqa: Reasoning over scientific plots. pp.\ 1527--1536, 2020

    2020

  69. [78]

    Opengvlab/internvl-chat-v1-2-sft-data, Jan 2024

    OpenGVLab. Opengvlab/internvl-chat-v1-2-sft-data, Jan 2024. URL https://huggingface.co/OpenGVLab/InternVL-Chat-V1-2

    2024

  70. [79]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  71. [80]

    Kosmos-2: Grounding multimodal large language models to the world

    Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  72. [81]

    A., Wang, L., Cervantes, C

    Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp.\ 2641-...

    2015

  73. [82]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. 35: 0 25278--25294, 2022 a

    2022

  74. [83]

    Laion coco: 600m synthetic captions from laion2b-en

    Schuhmann, C., K \"o pf, A., Vencu, R., Coombes, T., and Beaumont, R. Laion coco: 600m synthetic captions from laion2b-en. https://laion.ai/blog/laion-coco/, 2022 b

    2022

  75. [84]

    Solving geometry problems: Combining text and diagram interpretation

    Seo, M., Hajishirzi, H., Farhadi, A., Etzioni, O., and Malcolm, C. Solving geometry problems: Combining text and diagram interpretation. pp.\ 1466--1476, 2015

    2015

  76. [85]

    Shah, S., Mishra, A., Yadati, N., and Talukdar, P. P. Kvqa: Knowledge-aware visual question answering. volume 33, pp.\ 8876--8884, 2019

    2019

  77. [86]

    Objects365: A large-scale, high-quality dataset for object detection

    Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., and Sun, J. Objects365: A large-scale, high-quality dataset for object detection. pp.\ 8430--8439, 2019

    2019

  78. [87]

    Towards vqa models that can read

    Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. pp.\ 8317--8326, 2019 a

    2019

  79. [88]

    Towards vqa models that can read

    Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 8317--8326, 2019 b

    2019

  80. [89]

    Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text

    Singh, A., Pang, G., Toh, M., Huang, J., Galuba, W., and Hassner, T. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. pp.\ 8802--8812, 2021

    2021

  81. [90]

    H., Yu, F., Wan, X., and Wang, B

    Song, D., Chen, S., Chen, G. H., Yu, F., Wan, X., and Wang, B. Milebench: Benchmarking mllms in long context. arXiv preprint arXiv:2404.18532, 2024

    2024 arXiv

  82. [91]

    Transformer roadmap: 2

    Su, J. Transformer roadmap: 2. rotary position embedding, Mar 2021. URL https://spaces.ac.cn/archives/8265

    2021

  83. [92]

    C., Han, J., Ding, E., Liu, J., Karatzas, D., et al

    Sun, Y., Ni, Z., Chng, C.-K., Liu, Y., Luo, C., Ng, C. C., Han, J., Ding, E., Liu, J., Karatzas, D., et al. Icdar 2019 competition on large-scale street view text with partial labeling-rrc-lsvt. In ICDAR, pp.\ 1557--1562. IEEE, 2019

    2019

  84. [93]

    Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024

    Team, O. Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024. URL https://internvl.github.io/blog/2024-07-02-InternVL-2.0/

    2024

  85. [94]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  86. [95]

    Coco-text: Dataset and benchmark for text detection and recognition in natural images

    Veit, A., Matera, T., Neumann, L., Matas, J., and Belongie, S. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140, 2016

    2016 arXiv

  87. [96]

    V3det: Vast vocabulary visual detection dataset

    Wang, J., Zhang, P., Chu, T., Cao, Y., Zhou, Y., Wu, T., Wang, B., He, C., and Lin, D. V3det: Vast vocabulary visual detection dataset. pp.\ 19844--19854, 2023 a

    2023

  88. [97]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Wang, K., Pan, J., Shi, W., Lu, Z., Zhan, M., and Li, H. Measuring multimodal mathematical reasoning with math-vision dataset. arXiv preprint arXiv:2402.14804, 2024 a

    2024 arXiv

  89. [98]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv prepr...

    2024 arXiv

  90. [99]

    The all-seeing project: Towards panoptic visual recognition and understanding of the open world

    Wang, W., Shi, M., Li, Q., Wang, W., Huang, Z., Xing, L., Chen, Z., Li, H., Zhu, X., Cao, Z., et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. arXiv preprint arXiv:2308.01907, 2023 b

    2023 arXiv

  91. [100]

    Needle in a multimodal haystack

    Wang, W., Zhang, S., Ren, Y., Duan, Y., Li, T., Liu, S., Hu, M., Chen, Z., Zhang, K., Lu, L., et al. Needle in a multimodal haystack. arXiv preprint arXiv:2406.07230, 2024 c

    2024 arXiv

  92. [101]

    C., Luo, C., Jin, L., Chan, C

    Wang, X., Liu, Y., Shen, C., Ng, C. C., Luo, C., Jin, L., Chan, C. S., Hengel, A. v. d., and Wang, L. On the general value of evidence, and bilingual scene-text visual question answering. pp.\ 10126--10135, 2020

    2020

  93. [102]

    Pmc-casereport

    Wu, C. Pmc-casereport. https://huggingface.co/datasets/chaoyi-wu/PMC-CaseReport, 2023

    2023

  94. [103]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding

    Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024

    2024 arXiv

  95. [104]

    Efficient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  96. [105]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  97. [106]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models

    Ye, J., Xu, H., Liu, H., Hu, A., Yan, M., Qian, Q., Zhang, J., Huang, F., and Zhou, J. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024

    2024 arXiv

  98. [107]

    Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi

    Ying, K., Meng, F., Wang, J., Li, Z., Lin, H., Yang, Y., Zhang, H., Zhang, W., Lin, Y., Liu, S., et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.16006, 2024

    2024 arXiv

  99. [108]

    T., Li, Z., Weller, A., and Liu, W

    Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023

    2023 arXiv

  100. [109]

    A large chinese text dataset in the wild

    Yuan, T., Zhu, Z., Xu, K., Li, C., Mu, T., and Hu, S. A large chinese text dataset in the wild. 34 0 (3): 0 509--521, 2019

    2019

  101. [110]

    Syntax-aware network for handwritten mathematical expression recognition

    Yuan, Y., Liu, X., Dikubab, W., Liu, H., Ji, Z., Wu, Z., and Bai, X. Syntax-aware network for handwritten mathematical expression recognition. arXiv preprint arXiv:2203.01601, 2022

    2022 arXiv

  102. [111]

    Icdar 2019 robust reading challenge on reading chinese text on signboard

    Zhang, R., Zhou, Y., Jiang, Q., Song, Q., Li, N., Zhou, K., Wang, L., Wang, D., Liao, M., Yang, M., et al. Icdar 2019 robust reading challenge on reading chinese text on signboard. In ICDAR, pp.\ 1577--1581. IEEE, 2019

    2019

  103. [112]

    Pmc-vqa: Visual instruction tuning for medical visual question answering

    Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., and Xie, W. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415, 2023 a

    2023 arXiv

  104. [113]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding

    Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D., and Sun, T. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023 b

    2023 arXiv

  105. [114]

    Zheng, X., Burdick, D., Popa, L., Zhong, X., and Wang, N. X. R. Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context. pp.\ 697--706, 2021

    2021

  106. [115]

    Mmvp: Motion-matrix-based video prediction

    Zhong, Y., Liang, L., Zharkov, I., and Neumann, U. Mmvp: Motion-matrix-based video prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4273--4283, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.