Pith. sign in

REVIEW 4 major objections 5 minor 50 references

Rethinking Causal Mask Attention for Vision-Language Inference

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that allowing image tokens to see future context at prefill time improves vision-language models, even when the model was trained with strict causal attention.

desk verdict Plausible extension to VLMs, but the headline claim outruns the evidence: per-task calibration, tiny samples, and no control mean the gains are not established. read the letter →

arxiv 2505.18605 v1 pith:EWGQECJF submitted 2025-05-24 cs.CV cs.AI

Xiaohuan Pei , Tao Huang , YanXiang Ma , Chang Xu This is my paper
classification cs.CVcs.AI
keywords causalattentionvision-languagemodelsfuture-awaremaskingprefillinferencesinkmulti-imagereasoningtext-richVQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models inherit the left-to-right causal mask from text LLMs, which blocks every token from seeing later tokens. This paper argues that the mask is a wrong fit for image tokens: when the mask is relaxed during the prefill pass (the initial processing of the full image-and-question sequence), so each visual query can attend to future image tokens, future text tokens, or both, a causally trained VLM answers temporal multi-image, visual-relation, and text-rich VQA benchmarks more accurately. The paper then shows that pooling those future attention scores into a single early prefix token restores a strictly causal lower-triangular pattern and keeps decoding fast while retaining most of the accuracy gain. If the finding holds, part of a VLM's visual reasoning capacity is being suppressed by the default mask and can be recovered with no retraining.

What carries the argument

The central object is a family of future-aware causal masks defined on the multimodal attention matrix: $M^{f}$, $M^{v2v}$, and $M^{v2t}$, each obtained by changing selected $-\infty$ upper-triangular entries in visual query rows to $0$ while leaving text rows causal. The mechanism that carries the efficiency claim is a prefill-stage pooling operator that applies 1D kernel pooling over each visual query row's future attention scores and merges the pooled summary into the first past position, the attention-sink region. The resulting attention pattern is strictly lower-triangular again, so decoding can use the standard causal mask: the mask family supplies the accuracy gains, and the pooling operator supplies the latency reduction.

What would settle it

Run the same benchmark comparisons with all task-specific biases removed and with the relaxed mask applied to a model that has also been fine-tuned for even a few steps under that mask: if the inference-only gains vanish, the claim that a fixed causal model is 'using' future context fails. A second check: open a permuted or random set of future positions instead of the ordered future; equal gains would show the effect is not about semantic future context.

Watch

Extended reading notes

Core claim

Under the standard causal mask, the visual query at position $i$ can attend only to positions $j \leq i$. The paper's discovery is that replacing the $-\infty$ entries in the upper-triangular part of the mask for visual rows with zeros, while leaving text rows fully causal, improves accuracy on tasks where the decisive visual evidence appears later in the sequence. Three variants are defined and tested: $M^{f}$ opens the whole future, $M^{v2v}$ opens only future visual tokens, and $M^{v2t}$ opens only future text tokens. Each helps a different family of tasks: $M^{f}$ for temporal multi-image reasoning, $M^{v2v}$ for visual-relation inference, and $M^{v2t}$ for text-rich visual QA, with the best variant depending on which modality carries the task's information. The authors take this as evidence that strict causal masking conflicts with the holistic, non-sequential structure of visual information, and that future visual context carries semantic cues the model is otherwise forced to ignore.

Load-bearing premise

The fragile premise is that changing only the attention mask at inference time gives honest, useful hidden states for a model trained with strict causal attention, so the measured gains really come from the future context rather than from the distribution shift or from task-level evaluation biases.

Editorial extensions

If this is right

  • A causally trained VLM can be made more accurate on temporal, relational, and text-rich visual QA tasks using only inference-time changes, with no fine-tuning or weight updates.
  • Merging pooled future scores into a single prefix token preserves most of the accuracy gain while cutting decoding latency on the evaluated 7B model from up to about 83 ms/token to about 26 ms/token.
  • The best future-access pattern is task-dependent: full future for temporal multi-image tasks, future visual-only for visual relations, and future text-only for text-rich QA, while strict causality remains preferable for text-dominant and retrieval-style tasks.
  • Because the merged attention pattern is strictly causal, the gains can be used with the existing generation loop unchanged, opening the approach to standard causal-decoding optimizations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same prefill-time pooling recipe would likely transfer to video question answering and long-horizon embodied tasks, where all future frames are already available before generation begins; the bottleneck is the mask, not the weights.
  • If the mechanism is right, training-time mask design also matters: fine-tuning a VLM with a future-aware or sink-aware mask could make the gains larger and more stable than inference-only relaxation, a question this paper leaves open.
  • The single-prefix-token result suggests that KV-cache compression policies should treat the first token as a protected sink that can carry compressed future-visual context, rather than as a disposable padding token.
  • The mutual-information ratio in the appendix gives a testable layer-wise prediction: under $M^{f}$, intermediate layers should retain a larger fraction of output-relevant information than under $M^{c}$; layer-wise probing would confirm or refute that mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies whether the causal attention mask used in autoregressive vision-language models (VLMs) is appropriate for visual tokens during the prefill stage. It proposes three future-aware mask variants (full, visual-to-visual, visual-to-textual) that relax strict causality for visual queries, and a lightweight merging technique that pools future attention scores into a prefix token to retain autoregressive decoding. Experiments on LLaVA-7B/13B across a range of benchmarks report modest gains for the relaxed masks on some tasks and mixed performance on others. The paper argues that selectively exposing future context to visual queries improves inference, and that merging preserves most of these gains at lower decoding latency.

Significance. If the finding that future visual context improves VLM inference is robust, it would challenge a default design choice in decoder-only VLMs and could lead to cheap inference-time modifications. The paper's strengths include a systematic exploration of three mask families, a practical pooling-based merging scheme that keeps decoding causal, and a wide task suite covering temporal, relational, and text-rich reasoning. However, the empirical support is weakened by undisclosed evaluation biases, lack of statistical uncertainty quantification, and the absence of a control condition that isolates the semantic contribution of future tokens. These issues currently prevent the central claim from being considered established.

major comments (4)
  1. [Appendix A.1] The disclosed 'minor task-specific biases' (0.5 in EgocentricNavigation, 1.5 in SlideVQA) and 'top-k region ratios calibrated per dataset' are not tied to specific masks or selection procedures. Since several reported gains in Table 4 are at or below 0.02 (e.g., ActionP 0.515 vs 0.515, CLEVR 0.166 vs 0.156), a constant offset of 0.5 or 1.5 can fully determine whether a method appears superior. The paper must state exactly how these biases are applied (e.g., to which methods, at which stage) and justify that they are not introduced post hoc to inflate gains.
  2. [Tables 4 and 7] Each task uses approximately 200 samples (except Needle and Retrieval), yet all results are reported as point estimates with no standard errors, confidence intervals, or significance tests. A difference of 0.01–0.02 corresponds to 2–4 examples and is well within sampling variability. The paper's central claim rests on these deltas; without error bars, the existence of a true improvement is not established for many rows.
  3. [Sections 3 and 4] The comparisons only contrast strict causal masks with future-aware masks. There is no control that exposes the same number of additional tokens while stripping their future/semantic content (e.g., permuted or duplicated past tokens). Consequently, the improvements attributed to 'future context' could instead arise from simply giving visual queries more keys to attend to, which changes the effective context and softmax statistics. The conclusion that the semantic content of future tokens is responsible for the gains is not supported.
  4. [Section 5, Table 5, Eq. (10)] The paper groups tasks into 'Temporal Multi-Image', 'Semantic Multi-Image', etc., and reports per-group mask preferences, but these groupings and the mask selection per task are determined after observing the results. Since Eq. (10) states that µ is 'selected manually and fixed' per evaluation, the reported improvements are the result of choosing, for each benchmark, the mask that performs best in that benchmark. Without a predefined protocol or held-out validation, this constitutes post-hoc selection and inflates the apparent consistency of the findings.
minor comments (5)
  1. [Throughout] There are several typos: 'casual' for 'causal' in Figure 1 and Figure 2 captions; 'Sematic' in Table 5 heading; 'Haytack' for 'Haystack' in Table 5; 'expermental' in Section 5. These should be corrected.
  2. [Eq. (12)] The kernel pooling formula is unclear: the index ranges and the placement of the merging result (only into j=1?) are not fully specified, and the relationship between prefix size and the kernel size k is not explained. This makes the proposed merging method difficult to reproduce.
  3. [Table 6] The decoding latency comparison includes only the future-aware variants and their merged versions; the standard causal mask Mc is not reported. Since the paper's efficiency claim is that merging preserves decoding efficiency, a direct comparison against Mc is needed.
  4. [Appendix A.3] The theoretical properties A.1–A.5 rest on assumptions (causally isotropic representations, contextual decoder) and essentially restate the data processing inequality. They are not connected to the experimental results and do not provide a testable prediction; this section should either be substantially strengthened or presented as intuition.
  5. [Figure 1] Figure 1's caption says 'Breaking the casual masks of LLaVA-7b on the ALFRED benchmark' but does not explain what is plotted on the axes or what metric improves. The figure should be self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central claims are empirical comparisons against external benchmarks, and the mask definitions are independent of the reported accuracy values.

full rationale

The paper's claimed derivation chain is empirical rather than analytic. The future-aware masks (Definitions 3.1-3.3, Eqs. 7-9) are defined independently of the benchmark results, and Eqs. 10 and 14 merely re-express softmax attention with a chosen mask; no accuracy number in Table 4 is computed from these equations. The conclusion that future access helps vision queries is supported by external MILEBench evaluations under greedy decoding, not by a fitted parameter that defines the conclusion. The only self-citation, ref. [31] (Pei, Huang, Xu), supports the generic claim that inter-token context affects prediction and is not load-bearing. Appendix A.1 discloses per-dataset top-k region calibration and small task-specific biases (0.5 in EgocentricNavigation, 1.5 in SlideVQA); this is a potential test-set-tuning/validity concern, but it does not make the future-context claim true by construction, and the paper reports all three masks rather than selecting only favorable ones. The information-theoretic properties in Appendix A.3 are largely data-processing-inequality restatements, and they are not used to produce any fitted number in the main comparisons. No circular step can be exhibited by quoting an equation or a fitted-parameter-as-prediction reduction.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the validity of inference-time mask changes, on the vision-before-text layout, and on attention sinks as carriers of compressed future information. There are no newly postulated physical entities. The main input-side parameters are the per-task biases and top-k ratios fitted to the evaluation benchmarks, plus unstated kernel and prefix sizes. The information-theoretic appendix adds no independent evidence.

free parameters (4)
  • Per-task evaluation bias = 0.5 for EgocentricNavigation, 1.5 for SlideVQA
    Appendix A.1 states a fixed bias of 0.5 in EgocentricNavigation and 1.5 in SlideVQA; these values are chosen per benchmark and affect reported scores.
  • Kernel size k for future pooling = not reported
    Section 4, Eq. 12 applies 1D kernel pooling with a kernel size k, but the paper gives no value, sweep, or selection criterion for k.
  • Prefix size for merged attention = prefix size 1 in main experiments
    Table 4 uses prefix size 1; Figure 8 explores prefix ratios and concludes that a single sink token suffices, so the main configuration is informed by the same benchmark data.
  • Top-k region ratios for kernel merging = calibrated per dataset, values not reported
    Appendix A.1 says kernel-based attention merging strategies use top-k region ratios calibrated per dataset; exact values are absent, making part of the evaluation fitted.
assumptions (6)
  • domain assumption A frozen causally pretrained VLM can be evaluated with an inference-time mask different from the training mask, and the resulting hidden states remain useful.
    Section 3 and Figure 1: all improvements are obtained without fine-tuning. If the distribution shift from the mask mismatch invalidates the representations, the empirical conclusion would not follow.
  • domain assumption Vision tokens are placed before text tokens, and the analysis covers only this arrangement.
    Section 3: "we focus on the case where vision tokens are entered before text tokens." Interleaved or repeated image layouts are not tested.
  • domain assumption Attention-sink behavior from LLM studies transfers to VLM prefill and can carry pooled future information.
    Section 4 justifies merging future scores into the first vertical position by citing attention-sink work [15,42,43]; this transfer is assumed, not demonstrated inside the paper.
  • domain assumption Causal masking is necessary for text tokens even when relaxed for vision tokens.
    All mask variants keep text rows strictly causal and text-dominant tasks degrade under relaxed masks; the paper does not systematically test text-row relaxation.
  • domain assumption The pretrained vision encoder already encodes global image semantics, so causal restriction on visual tokens is redundant and restrictive.
    Section 2.2, Eq. 3 and the surrounding text argue that the vision encoder fixes visual context before the LLM, which motivates relaxing visual masks. This is an interpretation, not a proven fact.
  • standard math Data-processing inequality and standard information-theoretic monotonicity.
    Appendix A.5 derives Properties A.1-A.5 from DPI; these are standard and true for any mask, but they do not establish that future-aware masks improve downstream prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Causal Mask Attention for Vision-Language Inference." pith.science (2026). https://pith.science/paper/EWGQECJF

@misc{pith2026250518605,
  author       = {Pith},
  title        = {Pith review of: Rethinking Causal Mask Attention for Vision-Language Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EWGQECJF}},
  note         = {Machine review of arXiv:2505.18605}
}
read the original abstract

Causal attention has become a foundational mechanism in autoregressive vision-language models (VLMs), unifying textual and visual inputs under a single generative framework. However, existing causal mask-based strategies are inherited from large language models (LLMs) where they are tailored for text-only decoding, and their adaptation to vision tokens is insufficiently addressed in the prefill stage. Strictly masking future positions for vision queries introduces overly rigid constraints, which hinder the model's ability to leverage future context that often contains essential semantic cues for accurate inference. In this work, we empirically investigate how different causal masking strategies affect vision-language inference and then propose a family of future-aware attentions tailored for this setting. We first empirically analyze the effect of previewing future tokens for vision queries and demonstrate that rigid masking undermines the model's capacity to capture useful contextual semantic representations. Based on these findings, we propose a lightweight attention family that aggregates future visual context into past representations via pooling, effectively preserving the autoregressive structure while enhancing cross-token dependencies. We evaluate a range of causal masks across diverse vision-language inference settings and show that selectively compressing future semantic context into past representations benefits the inference.

Figures

Figures reproduced from arXiv: 2505.18605 by the authors.

Figure 1
Figure 1. Breaking the casual masks of LLaVA-7b on the AL￾FRED benchmark[35]. As a result, a critical question arises: Is causal attention truly a feasible mechanism for vision-language understanding? In this paper, we systematically explore the impacts of causal attention on textual and visual tokens and reveal a surprising finding: while breaking the causal masks between textual tokens sig￾nificantly disrupts model predicti… view at source ↗
Figure 2
Figure 2. An overview of our investigation into causal attention in vision-language inference. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An Example of Temporal Multi-Images Task, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: An Example of Visual Relation Tasks. Mask Visual Relation Tasks VCC VRE M 16.2 16.6 Mv2v 16.7(↑) 18.1(↑) [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: An Example of Text-Rich VQA Tasks Mask Text-Rich Image QA Tasks OCR-VQA TextVQA M 22.5 32.0 Mv2t 23.0(↑) 38.5(↑) [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: An overview of attention design for vision language inference. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Performance comparison of three causal masks ( [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Effect of prefix ratio of Light future aware attentions. 2. Future semantics can be utilized by merging them into attention sink regions in the past. To evaluate this, we define the prefix ratio as prefix size/L, where L is the total attention length, and the prefix si…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 9 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

  4. [4]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  6. [6]

    nuscenes: A multimodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020

    2020

  7. [7]

    Any-resolution training for high-resolution image synthesis

    Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-resolution training for high-resolution image synthesis. InEuropean conference on computer vision, pages 170–188. Springer, 2022

    2022

  8. [8]

    Webqa: Multihop and multimodal qa

    Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16495–16504, 2022

    2022

Show all 50 references
  1. [9]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101, 2024

    2024

  2. [10]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer visi...

    2024

  3. [11]

    Mobilevlm: A fast, strong and open vision language assistant for mobile devices.arXiv preprint arXiv:2312.16886, 2023

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, strong and open vision language assistant for mobile devices.arXiv preprint arXiv:2312.16886, 2023

    2023 arXiv

  4. [12]

    Tall: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. InProceedings of the IEEE international conference on computer vision, pages 5267–5275, 2017

    2017

  5. [13]

    Model tells you what to discard: Adaptive kv cache compression for llms.arXiv preprint arXiv:2310.01801, 2023

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms.arXiv preprint arXiv:2310.01801, 2023

    2023 arXiv

  6. [14]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    When attention sink emerges in language models: An empirical view.arXiv preprint arXiv:2410.10781, 2024

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view.arXiv preprint arXiv:2410.10781, 2024. 10

    2024 arXiv

  8. [16]

    Image change captioning by learning from an auxiliary task

    Mehrdad Hosseinzadeh and Yang Wang. Image change captioning by learning from an auxiliary task. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2725–2734, 2021

    2021

  9. [17]

    Movienet: A holistic dataset for movie understanding

    Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 709–727. Springer, 2020

    2020

  10. [18]

    Learning to describe differences between pairs of similar images.arXiv preprint arXiv:1808.10584, 2018

    Harsh Jhamtani and Taylor Berg-Kirkpatrick. Learning to describe differences between pairs of similar images.arXiv preprint arXiv:1808.10584, 2018

    2018 arXiv

  11. [19]

    Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension

    Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Han- naneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. InProceedings of the IEEE Conference on Computer Vision and Pattern re...

    2017

  12. [20]

    Beyond the nav- graph: Vision-and-language navigation in continuous environments

    Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav- graph: Vision-and-language navigation in continuous environments. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVIII 16, pages...

    2020

  13. [21]

    Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313, 2025

    Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313, 2025

    2025 arXiv

  14. [22]

    Contourlet residual for prompt learning enhanced infrared image super-resolution

    Xingyuan Li, Jinyuan Liu, Zhixin Chen, Yang Zou, Long Ma, Xin Fan, and Risheng Liu. Contourlet residual for prompt learning enhanced infrared image super-resolution. InEuropean Conference on Computer Vision, pages 270–288. Springer, 2024

    2024

  15. [23]

    From text to pixels: a context-aware semantic synergy solution for infrared and visible image fusion

    Xingyuan Li, Yang Zou, Jinyuan Liu, Zhiying Jiang, Long Ma, Xin Fan, and Risheng Liu. From text to pixels: a context-aware semantic synergy solution for infrared and visible image fusion. arXiv preprint arXiv:2401.00421, 2023

    2023 arXiv

  16. [24]

    Mmcoqa: Conversational question answering over text, tables, and images

    Yongqi Li, Wenjie Li, and Liqiang Nie. Mmcoqa: Conversational question answering over text, tables, and images. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4220–4231, 2022

    2022

  17. [25]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  18. [26]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY , USA, 2023. Curran Associates Inc

    2023

  19. [27]

    Intactkv: Improving large language model quantization by keeping pivot tokens intact.arXiv preprint arXiv:2403.01241, 2024

    Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact.arXiv preprint arXiv:2403.01241, 2024

    2024 arXiv

  20. [28]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021

    2021

  21. [29]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE, 2019

    2019

  22. [30]

    Perception test: A diagnostic benchmark for multimodal video models.Advances in Neural Information Processing Systems, 36, 2024

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models.Advances in Neural Information Processing Syste...

    2024

  23. [31]

    Cross-self kv cache pruning for efficient vision- language inference.arXiv preprint arXiv:2412.04652, 2024

    Xiaohuan Pei, Tao Huang, and Chang Xu. Cross-self kv cache pruning for efficient vision- language inference.arXiv preprint arXiv:2412.04652, 2024. 11

    2024 arXiv

  24. [32]

    Beyond semantics: Rediscovering spatial awareness in vision-language models.arXiv preprint arXiv:2503.17349, 2025

    Jianing Qi, Jiawei Liu, Hao Tang, and Zhigang Zhu. Beyond semantics: Rediscovering spatial awareness in vision-language models.arXiv preprint arXiv:2503.17349, 2025

    2025

  25. [33]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  26. [34]

    Gpr1200: a benchmark for general-purpose content-based image retrieval

    Konstantin Schall, Kai Uwe Barthel, Nico Hezel, and Klaus Jung. Gpr1200: a benchmark for general-purpose content-based image retrieval. InInternational Conference on Multimedia Modeling, pages 205–216. Springer, 2022

    2022

  27. [35]

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. InProceedings of the IEEE/CVF conference on computer vision and pattern...

    2020

  28. [36]

    Flava: A foundational language and vision alignment model

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15638–15650, 2022

    2022

  29. [37]

    Milebench: Benchmarking mllms in long context.arXiv preprint arXiv:2404.18532, 2024

    Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking mllms in long context.arXiv preprint arXiv:2404.18532, 2024

    2024 arXiv

  30. [38]

    Multimodalqa: Complex question answering over text, tables and images.arXiv preprint arXiv:2104.06039, 2021

    Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalqa: Complex question answering over text, tables and images.arXiv preprint arXiv:2104.06039, 2021

    2021 arXiv

  31. [39]

    Slidevqa: A dataset for document visual question answering on multiple images

    Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13636–13645, 2023

    2023

  32. [40]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  33. [41]

    Star: A benchmark for situated reasoning in real-world videos.arXiv preprint arXiv:2405.09711, 2024

    Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos.arXiv preprint arXiv:2405.09711, 2024

    2024 arXiv

  34. [42]

    Efficient streaming language models with attention sinks, 2024.URL https://arxiv

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024.URL https://arxiv. org/abs/2309.17453, page 1, 2024

    2024 arXiv

  35. [43]

    Seed-story: Multimodal long story generation with large language model.arXiv preprint arXiv:2407.08683, 2024

    Shuai Yang, Yuying Ge, Yang Li, Yukang Chen, Yixiao Ge, Ying Shan, and Yingcong Chen. Seed-story: Multimodal long story generation with large language model.arXiv preprint arXiv:2407.08683, 2024

    2024 arXiv

  36. [44]

    Enhancing web service anomaly detection via fine-grained multi-modal association and frequency domain analysis.arXiv preprint arXiv:2501.16875, 2025

    Xixuan Yang, Xin Huang, Chiming Duan, Tong Jia, Shandong Dong, Ying Li, and Gang Huang. Enhancing web service anomaly detection via fine-grained multi-modal association and frequency domain analysis.arXiv preprint arXiv:2501.16875, 2025

    2025 arXiv

  37. [45]

    Causal attention for vision-language tasks

    Xu Yang, Hanwang Zhang, Guojun Qi, and Jianfei Cai. Causal attention for vision-language tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9847–9857, June 2021

    2021

  38. [46]

    Filip: Fine-grained interactive language-image pre-training.arXiv preprint arXiv:2111.07783, 2021

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training.arXiv preprint arXiv:2111.07783, 2021

    2021 arXiv

  39. [47]

    Clevrer: Collision events for video representation and reasoning.arXiv preprint arXiv:1910.01442, 2019

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning.arXiv preprint arXiv:1910.01442, 2019. 12

    1910 arXiv

  40. [48]

    Stablemask: Refining causal masking in decoder-only transformer.arXiv preprint arXiv:2402.04779, 2024

    Qingyu Yin, Xuzheng He, Xiang Zhuang, Yu Zhao, Jianhua Yao, Xiaoyu Shen, and Qiang Zhang. Stablemask: Refining causal masking in decoder-only transformer.arXiv preprint arXiv:2402.04779, 2024

    2024 arXiv

  41. [49]

    Are transformers universal approximators of sequence-to-sequence functions?arXiv preprint arXiv:1912.10077, 2019

    Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions?arXiv preprint arXiv:1912.10077, 2019

    1912 arXiv

  42. [50]

    Contourlet refinement gate framework for thermal spectrum distribution regularized infrared image super-resolution.arXiv preprint arXiv:2411.12530, 2024

    Yang Zou, Zhixin Chen, Zhipeng Zhang, Xingyuan Li, Long Ma, Jinyuan Liu, Peng Wang, and Yanning Zhang. Contourlet refinement gate framework for thermal spectrum distribution regularized infrared image super-resolution.arXiv preprint arXiv:2411.12530, 2024. 13 A Technical Appen...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.