Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Demystifying the Visual Quality Paradox in Multimodal Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For multimodal LLMs, degraded images can beat clean ones.

desk verdict A real phenomenon buried under inconsistent baselines and missing statistics; the paradox deserves a serious look, but the VQ-TTT results as reported don't yet support the headline. read the letter →

arxiv 2506.15645 v1 pith:USO6E7V2 submitted 2025-06-18 cs.CV cs.AI

Shuo Xing , Lanqing Guo , Hongyuan Hua , Seoyoung Lee , Peiran Li , Yufei Wang , Zhangyang Wang , Zhengzhong Tu This is my paper
classification cs.CVcs.AI
keywords multimodallargelanguagemodelsvisualqualityparadoxtest-timetuningimagerestorationentropyminimizationLoRAadaptationVQArobustnessdegradations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that human-perceived image quality is not the right yardstick for multimodal large language models: over a systematic sweep of five degradation types and 13 vision-language datasets, model performance often improves when images are noisier, blurrier, or weather-corrupted, and off-the-shelf restoration can undo that benefit. The authors argue that each model and task has its own preferred input representation, and they support this with a lightweight test-time tuner, VQ-TTT, that reshapes each input image's frequency content and reports consistent average accuracy gains across the evaluated models and datasets with less than 1% additional parameters. If true, this reframes 'better' visual inputs as model-aligned rather than human-clean and motivates adaptive image preparation for AI consumers.

What carries the argument

The mechanism is a two-part test-time tuning module called VQ-TTT. First, a learnable kernel layer modulates the input image as $v' = (1+b)v - b\,(v * K_\sigma)$, where $b$ is a learnable blending coefficient and $K_\sigma$ a separable Gaussian kernel with learnable standard deviation; positive $b$ sharpens and negative $b$ blurs using only two parameters. Second, LoRA adapters inserted into the first two layers of the frozen CLIP vision encoder let the model adapt to the modulated input distribution. Both parts are tuned once per test image by minimizing the prediction entropy $L_{entropy} = -\sum_{y\in Y} p_\theta(y|v',t)\log p_\theta(y|v',t)$, which pushes the model toward confident answers. The same two components carry the whole argument: the kernel searches image space, the LoRA keeps the representation aligned, and the entropy loss selects the modulation without any labels.

What would settle it

Split a benchmark into a tuning half and a held-out half. Run VQ-TTT by minimizing entropy on the tuning half, then apply the learned per-image modulation to the held-out half and compare accuracy against no tuning; if the gains vanish or reverse while entropy still drops on the tuned half, the entropy proxy was overfitting the tuned inputs rather than improving correctness. A second check is to compare VQ-TTT against a version tuned with true answer labels on a small subsample; if label-tuned modulation yields different image filters and larger accuracy gains, entropy minimization is not finding the real model preference.

Watch

Extended reading notes

Core claim

The central claim is the visual-quality paradox: for many MLLMs, answers on vision-language questions can become more accurate when the input image deviates from human-perceived fidelity, depending on the model, the task, and even the individual instance. The paper demonstrates this with controlled Gaussian noise, motion and defocus blur, snow, and fog applied across MathVista, MMMU, ScienceQA, TextVQA, and MME, showing that reasoning-heavy tasks sometimes improve under degradation while recognition-heavy tasks drop. It then shows that standard restoration pipelines fail to recover the lost performance and can make it worse, because they optimize for perceptual metrics rather than the features the model actually uses. The positive proposal, VQ-TTT, inserts a learnable low-rank frequency-modulating kernel before the frozen vision encoder and fine-tunes shallow encoder layers with LoRA, minimizing prediction entropy on the test input itself; the paper reports that this lifts average accuracy across all evaluated models and datasets without external models, cached features, or extra training data.

Load-bearing premise

The load-bearing premise is that minimizing the model's prediction entropy on a test image is a valid proxy for getting the answer right, because the reported accuracy gains are measured on the very same inputs whose entropy was minimized.

Editorial extensions

If this is right

  • Off-the-shelf restoration pipelines, including strong transformer- and diffusion-based models, do not recover MLLM performance on degraded inputs and can actively hurt it, implying human perceptual quality metrics are misaligned with what MLLMs use.
  • VQ-TTT improves average accuracy across the evaluated MLLMs and all datasets, with under 1% of original model parameters, no external models, no cached features, and no extra training data.
  • Degradation can sharpen the model's relative attention on question-relevant image regions and improve semantic coherence on cognitively demanding tasks like MathVista and ScienceQA.
  • Each model and task has its own preferred visual input quality, so there is no universally 'clean' image optimum; input preparation should be adaptive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the paradox holds generally, image compression and delivery pipelines could be re-engineered to maximize downstream MLLM accuracy rather than human-rated fidelity, for example by preserving model-preferred frequency content and discarding what the model ignores.
  • A natural stress test is to apply VQ-TTT to open-ended generation tasks with no single correct answer, where confidence is not calibrated; the entropy proxy may fail there even if it works on multiple-choice VQA.
  • The per-instance nature of the paradox suggests that robustness benchmarks should report not only average degradation curves but also the distribution of instances where degradation helps, which could change how model robustness is scored.
  • The frequency-modulation idea is modality-agnostic in principle; a testable extension is to translate the learnable kernel to video frames or audio spectrograms for multimodal models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies how input image quality affects the performance of multimodal large language models (MLLMs). Through systematic degradation experiments across several MLLMs and vision-language benchmarks, it claims a "visual-quality paradox": models sometimes perform better on images that humans perceive as degraded. It then proposes Visual-Quality Test-Time Tuning (VQ-TTT), a lightweight module that inserts a learnable low-rank kernel before the frozen vision encoder and applies shallow-layer LoRA, tuned by entropy minimization on the test input. The paper reports accuracy gains from VQ-TTT and argues that restoration pipelines are insufficient because they optimize human-perceptual quality rather than model-preferred representations.

Significance. If the empirical claims were fully supported, the paper would be a valuable challenge to the common assumption that cleaner, higher-fidelity images always benefit MLLMs. The paper has notable strengths: it evaluates several models and many datasets, attempts mechanistic analysis via relative attention and logit lens, and proposes a parameter-efficient adaptation method that could be practically useful. However, the central evidence is currently undermined by an internal baseline inconsistency, a transductive evaluation protocol whose validity is not demonstrated, and the absence of variance or significance reporting. These issues must be resolved before the paradox or the VQ-TTT gains can be assessed reliably.

major comments (4)
  1. [Table 1 / Table 2] The 'original' baseline for LLaVA-v1.5-7B is not stable across the paper. Table 1 (and Table 7) report MME Perception 1510.28 and MME Cognition 357.85 for the original condition, while Table 2 reports MME Perception 1500.13 and MME Cognition 316.43 for the same model and condition. The 41.42-point discrepancy in MME Cognition is much larger than the claimed VQ-TTT delta on that metric (-8.6), and the 10.15-point discrepancy in MME Perception is comparable to the reported +12.0 gain. As a result, the reported improvement pattern is not uniquely determined by the data as presented. Please provide a single reproducible evaluation setting, including the exact checkpoint, decoding parameters, and metric implementation, and use it consistently for all baseline and VQ-TTT rows.
  2. [§4.1, Eq. (2)] VQ-TTT minimizes prediction entropy on the very test inputs on which accuracy is later reported. This transductive protocol can inflate apparent gains by making the model more confident without making it more correct. The paper does not show that lower entropy correlates with correct answers on these benchmarks, nor does it report accuracy under entropy-minimized modulation on a held-out set. Please evaluate on a separate split, or at least report the accuracy-entropy correlation and the accuracy of entropy-minimized predictions on a validation set, to rule out overconfident wrong predictions as the source of the reported gains.
  3. [§3.2 and Table 2] The headline claims of 'significant' gains and 'consistent improvements' are not backed by any variance information. No seeds, standard deviations, or significance tests are reported, and many deltas are small (e.g., +0.3, +0.2, +0.1 in Table 2). Please report multiple runs or bootstrap confidence intervals and apply a paired test across examples. Without this, the visual-quality paradox itself, which rests on average improvements of about one point on MathVista and smaller effects elsewhere, is not statistically distinguishable from noise.
  4. [Table 7] Table 7 contains unexplained rows and entries that prevent independent audit. 'VLM-R1' appears without any introduction in Section 3 or in the list of evaluated models, and Qwen-2.5-VL-3B-instruct's ScienceQAI accuracy collapses to 0.00 after every restoration pipeline while remaining nonzero under direct degradation; neither phenomenon is discussed. These entries need to be explained or removed, and the table should be reconciled with Table 1.
minor comments (5)
  1. [§3.2] The text says the results are presented in Table 7, but the main-text table is Table 1; please unify the cross-references.
  2. [Throughout] There are repeated formatting issues, including 'LLaV A' spacing, 'Visualizaion' in the Figure 1 caption, and 'traiditonal' in Section 5.3; a careful proofread is needed.
  3. [Table 1/Table 2] The column labels 'ScienceQA T' and 'ScienceQAI' are not defined; please clarify whether these are different subsets or a typo.
  4. [Abstract and §4.2] The claimed gain magnitude is inconsistent: the Abstract and Introduction state gains of up to 8.6%, while Section 4.2 reports gains of up to 4.5%; please reconcile these numbers.
  5. [Figures 2 and 3] The attention-entropy and logit-lens analyses are presented qualitatively or for single examples; please add quantitative summaries across samples to support the mechanistic claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paradox is an empirical measurement and VQ-TTT accuracy gains are evaluated with labels after label-free entropy tuning.

full rationale

I walked the paper's derivation chain and found no circular step that reduces a claimed prediction to its own inputs by construction. The visual-quality paradox is an empirical observation: the paper systematically degrades images and reports measured accuracy changes, so it is not derived from an assumption that already contains the conclusion. VQ-TTT's learning objective, Eq. (2), minimizes prediction entropy over the test input without using ground-truth answers; the reported accuracy in Table 2 is then computed with labels as an external evaluation. There is no equation or construction in the paper that forces low entropy to equal high accuracy, so the accuracy gain is not a fitted parameter renamed as a prediction. This is a standard transductive test-time adaptation evaluation: the test input is used for adaptation, but the measured outcome is independent of the tuning signal in the sense that the labels are never used during optimization. The paper's self-citations appear in related work and application listings, not as load-bearing evidence for the central claim, and no uniqueness theorem or ansatz is smuggled in through self-citation. I also checked the internal Table 1/Table 2 inconsistency: the LLaVA-v1.5-7B baseline differs between tables for MME Perception (1510.28 vs 1500.13) and MME Cognition (357.85 vs 316.43), and Table 7 contains an unexplained VLM-R1 row and some ScienceQA-I entries collapsed to 0.00. Those are reproducibility and correctness concerns, not circularity, because they do not exhibit a derivation that is equivalent to its input by definition. The paper's own Limitations section acknowledges the narrow evaluation scope and the compact method's limited capacity, which further supports treating the empirical claims as honestly scoped rather than circular. Overall, no step in the claimed derivation chain is self-definitional, no fitted input is relabeled as a prediction, and the central empirical findings stand as independent measurements.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the assumption that entropy minimization is a safe objective for test-time adaptation, that the two-parameter kernel and shallow LoRA are expressive enough, and that synthetic degradations stand in for real-world quality shifts. No free parameters are fixed to particular values in the paper, which limits reproduction.

free parameters (3)
  • b (blending coefficient in Eq. 1) = not reported
    Learned at test time via entropy minimization; controls the sharpening/blurring trade-off.
  • sigma (Gaussian kernel standard deviation) = not reported
    Learned at test time; controls the frequency content of the modulation.
  • LoRA configuration (rank, alpha, layers) = approx 0.1M parameters
    Fine-tunes shallow vision-encoder layers; exact hyperparameters are not provided.
assumptions (3)
  • domain assumption Entropy minimization is a valid proxy for prediction accuracy.
    Eq. 2 minimizes the entropy of the model's output distribution on the test input; the paper assumes this improves accuracy without providing evidence that lower entropy correlates with correct answers.
  • ad hoc to paper A learnable kernel with two scalar parameters plus shallow-layer LoRA can capture task-specific visual quality preferences.
    The paper's own Limitations section admits the compact design may be insufficient for complex or highly nonlinear degradations.
  • domain assumption Synthetic degradations from the imagecorruptions library are representative of real-world visual quality variation.
    The paper generalizes its conclusions to real-world settings based on synthetic noise, blur, snow, and fog.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Demystifying the Visual Quality Paradox in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/USO6E7V2

@misc{pith2026250615645,
  author       = {Pith},
  title        = {Pith review of: Demystifying the Visual Quality Paradox in Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/USO6E7V2}},
  note         = {Machine review of arXiv:2506.15645}
}
read the original abstract

Recent Multimodal Large Language Models (MLLMs) excel on benchmark vision-language tasks, yet little is known about how input visual quality shapes their responses. Does higher perceptual quality of images already translate to better MLLM understanding? We conduct the first systematic study spanning leading MLLMs and a suite of vision-language benchmarks, applying controlled degradations and stylistic shifts to each image. Surprisingly, we uncover a visual-quality paradox: model, task, and even individual-instance performance can improve when images deviate from human-perceived fidelity. Off-the-shelf restoration pipelines fail to reconcile these idiosyncratic preferences. To close the gap, we introduce Visual-Quality Test-Time Tuning (VQ-TTT)-a lightweight adaptation module that: (1) inserts a learnable, low-rank kernel before the frozen vision encoder to modulate frequency content; and (2) fine-tunes only shallow vision-encoder layers via LoRA. VQ-TTT dynamically adjusts each input image in a single forward pass, aligning it with task-specific model preferences. Across the evaluated MLLMs and all datasets, VQ-TTT lifts significant average accuracy, with no external models, cached features, or extra training data. These findings redefine ``better'' visual inputs for MLLMs and highlight the need for adaptive, rather than universally ``clean'', imagery, in the new era of AI being the main data customer.

Figures

Figures reproduced from arXiv: 2506.15645 by the authors.

Figure 1
Figure 1. Visualizaion of the relative attention of LLaVA-v1.5-7B on a image-question pair (image [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Changes in relative attention entropy of LLaVA-v1.5-7B [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The logit lens of LLaVA-v1.5-7B model with a image-question pair (image 180) of the [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Illustration of VQ-TTT framework. Learnable Kernel for Adaptive Vi￾sual Quality. To modulate image quality, i.e., learnable blurring or sharpening, we introduce a learnable kernel layer that adaptively interpo￾lates between the input image and its blurred version. Give…
Figure 5
Figure 5. Figure 5: Examples of the images with degradations and restorations. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Diagnosing Corruption-Induced Reliability Failures in Vision-Language Models

    cs.CV 2025-11 conditional novelty 6.0 of 10

    Mild visual corruption can boost a vision-language model's top-1 accuracy while its confidence–correctness alignment (measured by the new RAS score) degrades.

Reference graph

Works this paper leans on

80 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    Abdelhamed, S

    A. Abdelhamed, S. Lin, and M. S. Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1692--1700, 2018

    work page 2018

  2. [2]

    Abdin, J

    M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  3. [3]

    Abouelenin, A

    A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V. Chaudhary, C. Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025

    arXiv 2025

  4. [4]

    Agrawal, K

    H. Agrawal, K. Desai, Y. Wang, X. Chen, R. Jain, M. Johnson, D. Batra, D. Parikh, S. Lee, and P. Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948--8957, 2019

    work page 2019

  5. [5]

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023 a

    arXiv 2023

  6. [6]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  7. [7]

    Y. Bai, X. Zhang, Y. Wang, Y. Fu, L. Yuan, and J. Gao. Align before generate: Vision-language pretraining with contrastive learning and knowledge distillation. arXiv preprint arXiv:2302.14045, 2023 b

    arXiv 2023

  8. [8]

    Beyer, A

    L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024

    arXiv 2024

Show all 80 references
  1. [9]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  2. [10]

    K. Chen, S. Xie, Z. Ma, and K. Goldberg. Robo2vlm: Visual question answering from large-scale in-the-wild robot manipulation datasets. arXiv preprint arXiv:2505.15517, 2025

    2025 arXiv

  3. [11]

    L. Chen, X. Chu, X. Zhang, and J. Sun. Simple baselines for image restoration. In European conference on computer vision, pages 17--33. Springer, 2022

    2022

  4. [12]

    X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Doll \'a r, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015

    2015 arXiv

  5. [13]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  6. [14]

    C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using deep convolutional networks. In IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 38, pages 295--307. IEEE, 2016

    2016

  7. [15]

    C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y. Wu, and R. Ji. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models . arXiv, June 2023. doi:10.48550/arXiv.2306.13394

    2023 doi arXiv

  8. [16]

    Gurari, Q

    D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608--3617, 2018

    2018

  9. [17]

    C.-W. Hu, Y. Wang, S. Xing, C.-J. Chen, and Z. Tu. mrag: Elucidating the design space of multi-modal retrieval-augmented generation. arXiv preprint arXiv:2505.24073, 2025

    2025 arXiv

  10. [18]

    D. A. Hudson and C. D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700--6709, 2019

    2019

  11. [19]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  12. [20]

    Ledig, L

    C. Ledig, L. Theis, F. Husz \'a r, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi. Photo-realistic single image super-resolution using a generative adversarial network. CVPR, 2017

    2017

  13. [21]

    C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024 a

    2024

  14. [22]

    F. Li, Y. Wang, Q. Sun, L. Van Gool, and R. Timofte. Joint image restoration and recognition using a single model. In CVPR, 2023 a

    2023

  15. [23]

    F. Li, R. Zhang, H. Zhang, Y. Zhang, B. Li, W. Li, Z. Ma, and C. Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024 b

    2024 arXiv

  16. [24]

    H. Li, Z. Zhang, J. Liu, X. Wang, J. Wu, and Y. Wang. Visual prompt tuning for adapting vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024 c

    2024

  17. [25]

    J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888--12900. PMLR, 2022

    2022

  18. [26]

    J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR, 2023 b

    2023

  19. [27]

    P. Li, X. Zou, Z. Wu, R. Li, S. Xing, H. Zheng, Z. Hu, Y. Wang, H. Li, Q. Yuan, et al. Safeflow: A principled protocol for trustworthy and transactional autonomous agent systems. arXiv preprint arXiv:2506.07564, 2025

    2025 arXiv

  20. [28]

    X. Lin, J. He, Z. Chen, Z. Lyu, B. Dai, F. Yu, Y. Qiao, W. Ouyang, and C. Dong. Diffbir: Toward blind image restoration with generative diffusion prior. In European Conference on Computer Vision, pages 430--448. Springer, 2024

    2024

  21. [29]

    H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023. URL https://arxiv.org/abs/2304.08485

    2023 arXiv

  22. [30]

    H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024

    2024

  23. [31]

    H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, H. Yang, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525, 2024 a

    2024 arXiv

  24. [32]

    P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022

    2022

  25. [33]

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024 b

    2024

  26. [34]

    X. Luo, F. Yang, F. Ding, X. Gao, S. Xing, Y. Zhou, Z. Tu, and C. Liu. V2x-unipool: Unifying multimodal perception and knowledge reasoning for autonomous driving. arXiv preprint arXiv:2506.02580, 2025

    2025

  27. [35]

    o lund, and T. B. Sch \

    Z. Luo, F. K. Gustafsson, Z. Zhao, J. Sj \"o lund, and T. B. Sch \"o n. Controlling vision-language models for multi-task image restoration. arXiv preprint arXiv:2310.01018, 2023 a

    2023 arXiv

  28. [36]

    o lund, and T. B. Sch \

    Z. Luo, F. K. Gustafsson, Z. Zhao, J. Sj \"o lund, and T. B. Sch \"o n. Image restoration with mean-reverting stochastic differential equations. arXiv preprint arXiv:2301.11699, 2023 b

    2023 arXiv

  29. [37]

    Y. Ma, W. Ye, C. Cui, H. Zhang, S. Xing, F. Ke, J. Wang, C. Miao, J. Chen, H. Rezatofighi, et al. Position: Prospective of autonomous driving-multimodal llms world models embodied intelligence ai alignment and mamba. In Proceedings of the Winter Conference on Applications of C...

    2025

  30. [38]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models

    Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. 2024. URL https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/

    2024

  31. [39]

    Michaelis, B

    C. Michaelis, B. Mitzkus, R. Geirhos, E. Rusak, O. Bringmann, A. S. Ecker, M. Bethge, and W. Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv:1907.07484, 2019

    1907 arXiv

  32. [40]

    M. Moor, Q. Huang, S. Wu, M. Yasunaga, Y. Dalmia, J. Leskovec, C. Zakka, E. P. Reis, and P. Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pages 353--367. PMLR, 2023

    2023

  33. [41]

    S. Nah, T. Hyun Kim, and K. Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3883--3891, 2017

    2017

  34. [42]

    interpreting gpt: the logit lens

    nostalgebraist. interpreting gpt: the logit lens. 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  35. [43]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  36. [44]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748--8763. PMLR, 2021

    2021

  37. [45]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  38. [46]

    K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning. In 7th Annual Conference on Robot Learning, 2023

    2023

  39. [47]

    Roziere, J

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  40. [48]

    S. Saha, A. Sinha, and S. Bandyopadhyay. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In Pattern Recognition Letters, volume 138, pages 185--191. Elsevier, 2021

    2021

  41. [49]

    H. Shao, Y. Hu, L. Wang, G. Song, S. L. Waslander, Y. Liu, and H. Li. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15120--15130, 2024

    2024

  42. [50]

    C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, P. Luo, A. Geiger, and H. Li. Drivelm: Driving with graph visual question answering. arXiv preprint arXiv:2312.14150, 2023

    2023 arXiv

  43. [51]

    Singh, V

    A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317--8326, 2019 a

    2019

  44. [52]

    Singh, V

    A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317--8326, 2019 b

    2019

  45. [53]

    Q. Sun, A. Tzamarias, and B. Schiele. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning (ICML), pages 9229--9248, 2020

    2020

  46. [54]

    G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  47. [55]

    Q. Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/

    2024

  48. [56]

    X. Tian, J. Gu, B. Li, Y. Liu, C. Hu, Y. Wang, K. Zhan, P. Jia, X. Lang, and H. Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289, 2024

    2024 arXiv

  49. [57]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi \`e re, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  50. [58]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  51. [59]

    Ulyanov, A

    D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. CVPR, 2018

    2018

  52. [60]

    D. Wang, J. Bao, X. Dong, J.-Y. Zhu, and J. E. Gonzalez. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations (ICLR), 2021

    2021

  53. [61]

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  54. [62]

    Y. Wang, S. Xing, C. Can, R. Li, H. Hua, K. Tian, Z. Mo, X. Gao, K. Wu, S. Zhou, et al. Generative ai for autonomous driving: Frontiers and opportunities. arXiv preprint arXiv:2505.08854, 2025

    2025 arXiv

  55. [63]

    Z. Wu, X. Chen, Z. Pan, X. Liu, W. Liu, D. Dai, H. Gao, Y. Ma, C. Wu, B. Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024

    2024 arXiv

  56. [64]

    S. Xing, H. Hua, X. Gao, S. Zhu, R. Li, K. Tian, X. Li, H. Huang, T. Yang, Z. Wang, Y. Zhou, H. Yao, and Z. Tu. AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving . arXiv, Dec. 2024 a . doi:10.48550/arXiv.2412.15206

    2024 doi

  57. [65]

    S. Xing, C. Qian, Y. Wang, H. Hua, K. Tian, Y. Zhou, and Z. Tu. Openemma: Open-source multimodal model for end-to-end autonomous driving. arXiv, Dec. 2024 b . doi:10.48550/arXiv.2412.15208

    2024 doi arXiv

  58. [66]

    S. Xing, Z. Sun, S. Xie, K. Chen, Y. Huang, Y. Wang, J. Li, D. Song, and Z. Tu. Can large vision language models read maps like a human? arXiv preprint arXiv:2503.14607, 2025 a

    2025 arXiv

  59. [67]

    S. Xing, Y. Wang, P. Li, R. Bai, Y. Wang, C. Qian, H. Yao, and Z. Tu. Re-align: Aligning vision language models via retrieval-augmented direct preference optimization. arXiv preprint arXiv:2502.13146, 2025 b

    2025

  60. [68]

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. ...

    2024 arXiv

  61. [69]

    C. Yang, Q. Li, M. Luo, F. Wu, and C. Xu. Multi-task learning for image super-resolution with auxiliary tasks. In ECCV, 2020

    2020

  62. [70]

    Z. Yang, W. Ping, Z. Liu, V. Korthikanti, W. Nie, D.-A. Huang, L. Fan, Z. Yu, S. Lan, B. Li, et al. Re-vilm: Retrieval-augmented visual language model for zero and few-shot image captioning. arXiv preprint arXiv:2302.04858, 2023

    2023 arXiv

  63. [71]

    Yasunaga, A

    M. Yasunaga, A. Aghajanyan, W. Shi, R. James, J. Leskovec, P. Liang, M. Lewis, L. Zettlemoyer, and W.-t. Yih. Retrieval-augmented multimodal language modeling. arXiv preprint arXiv:2211.12561, 2022

    2022 arXiv

  64. [72]

    F. Yu, J. Gu, Z. Li, J. Hu, X. Kong, X. Wang, J. He, Y. Qiao, and C. Dong. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 25669--2...

    2024

  65. [73]

    X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark...

    2024

  66. [74]

    S. W. Zamir, A. Arora, S. H. Khan, M. Hayat, F. Khan, M. Yang, and L. Shao. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2021

    2021

  67. [75]

    Zhang, Y

    H. Zhang, Y. Liu, M. Tan, Q. Le, and A. A. Yu. Test-time adaptation for large language models via entropy minimization and alignment tuning. arXiv preprint arXiv:2305.07026, 2023 a

    2023 arXiv

  68. [76]

    Zhang, M

    J. Zhang, M. Khayatkhoei, P. Chhikara, and F. Ilievski. Mllms know where to look: Training-free perception of small visual details with multimodal llms. arXiv preprint arXiv:2502.17422, 2025

    2025 arXiv

  69. [77]

    Zhang, W

    K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26 0 (7): 0 3142--3155, 2017

    2017

  70. [78]

    Zhang, P

    Y. Zhang, P. Sun, Y. Jiang, D. Yu, C. Weng, Z. Yuan, P. Luo, and T. Kong. Bytetrack: Multi-object tracking by associating every detection box. In ECCV, 2022

    2022

  71. [79]

    Zhang, Q

    Y. Zhang, Q. Wang, P. Xie, J. Lin, and X. E. Wang. Test-time adaptation of vision-language models with cross-modal consistency. arXiv preprint arXiv:2310.10039, 2023 b

    2023 arXiv

  72. [80]

    R. Zhu, Z. Tu, J. Liu, A. C. Bovik, and Y. Fan. Mwformer: Multi-weather image restoration using degradation-aware transformers. IEEE Transactions on Image Processing, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.