Pith. sign in

REVIEW 3 major objections 5 minor 74 references

Pretrained vision encoders have rigid spectral profiles, and query-conditioned frequency injection restores the fine detail vision-language models miss.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 14:44 UTC pith:PYLWXLGR

load-bearing objection Spectral rigidity diagnosis is worth reading, but the missing VEL-Adapter-only control leaves the main claim—frequency injection drives the gains—unproven. the 3 major comments →

arxiv 2608.02124 v1 pith:PYLWXLGR submitted 2026-08-03 cs.CV cs.CL

HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models

Jin Cui , Chuanchang Su , Jiayi Lu , Xinyue Long , Boran Zhao , Pengju Ren This is my paper
classification cs.CV cs.CL
keywords spectral response rigidityfrequency injectionvision-language modelsdiscrete cosine transformhallucination robustnessfine-grained perceptiontext-conditioned attentionvisual enrichment adapter
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that large vision-language models fail on fine-grained questions because their pretrained image encoders have a fixed, per-architecture spectral fingerprint: layer by layer, the proportion of low-, mid-, and high-frequency energy barely changes with the input image, the task, or even full fine-tuning. Because the encoder sees only the image, the same picture produces the same visual tokens whether the question asks about global layout or small printed text; evidence in an under-represented frequency band never reaches the language model. To fix this, the authors add a parallel discrete-cosine-transform branch that splits the image into low-, mid-, and high-frequency tokens, and use text-conditioned cross-attention to retrieve and residually inject the bands each query needs at several encoder depths. They also add a small language-side adapter so shallow layers of the language backbone can use the enriched tokens. Across general VQA, text-rich benchmarks, and hallucination checks, the method improves accuracy on two 7-billion-parameter vision-language families without high-resolution input, and mechanistic plots show the frequency allocation becomes task-dependent while semantic attention is preserved.

Core claim

The central discovery is spectral response rigidity: pretrained vision encoders allocate frequency energy in a way that is almost fixed by pretraining, not by the image or the task, and downstream fine-tuning moves it only marginally. In a vision-language model this means the query cannot steer visual encoding: the same image yields identical visual tokens regardless of what the question asks. HAFI-VLM adds a query-conditioned frequency pathway—DCT tokens in three bands, text-modulated cross-attention with a spatial bias at multiple depths, and a text-conditioned band gate—so different questions about the same image retrieve different spectral evidence. A lightweight adapter on the first fou

What carries the argument

HAFI (Hierarchical Adaptive Frequency Injection): a parallel DCT branch turns the image into separate low-, mid-, and high-frequency token sets; at three selected encoder depths, cross-attention uses the visual tokens as queries and the frequency tokens as keys and values, with a soft spatial alignment prior and a text-conditioned band gate that weights the three bands per query, and the resulting delta is added residually to the pretrained visual representation. VEL-Adapter applies LoRA to the query and key projections of the first four LLM layers so the frozen language backbone can use the enriched visual tokens. Together they preserve the pretrained semantic pathway while making spectral

Load-bearing premise

The load-bearing premise is that the frequency pathway, not the added language-side adapter, drives the gains; the main ablation removes VEL-Adapter but never runs VEL-Adapter alone, so a language-adaptation-only explanation for part or most of the improvement cannot be excluded from the reported experiments.

What would settle it

Retrain the same staged pipeline with VEL-Adapter on the frozen base visual tokens but without any HAFI injection; if its scores match HAFI-VLM within a point on TextVQA and POPE, the claimed spectral mechanism is not the active ingredient. A second check: take a trained HAFI-VLM and randomly permute the DCT band tokens at inference; a large accuracy drop would show the retrieved frequencies matter, while no drop would mean the benefit comes from the adapter or training budget.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If spectral rigidity is real, increasing resolution or cropping regions attacks only the input side; the same encoder will still compress away task-relevant bands, so frequency-conditioned retrieval is complementary rather than redundant.
  • HAFI breaks the paper's identity equation: with text-conditioned gates, the same image can yield different visual tokens for different questions, removing a structural cause of hallucination.
  • Reported gains on OCR-style benchmarks and hallucination checks suggest a cheaper alternative to high-resolution pipelines: native resolution, single encoder pass.
  • The trained allocations become task-dependent, meaning the model learns to expose different spectral evidence per question type rather than simply amplifying all detail.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An untested corollary: combining HAFI with existing resolution or cropping methods should yield additive gains, since HAFI operates on the spectral axis rather than the spatial-detail axis.
  • A testable extension: the learned band-gate weights could serve as a probe for task difficulty—low gate entropy on a given question may flag cases where the model is about to fall back on linguistic priors.
  • If spectral rigidity is a general property of pretrained encoders, the same diagnosis and DCT-plus-text-gating fix may transfer to audio or video encoders with fixed spectral priors.
  • The reconstruction loss weights high-frequency coefficients most heavily, suggesting a concrete check: mid- and high-band gate activity should correlate with OCR accuracy if the mechanism is doing the claimed work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper identifies a phenomenon it calls spectral response rigidity: the layerwise distribution of low/mid/high frequency energy in pretrained vision encoders is nearly invariant across images, tasks, and fine-tuning checkpoints. It argues this limits VLMs because the vision encoder is not conditioned on the text query, so fine-grained evidence in bands attenuated by the pretrained profile is unavailable to the LLM. To address this, HAFI-VLM adds a parallel DCT frequency branch and hierarchical cross-attention modules that inject text-conditioned low/mid/high-frequency tokens at selected encoder depths, plus VEL-Adapter, a LoRA on the Q/K projections of the first four LLM layers. On LLaVA-1.5-7B and Qwen2.5-VL-7B the authors report gains on VQAv2, GQA, ScienceQA, TextVQA, DocVQA, ChartQA, MMBench, and POPE, and mechanistic plots show task-dependent gate weights and preserved semantic attention.

Significance. If the effect is real, this is a useful contribution: it is an inexpensive, query-aware way to expose fine-grained visual evidence without high-resolution re-encoding. The diagnostic part is the strongest: the rigidity measurements in Fig. 1 are clean, multi-encoder, and include full fine-tuning persistence, and the paper is careful to preserve the pretrained semantic path. The method includes sensible staged training and a reconstruction loss. However, the paper's central causal claim—that frequency enrichment, not LLM-side adaptation, drives the gains—is not established by the reported ablations, and the mechanistic analyses are partly circular. With an explicit no-HAFI adapter-only control and direct intervention tests, the paper could be a solid contribution.

major comments (3)
  1. [Ablation Studies, Table 2] The attribution of gains to frequency injection is not supported. Removing VEL-Adapter lowers performance by 6.5–9.3 points across the four benchmarks (Table 2), and the HAFI-only model falls below the LLaVA-1.5-7B baseline on VQAv2 (76.3 vs 78.5). The paper interprets this as a distributional shift that VEL-Adapter repairs, but no control trains VEL-Adapter (or the equivalent shallow-Q/K LoRA) without the DCT branch/HAFI. The 'Early 6 layers LoRa' row still includes HAFI, so it is not a control. Under the current evidence, H1 (HAFI supplies complementary spectral evidence) and H2 (all gains come from LoRA adaptation, which also happens to compensate for HAFI's perturbation) are indistinguishable. Please add the missing control trained under the same three stages with the frequency branch and HAFI modules disabled, and, ideally, a control with random/noise tokens in place of DCT tokens.
  2. [Mechanistic Analyses, Figs. 3–5] The claim that 'HAFI restores task-dependent spectral allocation' is supported by inspecting learned gate weights and injection strengths. Since these weights are trained with a text-conditioned input and an explicit gate-balancing regularizer, their task dependence is by construction and does not demonstrate a causal role of frequency evidence in the performance gain. The paper would need an intervention: e.g., freeze the gates to uniform or to a task-invariant schedule, swap gate patterns across tasks, or replace frequency tokens with shuffled/random band tokens, and show accuracy changes in the direction predicted by spectral content. Without such a test, Figs. 4 and 5 are descriptive of the optimized architecture, not evidence for the mechanism.
  3. [Main Results, Table 1] All benchmark numbers are single runs and many baselines are quoted from previous papers without matched training budgets. Some margins are small (Qwen GQA 72.9 vs 72.1; POPE 87.9 vs 86.6), so 'consistent improvements' is hard to evaluate. At minimum, report the mean and standard deviation over three runs for the main models and the matched controls (Finetuned-LLaVA/Finetuned-Qwen), and state explicitly which rows are reproduced under identical data/settings versus copied from the literature. This is a standard requirement for benchmark claims of this strength.
minor comments (5)
  1. [Figure 2] The overview figure uses 'VLE-Adapter' while the text and equations use 'VEL-Adapter'; please unify the terminology.
  2. [Eq. (17)] The reconstruction loss notation 'BNf 3b' is unclear. Please define the dimensions and indicate clearly that the norm is over band b.
  3. [Table 1 caption and 'Finetuned-LLaVA/Qwen' rows] The caption says results are from original papers or reproduced according to their specifications, but it is not stated which rows are reproduced and what 'Finetuned on the same settings' updates. Please specify whether these controls include the shallow Q/K LoRA, and list their training data and resolution.
  4. [Figure 6] The axis labels in Figure 6 are difficult to read ('Score' axis is repeated and 'Robust' overlaps with values). Please improve labeling and ensure the legend is legible.
  5. [Ablation Studies, Role of VEL-Adapter] The phrase 'average of 7.8 points' is consistent with Table 2, but the individual deltas (6.5, 9.3, 7.4, 7.9) should be listed together with the average to make the variance across benchmarks visible.

Circularity Check

1 steps flagged

Benchmarks are independent, but the mechanistic claim that HAFI restores task-dependent spectral allocation is partly by construction: the band gate is a text-conditioned fitted module, so observing task-dependent gate weights is a fitted-parameter readout.

specific steps
  1. fitted input called prediction [Method, Eq. (13) 'Text-conditioned band allocation'; Results, Figure 4 caption; Abstract]
    "Different tasks may require different combinations of spectral evidence. We therefore use a layer-level gate conditioned on both the current visual state and the same text representation used in Eq. (7): g_i = Softmax (MLP_gate,i([v_i;t])) ... The learned frequency injection weights vary across different benchmarks and across injection layers, demonstrating task- and depth-dependent spectral selection. ... Mechanistic analyses show that HAFI restores task-dependent spectral allocation while retaining semantic attention."

    The HAFI band gate (Eq. 13) takes the text query t as an explicit input and is trained with L_task on the task training mixtures. After fitting, the gate weights will naturally differ across tasks/benchmarks, so presenting that difference as evidence that HAFI 'restores task-dependent spectral allocation' is reading out the fitted conditioning mechanism rather than testing a prediction of the frequency-enrichment hypothesis. The gate-balancing regularizer (Eq. 18) additionally encourages diverse band usage, so the reported broadened cross-task band distributions are partly imposed by the loss. The benchmark numbers are external and non-circular, but this mechanistic 'distinct route' claim reduces to the architecture by construction.

full rationale

The headline results are external benchmark measurements (Table 1) and are therefore not circular; I found no load-bearing self-citation or imported-uniqueness argument. The missing VEL-Adapter-only control (Table 2, 'w/o VEL-Adapter' drops 7.8 points and falls below LLaVA-1.5-7B on VQAv2) is a genuine attribution gap that prevents distinguishing frequency-injection effects from LLM-adaptation effects, but that is a confounding/control problem, not a circular derivation, so it does not itself raise the circularity score. The one circular element is confined to the mechanistic analysis: since the band gate is defined as a function of the text query (Eq. 13) and trained on the same tasks, the observed task-dependent gate weights and band allocations are fitted-parameter readouts; the claim that HAFI 'restores task-dependent spectral allocation' is thus partly true by construction. The gate-balancing regularizer further guarantees diversity in band usage. Score 3 reflects one by-construction mechanistic step while the central performance claims retain independent content.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or mathematical entities. It does introduce a new architectural component (HAFI, VEL-Adapter) and a diagnostic concept (spectral response rigidity), but these are not 'entities' in the sense of the ledger. The free parameters are the hand-chosen band partitions, injection layers, loss weights, and learning rates, all tuned on downstream validation performance.

free parameters (6)
  • Frequency band partition boundaries = low: 0-15, mid: 16-39, high: 40-63 (zigzag order)
    Chosen by hand; determines what is considered low, mid, and high frequency evidence.
  • Injection layers S_inj = LLaVA: {6,12,18}; Qwen2.5-VL: {8,16,24}
    Selected empirically; ablations show exact layer choice within a depth range has minor effect.
  • Reconstruction loss band weights = w_low:w_mid:w_high = 1:2:4
    Ad hoc weighting to stabilize frequency branch pretraining.
  • Gate balancing coefficient = beta_bal = 0.01
    Regularizer strength chosen by hand.
  • DCT reconstruction coefficient = beta_rec = 0.1
    Auxiliary loss weight in Stage 2.
  • Bottleneck dimension D_s = not reported in paper
    Dimension of shared bottleneck in HAFI attention; unspecified, could affect capacity.
axioms (5)
  • domain assumption DCT coefficients after 8x8 block transform, pooled 2x2, retain sufficient spectral fidelity to serve as complementary evidence at 21x21 tokens.
    The method relies on the frequency branch preserving useful detail after pooling and projection; if this pooling discards high-frequency evidence, the approach may fail on text-rich tasks.
  • domain assumption A frozen CLIP text encoder (ViT-L/14) provides task semantics adequate for band gating.
    The text-conditioned gate uses this encoder's pooled representation; if the text encoder is not aligned with VLM tasks, the gating may be uninformative.
  • domain assumption Spectral response rigidity is measured via channel-averaged DCT energy ratios and this reflects a limitation of the encoder.
    The diagnosis equates the energy ratio trajectories with a perceptual limitation; other aspects of representation geometry may also matter.
  • ad hoc to paper LoRA on query/key projections of first four LLM layers is sufficient to adapt the LLM to enriched visual tokens.
    Choice of adapter scope is specific to this paper; the large ablation drop suggests the interface adaptation is critical.
  • standard math Standard autoregressive language modeling loss with the given dataset mixture is a sufficient training objective.
    Uses standard cross-entropy loss, but the dataset mixture is hand-crafted.

pith-pipeline@v1.3.0-daily-deepseek · 14068 in / 11351 out tokens · 75146 ms · 2026-08-04T14:44:21.655070+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models." pith.science (2026). https://pith.science/paper/PYLWXLGR

@misc{pith2026260802124,
  author       = {Pith},
  title        = {Pith review of: HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PYLWXLGR}},
  note         = {Machine review of arXiv:2608.02124}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Vision-language models (VLMs) remain unreliable when predictions require fine-grained visual evidence. We identify a previously overlooked cause: spectral response rigidity. Despite substantial frequency variation across images and tasks, pretrained vision encoders exhibit persistent, encoder-specific layerwise spectral profiles that change only marginally under downstream fine-tuning. Since pretrained vision encoders only receive images, they cannot adapt spectral extraction to the evidence required by the current query. We therefore propose HAFI-VLM, which introduces a task-conditioned frequency pathway while preserving the pretrained semantic representation. Hierarchical Adaptive Frequency Injection (HAFI) retrieves complementary low-, mid-, and high-frequency evidence at multiple encoder depths using text-modulated, spatially aligned cross-attention. A Visual Enrichment Layer Adapter further recalibrates shallow LLM attention to effectively utilize the enriched visual tokens. Experiments on LLaVA-1.5 and Qwen2.5-VL demonstrate consistent improvements in general VQA, text-rich understanding, and hallucination robustness, outperforming representation-level enhancement methods and most resolution- or cropping-based approaches without additional high-resolution encoding. Mechanistic analyses show that HAFI restores task-dependent spectral allocation while retaining semantic attention, establishing frequency enrichment as a distinct and effective route for improving VLM perception.

Figures

Figures reproduced from arXiv: 2608.02124 by Boran Zhao, Chuanchang Su, Jiayi Lu, Jin Cui, Pengju Ren, Xinyue Long.

Figure 1
Figure 1. Figure 1: Spectral response rigidity of pretrained vision encoders. Although the benchmarks exhibit substantially different input spectra in (b), the layerwise frequency trajectories remain highly consistent within each encoder in (a). (c–d) Effective rank and patch uniformity reveal encoder-specific representation compression that co-evolves with the spectral profiles. (e–f) Across TextVQA fine-tuning checkpoints, … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of HAFI-VLM. A DCT branch constructs multi-frequency tokens, which are retrieved and residually injected through text-conditioned HAFI modules at multiple vision encoder depths. VEL-Adapter adapts the shallow visual– language interface in the LLM backbone to the frequency-enriched visual tokens. resents high-resolution documents in the DCT domain, whereas Fourier Compressor exploits frequency redu… view at source ↗
Figure 3
Figure 3. Figure 3: Task-adaptive spectral rebalancing induced by HAFI. (a) HAFI produces structured, layer-dependent shifts across frequency bands. (b) Frequency enhancement yields more diverse sample-level and cross-task band distributions than the base model. (c) Frequency enhancement didn’t alter the frequency response characteristics established during pre-training; instead, it amplifies the sample-level variance in spec… view at source ↗
Figure 5
Figure 5. Figure 5: Text semantic guided frequency injection. Gate and injection strengths for different frequency bands vary across different queries, showing that different textual task types induce distinct frequency-injection patterns. ment methods: (1) representation-level approaches that fuse, reweight, or refine multi-layer visual features, and (2) input￾level approaches that acquire additional evidence through higher … view at source ↗
Figure 6
Figure 6. Figure 6: Ablations on frequency bands and injection depths. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Stage-wise composition of the training data. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Preservation of semantic grounding under frequency enrichment. Each row presents the original image and the visual attention distributions of the base model and HAFI-VLM under the same query. Attention is extracted from the last input token immediately before answer generation at LLM Layer 15, providing a fixed middle-to-deep-layer view of the visual evidence used for decoding. Despite the redistribution o… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

74 extracted references · 15 linked inside Pith

  1. [1]

    Bai, S.; qin Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; Zhong, H.; Zhu, Y.; Yang, M.; Li, Z.; Wan, J.; Wang, P.; Ding, W.; Fu, Z.; Xu, Y.; Ye, J.; Zhang, X.; Xie, T.; Cheng, Z.; Zhang, H.; Yang, Z.; Xu, H.; and Lin, J. 2025. Qwen2.5-VL Technical Report. ArXiv, abs/2502.13923

    Pith/arXiv arXiv 2025

  2. [2]

    Cao, Y.; Liu, Y.; Chen, Z.; Shi, G.; Wang, W.; Zhao, D.; and Lu, T. 2024. MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understanding. ArXiv, abs/2410.11829

    Pith/arXiv arXiv 2024

  3. [3]

    Dai, W.; Li, J.; Li, D.; Tiong, A. M. H.; Zhao, J.; Wang, W.; Li, B.; Fung, P.; and Hoi, S. C. H. 2023. InstructBLIP : Towards General-purpose Vision-Language Models with Instruction Tuning. In Advances in Neural Information Processing Systems, volume 36, 49250--49267

    2023

  4. [4]

    Feng, H.; Liu, Q.; Liu, H.; Tang, J.; Zhou, W.; Li, H.; and Huang, C. 2024. DocPedia : Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding. Science China Information Sciences, 67(12): 220106

    2024

  5. [6]

    Lin, C.; Su, S.; Luo, R.; Chen, Y.; Wang, Y.; Pavone, M.; and Miao, F. 2026. Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMs. ArXiv, abs/2601.03100

    arXiv 2026

  6. [7]

    Lin, Z.; Liu, Y.; Yang, Y.; Tao, L.; and Ye, D. 2025. AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition. ArXiv, abs/2512.03794

    arXiv 2025

  7. [8]

    Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2023. Improved Baselines with Visual Instruction Tuning. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 26286--26296

    2023

  8. [10]

    Luo, G.; Zhou, Y.; Zhang, Y.; Zheng, X.; Sun, X.; and Ji, R. 2024. Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models. ArXiv, abs/2403.03003

    Pith/arXiv arXiv 2024

  9. [12]

    N.; Namboodiri, V

    Patro, B. N.; Namboodiri, V. P.; and Agneeswaran, V. S. 2025. SpectFormer : Frequency and Attention Is What You Need in a Vision Transformer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 9543--9554

    2025

  10. [13]

    Qin, Z.; Zhang, P.; Wu, F.; and Li, X. 2021. FcaNet : Frequency Channel Attention Networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 783--792

    2021

  11. [15]

    E.; and Darrell, T

    Shi, B.; Wu, Z.; Mao, M.; Wang, X. E.; and Darrell, T. 2024. When Do We Not Need Larger Vision Models? ArXiv, abs/2403.13043

    Pith/arXiv arXiv 2024

  12. [16]

    Shi, Y.; Pei, X.; Dong, M.; and Xu, C. 2025. Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception. ArXiv, abs/2509.16944

    arXiv 2025

  13. [17]

    Shi, Y.; Pei, X.; Wen, L.; Dong, M.; and Xu, C. 2026. Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models. ArXiv, abs/2604.06912

    Pith/arXiv arXiv 2026

  14. [18]

    Wang, H.; Kai, J.; Bai, H.; Hou, L.; Jiang, B.; He, Z.; and Lin, Z. 2025. Fourier Compressor : Frequency-Domain Visual Token Compression for Vision-Language Models. arXiv preprint arXiv:2508.06038

    Pith/arXiv arXiv 2025

  15. [19]

    J.; Lai, X.; Yu, B.; Zhao, H.; and Jia, J

    Yang, S.; Li, J. J.; Lai, X.; Yu, B.; Zhao, H.; and Jia, J. 2025. VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning. ArXiv, abs/2507.13348

    Pith/arXiv arXiv 2025

  16. [20]

    Yao, H.; Wu, W.; Yang, T.; Song, Y.; Zhang, M.; Feng, H.; Sun, Y.; Li, Z.; Ouyang, W.; and Wang, J. 2024. Dense Connector for MLLMs. ArXiv, abs/2405.13800

    Pith/arXiv arXiv 2024

  17. [21]

    Zhang, J.; Khayatkhoei, M.; Chhikara, P.; and Ilievski, F. 2025. MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs. ArXiv, abs/2502.17422

    Pith/arXiv arXiv 2025

  18. [23]

    ArXiv , year=

    RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control , author=. ArXiv , year=

  19. [24]

    ArXiv , year=

    OpenVLA: An Open-Source Vision-Language-Action Model , author=. ArXiv , year=

  20. [25]

    ArXiv , year=

    0: A Vision-Language-Action Flow Model for General Robot Control , author=. ArXiv , year=

  21. [26]

    ArXiv , year=

    Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model , author=. ArXiv , year=

  22. [27]

    ArXiv , year=

    GLaD: Geometric Latent Distillation for Vision-Language-Action Models , author=. ArXiv , year=

  23. [28]

    ArXiv , year=

    PointVLA: Injecting the 3D World into Vision-Language-Action Models , author=. ArXiv , year=

  24. [29]

    ArXiv , year=

    SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model , author=. ArXiv , year=

  25. [30]

    ArXiv , year=

    3D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Tasks , author=. ArXiv , year=

  26. [31]

    ArXiv , year=

    GeoVLA: Empowering 3D Representations in Vision-Language-Action Models , author=. ArXiv , year=

  27. [32]

    ArXiv , year=

    ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Models , author=. ArXiv , year=

  28. [33]

    ArXiv , year=

    GeoAware-VLA: Implicit Geometry Aware Vision-Language-Action Model , author=. ArXiv , year=

  29. [34]

    2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    VGGT: Visual Geometry Grounded Transformer , author=. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    2025

  30. [35]

    ArXiv , year=

    Evo-0: Vision-Language-Action Model with Implicit Spatial Understanding , author=. ArXiv , year=

  31. [36]

    2026 , url=

    3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models , author=. 2026 , url=

    2026

  32. [37]

    ArXiv , year=

    AugVLA-3D: Depth-Driven Feature Augmentation for Vision-Language-Action Models , author=. ArXiv , year=

  33. [38]

    ArXiv , year=

    ABot-M0: VLA Foundation Model for Robotic Manipulation with Action Manifold Learning , author=. ArXiv , year=

  34. [39]

    2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    What’s in the Imageƒ A Deep-Dive into the Vision of Vision Language Models , author=. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    2025

  35. [40]

    2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens , author=. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    2025

  36. [41]

    ArXiv , year=

    The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering , author=. ArXiv , year=

  37. [42]

    CoRR , year=

    An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks , author=. CoRR , year=

  38. [43]

    LoRA Learns Less and Forgets Less , author=. Trans. Mach. Learn. Res. , year=

  39. [44]

    ArXiv , year=

    LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention , author=. ArXiv , year=

  40. [45]

    2023 IEEE/CVF International Conference on Computer Vision (ICCV) , year=

    Adding Conditional Control to Text-to-Image Diffusion Models , author=. 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , year=

    2023

  41. [46]

    2025 , url=

    Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models , author=. 2025 , url=

    2025

  42. [47]

    2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    Improved Baselines with Visual Instruction Tuning , author=. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    2024

  43. [48]

    ArXiv , year=

    Qwen2.5-VL Technical Report , author=. ArXiv , year=

  44. [49]

    ArXiv , year=

    Dense Connector for MLLMs , author=. ArXiv , year=

  45. [50]

    ArXiv , year=

    Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models , author=. ArXiv , year=

  46. [51]

    International Journal of Computer Vision , year=

    TokenPacker: Efficient Visual Projector for Multimodal LLM , author=. International Journal of Computer Vision , year=

  47. [52]

    2024 , url=

    LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformer , author=. 2024 , url=

    2024

  48. [53]

    ArXiv , year=

    A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models , author=. ArXiv , year=

  49. [54]

    ArXiv , year=

    HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models , author=. ArXiv , year=

  50. [55]

    ArXiv , year=

    VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning , author=. ArXiv , year=

  51. [56]

    ArXiv , year=

    AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition , author=. ArXiv , year=

  52. [57]

    ArXiv , year=

    Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception , author=. ArXiv , year=

  53. [58]

    ArXiv , year=

    Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models , author=. ArXiv , year=

  54. [59]

    ArXiv , year=

    When Do We Not Need Larger Vision Models? , author=. ArXiv , year=

  55. [60]

    ArXiv , year=

    MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs , author=. ArXiv , year=

  56. [61]

    ArXiv , year=

    Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMs , author=. ArXiv , year=

  57. [62]

    Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages =

    Qin, Zequn and Zhang, Pengyi and Wu, Fei and Li, Xi , title =. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages =. 2021 , doi =

    2021

  58. [63]

    and Namboodiri, Vinay P

    Patro, Badri N. and Namboodiri, Vinay P. and Agneeswaran, Vijay S. , title =. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages =. 2025 , doi =

    2025

  59. [64]

    Science China Information Sciences , volume =

    Feng, Hao and Liu, Qi and Liu, Hao and Tang, Jingqun and Zhou, Wengang and Li, Houqiang and Huang, Can , title =. Science China Information Sciences , volume =. 2024 , doi =

    2024

  60. [65]

    arXiv preprint arXiv:2508.06038 , year =

    Wang, Huanyu and Kai, Jushi and Bai, Haoli and Hou, Lu and Jiang, Bo and He, Ziwei and Lin, Zhouhan , title =. arXiv preprint arXiv:2508.06038 , year =. doi:10.48550/arXiv.2508.06038 , eprint =

    Pith/arXiv arXiv doi:10.48550/arxiv.2508.06038

  61. [66]

    Dai, Wenliang and Li, Junnan and Li, Dongxu and Tiong, Anthony Meng Huat and Zhao, Junqi and Wang, Weisheng and Li, Boyang and Fung, Pascale and Hoi, Steven C. H. , title =. Advances in Neural Information Processing Systems , volume =. 2023 , doi =

    2023

  62. [67]

    , title =

    Ruthardt, Jona and Gaur, Manu and Ramanan, Deva and Tapaswi, Makarand and Asano, Yuki M. , title =. arXiv preprint arXiv:2604.02327 , year =. 2604.02327 , archivePrefix =

    Pith/arXiv arXiv

  63. [68]

    arXiv preprint arXiv:2606.19584 , year =

    Mao, Chengzhi and Lin, Xudong and Chu, Wen-Sheng , title =. arXiv preprint arXiv:2606.19584 , year =. 2606.19584 , archivePrefix =

    Pith/arXiv arXiv

  64. [69]

    arXiv preprint arXiv:2603.00655 , year =

    Liu, Ying and Han, Yudong and Shi, Kean and Pan, Liyuan , title =. arXiv preprint arXiv:2603.00655 , year =. 2603.00655 , archivePrefix =

    Pith/arXiv arXiv

  65. [70]

    and Zhao, Yijun , title =

    Kitessa, Akayou A. and Zhao, Yijun , title =. arXiv preprint arXiv:2606.03795 , year =. 2606.03795 , archivePrefix =

    Pith/arXiv arXiv

  66. [71]

    arXiv preprint arXiv:2604.27875 , year =

    Zhou, Shuchang and Wu, Shangkun and Wei, Jiwei and Liu, Ke and Ran, Ran and Qin, Caiyan and Yang, Yang , title =. arXiv preprint arXiv:2604.27875 , year =. 2604.27875 , archivePrefix =

    Pith/arXiv arXiv

  67. [72]

    ArXiv , year=

    MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understanding , author=. ArXiv , year=

  68. [73]

    ArXiv , year=

    Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice , author=. ArXiv , year=

  69. [74]

    ArXiv , year=

    Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere , author=. ArXiv , year=

  70. [75]

    ArXiv , year=

    Learning Transferable Visual Models From Natural Language Supervision , author=. ArXiv , year=

  71. [76]

    Proceedings of the National Academy of Sciences of the United States of America , year=

    Prevalence of neural collapse during the terminal phase of deep learning training , author=. Proceedings of the National Academy of Sciences of the United States of America , year=

  72. [77]

    ArXiv , year=

    Understanding Dimensional Collapse in Contrastive Self-supervised Learning , author=. ArXiv , year=

  73. [78]

    2000 , url=

    Matrix Analysis and Applied Linear Algebra , author=. 2000 , url=

    2000

  74. [79]

    International Conference on Machine Learning , year=

    Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth , author=. International Conference on Machine Learning , year=