Pith. sign in

REVIEW 4 major objections 7 minor 38 references

From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Two-space LoRA beats LoRA-MoE at near-vanilla speed.

desk verdict A genuinely useful empirical adapter for EVIT, but the theoretical story oversells what the static Corollary 2 proves; worth a serious referee with major revisions. read the letter →

arxiv 2411.12787 v3 pith:ZRQQCHJT submitted 2024-11-19 cs.CV cs.AI

Pengkun Jiao , Bin Zhu , Jingjing Chen , Chong-Wah Ngo , Yu-Gang Jiang This is my paper · ORCID
classification cs.CVcs.AI
keywords DualLow-RankAdaptationVisualCueEnhancementEfficientInstructionFine-TuningMixtureofExpertsDataConflictMultimodalLargeLanguageModelsRankRectificationParameter-Efficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual instruction tuning of multimodal LLMs with low-rank adapters suffers data conflicts as tasks diversify: knowledge learned for one task, like ingredient recognition, can contradict knowledge for another, like recipe generation. Mixture-of-experts variants of LoRA mitigate this by activating different expert adapters per task, but the router plus multiple experts adds inference time. This paper claims a single adapter can get the benefit of localized expert activation by splitting into two low-rank spaces: a skill space that stores holistic, stable knowledge, and a task space that rectifies, per token, which parts of that knowledge are locally activated. It also adds a lightweight visual-cue module that folds multi-level local image features into the vision projection. The result is a method that, the paper reports, beats vanilla LoRA and LoRA-MoE baselines on food, science QA, and captioning benchmarks while adding only 1.16x the inference time of vanilla LoRA.

What carries the argument

The central objects are the skill space $S$ and task space $T$: $S$ is a low-rank matrix that accumulates stable, holistic domain knowledge across tasks, while $T$ is a second low-rank matrix whose output is passed through ReLU and elementwise-multiplied with the normalized skill output to produce a rank-rectified, token-dependent gate on the holistic knowledge. The update is applied through $B$ and scaled by $r/\alpha$. The other component, Visual Cue Enhancement (VCE), uses deformable attention to gather local features from selected intermediate ViT layers and adds them to the high-level anchor feature before projection. The key mechanism is the elementwise product of a holistic, normalized skill response with a per-token rectified task response, which the paper claims gives localized, task-specific activation without a router.

What would settle it

Train on a synthetic two-task dataset where the optimal task updates are rank-1 and have disjoint input supports; then check whether a single Dual-LoRA can match the loss of a two-expert LoRA-MoE, and if no $S$ and $T$ can achieve the per-token selection, the expressiveness claim is refuted. Alternatively, if removing the non-linear activation from the task space does not reduce task-specific activation (measured by entropy or gradient correlation), the rectification mechanism is not doing the claimed work.

Watch

Extended reading notes

Core claim

The paper's central claim is that the Dual-LoRA module $D(x)=\frac{r}{\alpha}B(\mathrm{Norm}(Sx)\odot\sigma(Tx))$ can replace the router-and-experts structure of LoRA-MoE: the skill matrix $S$ learns holistic cross-task knowledge, and the task matrix $T$, passed through ReLU, provides a per-token rectification that locally activates that knowledge. The paper argues via a corollary of a matrix decomposition result that a LoRA of the form $B(A\odot \sigma(T))$ is as expressive as a set of separate LoRA experts, and that this structure is more time-efficient because it requires no routing over multiple experts. On UniFood, ScienceQA, and Flickr30k the proposed method outperforms vanilla LoRA and three LoRA-MoE baselines, and the full Dual-LoRA+VCE system uses 1.16x vanilla LoRA inference time and 73% of the time of a 4-expert LoRA-MoE.

Load-bearing premise

The paper assumes that a static matrix factorization result about $B(A\odot \sigma(T))$ implies that the input-dependent update $B(Sx\odot \sigma(Tx))$ can simulate the per-token expert selection of LoRA-MoE, but it offers no proof for the dynamic, per-token version.

Editorial extensions

If this is right

  • If the dual-space adapter indeed reproduces expert-level local response, LoRA-MoE systems could be replaced by a single adapter with comparable accuracy and lower latency.
  • The skill-space/task-space split offers a target for interpretability: the rectified skill space shows lower entropy and localized high-energy bands, suggesting a measurable, task-specific activation pattern.
  • The VCE module's use of multi-level features works with frozen vision encoders, so existing efficient visual instruction tuning pipelines can adopt it without full fine-tuning.
  • On general benchmarks, the method ranks among the top two in most settings, implying the benefits of localized adaptation transfer beyond specialized food and science tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's expressiveness argument moves from a static decomposition (fixed $T$) to a per-token $Tx$; whether the dynamic version truly spans the same function class is not proven, so the strongest theoretical reading rests on an unstated assumption.
  • A natural test is to measure how task-specific the learned task space actually is: if the ReLU gates collapse to a single pattern across tasks, the local-response explanation would be weakened even if accuracy stays high.
  • The same rectification idea might generalize to other parameter-efficient methods, for instance as a drop-in replacement for adapter MoEs in text-only LLMs where data conflicts also occur.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Dual Low-Rank Adaptation (Dual-LoRA) and Visual Cue Enhancement (VCE) for efficient visual instruction tuning of multimodal LLMs. Dual-LoRA replaces a LoRA-MoE router-expert ensemble with a single parameterized module D(x) = (r/alpha) B(Norm(Sx) ⊙ σ(Tx)), where S is a holistic 'skill space' and T is a 'task space'; the authors argue via Corollary 2 that this structure can simulate the local, task-conditioned responses of LoRA-MoE. VCE enhances high-level vision features by aggregating local multi-level features through deformable attention. Experiments on UniFood, ScienceQA, Flickr30k, and general MLLM benchmarks report consistent improvements over vanilla LoRA and LoRA-MoE baselines, with inference overhead of 1.16× over vanilla LoRA and 73% of a 4-expert LoRA-MoE.

Significance. If the claims hold, the paper offers a practically attractive alternative to LoRA-MoE: a structurally simpler adapter with competitive accuracy, lower memory, and substantially lower inference cost. The VCE module is lightweight and ablation results suggest it helps. The paper is also honest in reporting efficiency numbers. However, the central conceptual justification—that Dual-LoRA inherits the local-response capability of LoRA-MoE—rests on a proof gap, and the empirical gains are reported without variance or multiple seeds, so the strength of the empirical claim is not yet fully established.

major comments (4)
  1. [Section 3.4.2, Eq. (10)-(12)] Corollary 2 is a statement about a fixed matrix product B(A ⊙ σ(T)), where T is a constant matrix of the same shape as A. The actual Dual-LoRA module is D(x) = B(Norm(Sx) ⊙ σ(Tx)), an input-dependent, per-token gated map. These are not the same object: (Sx) ⊙ σ(Tx) is not equal to (S ⊙ σ(T))x in general. The paper asserts in Section 3.4.2 that the task space T can be 'mapped through a non-linear activation to achieve rank rectification' and thereby simulate local responses, but no proof or formal argument connects the static decomposition of Corollary 2 to the per-token rectification in Eq. (10). This is load-bearing because the paper's central claim is that Dual-LoRA is a unified, efficient simulation of LoRA-MoE's localized activation; without this bridge, Dual-LoRA is an ad hoc gated adapter whose empirical success must stand on its own.
  2. [Tables 1, 2, 4, and 5] All experimental results are reported from a single run, without error bars, confidence intervals, or significance tests. Several head-to-head gaps are small: in Table 1, Dual-LoRA beats RoDE by only 0.6 IoU and 1.0 SacreBLEU, and in Table 3, adding VCE to vanilla LoRA improves IoU by only 0.1. For a paper whose main claim is consistent improvement across tasks, these differences could be within run-to-run variance. Please report at least three seeds with mean and standard deviation, or otherwise justify that the differences are reproducible.
  3. [Section 3.4.1, Proposition 1 and Corollary 1] The proposition and corollary state that a single higher-rank LoRA is at least as expressive as a sum of lower-rank LoRAs, which is true in a linear-algebra sense. However, the paper then argues that LoRA-MoE's advantage comes from 'differentiated local response capability' and that Corollary 2 provides the route to recovering this in a single module. Since the gap between Corollary 2 and the implemented input-dependent rectification is unresolved, the narrative from Proposition 1 through Corollary 2 to Eq. (10) does not yet constitute a rigorous motivation for the Dual-LoRA design. A precise statement of what class of per-token gating functions can be represented, or a constructive decomposition of a MoE forward pass into the Dual-LoRA form, would be needed.
  4. [Section 4.5, Figure 7] The memory-efficiency comparison in Figure 7 varies the rank of vanilla LoRA and Dual-LoRA while freezing outside adapters, but the caption says 'only the adapters are set as trainable,' which contradicts the main experimental setup where the vision projector and VCE are also trainable. It should be clarified whether VCE is included in the Dual-LoRA curve and what 'adapter memory' includes; otherwise the comparison of parameter efficiency is not airtight.
minor comments (7)
  1. [Table 2] The column header 'ScareBLEU' is a typo; it should be 'SacreBLEU'.
  2. [Section 4.2] The text says 'for ScienceQA, as illustrated in Fig 2', but the ScienceQA results appear in Table 2, not Figure 2; please correct the cross-reference.
  3. [Eq. (5)-(7)] The phrase 'indicate the interest ares' appears to be a typo for 'indicate the interest areas'.
  4. [Figure 6 caption] The caption uses 'it's rectified space'; this should be 'its rectified space'.
  5. [Section 4.6 heading] The heading 'Disscusion on General Benchmarks' should be 'Discussion on General Benchmarks'.
  6. [Section 3.3 vs Section 4.1] The VCE module size is reported as 5.53 MB in the introduction and 5.52 MB in the hyperparameters paragraph; please make the numbers consistent.
  7. [Section 4.1] The paper references Appendix B for implementation details, but the main text includes no appendix; please ensure the supplementary materials are available and referenced correctly at submission.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: headline results are external benchmark evaluations; the Corollary 2-to-Eq. (10) step is an expressiveness gap, not a definitional reduction.

full rationale

The paper's central claims are empirical: Dual-LoRA and VCE are tested against vanilla LoRA, LoRA-MoE variants, and RoDE on UniFood, ScienceQA, Flickr30k, and general MLLM benchmarks (Tables 1, 2, 5 and Fig. 8). These are direct test-set evaluations, not fitted parameters renamed as predictions. The theoretical motivation in Sections 3.4.1-3.4.2 cites Corollary 2, a static decomposition result for a fixed matrix B(A⊙σ(T)), and then asserts that the per-token rectified map D(x)=B(Sx⊙σ(Tx)) can 'simulate the local response' of LoRA-MoE. Corollary 2 and Eq. (10) are not the same object because (Sx)⊙(Tx) is not generally equal to (S⊙T)x, so the transition is unsupported; however, this is a correctness/expressiveness gap, not circularity, since Eq. (10) is not defined in terms of the empirical outcome and the experiments do not presuppose the conclusion. Self-citations to RoDE [13] are used for motivation and as a baseline, and the paper's own Tables 1 and 2 reproduce the cited claim that LoRA-MoE can outperform a single LoRA, so the self-citation is not load-bearing in a circular way. No fitted-input-called-prediction, no imported uniqueness theorem, and no renaming of a known result was found.

Assumptions & free parameters 4 free parameters · 3 assumptions · 2 invented entities

The central method depends on two hand-set hyperparameters (gamma, VCE layer indices), one architectural assumption that a static LoRA decomposition can be lifted to an input-dependent rectifier, and the domain assumption that data conflicts are the causal bottleneck. No new physical entities are proposed beyond the named skill/task matrices, which have no independent falsifiable handle.

free parameters (4)
  • VCE residual scaling gamma = 1.0
    Set to 1.0 for all tasks in Section 4.1; not tuned per task and not ablated.
  • VCE reference layer indices = 2nd, 8th, 14th, 20th to last layers of CLIP ViT-L, anchor 2nd
    Hand-selected in Section 4.1; no ablation or search is reported.
  • Dual-LoRA rank = 64
    Chosen in Section 4.1 to match the total rank of the LoRA-MoE baselines; a sweep over 32, 64, 128 is shown only for memory scaling.
  • LoRA alpha and dropout = alpha = 2 * rank, dropout 0.05
    Set in Section 4.1 following standard LoRA practice; not derived.
assumptions (3)
  • ad hoc to paper The static matrix decomposition in Corollary 2 implies that the input-dependent rectified Dual-LoRA D(x)=B(Sx elementwise-multiplied by sigma(Tx)) can simulate the local response of LoRA-MoE.
    Section 3.4.2 uses Corollary 2 to justify the task space design, but the corollary treats T as a fixed matrix while Eq. (10) applies T to each input x; the implication is not proved.
  • domain assumption Data conflicts in LoRA-based instruction tuning are the main cause of performance degradation and are mitigated by local response capability.
    Section 1 and Figure 4 motivate the method with this assumption; no controlled experiment isolates data conflict severity.
  • domain assumption One epoch of training on each dataset is sufficient to fairly compare adapter methods.
    Section 4.1 states all datasets are trained for one epoch; the paper does not show convergence behavior or multiple training runs.
invented entities (2)
  • Skill space S
    purpose: Stores holistic cross-task knowledge as a low-rank matrix; the paper claims it stabilizes adaptation.
    No falsifiable prediction outside the paper; its existence is inferred only from the proposed model's internal structure and downstream metrics.
  • Task space T
    purpose: Provides input-dependent, ReLU-rectified modulation of the skill space to achieve localized responses.
    No independent handle; the qualitative entropy analysis in Fig. 6 is post-hoc visualization, not a falsifiable prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning." pith.science (2026). https://pith.science/paper/ZRQQCHJT

@misc{pith2026241112787,
  author       = {Pith},
  title        = {Pith review of: From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZRQQCHJT}},
  note         = {Machine review of arXiv:2411.12787}
}
abstract

Efficient Visual Instruction Fine-Tuning (EVIT) seeks to adapt Multimodal Large Language Models (MLLMs) to downstream tasks with minimal computational overhead. However, as task diversity and complexity increase, EVIT faces significant challenges in resolving data conflicts. To address this limitation, we propose the Dual Low-Rank Adaptation (Dual-LoRA), a holistic-to-local framework that enhances the adapter's capacity to address data conflict through dual structural optimization. Specifically, we utilize two subspaces: a skill space for stable, holistic knowledge retention, and a rank-rectified task space that locally activates the holistic knowledge. Additionally, we introduce Visual Cue Enhancement (VCE), a multi-level local feature aggregation module designed to enrich the vision-language projection with local details. Our approach is both memory- and time-efficient, requiring only 1.16$\times$ the inference time of the standard LoRA method (with injection into the query and value projection layers), and just 73\% of the inference time of a 4-expert LoRA-MoE. Extensive experiments on various downstream tasks and general MLLM benchmarks validate the effectiveness of our proposed methods.

Figures

Figures reproduced from arXiv: 2411.12787 by the authors.

Figure 1
Figure 1. The motivation behind the proposed approach. (a) Hu [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison between the proposed Dual-LoRA and existing methods. (a) Mainstream MLLMs, [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of our two key components: Visual Cue Enhancement (VCE) module enhances high-level anchor features by aggregating local information from multi-level feature maps. The Dual Low-Rank Adaptation (Dual-LoRA) module projects the input feature into two low-rank subspaces: one for stable holistic domain knowledge (skill space) learning and the other for instruction condition (task space) learning. Dual-LoRA module… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Feature Map Visualization of Enhanced Visual Cue with [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Panels (a), (b), and (c) represent feature visualizations [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Performance vs. Trainable LLM Adapter Parameters for Vanilla LoRA and Our Proposed Dual-LoRA on the UniFood Dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Relative Inference Time Efficiency of Our Dual-LoRA [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 9 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  2. [2]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. Layer normalization. arXiv preprint arXiv:1607.06450,

    arXiv

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 1, 2, 4

    arXiv 2023

  4. [4]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 8

    arXiv 2023

  5. [5]

    Llava-mole: Sparse mixture of lora experts for mitigating data con- flicts in instruction finetuning mllms

    Shaoxiang Chen, Zequn Jie, and Lin Ma. Llava-mole: Sparse mixture of lora experts for mitigating data con- flicts in instruction finetuning mllms. arXiv preprint arXiv:2401.16160, 2024. 2, 3, 4, 5

    arXiv 2024

  6. [6]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6,

    2023

  7. [7]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  8. [8]

    Parameter-efficient fine-tuning of large-scale pre-trained language models.Nature Machine In- telligence, 5(3):220–235, 2023

    Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models.Nature Machine In- telligence, 5(3):220–235, 2023. 1

    2023

Show all 38 references
  1. [9]

    Mousi: Poly-visual-expert vision-language models

    Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, et al. Mousi: Poly-visual-expert vision-language models. arXiv preprint arXiv:2401.17221, 2024. 3

    2024 arXiv

  2. [10]

    Deep sparse rectifier neural networks

    Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the four- teenth international conference on artificial intelligence and statistics, pages 315–323. JMLR Workshop and Conference Proceedings, 2011. 5

    2011

  3. [11]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 1, 3, 6

    2021 arXiv

  4. [12]

    Harder tasks need more experts: Dynamic routing in moe models

    Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder tasks need more experts: Dynamic routing in moe models. arXiv preprint arXiv:2403.07652 ,

  5. [13]

    Rode: Linear rectified mixture of diverse experts for food large multi-modal models

    Pengkun Jiao, Xinlan Wu, Bin Zhu, Jingjing Chen, Chong- Wah Ngo, and Yugang Jiang. Rode: Linear rectified mixture of diverse experts for food large multi-modal models. arXiv preprint arXiv:2407.12730, 2024. 2, 3, 4, 5, 6, 8

    2024 arXiv

  6. [14]

    Unlocking textual and visual wisdom: Open-vocabulary 3d object detection enhanced by comprehensive guidance from text and image

    Pengkun Jiao, Na Zhao, Jingjing Chen, and Yu-Gang Jiang. Unlocking textual and visual wisdom: Open-vocabulary 3d object detection enhanced by comprehensive guidance from text and image. In European Conference on Computer Vi- sion, pages 376–392. Springer, 2024. 1

    2024

  7. [15]

    Don’t deceive me: Mitigating gaslight- ing through attention reallocation in lmms

    Pengkun Jiao, Bin Zhu, Jingjing Chen, Chong-Wah Ngo, and Yu-Gang Jiang. Don’t deceive me: Mitigating gaslight- ing through attention reallocation in lmms. arXiv preprint arXiv:2504.09456, 2025

    2025 arXiv

  8. [16]

    Lumen: Unleashing versa- tile vision-centric capabilities of large multimodal models

    Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Lumen: Unleashing versa- tile vision-centric capabilities of large multimodal models. Advances in Neural Information Processing Systems , 37: 81461–81488, 2025. 1

    2025

  9. [17]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. Advances in neural information processing systems , 25, 2012. 5

    2012

  10. [18]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 3

    2023

  11. [19]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74–81, 2004. 5

    2004

  12. [20]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 1, 2, 3, 4, 6

    2023

  13. [21]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 8

    2024

  14. [22]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 1

    2023 arXiv

  15. [23]

    Adamole: Fine-tuning large lan- guage models with adaptive mixture of low-rank adaptation experts

    Zefang Liu and Jiahua Luo. Adamole: Fine-tuning large lan- guage models with adaptive mixture of low-rank adaptation experts. arXiv preprint arXiv:2405.00361, 2024. 2

    2024 arXiv

  16. [24]

    Deepseek-vl: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525,

    arXiv

  17. [25]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 3

    2023 arXiv

  18. [26]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In Pro- ceedings of the IEEE international conference on computer ...

    2015

  19. [27]

    A call for clarity in reporting bleu scores

    Matt Post. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771, 2018. 5

    2018 arXiv

  20. [28]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  21. [29]

    Scienceqa: A novel resource for question answering on scholarly articles

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289–301, 2022. 5

    2022

  22. [30]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 3

    2024 arXiv

  23. [31]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 3

    2024

  24. [32]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1, 3

    2023 arXiv

  25. [33]

    Mixture of lora experts

    Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. arXiv preprint arXiv:2404.13628, 2024. 2, 3, 4, 6, 8

    2024 arXiv

  26. [34]

    Vision transformer with deformable attention

    Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 4794–4803, 2022. 4

    2022

  27. [35]

    Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images

    Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403.11703, 2024. 3

    2024 arXiv

  28. [36]

    Foodlmm: A versatile food assistant using large multi-modal model

    Yuehao Yin, Huiyan Qi, Bin Zhu, Jingjing Chen, Yu-Gang Jiang, and Chong-Wah Ngo. Foodlmm: A versatile food assistant using large multi-modal model. arXiv preprint arXiv:2312.14991, 2023. 5

    2023 arXiv

  29. [37]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 3

    2023 arXiv

  30. [38]

    Deformable detr: Deformable trans- formers for end-to-end object detection

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable trans- formers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020. 4

    2010 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.