Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation

Fanshuo Zeng; Jiawei Yao; Xiaofeng Zhang; Yihao Quan; Zheng Hui

REVIEW 5 major objections 6 minor 55 references

Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that text-relevant image tokens are the ones that carry reasoning in vision-language models, and that masking out the rest with a text-similarity filter improves accuracy on complex visual questions.

desk verdict New idea (text-conditioned token pruning) with a plausible mechanism, but test-set-tuned K and missing baselines keep the main effect from being fully trusted. read the letter →

arxiv 2412.09817 v1 pith:OEZYPUPO submitted 2024-12-13 cs.CV cs.CL

Xiaofeng Zhang , Fanshuo Zeng , Yihao Quan , Zheng Hui , Jiawei Yao This is my paper · ORCID

classification cs.CVcs.CL

keywords multimodallargelanguagemodelsimagetokenreductionvisualreasoningattentioninterpretabilitycosinesimilarityScienceQAchain-of-thoughtpruning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that when a vision-language model answers a complex reasoning question, the image patches that matter are exactly those whose embeddings sit closest to the text embeddings, and that deliberately ignoring the rest improves both accuracy and speed. It proposes Simignore, which computes the cosine similarity between image-token embeddings and text-token embeddings, keeps the K most similar image tokens, and zeroes out the attention mask of the rest. On ScienceQA image questions, applying Simignore to LLaVA1.5-7B raises accuracy from 65.15 to 68.02 percent, LLaVA1.5-13B from 72.09 to 73.23 percent, and Mipha-3B from 70.40 to 70.85 percent, while reducing inference time from 303 to 279 seconds at K=124 ignored tokens. Ablations show that ignoring high-similarity tokens hurts accuracy while ignoring low-similarity tokens helps, and that cosine similarity beats Euclidean and Manhattan distances for token selection.

What carries the argument

The load-bearing object is the cosine similarity matrix $S(i,j)$ between normalized image-token embeddings and normalized text-token embeddings (Eq. 11). Flattening and argsorting $S$ yields the indices of the $K$ image tokens with the strongest text affinity; the attention mask for every other image token is set to zero. The paper motivates this selection with an attention-convergence observation: overlaying decoder attention scores on the input image shows concentrated attention on text-related objects, such as a mushroom or copepod mentioned in the options.

What would settle it

On a fixed set of ScienceQA image questions, compare Simignore's selected K tokens with the K tokens whose individual masking actually changes the model's predicted answer. If the two sets do not overlap substantially, the accuracy gain cannot be attributed to removing causally relevant tokens.

Watch

Extended reading notes

Core claim

The paper's central claim is that in large vision-language models such as LLaVA1.5, image tokens semantically related to the prompt receive higher attention scores in the LLM decoding layers, a phenomenon the paper calls information-flow convergence, while unrelated tokens receive very little attention. Simignore exploits this by treating the top-K image tokens under normalized cosine similarity to the text as the essential visual context and masking out the rest before decoding. The authors report that this masking is not only a speed optimization: it improves complex reasoning accuracy across three backbones, and ablations show the effect depends on the similarity ranking, since dropping important tokens degrades accuracy and cosine similarity outperforms distance-based alternatives.

Load-bearing premise

The load-bearing premise is that text similarity is a faithful proxy for causal importance: the image tokens with the highest cosine similarity to the prompt are the ones that must stay for the model to answer correctly.

Editorial extensions

If this is right

At K=124 of 576 image tokens ignored, ScienceQA image accuracy rises from 65.15 to 68.02 for LLaVA1.5-7B, from 72.09 to 73.23 for LLaVA1.5-13B, and from 70.40 to 70.85 for Mipha-3B.
Runtime on the LLaVA1.5-7B evaluation drops monotonically as more tokens are ignored, from 303 seconds at baseline to 279 seconds at the best-accuracy setting.
Ablations indicate that the benefit comes from removing low-similarity tokens: ignoring the most important tokens drops accuracy to 61.73, while ignoring unimportant tokens gives 68.02, and cosine similarity outperforms Euclidean and Manhattan distance.
A case study suggests that low-similarity image tokens can form a distinct embedding cluster whose removal can flip a wrong answer to the correct one.
Simignore requires no fine-tuning or training, operating purely through attention masks at inference time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because Simignore needs no training, its selected tokens could double as a zero-shot saliency map for diagnosing which image region a model used, though the paper does not evaluate this use.
The random-ignoring ablation (65.12 percent at K=124) is close to baseline, so the gain is not merely length regularization; an independent test would be to compare Simignore against a mask that removes tokens with the lowest attention scores rather than lowest text similarity.
An adaptive K, chosen per question from the similarity distribution or the model's confidence, could improve the accuracy-runtime frontier beyond the fixed K=124 setting the paper reports.
The method's reliance on text-embedding similarity may transfer poorly to questions where the relevant visual detail is not lexicalized in the prompt; benchmarks with implicit visual reasoning would test that boundary.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing you should know: this paper proposes a genuinely different token-reduction rule—mask image tokens whose embeddings have low cosine similarity to the text prompt—and reports consistent accuracy gains on ScienceQA across three LVLMs (e.g., LLaVA-1.5-7B from 65.15 to 68.02). That's a real departure from FastV (attention-based) and LLaVA-PruMerge (CLS-based), and the idea is simple enough to be deployment-friendly. The ablations are directionally coherent: ignoring low-similarity tokens helps more than ignoring random or high-similarity ones, and the paper includes a nice case study with clustering.

The soft spots are three, and the first two are load-bearing. One: K (the number of ignored tokens) is chosen on the test set. Table 2 sweeps ten values and picks the best accuracy on ScienceQA; that inflates the reported gain. There's no validation split or principled criterion. Two: no error bars or multiple seeds, so we don't know if 68.02 vs. 65.15 is noise. Three: the paper cites FastV and LLaVA-PruMerge but never compares against them. Without that, we can't tell whether the gain comes from text-conditioning or just from dropping a fixed number of tokens.

The stress-test note worries that the causal story is unverified—the paper never shows that the masked tokens are exactly the ones whose removal flips the answer. That's fair as a secondary concern, but I don't think it's fatal: the paper's primary claim is that the method improves accuracy, and the ablations at least support the direction of the effect. The mechanism explanation ("spy tokens") is speculative, but the paper flags it as conjecture.

The writing is rough in places (typos, some unclear notation), and the evaluation is single-benchmark. Still, the core idea is new and the experiments, while under-controlled, are clearly described. For anyone working on token reduction or LVLM efficiency, this is a useful data point. This is exactly the kind of paper that should get a serious referee: the method is worth testing, but it needs a validation split, seeds, and baseline comparisons before the numbers can be trusted. I'd send it to review with a request for major revision.

Referee Report

5 major / 6 minor

Summary. The paper proposes Simignore, a training-free image token reduction method for large vision-language models (LVLMs). During inference, it computes cosine similarity between image token embeddings and text token embeddings, then masks out the K image tokens with the lowest similarity to the text, aiming to improve complex reasoning accuracy by removing irrelevant or unimportant visual information. The method is evaluated on the ScienceQA-image benchmark with LLaVA1.5-7B, LLaVA1.5-13B, and Mipha-3B, reporting accuracy gains (e.g., LLaVA1.5-7B from 65.15% to 68.02%) and reduced runtime. The paper also includes attention visualization, a k-means clustering case study, and ablations on the number of ignored tokens, similarity algorithms, and importance levels.

Significance. If the reported gains are robust, Simignore offers an attractive zero-training plug-in for improving LVLMs on visual reasoning tasks, and the information-flow observation provides a useful interpretability perspective. The method is conceptually simple and has a single hyperparameter K. The paper promises public code, which would aid reproducibility. However, the significance is currently tempered by several issues: evaluation is limited to a single benchmark, no comparison is made with prior token-reduction methods, K is selected on the test set, and the causal link between similarity-based token removal and accuracy improvement is not directly verified. The central claim is plausible but not yet established; the paper reads as a promising workshop-level contribution rather than a fully validated research result.

major comments (5)

[Ablation Study, Table 2] The hyperparameter K=124 is selected as the best value from a ten-point scan over the ScienceQA-image test set, and the reported improvement (65.15% to 68.02%) is the best-case result from that scan. Because no separate validation split is used for hyperparameter selection, the reported accuracy is an optimistic estimate and does not reflect the performance of the method with a fixed K chosen a priori. Please report results on a held-out validation set for selecting K, or provide a clear selection rule and discuss how the reported gain changes under that rule.
[Experiment, Table 1] The paper cites FastV and LLaVA-PruMerge as prior token-reduction methods in the Related Work, but Table 1 only compares against whole-model baselines and does not include any token-reduction baseline. Since the central contribution is a token-reduction method, the claim of effectiveness requires direct comparison with the most relevant prior work under the same evaluation protocol. Please add these comparisons, as their absence leaves the relative improvement unclear.
[Method / Ablation Study, Table 3] The causal mechanism is not validated. The paper claims Simignore works by ignoring irrelevant image tokens, but the evidence is correlational: Table 3 contrasts ignoring tokens with high versus low similarity, using the same similarity score for selection and evaluation. This does not establish that the selected tokens are the ones whose removal is causally responsible for better answers. A direct test would be to check whether masking the top-K similar tokens changes the model's answer more often than masking K random tokens or K low-attention tokens, and whether the tokens that actually flip the answer coincide with the low-similarity tokens. As it stands, the accuracy gain could arise from a different mechanism such as noise reduction or length regularization.
[Method, Eqs. (11)-(15)] The token-selection procedure is under-specified and potentially flawed. The flatten-and-argsort in Eqs. (13)-(14) selects the top-K image-text pairs, not the top-K distinct image tokens; an image token can appear multiple times among the top-K pairs, so the final attention mask may ignore fewer than K distinct tokens. Additionally, the paper does not specify which embeddings are used for the similarity computation (e.g., the LLM input embeddings after the projector, or the vision encoder outputs before projection) and how the feature-alignment function F⅁ is implemented for LLaVA. Please clarify the selection procedure so that exactly K distinct image tokens are ignored, and specify the exact embedding sources used.
[Experiment, Tables 1-4] All accuracy numbers are reported as single values without variance, error bars, or significance tests. The gains are small (e.g., 0.45 points for Mipha-3B and 1.14 points for LLaVA1.5-13B) and could be within run-to-run variation. Please report the mean and standard deviation over multiple runs (at least three random seeds) and, if feasible, a paired significance test for the main comparisons.

minor comments (6)

[Title] The title uses 'Complex Reason'; it should be 'Complex Reasoning'.
[Method, Eq. (3)] The left-hand side λj_img is indexed by j, but the right-hand side sums over j ∈ I, so the index is inconsistent; the left-hand side should likely be λ_i_img or another index.
[Method, paragraph after Eq. (3)] The word 'unsequenced' appears to be a typo for 'unsqueezed' or 'reshaped' when describing the attention matrix transformation.
[Figure 3 caption] The caption uses 'regularization' where 'normalization' is meant.
[References] The reference list contains an incomplete entry: 'Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models' has '????' instead of a year, and the citation is malformed.
[Discussion and Limitations] The limitations section only mentions future work on adaptively choosing K; it should also acknowledge the test-set selection of K, the lack of comparison with token-reduction baselines, and the correlational nature of the evidence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Simignore's accuracy gains are empirical measurements against base models, and no equation or fitted parameter is reused as its own validation.

full rationale

The paper's derivation chain is observational and empirical rather than definitional. The method (Eqs. 8-15) computes cosine similarity between image and text embeddings, selects top-K similar image tokens, and masks the rest; the reported gains (e.g., LLaVA1.5-7B 65.15 to 68.02) are measured on ScienceQA accuracy, which is external to the similarity computation. The 'importance' ablation in Table 3 labels tokens by the same similarity score used for selection, but the conclusion that low-similarity tokens are harmful is an empirical accuracy comparison, not a consequence of the definition. The information-flow motivation cites prior work by the same group (Zhang et al. 2024a,b), but Fig. 2 provides direct attention-score evidence in this paper, so the self-citation is supplementary rather than load-bearing. The main validity concern is that K=124 is chosen from a ten-point sweep on the same ScienceQA-image set (Table 2), and the Discussion and Limitations section acknowledges that adaptive K selection is future work; this makes the headline number a post-selected outcome, but this is an overfitting/generalization limitation, not a circularity, because the reported accuracy is not a fitted quantity renamed as a prediction. No uniqueness theorem, ansatz-by-citation, or renaming of a known result is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The method rests on a small number of unverified domain assumptions about what cosine similarity and attention reveal, plus one free parameter K selected on the test set. No new physical or architectural entities are introduced; 'information flow convergence' is a descriptive term for an attention pattern, not a separate postulated entity.

free parameters (1)

K (number of ignored image tokens) = 124 out of 576
Chosen as the accuracy-maximizing value in an ablation scan over ten values on the ScienceQA-image test split (Table 2); no held-out validation split is reported.

assumptions (4)

domain assumption Cosine similarity between normalized image and text embeddings identifies image tokens that are relevant to the correct answer.
The whole selection mechanism in Eq. 9-15 depends on this. The paper supports it with qualitative attention maps (Fig. 2) and one cluster case study (Fig. 7), not with a quantitative correlation.
domain assumption Image tokens with low similarity to the text can be safely masked without removing answer-critical information.
Central to the pruning claim. Tested only indirectly through accuracy differences on ScienceQA; no per-question verification that masked tokens never contain the answer.
domain assumption Attention scores of the output token on image tokens measure information flow convergence.
Defined informally in 'Influence Rate of Image Token on Output Token' (Eq. 1-3); no formal metric or causal analysis is provided.
domain assumption ScienceQA image split is representative of multimodal complex reasoning.
The paper states ScienceQA is 'the only dataset available for complex reasoning' and uses only the image questions, so generalization to other reasoning benchmarks is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation." pith.science (2026). https://pith.science/paper/OEZYPUPO

@misc{pith2026241209817,
  author       = {Pith},
  title        = {Pith review of: Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OEZYPUPO}},
  note         = {Machine review of arXiv:2412.09817}
}

read the original abstract

Multimodal large language models have experienced rapid growth, and numerous different models have emerged. The interpretability of LVLMs remains an under-explored area. Especially when faced with more complex tasks such as chain-of-thought reasoning, its internal mechanisms still resemble a black box that is difficult to decipher. By studying the interaction and information flow between images and text, we noticed that in models such as LLaVA1.5, image tokens that are semantically related to text are more likely to have information flow convergence in the LLM decoding layer, and these image tokens receive higher attention scores. However, those image tokens that are less relevant to the text do not have information flow convergence, and they only get very small attention scores. To efficiently utilize the image information, we propose a new image token reduction method, Simignore, which aims to improve the complex reasoning ability of LVLMs by computing the similarity between image and text embeddings and ignoring image tokens that are irrelevant and unimportant to the text. Through extensive experiments, we demonstrate the effectiveness of our method for complex reasoning tasks. The paper's source code can be accessed from \url{https://github.com/FanshuoZeng/Simignore}.

Figures

Figures reproduced from arXiv: 2412.09817 by the authors.

**Figure 1.** The simplified structure of our method. extra image tokens lead to an increase in computation, and some studies (Chu et al. 2023a; Yuan, Li, and Sun 2023) have reduced the inference cost by using smaller LLMs with fewer parameters, however, these approaches also lead to a decrease in the inference power of LLMs (Chu et al. 2024). Therefore, a better approach is to reduce the computational cost by decreasing the leng… view at source ↗

**Figure 2.** We find that the information flow converges in re [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** The holistic framework for an approach to enhance complex reasoning in multimodal large language models through [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Influence rate of attention score about image to [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Distribution of image token and prompt token in [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: This is a question closely related to the information [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: Fig.(a) represents the distribution of image tokens. Where the red dots indicate the distribution position of tokens [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 17 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

work page 2023
[4]

Chen, K.; Zhang, Z.; Zeng, W.; Zhang, R.; Zhu, F.; and Zhao, R. 2023. Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic. arXiv preprint arXiv:2306.15195

arXiv 2023
[5]

Chen, L.; Zhao, H.; Liu, T.; Bai, S.; Lin, J.; Zhou, C.; and Chang, B. 2024. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. arXiv preprint arXiv:2403.06764

arXiv 2024
[6]

Chen, X.; Pun, C.-M.; and Wang, S. 2024. Medprompt: Cross-modal prompting for multi-task medical image translation. In PRCV, 61--75

work page 2024
[8]

Chu, X.; Qiao, L.; Lin, X.; Xu, S.; Yang, Y.; Hu, Y.; Wei, F.; Zhang, X.; Zhang, B.; Wei, X.; et al. 2023 b . Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886

arXiv 2023
[9]

Chu, X.; Qiao, L.; Zhang, X.; Xu, S.; Wei, F.; Yang, Y.; Sun, X.; Hu, Y.; Lin, X.; Zhang, B.; et al. 2024. MobileVLM V2: Faster and Stronger Baseline for Vision Language Model. arXiv preprint arXiv:2402.03766

arXiv 2024

Show all 55 references

[10]

Dai, W.; Li, J.; Li, D.; Tiong, A. M. H.; Zhao, J.; Wang, W.; Li, B.; Fung, P. N.; and Hoi, S. 2024. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36

2024
[11]

Diao, H.; Zhang, Y.; Ma, L.; and Lu, H. 2021. Similarity reasoning and filtration for image-text matching. In Proceedings of the AAAI conference on artificial intelligence, volume 35, 1218--1226

2021
[12]

K.-p.; Qian, T.; and Wang, S

Dong, Y.; Chen, X.; Shen, Y.; Ng, M. K.-p.; Qian, T.; and Wang, S. 2025. Multi-modal Mood Reader: Pre-trained Model Empowers Cross-Subject Emotion Recognition. In NCAA, 178--192

2025
[13]

J.; Kiros, J

Faghri, F.; Fleet, D. J.; Kiros, J. R.; and Fidler, S. 2017. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612

2017 arXiv
[14]

S.; Shlens, J.; Bengio, S.; Dean, J.; Ranzato, M.; and Mikolov, T

Frome, A.; Corrado, G. S.; Shlens, J.; Bengio, S.; Dean, J.; Ranzato, M.; and Mikolov, T. 2013. Devise: A deep visual-semantic embedding model. Advances in neural information processing systems, 26

2013
[15]

Hu, M.; Xia, P.; Wang, L.; Yan, S.; Tang, F.; Xu, Z.; Luo, Y.; Song, K.; Leitner, J.; Cheng, X.; et al. 2025. Ophnet: A large-scale video benchmark for ophthalmic surgical workflow understanding. In ECCV

2025
[16]

Hu, M.; Yuan, K.; Shen, Y.; Tang, F.; Xu, X.; Zhou, L.; Li, W.; Chen, Y.; Xu, Z.; Peng, Z.; et al. 2024. OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretraining. arXiv preprint arXiv:2411.15421

2024 arXiv
[17]

Huo, Y.; Huang, G.; Cheng, L.; He, J.; Chen, X.; Yuan, X.; Zhong, G.; and Pun, C.-M. 2024. IMAN: An Adaptive Network for Robust NPC Mortality Prediction with Missing Modalities. In BIBM

2024
[18]

Kiros, R.; Salakhutdinov, R.; and Zemel, R. S. 2014. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539

2014 arXiv
[19]

Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023 a . Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 19730--19742. PMLR

2023
[20]

Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, 12888--12900. PMLR

2022
[21]

Li, L.; Yin, Y.; Li, S.; Chen, L.; Wang, P.; Ren, S.; Li, M.; Yang, Y.; Xu, J.; Sun, X.; et al. 2023 b . A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning. arXiv preprint arXiv:2306.04387

2023 arXiv
[22]

Li, M.; Sun, H.; Lei, Y.; Zhang, X.; Dong, Y.; Zhou, Y.; Li, Z.; and Chen, X. 2024. High-Fidelity Document Stain Removal via A Large-Scale Real-World Dataset and A Memory-Augmented Transformer. In WACV

2024
[23]

Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2023. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744

2023 arXiv
[24]

Liu, Y.; Zhou, L.; Bai, X.; Huang, Y.; Gu, L.; Zhou, J.; and Harada, T. 2021. Goal-oriented gaze estimation for zero-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 3794--3803

2021
[25]

Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.-C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507--2521

2022
[26]

W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PMLR

2021
[27]

J.; and Yan, Y

Shang, Y.; Cai, M.; Xu, B.; Lee, Y. J.; and Yan, Y. 2024. LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models. arXiv preprint arXiv:2403.15388

2024
[28]

Shen, F.; Jiang, X.; He, X.; Ye, H.; Wang, C.; Du, X.; Li, Z.; and Tang, J. 2024 a . Imagdressing-v1: Customizable virtual dressing. arXiv preprint arXiv:2407.12705

2024 arXiv
[29]

Shen, F.; and Tang, J. 2024. IMAGPose: A Unified Conditional Framework for Pose-Guided Person Generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

2024
[30]

Shen, F.; Ye, H.; Liu, S.; Zhang, J.; Wang, C.; Han, X.; and Yang, W. 2024 b . Boosting consistency in story visualization with rich-contextual conditional diffusion models. arXiv preprint arXiv:2407.02482

2024 arXiv
[31]

???? Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models

Shen, F.; Ye, H.; Zhang, J.; Wang, C.; Han, X.; and Wei, Y. ???? Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models. In The Twelfth International Conference on Learning Representations
[32]

Sun, H.-L.; Zhou, D.-W.; Li, Y.; Lu, S.; Yi, C.; Chen, Q.-G.; Xu, Z.; Luo, W.; Zhang, K.; Zhan, D.-C.; et al. 2024. Parrot: Multilingual Visual Instruction Tuning. arXiv preprint arXiv:2406.02539

2024 arXiv
[33]

Sun, H.-L.; Zhou, D.-W.; Ye, H.-J.; and Zhan, D.-C. 2023. Pilot: A pre-trained model-based continual learning toolbox. arXiv preprint arXiv:2309.07117

2023 arXiv
[34]

Sun, H.-L.; Zhou, D.-W.; Zhao, H.; Gan, L.; Zhan, D.-C.; and Ye, H.-J. 2025. MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learning. In AAAI

2025
[35]

Tang, J.; Lin, C.; Zhao, Z.; Wei, S.; Wu, B.; Liu, Q.; Feng, H.; Li, Y.; Wang, S.; Liao, L.; et al. 2024 a . TextSquare: Scaling up Text-Centric Visual Instruction Tuning. arXiv preprint arXiv:2404.12803

2024 arXiv
[36]

Tang, J.; Liu, Q.; Ye, Y.; Lu, J.; Wei, S.; Lin, C.; Li, W.; Mahmood, M. F. F. B.; Feng, H.; Zhao, Z.; Wang, Y.; Liu, Y.; Liu, H.; Bai, X.; and Huang, C. 2024 b . MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering. arXiv:2405.11985

2024 arXiv
[37]

Wang, A.-L.; Shan, B.; Shi, W.; Lin, K.-Y.; Fei, X.; Tang, G.; Liao, L.; Tang, J.; Huang, C.; and Zheng, W.-S. 2025. ParGo: Bridging Vision-Language with Partial and Global Views

2025
[38]

Wang, C.; Pan, J.; Lin, W.; Dong, J.; Wang, W.; and Wu, X.-M. 2024. Selfpromer: Self-prompt dehazing transformers with depth-consistency. In AAAI, volume 38, 5327--5335

2024
[39]

Wang, C.; Pan, J.; Wang, W.; Dong, J.; Wang, M.; Ju, Y.; and Chen, J. 2023 a . PromptRestorer: A Prompting Image Restoration Method with Degradation Perception. In NeurIPS

2023
[40]

Wang, J.; Huang, Q.; Tang, F.; Meng, J.; Su, J.; and Song, S. 2022. Stepwise feature fusion: Local guides global. In MICCAI. Springer

2022
[41]

Wang, W.; Lv, Q.; Yu, W.; Hong, W.; Qi, J.; Wang, Y.; Ji, J.; Yang, Z.; Zhao, L.; Song, X.; et al. 2023 b . Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079

2023 arXiv
[42]

Wei, J.; and Zhang, X. 2024. Dopra: Decoding over-accumulation penalization and re-allocation in specific weighting layer. In Proceedings of the 32nd ACM International Conference on Multimedia, 7065--7074

2024
[43]

Xu, Z.; Tang, F.; Chen, Z.; Zhou, Z.; Wu, W.; Yang, Y.; Liang, Y.; Jiang, J.; Cai, X.; and Su, J. 2024. Polyp-Mamba: Polyp Segmentation with Visual Mamba. In MICCAI. Springer

2024
[44]

Yang, J.; Li, C.; Zhang, P.; Xiao, B.; Liu, C.; Yuan, L.; and Gao, J. 2022. Unified contrastive learning in image-text-label space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19163--19173

2022
[45]

Yuan, X.; Shen, C.; Yan, S.; Zhang, X.; Xie, L.; Wang, W.; Guan, R.; Wang, Y.; and Ye, J. 2024. Instance-adaptive Zero-shot Chain-of-Thought Prompting. arXiv preprint arXiv:2409.20441

2024 arXiv
[46]

Yuan, Z.; Li, Z.; and Sun, L. 2023. Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint arXiv:2312.16862

2023 arXiv
[47]

Zeng, R.; Ma, W.; Wu, X.; Liu, W.; and Liu, J. 2024. Image--Text Cross-Modal Retrieval with Instance Contrastive Embedding. Electronics, 13(2): 300

2024
[48]

Zhai, X.; Mustafa, B.; Kolesnikov, A.; and Beyer, L. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 11975--11986

2023
[49]

Zhang, X.; Quan, Y.; Gu, C.; Shen, C.; Yuan, X.; Yan, S.; Cheng, H.; Wu, K.; and Ye, J. 2024 a . Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs. arXiv preprint arXiv:2411.09968

2024 arXiv
[50]

Zhang, X.; Shen, C.; Yuan, X.; Yan, S.; Xie, L.; Wang, W.; Gu, C.; Tang, H.; and Ye, J. 2024 b . From Redundancy to Relevance: Enhancing Explainability in Multimodal Large Language Models. arXiv preprint arXiv:2406.06579

2024 arXiv
[51]

Zhao, H.; Cai, Z.; Si, S.; Ma, X.; An, K.; Chen, L.; Liu, Z.; Wang, S.; Han, W.; and Chang, B. 2023. Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv preprint arXiv:2309.07915

2023 arXiv
[52]

Zhao, Z.; Tang, J.; Lin, C.; Wu, B.; Huang, C.; Liu, H.; Tan, X.; Zhang, Z.; and Xie, Y. 2024 a . Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15567--15576

2024
[53]

Zhao, Z.; Tang, J.; Wu, B.; Lin, C.; Wei, S.; Liu, H.; Tan, X.; Zhang, Z.; Huang, C.; and Xie, Y. 2024 b . Harmonizing Visual Text Comprehension and Generation

2024
[54]

Zheng, G.; Yang, B.; Tang, J.; Zhou, H.-Y.; and Yang, S. 2023. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36: 5168--5191

2023
[55]

Zhou, D.-W.; Sun, H.-L.; Ye, H.-J.; and Zhan, D.-C. 2024. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 23554--23564

2024
[56]

Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

work page 2023

[4] [4]

Chen, K.; Zhang, Z.; Zeng, W.; Zhang, R.; Zhu, F.; and Zhao, R. 2023. Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic. arXiv preprint arXiv:2306.15195

arXiv 2023

[5] [5]

Chen, L.; Zhao, H.; Liu, T.; Bai, S.; Lin, J.; Zhou, C.; and Chang, B. 2024. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. arXiv preprint arXiv:2403.06764

arXiv 2024

[6] [6]

Chen, X.; Pun, C.-M.; and Wang, S. 2024. Medprompt: Cross-modal prompting for multi-task medical image translation. In PRCV, 61--75

work page 2024

[7] [8]

Chu, X.; Qiao, L.; Lin, X.; Xu, S.; Yang, Y.; Hu, Y.; Wei, F.; Zhang, X.; Zhang, B.; Wei, X.; et al. 2023 b . Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886

arXiv 2023

[8] [9]

Chu, X.; Qiao, L.; Zhang, X.; Xu, S.; Wei, F.; Yang, Y.; Sun, X.; Hu, Y.; Lin, X.; Zhang, B.; et al. 2024. MobileVLM V2: Faster and Stronger Baseline for Vision Language Model. arXiv preprint arXiv:2402.03766

arXiv 2024