Dynamic Modality Scheduling for Multimodal Large Models via Confidence, Uncertainty, and Semantic Consistency

Anika Rao; Hana Satou; Hiroshi Tanaka; Michael Johnson; Sofia Garc\'ia

REVIEW 4 major objections 5 minor 18 references

Dynamic Modality Scheduling for Multimodal Large Models via Confidence, Uncertainty, and Semantic Consistency

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Per-sample modality weights, set by confidence, uncertainty, and semantic alignment, lift VQA accuracy by 2.3 points and halve noise-induced degradation compared with static fusion.

desk verdict Load-bearing integration gap plus a vacuous consistency loss sink an otherwise plausible adaptive-fusion idea. read the letter →

arxiv 2506.12724 v1 pith:X7UIQARN submitted 2025-06-15 cs.CV

Hiroshi Tanaka , Anika Rao , Hana Satou , Michael Johnson , Sofia García This is my paper

classification cs.CV

keywords dynamicmodalityschedulingmultimodalfusionconfidenceestimationMonteCarlodropoutuncertaintyquantificationsemanticconsistencyvisualquestionansweringrobustness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dynamic Modality Scheduling (DMS) is a per-sample reweighting rule: for each input stream, the paper computes a confidence score from predictive entropy, an epistemic-uncertainty score from Monte Carlo dropout, and a semantic-consistency score from cosine similarity to the other modalities, then sets softmax weights $\omega_m \propto \exp(\alpha c_m - \beta u_m + \gamma s_m)$ and forms $h=\sum_m \omega_m f^{(m)}$. The paper claims that inserting this scheduler at the encoder-output level improves vision-language tasks: VQA accuracy rises by 2.3 points, image-text retrieval Recall@1 rises by 3.1 points, and degradation under image or text noise is roughly halved relative to static fusion. If the claim holds, instance-level soft weighting is a lightweight way to make multimodal large models robust to noisy or missing modalities, and no retraining of the underlying backbones is required.

What carries the argument

The load-bearing object is the weighted-sum fusion rule $h=\sum_m \omega_m f^{(m)}$ where $f^{(m)}$ is the encoder output for modality $m$ and $\omega_m$ comes from a softmax over $\alpha c_m - \beta u_m + \gamma s_m$. Here $c_m$ is $1-H(p^{(m)})$ with $H$ the entropy of the modality's softmax prediction, $u_m$ is the mean variance across $K$ classes and $T$ Monte Carlo dropout forward passes, and $s_m$ is the cosine similarity between $f^{(m)}$ and the average of the other modalities. The same weights feed a Modality Weight Consistency Loss $\mathcal{L}_{\text{mwcl}}=\sum_m \omega_m \lVert h - f^{(m)}\rVert_2^2$, which the paper derives as variance-reducing regularization that keeps the fusion near each unimodal embedding in proportion to its assigned weight. This rule is the entire mechanism: it converts three per-modality quality estimates into one normalized weight vector, and the paper's theoretical sections give a Lipschitz-based bound on deviation from oracle static fusion and a Rademacher-complexity generalization bound for the weighted hypothesis class.

What would settle it

Run the exact pipeline on a standard BLIP-2 checkpoint without removing the Q-Former: if inserting DMS weights before the multimodal transformer does not reproduce roughly 74.4 VQA accuracy and the reduced noise degradation, the claim fails. A sharper test: take a sample with a clean image and heavily corrupted text, and measure $\omega_{\text{image}}$ and $\omega_{\text{text}}$ as corruption increases; the mechanism predicts a monotone shift toward the image, while weights that stay near uniform would show DMS contributes no dynamic behavior.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a soft, data-dependent weighting of pretrained unimodal embeddings, driven by three complementary signals, outperforms static blending and attention-based fusion on both clean and corrupted inputs. DMS posits that a modality which is confident (low predictive entropy), stable under stochastic dropout passes (low Monte Carlo dropout variance), and semantically aligned with the other modality should dominate the fused representation, while a noisy or contradictory modality should be downweighted. Experiments on VQA v2, MSCOCO captioning, and Flickr30K retrieval report DMS at 74.4 VQA accuracy versus 72.1 for static BLIP-2, and 61.5 versus 58.4 Recall@1; under Gaussian blur or text corruption, DMS retains 88.5% of clean VQA accuracy while the static baseline retains 78.6%. The authors present this as evidence that dynamic scheduling, rather than a change in fusion architecture, carries the robustness gain.

Load-bearing premise

The load-bearing premise is that an MLLM's fusion reduces to a weighted sum $h=\sum_m \omega_m f^{(m)}$ of unimodal encoder outputs, but BLIP-2 fuses through Q-Former cross-attention and LLaVA injects image tokens into the LLM, and the paper never shows how those architectures become such a weighted sum.

Editorial extensions

If this is right

On standard benchmarks, DMS reports VQA accuracy 74.4 versus 72.1 for static BLIP-2, and Recall@1 61.5 versus 58.4, so instance-level weighting alone yields these gains.
Under image noise, DMS keeps 88.5% of clean VQA accuracy versus 78.6% for static fusion; under text noise degradation drops from -18.0% to -9.4%, implying the scheduler detects and discounts corrupted inputs.
Ablations attribute part of the gain to each factor: removing semantic alignment costs the most on retrieval (R@1 falls from 61.5 to 57.7), so cross-modal consistency carries a large part of the retrieval benefit.
The same scheduler transfers to LLaVA, raising factuality from 72.4 to 78.9 in open-ended question answering, supporting the paper's model-agnostic claim.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that the confidence term $c_m$ presupposes a per-modality classification head with a softmax over $K$ classes, which does not exist naturally for generative tasks such as captioning; making DMS fully model-agnostic would require auxiliary heads or proxy confidence from the language model's token probabilities.
Because $\mathcal{L}_{\text{mwcl}}$ pulls the fused $h$ toward each unimodal embedding with weight $\omega_m$, settings where one modality is almost completely dominant could over-regularize the fusion toward that modality and suppress cross-modal complementarity; this is testable by checking whether DMS underperforms static fusion when both modalities are clean but semantically redundant.
An economical extension would replace the $T$ Monte Carlo dropout passes with test-time augmentation or a lightweight variance proxy, cutting per-sample compute cost; the reported robustness gains suggest uncertainty estimates, not the specific estimator, are the important signal.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, the core mechanism—softmax weighting over confidence, MC-dropout uncertainty, and cosine alignment—is a reasonable composition of known tools, and as a generic module for architectures that literally sum unimodal embeddings it could work. Second, the paper as written does not support its headline claims: the consistency loss is a tautology, the theory section contains no proof where one is needed, and the mapping onto BLIP-2/LLaVA is never specified.

What is actually new: the specific combination of the three scheduling signals is not in the cited literature, and the ablation study is sensibly structured. If you ignore the claims about BLIP-2/LLaVA, the formulation is clear and the motivation—instance-level reweighting for noisy inputs—is reasonable.

The soft spots are structural, not cosmetic. Eq. (3.1) defines h as the weighted sum of the f(m), and Eq. (3.4) defines the MWCL as the same weights times squared distances to those f(m). That is the weighted variance of the embeddings around their own weighted mean. It has no adjustable parameters under the stated definition, and Proposition 4.2 merely restates that identity. If the weights were learnable, the trivial minimum would put all mass on one modality. So the regularizer cannot do what the paper claims. Theorem 4.3 is a generic Rademacher bound with no proof and no DMS-specific content; Proposition 4.1 is a simple Lipschitz argument that says little.

The integration claim is the load-bearing flaw. BLIP-2 fuses via Q-Former cross-attention; LLaVA concatenates projected image tokens with text embeddings before the LLM. Neither produces a common vector f^(image) and f^(text) whose weighted sum is h. The paper never says how token sequences are pooled, how p^(m) is computed for images, or how h replaces cross-attention or concatenation. Without that mapping, the 'DMS (Ours)' rows in Tables 1–4 cannot be verified as the described method. This is not a minor omission; it is the only bridge between the mechanism and the results.

The experimental reporting is also below the bar: no code, no seeds, no hyperparameters, no error bars, and the text contains literal placeholders ("Figure??" in 5.4, "MMBT [?]" in 2.4). I am not inclined to trust numbers that cannot be reproduced or even tied to a concrete architecture.

Who gets value out of this? A reader working on adaptive fusion in simple dual-encoder models might find a starting point, but they would have to reimplement everything and fix the loss themselves. The paper does not deserve a serious referee in its current form; I would desk-reject it and invite a resubmission that specifies the architectural mapping, removes or redefines the consistency loss, and provides reproducible experiments.

Referee Report

4 major / 5 minor

Summary. The paper proposes Dynamic Modality Scheduling (DMS), a per-sample weighting scheme for multimodal fusion. Each modality's embedding is weighted by a softmax over confidence (1 minus predictive entropy), negative Monte-Carlo dropout uncertainty, and semantic alignment (cosine similarity among unimodal embeddings). The authors add a Modality Weight Consistency Loss (MWCL) that penalizes distance between the fused representation and modality embeddings, weighted by the same soft weights. They claim DMS is model-agnostic and can be applied to BLIP-2 and LLaVA, and they report improvements on VQA v2, MSCOCO Captioning, and Flickr30K Retrieval, plus robustness gains under synthetic image/text corruption. A theoretical section gives an approximation bound, a MWCL variance identity, and a Rademacher generalization bound.

Significance. If the central claim were established, DMS would offer a simple, interpretable alternative to static fusion for instance-level robustness, with plausible practical value because it requires no architectural overhaul. However, the manuscript's two load-bearing pieces—the claimed applicability to BLIP-2/LLaVA and the theoretical justification—are not supported as written. The experiments are presented without reproducibility details. The paper does include a falsifiable empirical setup (Tables 1-4), but the missing integration mapping and the vacuous Proposition 4.2 prevent this from being a verifiable contribution. The theoretical section is either trivial or generic and does not validate the method's specifics.

major comments (4)

[§3.1, §3.6, §5.1–§5.6] The method, as defined in Eq. (3.1), computes the fused representation as h = Σ_m ω_m f^(m)(x^(m)), a convex combination of unimodal embeddings, and requires per-modality softmax probabilities p^(m) to compute confidence. Neither BLIP-2 nor LLaVA fuses modalities in this way: BLIP-2 uses a Q-Former whose fixed queries cross-attend to image features and then feed the LLM, with text handled inside the LLM; LLaVA projects image patches with an MLP and concatenates the resulting token embeddings with text tokens before the LLM. There is no common f^(image) and f^(text) whose convex combination h is the fusion, and the image encoder has no softmax classification head to supply p^(m). The paper does not specify how token sequences are pooled into f^(m), how p^(m) is computed for images or for the generative language side, or how h replaces cross-attention or concatenation. Without that mapping, the reported 'DMS (Ours)' rows in Tables 1–4 cannot be verified as implementing the described method. This is the central bridge between the proposed mechanism and the claimed results, and it is missing.
[§4.3, Proposition 4.2] Proposition 4.2 states E_m[||h − f^(m)||_2^2] = Var(f^(m)) − ||h − f̄||_2^2, but by the paper's own definition in Eq. (3.1), h = f̄ = Σ_m ω_m f^(m). Therefore h − f̄ = 0 exactly, and the identity reduces to E_m[||h − f^(m)||_2^2] = Var(f^(m)) with no free parameters. The MWCL term is simply the weighted variance of the unimodal embeddings around their weighted mean; it imposes no additional consistency constraint beyond the definition of h. The claimed 'regularization effect' is therefore a restatement of the fusion definition, not a theoretical justification. This is load-bearing for the paper's claim of a novel consistency loss.
[§4.4, Theorem 4.3] Theorem 4.3 is stated without proof, and as written it is a generic Rademacher bound R(f) ≤ R̂(f) + 2L·R_n(F) + sqrt(log(1/δ)/2n) that does not depend on the DMS scheduling signals or on the specific form of ω_m(x). The subsequent claim that 'our dynamic weighting mechanism indirectly controls R_n(F)' is not supported by any calculation, and the bound would apply identically to any hypothesis class over the same feature space. Since the theoretical section is offered as a contribution, the missing proof and the non-specificity of the bound are material.
[§5.1–§5.6] The experimental section reports no error bars, no number of random seeds, no hyperparameter values for α, β, γ, λ, or T, and no code release. Table 1 and Table 3 report differences as small as 0.6–1.3 points with no indication of variance; Table 4 reports 'Helpfulness (score)' without defining the metric or its scale; and Section 5.4 refers to a non-existent 'Figure??'. As a result, the referee cannot assess whether the reported gains are statistically meaningful or reproducible, and the claimed 2.3% VQA improvement and 3.1% R@1 improvement are not verifiable from the information provided.

minor comments (5)

[§2.4] The citation 'MMBT [?]' is unresolved; the reference list has no corresponding entry. Please either cite the actual MMBT paper or remove the citation.
[§5.4] The text refers to 'Figure??' for the modality weight visualization; the figure is missing or the cross-reference was never filled in.
[References] The LLaVA citation (reference [4]) lists incorrect author names; the authors of 'Visual Instruction Tuning' are Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
[§5.1] The paper does not state which split of VQA v2 is used (e.g., train/val or test-dev), which split of Flickr30K is used for retrieval evaluation, or the evaluation protocol for captioning (e.g., beam size, length penalty). These details are needed for reproducibility.
[§1 and §3.2] The term 'epistemic uncertainty' is used for MC dropout variance, but no distinction is made between aleatoric and epistemic uncertainty; this is fine, but the paper should clarify that u_m is a proxy, not a calibrated uncertainty estimate.

Circularity Check

1 steps flagged · score 6.0 of 10

Proposition 4.2 restates the definition of h as a weighted sum, so the claimed MWCL regularization effect is a tautology; the remaining theoretical and empirical claims rest on unsupported application details rather than on circular reasoning.

self definitional [Section 3.1 (fused representation h); Section 4.3, Proposition 4.2]
"the final fused representation is computed as: h = sum_{m=1}^M omega_m(x) * f^{(m)}(x^{(m)}) ... E_m[||h - f^{(m)}||_2^2] = Var(f^{(m)}) - ||h - \bar f||_2^2 where \bar f = sum_m omega_m f^{(m)}"

In Proposition 4.2, \bar f is defined as sum_m omega_m f^{(m)}, exactly the same expression used to define h in Section 3.1. Therefore ||h - \bar f||_2^2 = 0 identically, and the proposition reduces to E_m||h - f^{(m)}||_2^2 = Var(f^{(m)}), which is the variance of the unimodal embeddings around their weighted mean. This decomposition holds for any choice of nonnegative weights summing to 1, independent of c_m, u_m, s_m, the softmax scheduler, or the MWCL itself. The claimed 'regularization effect' is thus a restatement of the definition of h, not a derived property of the DMS weighting mechanism.

full rationale

The only circular step I can exhibit is the theoretical 'regularization effect' of the Modality Weight Consistency Loss: Proposition 4.2's \bar f is literally h from Eq. (3.1), so the stated variance identity is true by definition and does not validate DMS. I did not find a self-citation chain: the Gong et al. references are external authors, not overlapping with this paper's author list, and no uniqueness theorem is imported. The generalization bound (Theorem 4.3) is standard and generic, which is weak support but not circular. The larger concern that BLIP-2 and LLaVA are not reduced to the h = sum omega f^{(m)} fusion is a correctness and verification gap, not circularity, because the paper never claims to derive that reduction from the backbones' definitions. Score 6 reflects one load-bearing theoretical result that reduces by construction; the empirical tables are not internally forced by a fit.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the assumption that reliability can be read from entropy, dropout variance, and cosine alignment, and that MLLM fusion is a weighted sum of embeddings; these are not justified. Hyperparameters alpha, beta, gamma, lambda, and T are free and unreported. The scheduler and loss are parameterized components, not standalone entities.

free parameters (5)

alpha (confidence weight)
Coefficient for the confidence score in the softmax scheduler. Value not reported, likely tuned.
beta (uncertainty weight)
Coefficient for the negative uncertainty in the softmax scheduler. Value not reported.
gamma (alignment weight)
Coefficient for the semantic alignment score. Value not reported.
lambda (MWCL loss weight)
Trade-off between task loss and the consistency loss. Value not reported.
T (MC dropout forward passes)
Number of stochastic forward passes used to estimate epistemic uncertainty. Not specified.

assumptions (4)

ad hoc to paper Entropy, MC-dropout variance, and cross-modal cosine similarity are valid reliability signals for modality weighting.
Introduced in Section 3.2 without empirical or theoretical justification; the entire scheduler depends on this.
ad hoc to paper Fusion can be represented as a convex combination of unimodal embeddings.
Equation in Section 3.1 defines h as sum_m omega_m f(m); the claimed backbones use cross-attention, not this weighted sum.
standard math Encoders are L-Lipschitz and the loss is bounded for the stated bounds.
Assumed in Proposition 4.1 and Theorem 4.3; not verified for the actual models.
standard math Standard i.i.d. and uniform-convergence framework for the generalization bound.
Invoked in Theorem 4.3 without adaptation to the DMS-specific function class.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Dynamic Modality Scheduling for Multimodal Large Models via Confidence, Uncertainty, and Semantic Consistency." pith.science (2026). https://pith.science/paper/X7UIQARN

@misc{pith2026250612724,
  author       = {Pith},
  title        = {Pith review of: Dynamic Modality Scheduling for Multimodal Large Models via Confidence, Uncertainty, and Semantic Consistency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X7UIQARN}},
  note         = {Machine review of arXiv:2506.12724}
}

read the original abstract

Multimodal Large Models (MLLMs) have achieved remarkable progress in vision-language understanding and generation tasks. However, existing MLLMs typically rely on static modality fusion strategies, which treat all modalities equally regardless of their instance-level reliability or semantic contribution. This often leads to suboptimal performance, especially in scenarios with noisy, missing, or misaligned modalities. In this paper, we propose Dynamic Modality Scheduling (DMS), a novel framework that adaptively adjusts the contribution of each modality at a per-sample level. DMS evaluates each modality based on three key factors: (1) \textit{confidence}, estimated from predictive entropy; (2) \textit{uncertainty}, obtained via Monte Carlo dropout; and (3) \textit{semantic consistency}, computed through inter-modal similarity. These signals are combined through a learnable or rule-based scheduler to generate soft modality weights used in downstream fusion.To ensure stable training, we further introduce a \textit{Modality Weight Consistency Loss}, which regularizes the fused representation to stay close to unimodal embeddings proportionally to their assigned weights. Our method is model-agnostic and can be integrated into existing MLLMs such as BLIP-2 and LLaVA. Experimental results on VQA, image-text retrieval, and captioning tasks show that DMS significantly improves both clean and robust performance, especially under modality corruption or dropout conditions. This work provides a general and effective mechanism to enable instance-aware and robustness-enhanced multimodal modeling.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 9 canonical work pages

[1]

Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

Junnan Li, Dongxu Li, Chunyuan Xiong, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023. 1, 2, 5

arXiv 2023
[2]

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Paul Luc, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022. 1, 2

arXiv 2022
[3]

Pali: A jointly-scaled multilingual language-image model.arXiv preprint arXiv:2304.12995,

Yen-Chun Chen et al. Pali: A jointly-scaled multilingual language-image model.arXiv preprint arXiv:2304.12995,

arXiv
[4]

Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

Haotian Liu, Chunyuan Zhang, Yuheng Du, and Jian- feng Tang. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023. 1, 2, 5

arXiv 2023
[5]

Learning transferable visual models from natural language supervision.ICML,

Alec Radford, Jong Wook Kim, et al. Learning transferable visual models from natural language supervision.ICML,

work page
[6]

Elimi- nate deviation with deviation for data augmentation and a general multi-modal data learning method.arXiv preprint arXiv:2101.08533, 2021

Yunpeng Gong, Liqing Huang, and Lifei Chen. Elimi- nate deviation with deviation for data augmentation and a general multi-modal data learning method.arXiv preprint arXiv:2101.08533, 2021. 1, 2

arXiv 2021
[7]

Beyond dropout: Robust convolutional neural networks based on local feature masking

Yunpeng Gong, Chuangliang Zhang, Yongjie Hou, Lifei Chen, and Min Jiang. Beyond dropout: Robust convolutional neural networks based on local feature masking. In2024 In- ternational Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2024. 1, 2

work page 2024
[8]

Adversarial learning for neural pde solvers with sparse data.arXiv preprint arXiv:2409.02431, 2024

Yunpeng Gong, Yongjie Hou, Zhenzhong Wang, Zexin Lin, and Min Jiang. Adversarial learning for neural pde solvers with sparse data.arXiv preprint arXiv:2409.02431, 2024. 1, 2

arXiv 2024

Show all 18 references

[9]

Cross-modality perturbation synergy attack for person re-identification

Yunpeng Gong, Zhun Zhong, Yansong Qu, Zhiming Luo, Rongrong Ji, and Min Jiang. Cross-modality perturbation synergy attack for person re-identification. InThe Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. 1, 2

2024
[10]

Person re- identification method based on color attack and joint de- fence

Yunpeng Gong, Liqing Huang, and Lifei Chen. Person re- identification method based on color attack and joint de- fence. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4313–4322,
[11]

Ex- ploring color invariance through image-level ensemble learn- ing.arXiv preprint arXiv:2401.10512, 2024

Yunpeng Gong, Jiaquan Li, Lifei Chen, and Min Jiang. Ex- ploring color invariance through image-level ensemble learn- ing.arXiv preprint arXiv:2401.10512, 2024. 1, 2

2024 arXiv
[12]

Gemini: A family of highly capable mul- timodal models.arXiv preprint arXiv:2312.11805, 2023

Google DeepMind. Gemini: A family of highly capable mul- timodal models.arXiv preprint arXiv:2312.11805, 2023. 2

2023 arXiv
[13]

Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. InNeurIPS, 2019. 2, 3

2019
[14]

Beyond augmentation: Empowering model robustness under extreme capture environments

Yunpeng Gong, Yongjie Hou, Chuangliang Zhang, and Min Jiang. Beyond augmentation: Empowering model robustness under extreme capture environments. In2024 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2024. 2

2024
[15]

Dropout as a bayesian approximation: Representing model uncertainty in deep learning.ICML, 2016

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning.ICML, 2016. 2

2016
[16]

Cross-modality attack boosted by gradient-evolutionary multiform optimization.arXiv preprint arXiv:2409.17977, 2024

Yunpeng Gong, Qingyuan Zeng, Dejun Xu, Zhenzhong Wang, and Min Jiang. Cross-modality attack boosted by gradient-evolutionary multiform optimization.arXiv preprint arXiv:2409.17977, 2024. 2

2024 arXiv
[17]

Gshard: Scaling giant models with conditional computation and automatic shard- ing

Dmitry Lepikhin, Noam Shazeer, et al. Gshard: Scaling giant models with conditional computation and automatic shard- ing. InICLR, 2020. 3

2020
[18]

M3ae: Multi-modal model pre- training with masked autoencoders

Han Zhang et al. M3ae: Multi-modal model pre- training with masked autoencoders. InarXiv preprint arXiv:2111.07783, 2021. 3 8

2021 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

Junnan Li, Dongxu Li, Chunyuan Xiong, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023. 1, 2, 5

arXiv 2023

[2] [2]

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Paul Luc, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022. 1, 2

arXiv 2022

[3] [3]

Pali: A jointly-scaled multilingual language-image model.arXiv preprint arXiv:2304.12995,

Yen-Chun Chen et al. Pali: A jointly-scaled multilingual language-image model.arXiv preprint arXiv:2304.12995,

arXiv

[4] [4]

Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

Haotian Liu, Chunyuan Zhang, Yuheng Du, and Jian- feng Tang. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023. 1, 2, 5

arXiv 2023

[5] [5]

Learning transferable visual models from natural language supervision.ICML,

Alec Radford, Jong Wook Kim, et al. Learning transferable visual models from natural language supervision.ICML,

work page

[6] [6]

Elimi- nate deviation with deviation for data augmentation and a general multi-modal data learning method.arXiv preprint arXiv:2101.08533, 2021

Yunpeng Gong, Liqing Huang, and Lifei Chen. Elimi- nate deviation with deviation for data augmentation and a general multi-modal data learning method.arXiv preprint arXiv:2101.08533, 2021. 1, 2

arXiv 2021

[7] [7]

Beyond dropout: Robust convolutional neural networks based on local feature masking

Yunpeng Gong, Chuangliang Zhang, Yongjie Hou, Lifei Chen, and Min Jiang. Beyond dropout: Robust convolutional neural networks based on local feature masking. In2024 In- ternational Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2024. 1, 2

work page 2024

[8] [8]

Adversarial learning for neural pde solvers with sparse data.arXiv preprint arXiv:2409.02431, 2024

Yunpeng Gong, Yongjie Hou, Zhenzhong Wang, Zexin Lin, and Min Jiang. Adversarial learning for neural pde solvers with sparse data.arXiv preprint arXiv:2409.02431, 2024. 1, 2

arXiv 2024