Pith. sign in

REVIEW 6 major objections 5 minor 49 references

Asymmetric Reinforcing against Multi-modal Representation Bias

T0 review · 6 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A mutual-information valuation of each modality's per-sample contribution lets a training method reinforce weak modalities while keeping dominant ones strong.

desk verdict A novel combination of MI-based valuation and asymmetric reinforcement that shows real empirical gains, but the valuation math is not well-defined; as written, the central mechanism doesn't hold together. read the letter →

arxiv 2501.01240 v1 pith:JOVUX6NL submitted 2025-01-02 cs.CV

Xiyuan Gao , Bing Cao , Pengfei Zhu , Nannan Wang , Qinghua Hu This is my paper · ORCID
classification cs.CV
keywords multimodallearningmodalityimbalancemutualinformationconditionalasymmetricreinforcementforgettingdynamicfeaturefusionsampleresampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ARM, a training procedure that rebalances multimodal learning by measuring, sample by sample, how much each modality contributes to the model's correct prediction. The central claim is that mutual information between each modality's features and the fused output, combined with conditional mutual information between modalities, gives a reliable per-sample valuation of marginal and joint contribution; ARM then uses this valuation to dynamically weight the fusion, add a balanced min-max loss, and resample low-contribution samples more often. The paper argues that this asymmetric reinforcement narrows the contribution gap between modalities without degrading the dominant one, avoiding the modality forgetting that prior weak-modality-boosting methods exhibit. On Kinetics Sounds, UCF-51, and Food-101, the method reports the best accuracies among the compared methods, 66.52%, 75.60%, and 93.36%, and shows that the joint contribution of all modalities increases steadily during training.

What carries the argument

The central object is the mutual-information valuation metric (MIV), which converts model outputs into per-sample contribution scores. For each modality it computes normalized mutual information $I(f_Y; f_{x_i})$ between the fused feature and that modality's feature, scaled by $p(f_Y\to y)$, the probability that the fusion predicts the true label; the minimum over modalities gives a lower bound on the joint contribution (Eq. 7). The asymmetric marginal contribution adds interaction information $\mathrm{II}(f_Y; f_{x_j}; f_{x_i})$ between modality pairs, so each modality's score reflects both its own information and the extra information another modality provides given it (Eq. 11). These scores are the inputs to the three reinforcement mechanisms: fusion weights (Eq. 13), the balanced min-max loss (Eqs. 14-15) with a log-sum-exp smooth approximation of the min (Eq. 16), and the resampling schedule (Eq. 18). The min operator is made differentiable exactly so that the lower-bound loss can be optimized by gradient descent.

What would settle it

Compare the per-sample contribution scores $\phi_{\mathrm{CMI}}(x_i)$ that ARM computes against the actual accuracy drop observed when modality $i$ is removed (e.g., zeroed or ablated) at test time on Kinetics Sounds or Food-101; if the scores do not rank-order the ablation-induced drops, the valuation metric is not tracking true modality contribution and the whole reinforcement mechanism is optimizing a proxy.

Watch

Extended reading notes

Core claim

ARM's central claim is that modality imbalance can be corrected from a multimodal perspective rather than by strengthening weak modalities in isolation. The paper defines a mutual-information valuation metric (MIV): the marginal contribution of modality $i$ is $\phi_{\mathrm{MI}}(x_i) = p(f_Y \to y)\,I(f_Y; f_{x_i})$, with $I$ the normalized mutual information between the fused feature $f_Y$ and the modality feature, and the joint contribution of all modalities is lower-bounded by $\phi_{\mathrm{MI}}(X) = p(f_Y \to y)\,\min_i I(f_Y; f_{x_i})$. To capture inter-modality interaction, it adds an asymmetric term built from conditional mutual information, $\phi_{\mathrm{CMI}}(x_i) = p(f_Y \to y)I(f_Y; f_{x_i}) + \sum_{j\neq i} p(f_{x_j}\to y)\,\mathrm{II}(f_Y; f_{x_j}; f_{x_i})$, where $\mathrm{II}$ is the interaction information $I(f_Y;f_{x_j}) - \mathrm{NCMI}(f_Y; f_{x_j}\mid f_{x_i})$. These valuations drive three mechanisms: dynamic feature-level fusion weights $FW_i = \phi_{\mathrm{CMI}}(x_i)/\phi_{\mathrm{CMI}}(X)$, a balanced min-max loss $L = L_{\mathrm{CE}} + \lambda_1 (1 - \phi_{\mathrm{MI}}(X)) + \lambda_2 \frac{\sum_i |\phi_{\mathrm{CMI}}(x_i)-\phi_{\mathrm{CMI}}(X)|}{\phi_{\mathrm{CMI}}(X)}$, and dynamic sample-level resampling with frequency $s(X)=F_s(\phi_{\mathrm{CMI}}(X))$ decreasing in joint contribution. The paper reports that on audio-visual (Kinetics Sounds, UCF-51) and image-text (Food-101) benchmarks this yields accuracies of 66.52%, 75.60%, and 93.36%, exceeding prior imbalanced-learning and fusion methods while keeping dominant-modality accuracy above the concatenation and summation baselines.

Load-bearing premise

The load-bearing premise is that the per-sample mutual-information scores computed from the model's own outputs genuinely capture how much each modality contributes to getting the label right; if those scores are measuring something else, the fusion weights, losses, and resampling are all tuned to the wrong quantity.

Editorial extensions

If this is right

  • Adding ARM to standard concatenation or summation fusion raises accuracy by roughly 6 to 11 points across the three benchmarks (KS +6.91, UCF-51 +7.37, Food-101 +10.98), according to the paper's table of fusion-backbone gains.
  • The ablation study attributes gains to all three components: dynamic feature-level fusion alone improves KS from 59.61% to 64.34%, adding balanced min-max loss raises it further, and dynamic sample-level resampling gives the largest single jump on UCF-51 (to 74.29%).
  • ARM can be attached to other fusion backbones such as MMTM and CentralNet, improving their accuracy on all three datasets by 3.5 to 5.7 points, which suggests the reinforcement strategy is architecture-agnostic.
  • The appendix reports that more aggressive resampling (k = -3.0) yields even higher accuracy than the main results (e.g., 67.41% on KS), implying the headline numbers may not be the method's ceiling.
  • Contribution-gap curves and single-modality accuracy tables indicate that ARM narrows the gap between modalities while keeping the dominant modality's accuracy above the concatenation or summation baseline, i.e., no modality forgetting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the valuation operates on fused and per-modality features rather than on class labels, the same reinforcement scheme could in principle be applied to multimodal regression, retrieval, or generation, though the paper only evaluates classification.
  • The lower-bound identity $\phi_{\mathrm{MI}}(X) = p(f_Y\to y)\min_i I(f_Y; f_{x_i})$ is a strong structural assumption; the paper does not empirically verify that this min equals the true joint contribution, so a natural test is to compare ARM's scores with leave-one-modality-out accuracy drops.
  • If the valuation metric is faithful, the method suggests that balancing should mean equalizing marginal contributions rather than equalizing loss magnitudes or gradient norms, which are the targets of earlier gradient-modulation methods.
  • The reported sensitivity to the resampling slope k (best at k=-3.0 in the appendix) indicates that the main paper's default k=-2.0 was a conservative choice; readers should expect the optimal k to be dataset- and backbone-dependent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper proposes ARM, an asymmetric reinforcement method for imbalanced multimodal learning. The method defines a mutual-information-based valuation metric (MIV) to estimate each modality's marginal contribution and the joint contribution of all modalities per sample, and then uses these scores to (i) set dynamic feature-level fusion weights, (ii) define a balanced min-max loss comprising L_phiMI and L_phiCMI, and (iii) perform dynamic sample-level resampling. Experiments on Kinetics Sounds, UCF-51, and Food-101 report accuracy improvements over several imbalanced-learning, fusion, and MI-based baselines, with ablations of the loss terms and of the three ARM components.

Significance. The paper has strengths that should be acknowledged: a broad empirical evaluation across three datasets, comparison with many recent baselines, ablation studies of the loss terms and components, integration with existing fusion backbones, and a public code link. If the MIV scores were well-defined and correctly derived, the approach of directly balancing modality contributions through information-theoretic quantities would be of interest. However, the central theoretical foundation is not established: several load-bearing definitions are mathematically inconsistent or incomplete, and the claimed proof does not prove the stated theorem. The empirical gains may be real, but the paper's core claim that the method optimizes true modality contributions is not supported by the current text.

major comments (6)
  1. [Methods, Eq. (4)] The quantity defined in Eq. (4), I(fY=y; fX) = sum_x P(x|y) log(P(y|x)/P(y)), is not the mutual information of Eq. (1); the summand is a pointwise log-ratio and can be negative. The sentence following Eq. (1) states that MI is non-negative and hence has a positive contribution, but that property does not apply to the expression in Eq. (4). Since Theorem 1 and the Appendix proof in Eq. (19) use the same I(fY=y; .) notation, the theorem is not stated about a well-defined non-negative information measure.
  2. [Methods, Eqs. (5) and (10)] Eq. (5) is vacuous: because sum_hat-y p(fY -> hat-y) = 1, the right-hand side is identically I(fY; fxi), so it does not express an expected contribution over all possible predictions. In addition, the text announces that 'we use I to represent NMI' after Eq. (5), and then Eq. (10) defines II as I(fY; fxj) - NCMI(fY; fxj | fxi). Interaction information requires subtracting ordinary CMI, not a normalized version, so the resulting quantity is not the interaction information claimed and has no established information-theoretic meaning. The fusion weights, the loss in Eq. (15), and the resampling frequency in Eq. (18) all inherit this quantity.
  3. [Methods, Eq. (11)] The term p(fxj -> y) in Eq. (11) is never defined. No per-modality classifier, probability distribution over y given fxj, or estimation procedure is specified anywhere in the paper. Because Eq. (11) is the definition of phi_CMI(xi), the dynamic fusion weight in Eq. (13), the MAE loss in Eq. (15), and the resampling frequency in Eq. (18) all depend on an undefined quantity. As written, the core modules of ARM are not reproducible.
  4. [Methods, Eq. (7)] The 'lower bound of joint contribution' is asserted without proof. No definition of joint contribution is given before Eq. (7), and no argument shows that p(fY -> y) * min_i I(fY; fxi) bounds the contribution of the complete modality set to accurate prediction. The subsequent properties of phi_MI and the claim that maximizing phi_MI via the loss in Eq. (14) increases a lower bound of joint contribution are therefore unsupported.
  5. [Appendix A, Theorem 1] The proof of Theorem 1 does not establish the stated result. The derivation in Eq. (19) shows only that I(Y; A) - I(Y; B) >= 0 for sets A and B with B subset of A, via a KL divergence. It does not prove strict inequality, and it does not prove the pairwise assertion I(fY=y; fxi) != I(fY=y; fxj) for arbitrary i != j. The proof also uses the notation I(Y=y; .) inherited from Eq. (4), which, as noted above, is not a well-defined mutual information.
  6. [Appendix C, Table 5] Table 5 reports that DSR with k = -3.0 attains 67.41% on KS, 76.83% on UCF-51, and 93.69% on Food-101, all higher than the ARM results used in Tables 1 and 2 (66.52, 75.60, and 93.36, respectively). The main text does not explain why the reported ARM configuration uses k = -2.0 instead of the better-performing k = -3.0. This inconsistency affects the state-of-the-art claims and the choice of the final model.
minor comments (5)
  1. [Methods, Eqs. (2) and (9)] The denominators in Eqs. (2) and (9) appear to contain a typesetting error: they should be sqrt(H(X)H(Y)) and sqrt(H(X|Z)H(Y|Z)), respectively, rather than the current 'p H(...)' notation.
  2. [Appendix B, Algorithm 1] In Algorithm 1, 'Initialize Drs = D' appears inside the epoch loop, which resets the resampled dataset at the start of every epoch. Please clarify whether this is intended and, if so, how resampling frequencies from previous epochs are accumulated or reinitialized.
  3. [Experiments, Tables 1 and 3] On UCF-51, MLA achieves 76.01%, which is higher than ARM's 75.60%. The statement that ARM 'consistently outperforms other state-of-the-art methods' is therefore inaccurate; the later sentence acknowledging MLA's strength on UCF-51 should be reconciled with that claim.
  4. [Experiments, Table 4] The LCE-only baseline in Table 4 reports 63.88% on KS, whereas the Concatenation baseline in Table 1 reports 59.61%. Please state what fusion or training setup the ablation baseline uses so that the reader can interpret the ablation gains.
  5. [Methods, Eq. (16)] The log-sum-exp expression in Eq. (16) is a standard smooth approximation to max_i(-I_i), but the text describes it as making the min function differentiable. The sign convention should be stated explicitly to avoid confusion: max_i(-I_i) = -min_i I_i, and the log-sum-exp approximates the former.

Circularity Check

2 steps flagged · score 6.0 of 10

Two self-definitional reductions: the 'joint contribution' and 'contribution gap' curves are the paper's own loss objectives, so the claimed gap-narrowing and contribution-enhancement evidence is built into the optimization; benchmark accuracy remains external.

  1. self definitional [Methods > Lower bound of joint contribution ϕ(X), Eq. (7); Balanced Min-Max Loss, Eq. (14); Experiments > Multimodal Cooperation, Fig. 4 analysis]
    "ϕM I(X ) = p (fY → y) min i∈{1,...m} I (fY ; fxi ) . (7) ... LϕM I= 1 − ϕM I(X ), (14) ... in the training phase, by incorporating ϕM I into the loss function and using gradient descent to maximize ϕM I, thus each iteration moves towards increasing mutual information, ensuring the convergence of the lower bound. ... ARM exhibits a consistent and substantial increase in the multimodal average contribution, the chart shows that ARM not only achieves a higher overall contribution but also demonstrates a stable and continuous growth trend."

    The paper defines its 'joint contribution' as ϕMI(X), then defines LϕMI as 1 − ϕMI(X), so gradient descent directly maximizes the same quantity. The later claim that ARM produces 'a consistent and substantial increase in the multimodal average contribution' is therefore a restatement of the training objective, not an independent measurement. The increase in the plotted contribution curve is imposed by the loss by construction. The held-out accuracy in Tables 1–3 is separate evidence and prevents the whole paper from being circular, but this particular contribution-enhancement claim reduces to the definition of the loss.

  2. self definitional [Methods > Balanced Min-Max Loss, Eqs. (11)–(12), (15); Experiments > Comparison with Imbalanced Multimodal Learning Methods, Fig. 3 analysis]
    "for the latter, we can use the Mean Absolute Error (MAE) to minimize M AE(ϕCM I), thereby narrowing the marginal contribution gap between modalities. LϕCM I= P m i=1 |ϕCM I(xi) − ϕCM I(X )| / ϕCM I(X ) . (15) ... In contrast, ARM achieves a marked and consistent reduction in modality contribution gaps, indicating a more balanced learning process."

    The 'contribution gap' displayed in Fig. 3 is measured as the spread of the ϕCMI values, and Eq. (15) is precisely the normalized mean absolute error of those same values. Minimizing LϕCMI therefore narrows the plotted gap by definition. The paper's evidence that ARM 'narrows contribution gaps' is the objective itself, not an independent result about an externally defined modality contribution. This is a partial circularity: the loss controls the metric, while the final classification accuracy remains an external benchmark.

full rationale

The strongest circularity in ARM is that its two headline mechanistic claims are written into its losses. Eq. (7) defines the joint contribution as ϕMI and Eq. (14) maximizes it; Eqs. (11)–(12) define per-modality and joint ϕCMI and Eq. (15) minimizes their normalized spread. Consequently, the increasing 'average joint contribution' in Fig. 4 and the decreasing 'contribution gap' in Fig. 3 are the training objectives themselves, not independent confirmations of the method's mechanism. I score this as partial circularity (6) rather than higher because Tables 1–3 report held-out accuracy against external baselines, and Table 4's ablations are externally checkable; the empirical claim of improved accuracy does not reduce to the definitions of ϕMI or ϕCMI. I did not count the undefined p(fxj→y) in Eq. (11) or the mixed normalized/un-normalized subtraction in Eq. (10) as circularity; those are definitional and validity defects affecting whether the optimized quantity actually is a modality contribution, but they do not by themselves make the claimed accuracy result an input of the derivation. Similarly, Appendix A's proof of Theorem 1 establishes only set-wise monotonicity in Eq. (19), not the stated pairwise inequality I(fY=y;fxi) ≠ I(fY=y;fxj); that is a missing proof for a premise rather than a circular step. The paper's own limitations about discretizing continuous data and unresolved modality conflicts are acknowledged caveats, not circularity. No load-bearing self-citation chain appears in the derivation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a purpose-built mutual-information valuation framework whose key quantities are defined only loosely. No external constants or entities are introduced; instead, the method postulates that model confidences and normalized MI terms can be combined into modality contribution scores. The main free choices are the loss weights lambda_1, lambda_2 and the resampling slope k, with k tuned per dataset.

free parameters (2)
  • lambda_1, lambda_2 = not reported
    Trade-off weights in Eq. (17) are not given in the paper; their values are needed to reproduce the loss and were presumably tuned on data.
  • k (DSR slope) = main results use k=-2.0; k=-3.0 gives higher accuracy in Table 5
    Sampling frequency slope in Eq. (18) and Appendix C; grid-searched per dataset in Table 5, with k=-3.0 superior on all three datasets but not used in main tables, indicating test-set selection.
assumptions (4)
  • domain assumption Mutual information computed between model features is non-negative and measures a modality's contribution to accurate prediction.
    Eqs. (4)-(7) treat pointwise MI conditional on the true label as a positive contribution; this can be negative and is not the standard MI of Eq. (1).
  • ad hoc to paper The lower bound of joint contribution is min_i NMI(fY;fxi) scaled by p(fY->y).
    Eq. (7) asserts that joint contribution equals the minimum single-modality MI; no derivation shows why the minimum, rather than the union or interaction, is the joint contribution.
  • domain assumption A partial information decomposition applies to this setting and supports per-modality positive and unique contributions.
    Theorem 1 and Eq. (10) rely on decomposing multivariate MI into per-modality positive and unique contributions, which is a contested decomposition with non-unique solutions.
  • ad hoc to paper Softmax outputs p(fY->y) and p(fxj->y) quantify contribution to accurate prediction.
    p(fY->y) is the probability of the true class, and p(fxj->y) is never defined; these outputs are used as weights in Eqs. (6), (11), and (13).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Asymmetric Reinforcing against Multi-modal Representation Bias." pith.science (2026). https://pith.science/paper/JOVUX6NL

@misc{pith2026250101240,
  author       = {Pith},
  title        = {Pith review of: Asymmetric Reinforcing against Multi-modal Representation Bias},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JOVUX6NL}},
  note         = {Machine review of arXiv:2501.01240}
}
read the original abstract

The strength of multimodal learning lies in its ability to integrate information from various sources, providing rich and comprehensive insights. However, in real-world scenarios, multi-modal systems often face the challenge of dynamic modality contributions, the dominance of different modalities may change with the environments, leading to suboptimal performance in multimodal learning. Current methods mainly enhance weak modalities to balance multimodal representation bias, which inevitably optimizes from a partialmodality perspective, easily leading to performance descending for dominant modalities. To address this problem, we propose an Asymmetric Reinforcing method against Multimodal representation bias (ARM). Our ARM dynamically reinforces the weak modalities while maintaining the ability to represent dominant modalities through conditional mutual information. Moreover, we provide an in-depth analysis that optimizing certain modalities could cause information loss and prevent leveraging the full advantages of multimodal data. By exploring the dominance and narrowing the contribution gaps between modalities, we have significantly improved the performance of multimodal learning, making notable progress in mitigating imbalanced multimodal learning.

Figures

Figures reproduced from arXiv: 2501.01240 by the authors.

Figure 1
Figure 1. Accuracy curve of dominant modality compared [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Left: The Lower Bound joint contribution (MIV-LB) of all modalities and the Asymmetric marginal contribution (MIV-Asym) of each modality are estimated by ϕM I and ϕ CM I , respectively, serving as the basis for asymmetric reinforcement. fY is feature-level fusion result, p is the accurate production. Right: Representation of features in the latent space. We minimize the diversities in ϕ CM I to balance multimodal le… view at source ↗
Figure 3
Figure 3. Comparison of the narrowing trend of uni [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Average joint contribution of all modalities overall [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Curve of Balanced Min-Max Loss: the values are [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The per-class accuracy (%) of recognition on UCF￾51 comparing ARM with AMID. Visualizations in [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Visual feature distribution of MMTM, MMTM-ARM and CentralNet, CentralNet-ARM visualized by t-SNE (Van der [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Visualization of Audio-visual samples from Kinetics Sounds dataset. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Contribution improvement compared. Other imbalanced multimodal learning methods: Greedy (Wu et al. 2022), [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 37 canonical work pages

  1. [1]

    Arandjelovic, R.; and Zisserman, A. 2017. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, 609--617

    work page 2017

  2. [2]

    Bi, X.-a.; Hu, X.; Wu, H.; and Wang, Y. 2020. Multimodal data analysis of Alzheimer's disease based on clustering evolutionary random forest. IEEE Journal of Biomedical and Health Informatics, 24: 2973--2983

    work page 2020

  3. [3]

    Chen, M.; Xing, L.; Wang, Y.; and Zhang, Y. 2023. Enhanced multimodal representation learning with cross-modal kd. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11766--11775

    work page 2023

  4. [4]

    Chen, X.; Duan, Y.; Houthooft, R.; Schulman, J.; Sutskever, I.; and Abbeel, P. 2016. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29

    work page 2016

  5. [5]

    Colombo, P.; Chapuis, E.; Labeau, M.; and Clavel, C. 2021. Improving multimodal fusion via mutual dependency maximisation. arXiv preprint arXiv:2109.00922

    work page Pith review arXiv 2021

  6. [6]

    Cover, T. M. 1999. Elements of information theory. John Wiley & Sons

    1999

  7. [7]

    C.; Qiu, W.; Lu, M.; Kim, N

    Covert, I. C.; Qiu, W.; Lu, M.; Kim, N. Y.; White, N. J.; and Lee, S.-I. 2023. Learning to maximize mutual information for dynamic feature selection. In International Conference on Machine Learning, 6424--6447. PMLR

    work page 2023

  8. [8]

    Das, A.; Das, S.; Sistu, G.; Horgan, J.; Bhattacharya, U.; Jones, E.; Glavin, M.; and Eising, C. 2023. Revisiting modality imbalance in multimodal pedestrian detection. In 2023 IEEE International Conference on Image Processing (ICIP), 1755--1759. IEEE

    work page 2023

Show all 49 references
  1. [9]

    Du, C.; Li, T.; Liu, Y.; Wen, Z.; Hua, T.; Wang, Y.; and Zhao, H. 2021. Improving multi-modal learning with uni-modal teachers. arXiv preprint arXiv:2106.11059

    2021 arXiv

  2. [10]

    Fan, Y.; Xu, W.; Wang, H.; Wang, J.; and Guo, S. 2023. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 20029--20038

    2023

  3. [11]

    Fu, J.; Gao, J.; Bao, B.-K.; and Xu, C. 2023. Multimodal imbalance-aware gradient modulation for weakly-supervised audio-visual video parsing. IEEE Transactions on Circuits and Systems for Video Technology

    2023

  4. [12]

    F.; Rashidi, B.; and Baji \'c , I

    Hadizadeh, H.; Yeganli, S. F.; Rashidi, B.; and Baji \'c , I. V. 2024. Mutual Information Analysis in Multimodal Learning Systems. arXiv preprint arXiv:2405.12456

    2024 arXiv

  5. [13]

    M.; and Dziugaite, G

    Haghifam, M.; Negrea, J.; Khisti, A.; Roy, D. M.; and Dziugaite, G. K. 2020. Sharpened generalization bounds based on conditional mutual information and an application to noisy, iterative algorithms. Advances in Neural Information Processing Systems, 33: 9925--9935

    2020

  6. [14]

    Han, W.; Chen, H.; and Poria, S. 2021. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. arXiv preprint arXiv:2109.00412

    2021 arXiv

  7. [15]

    Hu, P.; Li, X.; and Zhou, Y. 2022. Shape: An unified approach to evaluate the contribution and cooperation of individual modalities. arXiv preprint arXiv:2205.00302

    2022 arXiv

  8. [16]

    Huang, Y.; Du, C.; Xue, Z.; Chen, X.; Zhao, H.; and Huang, L. 2021. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 34: 10944--10956

    2021

  9. [17]

    Huang, Y.; Lin, J.; Zhou, C.; Yang, H.; and Huang, L. 2022. Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably). In International conference on machine learning, 9226--9259. PMLR

    2022

  10. [18]

    Imfeld, M.; Graldi, J.; Giordano, M.; Hofmann, T.; Anagnostidis, S.; and Singh, S. P. 2023. Transformer fusion with optimal transport. arXiv preprint arXiv:2310.05719

    2023 arXiv

  11. [19]

    Kiela, D.; Grave, E.; Joulin, A.; and Mikolov, T. 2018. Efficient large-scale multi-modal classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32

    2018

  12. [20]

    M.; and Lee, S.-W

    Kim, J.-H.; Kim, Y.; Lee, J.; Yoo, K. M.; and Lee, S.-W. 2022. Mutual information divergence: A unified metric for multimodal generative models. Advances in Neural Information Processing Systems, 35: 35072--35086

    2022

  13. [21]

    Y.; Lo, R.; Jang, L.; Duvvur, V.; Lim, M

    Koh, J. Y.; Lo, R.; Jang, L.; Duvvur, V.; Lim, M. C.; Huang, P.-Y.; Neubig, G.; Zhou, S.; Salakhutdinov, R.; and Fried, D. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649

    2024 arXiv

  14. [22]

    Larsson, M.; Stenborg, E.; Toft, C.; Hammarstrand, L.; Sattler, T.; and Kahl, F. 2019. Fine-grained segmentation networks: Self-supervised segmentation for improved long-term visual localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 31--41

    2019

  15. [23]

    K.; Lee, J.; and Kim, S

    Lee, H. K.; Lee, J.; and Kim, S. B. 2022. Boundary-focused generative adversarial networks for imbalanced and multimodal time series. IEEE Transactions on Knowledge and Data Engineering, 34: 4102--4118

    2022

  16. [24]

    Li, H.; Li, X.; Hu, P.; Lei, Y.; Li, C.; and Zhou, Y. 2023 a . Boosting Multi-modal Model Performance with Adaptive Gradient Modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 22214--22224

    2023

  17. [25]

    Li, Y.; Quan, R.; Zhu, L.; and Yang, Y. 2023 b . Efficient multimodal fusion via interactive prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2604--2613

    2023

  18. [26]

    Liao, R.; Moyer, D.; Cha, M.; Quigley, K.; Berkowitz, S.; Horng, S.; Golland, P.; and Wells, W. M. 2021. Multimodal representation learning via maximization of local mutual information. In Medical Image Computing and Computer Assisted Intervention--MICCAI 2021: 24th Internatio...

    2021

  19. [27]

    K.; Mikkelsen, I

    Livne, M.; Boldsen, J. K.; Mikkelsen, I. K.; Fiebach, J. B.; Sobesky, J.; and Mouridsen, K. 2018. Boosted tree model reforms multimodal magnetic resonance imaging infarct prediction in acute stroke. Stroke, 49: 912--918

    2018

  20. [28]

    Nielsen, F.; and Sun, K. 2016. Guaranteed bounds on information-theoretic measures of univariate mixtures using piecewise log-sum-exp inequalities. Entropy, 18: 442

    2016

  21. [29]

    Pan, W.; Long, F.; and Pan, J. 2023. ScInfoVAE: interpretable dimensional reduction of single cell transcription data with variational autoencoders and extended mutual information regularization. BioData Mining, 16: 17

    2023

  22. [30]

    Pan, Y.; Jiang, J.; Jiang, K.; and Liu, X. 2024. Disentangled-Multimodal Privileged Knowledge Distillation for Depression Recognition with Incomplete Multimodal Data. In ACM Multimedia

    2024

  23. [31]

    Peng, X.; Wei, Y.; Deng, A.; Wang, D.; and Hu, D. 2022. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8238--8247

    2022

  24. [32]

    Perez, E.; Strub, F.; De Vries, H.; Dumoulin, V.; and Courville, A. 2018. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32

    2018

  25. [33]

    Rahate, A.; Walambe, R.; Ramanna, S.; and Kotecha, K. 2022. Multimodal co-learning: Challenges, applications with datasets, recent advances and future directions. Information Fusion, 81: 203--239

    2022

  26. [34]

    R.; and Shah, M

    Soomro, K.; Zamir, A. R.; and Shah, M. 2012. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402

    2012 arXiv

  27. [35]

    C.; Erricolo, D.; Ravi, S.; Tulabandhula, T.; and Trivedi, A

    Stutts, A. C.; Erricolo, D.; Ravi, S.; Tulabandhula, T.; and Trivedi, A. R. 2023. Mutual information-calibrated conformal feature fusion for uncertainty-aware multimodal 3d object detection at the edge. arXiv preprint arXiv:2309.09593

    2023 arXiv

  28. [36]

    Sun, Z.; Sarma, P.; Sethares, W.; and Liang, Y. 2020. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. In Proceedings of the AAAI conference on artificial intelligence, volume 34, 8992--8999

    2020

  29. [37]

    Van der Maaten, L.; and Hinton, G. 2008. Visualizing data using t-SNE. Journal of machine learning research, 9

    2008

  30. [38]

    Wang, X.; Kumar, D.; Thome, N.; Cord, M.; and Precioso, F. 2015. Recipe recognition with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 1--6. IEEE

    2015

  31. [39]

    Wei, Y.; Feng, R.; Wang, Z.; and Hu, D. 2024. Enhancing multimodal cooperation via sample-level modality valuation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 27338--27347

    2024

  32. [40]

    Wen, L.; Chen, S.; Xie, M.; Liu, C.; and Zheng, L. 2024. Training multi-source domain adaptation network by mutual information estimation and minimization. Neural Networks, 171: 353--361

    2024

  33. [41]

    L.; and Beer, R

    Williams, P. L.; and Beer, R. D. 2010. Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515

    2010 arXiv

  34. [42]

    Wu, N.; Jastrzebski, S.; Cho, K.; and Geras, K. J. 2022. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In International Conference on Machine Learning, 24043--24055. PMLR

    2022

  35. [43]

    Xue, Z.; and Marculescu, R. 2023. Dynamic multimodal fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2575--2584

    2023

  36. [44]

    Yang, Z.; Wei, Y.; Liang, C.; and Hu, D. 2024. Quantifying and Enhancing Multi-modal Robustness with Modality Preference. In The Twelfth International Conference on Learning Representations

    2024

  37. [45]

    Yu, W.; Yang, Z.; Li, L.; Wang, J.; Lin, K.; Liu, Z.; Wang, X.; and Wang, L. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490

    2023 arXiv

  38. [46]

    T.; and Peng, X

    Zhang, Q.; Wu, H.; Zhang, C.; Hu, Q.; Fu, H.; Zhou, J. T.; and Peng, X. 2023. Provable Dynamic Fusion for Low-Quality Multimodal Data. In International Conference on Machine Learning

    2023

  39. [47]

    Zhang, X.; Yoon, J.; Bansal, M.; and Yao, H. 2024. Multimodal representation learning by alternating unimodal adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 27456--27466

    2024

  40. [48]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.