Learning Optimal Multimodal Information Bottleneck Representations

Jun Wang; Qilong Wu; Xiaobo Sun; Yiyang Shao

REVIEW 4 major objections 4 minor 16 references

Learning Optimal Multimodal Information Bottleneck Representations

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that a single regularization weight, chosen inside a provable interval, makes a multimodal bottleneck representation achieve the theoretical optimum: it keeps all task-relevant information and nothing else.

desk verdict A sensible heuristic with a broken proof: the claimed guarantee on optimal MIB fails because the variational bound drops a positive term and the appendix changes the objective. read the letter →

arxiv 2505.19996 v1 pith:B5JGPCMI submitted 2025-05-26 cs.LG

Qilong Wu , Yiyang Shao , Jun Wang , Xiaobo Sun This is my paper · ORCID

classification cs.LG

keywords multimodallearninginformationbottleneckregularizationweightboundvariationalinferencemutualestimationmodalityimbalancerepresentation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal information bottleneck (MIB) methods try to learn a fused representation that keeps what is needed to predict a task and discards the rest, but they tune the strength of the 'discard' term by hand and can drop a weak modality's crucial information. This paper tries to close that gap by proving that a single regularization weight $\beta$, set inside the bound $M_u = 1/((1+r)(H(v_1)+H(v_2)-I(v_1;v_2)))$, forces the fused representation to contain exactly the task-relevant information $\{a_0, a_1, a_2\}$ and no superfluous information $\{b_0, b_1, b_2\}$. The framework it proposes, OMIB, pairs that objective with a per-modality weight $r$ that shrinks automatically when one modality still holds undigested task information, so imbalanced modalities are not regularized out. If the proof holds, the bottleneck strength stops being an ad hoc knob: both ends of the valid range are computable from the training data before training, via mutual information estimates. The paper validates the bound on controlled Gaussian data, where accuracy peaks just below $M_u$, and reports downstream gains on emotion recognition, multimodal sentiment analysis, and anomalous tissue detection.

What carries the argument

The load-bearing object is the objective of Equation (17), $\min_\xi \ell(\xi) = -I(\xi;y) + \beta(I(\xi;z_1) + r I(\xi;z_2))$, together with the bound $M_u = 1/((1+r)(H(v_1)+H(v_2)-I(v_1;v_2)))$ on the redundancy weight $\beta$. Assumption 5.3 supplies the Venn-diagram decomposition of the two modalities into task-relevant atoms $a_0, a_1, a_2$ and superfluous atoms $b_0, b_1, b_2$; this is what lets the proof treat missing task information as an atom disjoint from the current representation and show that adding it strictly lowers the loss whenever $\beta < M_u$. A second device carries the proof: the derivations of Lemmas 5.5 and 5.6 replace $I(\xi;y)$ with the relative mutual information $\hat{I}(\xi;y) = I(\xi;y)/(H(\xi)+H(y)-I(\xi;y))$, arguing that this normalized quantity better tracks the share of task-relevant information in the total. The objective actually trained is the variational upper bound of Equation (10), which converts $I(\xi;y)$ into a cross-entropy term and each $I(\xi;z_i)$ into a KL divergence against a standard Gaussian.

What would settle it

On the paper's own synthetic Gaussian datasets (SIM-I/II/III), after training compute the dropped conditional-information term $I(\zeta_2; z_1 | \zeta_1)$: wherever the modalities share the blocks $a_0$ and $b_0$, this term is positive, and if it is positive the inequality $I(\xi;z_1) \le I(\zeta_1;z_1)$ used in Proposition 5.1 is not established. A second check targets the relative-information substitution directly: train with a large shared task-relevant block $a_0$ and $\beta$ just below $M_u$, then measure whether the learned $\xi$ retains all of $a_0$ and none of $b_0$; a regime where the relative and ordinary objectives disagree would show a representation that omits part of $a_0$ or keeps part of $b_0$, contradicting Proposition 5.7.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the optimal multimodal bottleneck is achievable, not merely approachable. Under Assumption 5.3, which partitions each modality's information into task-relevant atoms (the shared $a_0$ and the modality-specific $a_1, a_2$) and superfluous atoms ($b_0$ shared, $b_1, b_2$ specific), the optimal MIB is defined by $F(\xi_{\mathrm{opt}}) = \{a_0, a_1, a_2\}$. The paper proves that minimizing $\ell(\xi) = -I(\xi;y) + \beta(I(\xi;z_1) + r I(\xi;z_2))$ with $\beta \in (0, M_u]$ both pulls all task-relevant information in (Lemma 5.5) and pushes all superfluous information out (Lemma 5.6), so that $F(\xi) = \{a_0, a_1, a_2\}$ (Proposition 5.7). The dynamic weight $r$ is computed during training from the ratio of the two modalities' remaining task-relevant information, which keeps the weaker modality's complementary information (say $a_2$) from being penalized away. The whole objective is implemented as a variational upper bound, so the intractable mutual informations become a cross-entropy term plus KL divergences.

Load-bearing premise

The guarantee rests on assuming the two modality encodings are conditionally independent, $\zeta_1 \perp z_2$ and $\zeta_2 \perp z_1$, when the variational bound is derived; the paper's own setup violates this because the modalities share the $a_0$ and $b_0$ blocks, and the proofs of the two lemmas also swap in the relative mutual information without showing that it shares the original objective's optimum.

Editorial extensions

If this is right

Bottleneck strength stops being a free knob: with $M_l$ and $M_u$ estimated from the training data by a neural mutual-information estimator, a practitioner can set $\beta$ in the guaranteed range before training and get the optimal representation without per-task search.
Imbalanced modalities are protected: because $r$ shrinks when one modality still holds undigested task-relevant information, a modality with minor but crucial information (such as $a_2$ in the paper's Figure 1) is not regularized away, so the fused representation stays sufficient.
The guarantee extends to three or more modalities through the generalized bound, so the argument covers fusion beyond the pairwise case treated in the main text.
Because the optimizable loss is a variational upper bound, the theoretical optimum is approached by minimizing a standard cross-entropy plus KL-divergence objective, so the recipe drops into existing deep learning pipelines.
On the paper's reported tasks, following the bound produces representations that outperform fixed-weight MIB baselines on emotion recognition, sentiment analysis, and anomalous tissue detection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The atomized Venn-diagram model ($a_0/a_1/a_2$ versus $b_0/b_1/b_2$) is an idealization; for real modalities with graded overlaps the practical content of Proposition 5.7 is a regime statement, and a testable extension is to intervene on the $a_0$ and $b_0$ blocks of the synthetic data to map how quickly the guarantee degrades as the shared superfluous block $b_0$ grows.
The proof's unstated swap to relative mutual information suggests a cleaner route: if the objective were written directly in terms of the normalized measure, the two-objective gap would disappear, and one could check empirically whether the OMIB bound still holds under that rewrite.
The bound $M_u$ is derived from $F(\xi,y) \subseteq F(v_1,v_2)$; computing the slack $F(v_1,v_2) - F(\xi,y)$ during training would give an online measure of how far the current representation is from the claimed optimum, effectively a certificate that the representation has not yet converged.
If the conditional-independence violation cannot be patched, a natural repair is to add the dropped term $I(\zeta_2;z_1|\zeta_1)$ as an explicit penalty, and to test whether the interval for $\beta$ widens or narrows as a result.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This paper proposes OMIB, a multimodal information bottleneck method that adapts the per-modality regularization weight and derives a data-dependent upper bound on the redundancy coefficient β. The formulation is sensible: rather than tuning β and the modality weights by hand, compute them from the data. The empirical results on CREMA-D, MOSI, and the tissue-detection benchmarks are competitive, and the synthetic experiment showing a peak in accuracy near the proposed β bound is interesting.

What's actually new is the dynamic weight r (Eq. 11) that grows when a modality still has task-relevant information missing from the fused representation, and the explicit Mu bound. I don't see those in the cited MIB papers. The paper also tries to connect the objective to the five information-theoretic factors, which is a useful framing.

The soft spots, however, are load-bearing. The proof of Proposition 5.1 in Appendix B assumes a 'Markov property' ζ1⊥z2 and ζ2⊥z1, and drops the term I(ζ2; z1|ζ1). That independence is not implied by the generative model. Since v1 and v2 share a0 and b0, the encodings z1 and z2 are correlated, and the reparameterized ζ1, ζ2 inherit that correlation. The dropped term is positive, so L_OMF is a lower bound, not an upper bound, on the objective in Eq. (17). The β interval (0, Mu] derived for that objective therefore doesn't control what's actually minimized. The paper's central claim rests on this bridge, and it's broken.

Separately, the proofs of Lemmas 5.5 and 5.6 in Appendix C replace I(ξ;y) with the relative mutual information Ĩ(ξ;y) and then analyze that modified objective. They never show that the two objectives share the same optimizer. So the achievability theorem is about a different loss than the one trained. The set-algebra treatment of information atoms is also informal; it would need to be made rigorous.

The empirical reporting has issues. In Table 3, the best non-MIB method is BiGated (58.4), not concatenation (53.2), so the claimed improvement margin is wrong. In Table 4, OMIB ties E-MIB on Acc7, yet the text says it outperforms all baselines on all metrics. No error bars are given, and the mean tissue-detection numbers mask large per-dataset variation.

That said, the heuristic itself may have value. The dynamic r is a reasonable response to modality imbalance, and the gains on CREMA-D and MOSI are real. The synthetic validation is a nice idea, but it doesn't rescue the proof.

Bottom line: a sensible engineering idea with a broken proof of its central guarantee. The REJECT verdict is correct. I'd still send it to peer review, because the flaws are substantive and a careful referee could identify them clearly, possibly pointing the authors toward a fix or a weaker claim. It's also a good case study for a reading group on how a false independence assumption can invalidate a variational bound.

Referee Report

4 major / 4 minor

Summary. The paper proposes OMIB, a multimodal information bottleneck framework whose optimization objective allegedly guarantees the achievability of an optimal multimodal information bottleneck. The central theoretical claim is Proposition 5.7: under Assumption 5.3, optimizing the objective in Eq. (17) with a regularization weight β in (0, Mu], where Mu is derived from the entropies and mutual information of the input modalities, yields a representation ξ whose information set is exactly {a0, a1, a2}. The method adds a dynamically computed per-modality weight r and implements the objective via variational approximations (Eq. (10)). The paper reports experiments on synthetic data and several real-world multimodal benchmarks.

Significance. If the central guarantee were correct, the contribution would be valuable: it would replace ad hoc selection of IB regularization weights with a computable a priori bound and would explicitly address imbalance in task-relevant information across modalities. The synthetic experiment in Section 6.2 directly tests the β bound and shows accuracy peaking near Mu, which is a good falsifiable check, and the empirical comparisons are extensive. However, the theoretical bridge from Eq. (17) to the implemented loss (10) and to the proofs of Lemmas 5.5 and 5.6 contains load-bearing gaps, so the main guarantee is not established as written.

major comments (4)

[Appendix B, Eq. (40)] The derivation of the upper bound I(ξ; z1) ≤ I(ζ1; z1) rests on the asserted Markov property ζ2 ⊥ z1, but this independence is not implied by the generative model: under Assumption 5.3, v1 and v2 share a0 and b0, and after the reparameterization in Eq. (6) both ζ1 and ζ2 are noisy functions of these shared components, so I(ζ2; z1|ζ1) > 0 in general. Dropping this term makes L_OMF in Eq. (10) strictly smaller than the claimed variational upper bound on Eq. (17), so the β interval (0, Mu] derived for Eq. (17) cannot be applied to the objective that is actually minimized during training. This breaks the theory-to-objective link on which Proposition 5.7 relies.
[Appendix C, Definition C.1 and Lemmas C.2/C.4] The proofs of Lemmas 5.5 and 5.6 explicitly replace I(ξ; y) with the relative mutual information Î(ξ; y) "in the following analysis," but no argument is given that the two objectives share the same optima or that the same β bound Mu applies to the modified objective. Proposition 5.7 is stated for Eq. (17), which uses I(ξ; y), while the proof establishes properties of a different objective. This is a gap at the core of the achievability claim and is acknowledged in the manuscript's own text.
[Section 5.1, Proposition 5.2 and Eq. (11)] The dynamic weight r is computed from KL divergences between the model's own predictive distributions p(ŷ_i|ξ,z_i) and p(ŷ|ξ). Since these same distributions are being optimized in Eq. (10), the proof that Eq. (11) satisfies the proportionality in Eq. (18) assumes access to the true posterior p(y|·), which the model only approximates. The derived r is therefore self-referential; Proposition 5.2 does not establish the claimed behavior for the quantity actually evaluated during training.
[Appendix B, Eqs. (42)–(45), and Lemma 5.6] The variational loss in Eq. (10) regularizes the KL divergence of ζ_i with N(0,I), but the objective in Eq. (17) is written in terms of I(ξ; z_i). The injected noise ϵ in the reparameterization (6) can propagate to ξ without increasing either I(ξ; z1) or I(ξ; z2), so the exclusiveness proof in Lemma 5.6, which only considers superfluous information contained in v1 or v2, does not rule out noise-only components in ξ. Hence the claimed equality F(ξ) = {a0,a1,a2} is not enforced by the optimized loss.

minor comments (4)

[Section 5.2] The symbol Mu is used both in Lemma 5.5 as 1/((1+r)(H(v1)+H(v2)-I(v1;v2))) and then redefined in Section 5.2 as 1/(3(H(v1)+H(v2)-I(v1;v2))) without a distinct name; this will confuse readers about which bound Proposition 5.7 refers to.
[Section 5.1] There are typos: 'Langrange' should be 'Lagrange', and 'tahn' should be 'tanh'.
[Figure 3] The caption lists case iii as having F_rel(v1)=F_rel(v2) with d11(300)=d21(300), but the text in Section 6.2 says the balanced dataset is SIM-III; please make the correspondence between the three cases and the SIM datasets explicit in the caption.
[Equation (10)] The KL terms in Eq. (10) are written without a batch average; if they are averaged over the batch as the empirical expectation in the preceding line suggests, the notation should be consistent.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the beta bound is computed a priori, the r update is a variational estimator rather than a fitted prediction, and the Appendix C substitution of \hat I for I is a proof gap rather than a reduction to the paper's own inputs.

full rationale

The central claim (Proposition 5.7) is that optimizing Equation (17) with beta in (0, Mu], where Mu := 1/((1+r)(H(v1)+H(v2)-I(v1;v2))), achieves F(xi) = {a0,a1,a2}. This bound is obtained from entropy and mutual information estimates computed a priori from the training data using MINE, and the synthetic experiments test accuracy as beta crosses Mu; the bound is not fitted to the optimal-MIB outcome, so the achievability claim is not circular by construction. The dynamic weight r in Equation (11) is defined from the model's own predictive KL divergences, and Proposition 5.2's proof first derives an exact identity involving conditional mutual information and then replaces the true posteriors with model predictions; this is a variational estimator, not a fitted parameter renamed as a prediction, and the beta bound does not depend on r being an exact estimate. The clearest weakness is in Appendix C: the text states 'Consequently, we replace I(xi;y) with \hat I(xi;y) in Equation (17) in the following analysis,' and Lemmas 5.5 and 5.6 are proved for this modified relative-mutual-information objective. Because the paper does not show that the optima of I(xi;y) and \hat I(xi;y) coincide, the proof of Proposition 5.7 has a gap; however, this is an omitted-support / correctness problem, not a self-referential reduction, since \hat I is an independently defined quantity rather than an input to the derivation. Self-citations to the authors' prior work appear mainly for datasets, benchmarks, and implementation details (e.g., Xu et al. 2024b, Xu et al. 2025, Du et al. 2025, Li et al. 2024); none of these supply a load-bearing uniqueness theorem or ansatz that forces the paper's conclusions. The derivation chain is therefore not circular, though it is not fully sound.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The claimed guarantee depends on an idealized decomposition of modality information (Assumption 5.3), an exact sufficiency of the warmed-up encoders, an unproved independence between modality encodings, and an unproved equivalence of two different objective functions in the proof. These are load-bearing premises that are neither formally justified nor verified empirically.

free parameters (2)

u (upper bound for r) = 2
Set by hand in Section 5.1 to define the range of the dynamic weight r via a tanh mapping.
Tighter bound factor (1/3) = 1/3
Section 5.2 replaces the theoretical Mu with 1/(3(H(v1)+H(v2)-I(v1;v2))) and sets Ml = 1/(3(H(v1)+H(v2))) with no derivation; these hand-chosen constants define the beta sampling interval.

assumptions (5)

domain assumption Assumption 5.3: information-set decomposition into a0,a1,a2,b0,b1,b2 with y determined by {a0,a1,a2}.
This idealized Venn-diagram structure is assumed for the theory; real modalities may not decompose this way.
domain assumption Sufficiency of z_i: I(z_i;y)=I(v_i;y) is achieved by warm-up training.
Equation (2) states this as a constraint, but no guarantee that the trained encoder satisfies it exactly.
ad hoc to paper Markov property zeta1 perp z2 and zeta2 perp z1 in Proposition 5.1.
Used to drop cross terms in the variational bound; not justified when modalities are correlated.
ad hoc to paper Replacement of I(xi;y) by hat(I)(xi;y) in the proofs of Lemmas 5.5 and 5.6.
The text states this substitution 'in the following analysis' without proving equivalence of optima.
domain assumption MINE provides accurate estimates of H(v_i) and I(v1;v2).
The bound Mu is computed from MINE estimates; MINE is a neural estimator with approximation error.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Learning Optimal Multimodal Information Bottleneck Representations." pith.science (2026). https://pith.science/paper/B5JGPCMI

@misc{pith2026250519996,
  author       = {Pith},
  title        = {Pith review of: Learning Optimal Multimodal Information Bottleneck Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B5JGPCMI}},
  note         = {Machine review of arXiv:2505.19996}
}

read the original abstract

Leveraging high-quality joint representations from multimodal data can greatly enhance model performance in various machine-learning based applications. Recent multimodal learning methods, based on the multimodal information bottleneck (MIB) principle, aim to generate optimal MIB with maximal task-relevant information and minimal superfluous information via regularization. However, these methods often set ad hoc regularization weights and overlook imbalanced task-relevant information across modalities, limiting their ability to achieve optimal MIB. To address this gap, we propose a novel multimodal learning framework, Optimal Multimodal Information Bottleneck (OMIB), whose optimization objective guarantees the achievability of optimal MIB by setting the regularization weight within a theoretically derived bound. OMIB further addresses imbalanced task-relevant information by dynamically adjusting regularization weights per modality, promoting the inclusion of all task-relevant information. Moreover, we establish a solid information-theoretical foundation for OMIB's optimization and implement it under the variational approximation framework for computational efficiency. Finally, we empirically validate the OMIB's theoretical properties on synthetic data and demonstrate its superiority over the state-of-the-art benchmark methods in various downstream tasks.

Figures

Figures reproduced from arXiv: 2505.19996 by the authors.

**Figure 1.** a) Venn diagrams for two data modalities (v1 and v2). The gridded area represents consistent information, while the non-gridded area denotes modality-specific information. Task-relevant information is highlighted in green, whereas superfluous information is shown in blue. b) An optimal MIB should exclusively contain task-relevant, nonsuperfluous information (i.e., a0, a1 and a2) to be utilized in downstream tasks … view at source ↗

**Figure 2.** OMIB Framework. Here, ’C’ represents the concatenation operation. For the definitions of other notations, refer to the Section 4 and [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** The impact of β values on classification accuracy on synthetic data. v1 and v2 represent sample vectors of two modalities, respectively. Frel(·) denotes task-relevant information. “a” sub-vectors denote task-relevant information, while “b” superfluous information. d11 and d21 denote the dimensions of modality-specific a1 and a2. Mu is the computed β upper bound. . Lemma 5.5 (Inclusiveness of task-relevant informatio… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Runtime per epoch during warm-up and main [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Venn diagrams for three data modalities ( [PITH_FULL_IMAGE:figures/full_fig_p023_5.png]

**Figure 6.** Figure 6: Genomic multi-modal applications. Genomic data can be divided into two modalities: the histology modality [PITH_FULL_IMAGE:figures/full_fig_p036_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 14 canonical work pages

[3]

10x-hNB-{A-H}& 10x-hBC-{A-D}. The 10x-hNB-{A-H} datasets comprises eight datasets derived from healthy human breast tissues, while the 10x-hBC-{A-D} datasets contain four datasets from human breast cancer tissues (Xu et al., 2024b). As shown in Figure 6, each dataset corresponds to a tissue section and include gene expression and histology modalities. For...

work page 2018
[6]

Majority vote of diverse classifiers for late fusion

Morvant, E., Habrard, A., and Ayache, S. Majority vote of diverse classifiers for late fusion. In Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR Inter- national Workshop, S+ SSPR 2014, Joensuu, Finland, August 20-22,

work page 2014
[8]

C., and Bialek, W

Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method. arXiv preprint physics/0004057,

arXiv
[9]

Deep multi-view information bottleneck

Wang, Q., Boudreau, C., Luo, Q., Tan, P.-N., and Zhou, J. Deep multi-view information bottleneck. In Proceed- ings of the 2019 SIAM International Conference on Data Mining, pp. 37–45. SIAM,

work page 2019
[11]

iii) I(x1; x2; · · ·; xn+1) = I(x1; · · ·; xn) − I(x1; · · ·; xn|xn+1)

ii) I(x; y, z) = I(x; y) + I(x; z|y). iii) I(x1; x2; · · ·; xn+1) = I(x1; · · ·; xn) − I(x1; · · ·; xn|xn+1). iv) If F (x1) ∩ F (x2) = ∅ − →I(x1; x3|x2) = I(x1; x3) v) If F (v2) ⊆ F (v1) − →I(v1; v2) = H(v2), H(v1, v2) = F (v1) ∪ F (v2) = F (v1) = H(v1) vi) If H(v2) ∩ H(v1) ̸= ∅ − →H(v1, v2) = H(v1) + H(v2) − I(v1; v2) vii) If H(v2) ∩ H(v1) = ∅ − →H(v1, v...

work page 1999
[12]

to estimate the information entropy of each data modality and the mutual information between data modalities. Given two modalities X and Z, MINE employs a neural network, implemented as a two-layer Multi-Layer Perceptron (MLP) network with ReLU activation function (Belghazi et al., 2018), to learn a set of functions {Tθ}θ∈Θ. Each function Tθ : X × Z → R m...

work page 2018
[100]

The TRB loss LT RBis cross-entropy; • The CMU-MOSI dataset: The prediction head is implemented as a single linear layer MLP (input X 50 X 1), outputting a single real value

followed by a softmax layer for classification, producing ak-dimensional output, where k is the number of classification types. The TRB loss LT RBis cross-entropy; • The CMU-MOSI dataset: The prediction head is implemented as a single linear layer MLP (input X 50 X 1), outputting a single real value. LT RBis mean squared error; • The 10x-hNB-{A-H} and 10x...

work page 2018
[200]

This configuration implies that Modality I dominates Modality II in terms of task-relevant information

For SIM-I used in case i, we set d11(500) ≫ d21(100) so that a1 has a significantly greater impact on determining y, compared to a2. This configuration implies that Modality I dominates Modality II in terms of task-relevant information. For SIM-II used in case ii, we switch the setting of d11 and d12, making Modality II dominant over Modality I. Finally, ...

work page 2014

Show all 16 references

[256]

with LeakyReLU activation functions, producing 256-dimensional latent multimodal representations. LT RBis defined as: LT RB= 1 N NX i=1 ∥ˆy − c∥2 + λ · R(Θ), c = 1 N NX i=1 ˆy, (104) where ˆy denotes the output of the prediction head, c the center of the hypersphere,R(Θ) the f...

2019
[2002]

Dual advancement of representation learning and clustering for sparse and noisy images

Li, W., Xu, Y ., Zheng, X., Han, S., Wang, J., and Sun, X. Dual advancement of representation learning and clustering for sparse and noisy images. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1934–1942,

1934
[2015]

Temporal graph convolutional network for multimodal sentiment analysis

Huang, J., Lin, Z., Yang, Z., and Liu, W. Temporal graph convolutional network for multimodal sentiment analysis. In Proceedings of the 2021 International Conference on Multimodal Interaction, pp. 239–247,

2021
[2016]

Tensor fusion network for multimodal sentiment analysis

Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.-P. Tensor fusion network for multimodal sentiment analysis. In Palmer, M., Hwa, R., and Riedel, S. (eds.), Proceedings of the 2017 Conference on Empirical Meth- ods in Natural Language Processing , pp. 1103–1114, Cop...

2017
[2018]

Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114,

arXiv
[2021]

and Zaslavsky, N

Tishby, N. and Zaslavsky, N. Deep learning and the infor- mation bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1–5. IEEE,

2015
[2023]

Dynamic multimodal infor- mation bottleneck for multimodality classification

9 Learning Optimal Multimodal Information Bottleneck Representations Fang, Y ., Wu, S., Zhang, S., Huang, C., Zeng, T., Xing, X., Walsh, S., and Yang, G. Dynamic multimodal infor- mation bottleneck for multimodality classification. In 2024 IEEE/CVF Winter Conference on Applica...

2024
[2024]

A methodolog- ical framework for measuring spatial labeling similarity

Du, Y ., Hu, J., Hou, S., Ding, Y ., and Sun, X. A methodolog- ical framework for measuring spatial labeling similarity. arXiv preprint arXiv:2505.14128,

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [3]

10x-hNB-{A-H}& 10x-hBC-{A-D}. The 10x-hNB-{A-H} datasets comprises eight datasets derived from healthy human breast tissues, while the 10x-hBC-{A-D} datasets contain four datasets from human breast cancer tissues (Xu et al., 2024b). As shown in Figure 6, each dataset corresponds to a tissue section and include gene expression and histology modalities. For...

work page 2018

[2] [6]

Majority vote of diverse classifiers for late fusion

Morvant, E., Habrard, A., and Ayache, S. Majority vote of diverse classifiers for late fusion. In Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR Inter- national Workshop, S+ SSPR 2014, Joensuu, Finland, August 20-22,

work page 2014

[3] [8]

C., and Bialek, W

Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method. arXiv preprint physics/0004057,

arXiv

[4] [9]

Deep multi-view information bottleneck

Wang, Q., Boudreau, C., Luo, Q., Tan, P.-N., and Zhou, J. Deep multi-view information bottleneck. In Proceed- ings of the 2019 SIAM International Conference on Data Mining, pp. 37–45. SIAM,

work page 2019

[5] [11]

iii) I(x1; x2; · · ·; xn+1) = I(x1; · · ·; xn) − I(x1; · · ·; xn|xn+1)

ii) I(x; y, z) = I(x; y) + I(x; z|y). iii) I(x1; x2; · · ·; xn+1) = I(x1; · · ·; xn) − I(x1; · · ·; xn|xn+1). iv) If F (x1) ∩ F (x2) = ∅ − →I(x1; x3|x2) = I(x1; x3) v) If F (v2) ⊆ F (v1) − →I(v1; v2) = H(v2), H(v1, v2) = F (v1) ∪ F (v2) = F (v1) = H(v1) vi) If H(v2) ∩ H(v1) ̸= ∅ − →H(v1, v2) = H(v1) + H(v2) − I(v1; v2) vii) If H(v2) ∩ H(v1) = ∅ − →H(v1, v...

work page 1999

[6] [12]

to estimate the information entropy of each data modality and the mutual information between data modalities. Given two modalities X and Z, MINE employs a neural network, implemented as a two-layer Multi-Layer Perceptron (MLP) network with ReLU activation function (Belghazi et al., 2018), to learn a set of functions {Tθ}θ∈Θ. Each function Tθ : X × Z → R m...

work page 2018

[7] [100]

The TRB loss LT RBis cross-entropy; • The CMU-MOSI dataset: The prediction head is implemented as a single linear layer MLP (input X 50 X 1), outputting a single real value

followed by a softmax layer for classification, producing ak-dimensional output, where k is the number of classification types. The TRB loss LT RBis cross-entropy; • The CMU-MOSI dataset: The prediction head is implemented as a single linear layer MLP (input X 50 X 1), outputting a single real value. LT RBis mean squared error; • The 10x-hNB-{A-H} and 10x...

work page 2018

[8] [200]

This configuration implies that Modality I dominates Modality II in terms of task-relevant information

For SIM-I used in case i, we set d11(500) ≫ d21(100) so that a1 has a significantly greater impact on determining y, compared to a2. This configuration implies that Modality I dominates Modality II in terms of task-relevant information. For SIM-II used in case ii, we switch the setting of d11 and d12, making Modality II dominant over Modality I. Finally, ...

work page 2014