{"id":"69237213-4245-4837-b442-40eb89989b96","arxiv_id":"2501.01240","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"ARM, a mutual-information-based asymmetric reinforcement method, narrows modality contribution gaps and reports improved accuracy on three multimodal classification datasets.","lead":"This paper presents ARM, a method that measures how much each input modality contributes to a multimodal model using mutual information, then rebalances the model by adjusting fusion weights, losses, and sampling. On three benchmarks, ARM reports accuracy gains over previous imbalanced multimodal learning methods, but its theoretical derivation contains unresolved gaps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The valuation metric is not well-defined: Eq. (11) introduces p(fxj→y) without defining it, and Eq. (10) subtracts normalized CMI from unnormalized MI, so phi_CMI(xi) need not measure marginal contribution; DFF, BMML, and DSR all inherit this undefined quantity.","rationale":"The reader's weakest_assumption correctly identifies the same load-bearing concern: the valuation metric must reflect true modality contribution, and Eq. (11) is the place where this fails. The paper provides code, ablations, and results on three datasets, which are real evidence of an empirical method, but the theoretical valuation on which the method is built is internally inconsistent. Specifically, p(fxj→y) is undefined, Eq. (10) mixes normalized and unnormalized information measures, and Eq. (7)'s min-based 'lower bound' is asserted rather than derived. These issues are not stylistic; they affect every downstream component: dynamic feature fusion, balanced min-max loss, and dynamic resampling all take phi_CMI as input. If phi_CMI does not measure marginal contribution, the method is not doing what the paper claims. Theorem 1's proof also fails to establish pairwise distinctness of modal contributions, but the more fundamental problem is Eq. (11), because even a correct theorem would not define p(fxj→y). I therefore agree with the reader's REJECT verdict and see no reason to adjust it. The revision path remains as the reader stated: define the estimator, fix the proof, report variance, and clarify how p(fxj→y) is obtained in the released implementation.","tokens_in":16735,"tokens_out":4875,"duration_ms":48445,"concrete_test":"Download the released code and trace the exact computation of Eq. (11) on one training batch. Identify where p(fxj→y) is computed and verify that it is a probability derived from a defined per-modality head or an explicitly documented alternative; also verify that I(fY;fxi) and NCMI(fY;fxj|fxi) are normalized on the same scale and that phi_CMI(xi) values fall in the claimed [0,m] range. If p(fxj→y) cannot be located or is replaced by an undocumented heuristic, or if phi_CMI(xi) is negative or outside [0,m], Eq. (11) is not the implemented valuation and the central claim of the paper fails as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MIV estimates each modality's marginal contribution and the joint contribution per sample, and that optimizing the losses in Eqs. (14) and (15) plus dynamic resampling in Eq. (18) narrows contribution gaps while preserving dominant-modality performance. The least secure point is Eq. (11): phi_CMI(xi) = p(fY→y) I(fY;fxi) + sum_{j≠i} p(fxj→y) II(fY;fxj;fxi). The term p(fxj→y) is never defined; no per-modality classifier or probability distribution over y given fxj is specified anywhere in the paper. Without such a definition, phi_CMI(xi) is not an operational estimator. In addition, Eq. (10) defines II as I(fY;fxj) − NCMI(fY;fxj|fxi), but standard interaction information subtracts CMI, not normalized CMI, and the paper first says 'we use I to represent NMI', so the subtraction mixes normalized and unnormalized scales. Consequently, phi_CMI(X) in Eq. (12) is not a well-defined information-theoretic quantity, and the fusion weights in Eq. (13), the MAE loss in Eq. (15), and the resampling frequency in Eq. (18) all optimize a quantity that may be unrelated to true modality contribution. Theorem 1's proof also fails to establish the stated pairwise inequality—it only shows monotonicity of set-wise mutual information—but even if Theorem 1 were true, Eq. (11) would still need a definition before MIV can be used. This is an internal inconsistency, not merely a disagreement with prior methods; the empirical gains could plausibly come from the regularization or resampling effects rather than from the claimed contribution valuation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ARM, an asymmetric reinforcement method for imbalanced multimodal learning. The method defines a mutual-information-based valuation metric (MIV) to estimate each modality's marginal contribution and the joint contribution of all modalities per sample, and then uses these scores to (i) set dynamic feature-level fusion weights, (ii) define a balanced min-max loss comprising L_phiMI and L_phiCMI, and (iii) perform dynamic sample-level resampling. Experiments on Kinetics Sounds, UCF-51, and Food-101 report accuracy improvements over several imbalanced-learning, fusion, and MI-based baselines, with ablations of the loss terms and of the three ARM components.","tokens_in":17163,"tokens_out":10243,"duration_ms":103503,"significance":"The paper has strengths that should be acknowledged: a broad empirical evaluation across three datasets, comparison with many recent baselines, ablation studies of the loss terms and components, integration with existing fusion backbones, and a public code link. If the MIV scores were well-defined and correctly derived, the approach of directly balancing modality contributions through information-theoretic quantities would be of interest. However, the central theoretical foundation is not established: several load-bearing definitions are mathematically inconsistent or incomplete, and the claimed proof does not prove the stated theorem. The empirical gains may be real, but the paper's core claim that the method optimizes true modality contributions is not supported by the current text.","major_comments":[{"comment":"The quantity defined in Eq. (4), I(fY=y; fX) = sum_x P(x|y) log(P(y|x)/P(y)), is not the mutual information of Eq. (1); the summand is a pointwise log-ratio and can be negative. The sentence following Eq. (1) states that MI is non-negative and hence has a positive contribution, but that property does not apply to the expression in Eq. (4). Since Theorem 1 and the Appendix proof in Eq. (19) use the same I(fY=y; .) notation, the theorem is not stated about a well-defined non-negative information measure.","section":"Methods, Eq. (4)"},{"comment":"Eq. (5) is vacuous: because sum_hat-y p(fY -> hat-y) = 1, the right-hand side is identically I(fY; fxi), so it does not express an expected contribution over all possible predictions. In addition, the text announces that 'we use I to represent NMI' after Eq. (5), and then Eq. (10) defines II as I(fY; fxj) - NCMI(fY; fxj | fxi). Interaction information requires subtracting ordinary CMI, not a normalized version, so the resulting quantity is not the interaction information claimed and has no established information-theoretic meaning. The fusion weights, the loss in Eq. (15), and the resampling frequency in Eq. (18) all inherit this quantity.","section":"Methods, Eqs. (5) and (10)"},{"comment":"The term p(fxj -> y) in Eq. (11) is never defined. No per-modality classifier, probability distribution over y given fxj, or estimation procedure is specified anywhere in the paper. Because Eq. (11) is the definition of phi_CMI(xi), the dynamic fusion weight in Eq. (13), the MAE loss in Eq. (15), and the resampling frequency in Eq. (18) all depend on an undefined quantity. As written, the core modules of ARM are not reproducible.","section":"Methods, Eq. (11)"},{"comment":"The 'lower bound of joint contribution' is asserted without proof. No definition of joint contribution is given before Eq. (7), and no argument shows that p(fY -> y) * min_i I(fY; fxi) bounds the contribution of the complete modality set to accurate prediction. The subsequent properties of phi_MI and the claim that maximizing phi_MI via the loss in Eq. (14) increases a lower bound of joint contribution are therefore unsupported.","section":"Methods, Eq. (7)"},{"comment":"The proof of Theorem 1 does not establish the stated result. The derivation in Eq. (19) shows only that I(Y; A) - I(Y; B) >= 0 for sets A and B with B subset of A, via a KL divergence. It does not prove strict inequality, and it does not prove the pairwise assertion I(fY=y; fxi) != I(fY=y; fxj) for arbitrary i != j. The proof also uses the notation I(Y=y; .) inherited from Eq. (4), which, as noted above, is not a well-defined mutual information.","section":"Appendix A, Theorem 1"},{"comment":"Table 5 reports that DSR with k = -3.0 attains 67.41% on KS, 76.83% on UCF-51, and 93.69% on Food-101, all higher than the ARM results used in Tables 1 and 2 (66.52, 75.60, and 93.36, respectively). The main text does not explain why the reported ARM configuration uses k = -2.0 instead of the better-performing k = -3.0. This inconsistency affects the state-of-the-art claims and the choice of the final model.","section":"Appendix C, Table 5"}],"minor_comments":[{"comment":"The denominators in Eqs. (2) and (9) appear to contain a typesetting error: they should be sqrt(H(X)H(Y)) and sqrt(H(X|Z)H(Y|Z)), respectively, rather than the current 'p H(...)' notation.","section":"Methods, Eqs. (2) and (9)"},{"comment":"In Algorithm 1, 'Initialize Drs = D' appears inside the epoch loop, which resets the resampled dataset at the start of every epoch. Please clarify whether this is intended and, if so, how resampling frequencies from previous epochs are accumulated or reinitialized.","section":"Appendix B, Algorithm 1"},{"comment":"On UCF-51, MLA achieves 76.01%, which is higher than ARM's 75.60%. The statement that ARM 'consistently outperforms other state-of-the-art methods' is therefore inaccurate; the later sentence acknowledging MLA's strength on UCF-51 should be reconciled with that claim.","section":"Experiments, Tables 1 and 3"},{"comment":"The LCE-only baseline in Table 4 reports 63.88% on KS, whereas the Concatenation baseline in Table 1 reports 59.61%. Please state what fusion or training setup the ablation baseline uses so that the reader can interpret the ablation gains.","section":"Experiments, Table 4"},{"comment":"The log-sum-exp expression in Eq. (16) is a standard smooth approximation to max_i(-I_i), but the text describes it as making the min function differentiable. The sign convention should be stated explicitly to avoid confusion: max_i(-I_i) = -min_i I_i, and the log-sum-exp approximates the former.","section":"Methods, Eq. (16)"}],"recommendation":"reject","confidential_remarks":"The empirical study is broad and the authors provide code, but the central theoretical apparatus is not sound as submitted. The undefined p(fxj -> y) alone prevents the core modules from being reproduced, and the other issues in Eqs. (4), (5), (7), and (10) are load-bearing rather than cosmetic. If the authors substantially rework the valuation metric, add operational definitions, and correct the proof, a future version could be considered; in its current form I cannot recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What should you know about this paper? It proposes ARM, a plug-in for imbalanced multimodal learning that combines mutual information valuation, conditional MI interaction, dynamic feature fusion, balanced min-max loss, and resampling. It reports consistent gains over strong baselines on KS, UCF-51, and Food-101, and also improves other fusion frameworks. If the valuation were sound, this would be a useful contribution to the subfield.\n\nThe genuinely new piece is the combination of these ingredients: the idea of using interaction information to estimate each modality's marginal contribution per sample and then asymmetrically reinforce weak modalities is a reasonable design intuition, and the experimental coverage is solid for a paper of this type. The code is out, which helps. Credit is also due for the integration experiments with MMTM, CentralNet, and other fusion methods; those improvements are consistent and nontrivial.\n\nThe soft spots are not minor. The valuation metric, which every component depends on, has definitional problems:\n\n- Eq. (5) is vacuous: it multiplies the NMI by N, and the claimed identity doesn't hold.\n- Eq. (10) subtracts normalized CMI from unnormalized MI (or NMI, depending on how you read the text), so the interaction information is not a well-defined information-theoretic quantity.\n- Eq. (11) uses p(fxj→y), a modality-specific probability that is never defined anywhere in the paper. Without it, phi_CMI is not an operational estimator.\n- The proof of Theorem 1 only shows that adding a modality to a set increases the set-level quantity; it does not show pairwise inequality between two individual modalities, which is what the theorem claims.\n- The smooth-min approximation in Eq. (16) is missing a minus sign; min_i I_i ≈ -log(∑ e^{-I_i}), not log(∑ e^{-I_i}).\n- The reported main results use k=-2.0 in the resampling function, but Table 5 shows k=-3.0 gives better numbers on all three datasets. That discrepancy, absent error bars, suggests the hyperparameter is being chosen on test accuracy.\n\nThere is also a circularity concern: L_phiMI includes the true-class softmax probability, so part of the ablation gain is just confidence maximization, not valuation-driven balancing. That said, the accuracy gains are large and consistent across backbones, so I don't think the empirical story is fake; it's just not yet supported by the claimed mechanism.\n\nWho is this for? Researchers working on modality imbalance will find the empirical results interesting and the flaws instructive. The paper deserves a serious referee, not because the math is sound, but because the idea and experiments are worth sorting out. A thorough review should ask for a grounded estimator, a correct proof or a weaker claim, hyperparameter selection on validation, and error bars.\n\nRecommendation: send it to peer review and let the referees push for the necessary corrections. As is, it's not acceptable; but it's a close call, and the fix path is clear.","headline":"A novel combination of MI-based valuation and asymmetric reinforcement that shows real empirical gains, but the valuation math is not well-defined; as written, the central mechanism doesn't hold together.","tokens_in":17658,"tokens_out":7262,"would_cite":false,"duration_ms":63131,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A mutual-information valuation of each modality's per-sample contribution lets a training method reinforce weak modalities while keeping dominant ones strong.","keywords":["multimodal learning","modality imbalance","mutual information","conditional mutual information","asymmetric reinforcement","modality forgetting","dynamic feature fusion","sample resampling"],"falsifier":"Compare the per-sample contribution scores $\\phi_{\\mathrm{CMI}}(x_i)$ that ARM computes against the actual accuracy drop observed when modality $i$ is removed (e.g., zeroed or ablated) at test time on Kinetics Sounds or Food-101; if the scores do not rank-order the ablation-induced drops, the valuation metric is not tracking true modality contribution and the whole reinforcement mechanism is optimizing a proxy.","tokens_in":16526,"feed_emoji":"⚖️","tokens_out":9913,"duration_ms":79974,"temperature":0.7,"pith_summary":"This paper introduces ARM, a training procedure that rebalances multimodal learning by measuring, sample by sample, how much each modality contributes to the model's correct prediction. The central claim is that mutual information between each modality's features and the fused output, combined with conditional mutual information between modalities, gives a reliable per-sample valuation of marginal and joint contribution; ARM then uses this valuation to dynamically weight the fusion, add a balanced min-max loss, and resample low-contribution samples more often. The paper argues that this asymmetric reinforcement narrows the contribution gap between modalities without degrading the dominant one, avoiding the modality forgetting that prior weak-modality-boosting methods exhibit. On Kinetics Sounds, UCF-51, and Food-101, the method reports the best accuracies among the compared methods, 66.52%, 75.60%, and 93.36%, and shows that the joint contribution of all modalities increases steadily during training.","feed_headline":"Mutual-information training balances modalities and lifts accuracy","feed_subtitle":"On audio-video and image-text benchmarks, ARM keeps dominant-modal performance while narrowing contribution gaps.","key_machinery":"The central object is the mutual-information valuation metric (MIV), which converts model outputs into per-sample contribution scores. For each modality it computes normalized mutual information $I(f_Y; f_{x_i})$ between the fused feature and that modality's feature, scaled by $p(f_Y\\to y)$, the probability that the fusion predicts the true label; the minimum over modalities gives a lower bound on the joint contribution (Eq. 7). The asymmetric marginal contribution adds interaction information $\\mathrm{II}(f_Y; f_{x_j}; f_{x_i})$ between modality pairs, so each modality's score reflects both its own information and the extra information another modality provides given it (Eq. 11). These scores are the inputs to the three reinforcement mechanisms: fusion weights (Eq. 13), the balanced min-max loss (Eqs. 14-15) with a log-sum-exp smooth approximation of the min (Eq. 16), and the resampling schedule (Eq. 18). The min operator is made differentiable exactly so that the lower-bound loss can be optimized by gradient descent.","core_discovery":"ARM's central claim is that modality imbalance can be corrected from a multimodal perspective rather than by strengthening weak modalities in isolation. The paper defines a mutual-information valuation metric (MIV): the marginal contribution of modality $i$ is $\\phi_{\\mathrm{MI}}(x_i) = p(f_Y \\to y)\\,I(f_Y; f_{x_i})$, with $I$ the normalized mutual information between the fused feature $f_Y$ and the modality feature, and the joint contribution of all modalities is lower-bounded by $\\phi_{\\mathrm{MI}}(X) = p(f_Y \\to y)\\,\\min_i I(f_Y; f_{x_i})$. To capture inter-modality interaction, it adds an asymmetric term built from conditional mutual information, $\\phi_{\\mathrm{CMI}}(x_i) = p(f_Y \\to y)I(f_Y; f_{x_i}) + \\sum_{j\\neq i} p(f_{x_j}\\to y)\\,\\mathrm{II}(f_Y; f_{x_j}; f_{x_i})$, where $\\mathrm{II}$ is the interaction information $I(f_Y;f_{x_j}) - \\mathrm{NCMI}(f_Y; f_{x_j}\\mid f_{x_i})$. These valuations drive three mechanisms: dynamic feature-level fusion weights $FW_i = \\phi_{\\mathrm{CMI}}(x_i)/\\phi_{\\mathrm{CMI}}(X)$, a balanced min-max loss $L = L_{\\mathrm{CE}} + \\lambda_1 (1 - \\phi_{\\mathrm{MI}}(X)) + \\lambda_2 \\frac{\\sum_i |\\phi_{\\mathrm{CMI}}(x_i)-\\phi_{\\mathrm{CMI}}(X)|}{\\phi_{\\mathrm{CMI}}(X)}$, and dynamic sample-level resampling with frequency $s(X)=F_s(\\phi_{\\mathrm{CMI}}(X))$ decreasing in joint contribution. The paper reports that on audio-visual (Kinetics Sounds, UCF-51) and image-text (Food-101) benchmarks this yields accuracies of 66.52%, 75.60%, and 93.36%, exceeding prior imbalanced-learning and fusion methods while keeping dominant-modality accuracy above the concatenation and summation baselines.","pith_inferences":["Because the valuation operates on fused and per-modality features rather than on class labels, the same reinforcement scheme could in principle be applied to multimodal regression, retrieval, or generation, though the paper only evaluates classification.","The lower-bound identity $\\phi_{\\mathrm{MI}}(X) = p(f_Y\\to y)\\min_i I(f_Y; f_{x_i})$ is a strong structural assumption; the paper does not empirically verify that this min equals the true joint contribution, so a natural test is to compare ARM's scores with leave-one-modality-out accuracy drops.","If the valuation metric is faithful, the method suggests that balancing should mean equalizing marginal contributions rather than equalizing loss magnitudes or gradient norms, which are the targets of earlier gradient-modulation methods.","The reported sensitivity to the resampling slope k (best at k=-3.0 in the appendix) indicates that the main paper's default k=-2.0 was a conservative choice; readers should expect the optimal k to be dataset- and backbone-dependent."],"forward_implications":["Adding ARM to standard concatenation or summation fusion raises accuracy by roughly 6 to 11 points across the three benchmarks (KS +6.91, UCF-51 +7.37, Food-101 +10.98), according to the paper's table of fusion-backbone gains.","The ablation study attributes gains to all three components: dynamic feature-level fusion alone improves KS from 59.61% to 64.34%, adding balanced min-max loss raises it further, and dynamic sample-level resampling gives the largest single jump on UCF-51 (to 74.29%).","ARM can be attached to other fusion backbones such as MMTM and CentralNet, improving their accuracy on all three datasets by 3.5 to 5.7 points, which suggests the reinforcement strategy is architecture-agnostic.","The appendix reports that more aggressive resampling (k = -3.0) yields even higher accuracy than the main results (e.g., 67.41% on KS), implying the headline numbers may not be the method's ceiling.","Contribution-gap curves and single-modality accuracy tables indicate that ARM narrows the gap between modalities while keeping the dominant modality's accuracy above the concatenation or summation baseline, i.e., no modality forgetting."],"supporting_citations":[{"why":"Supplies the entropy and mutual-information definitions (Eqs. 1-2) on which the whole valuation metric is built.","marker":"(Cover 1999)"},{"why":"Provides the multivariate information-decomposition perspective that justifies computing MI between the fusion output and each modality separately rather than as one joint expression.","marker":"(Williams and Beer 2010)"},{"why":"Introduces sample-level modality valuation and inverse contribution-based resampling, which ARM extends from a unimodal to a multimodal perspective with its own valuation metric.","marker":"(Wei et al. 2024)"},{"why":"Provides the log-sum-exp inequality used to make the min operator in Eq. (7) differentiable in Eq. (16), without which the lower-bound loss cannot be optimized by backpropagation.","marker":"(Nielsen and Sun 2016)"},{"why":"Characterizes the greedy nature of multimodal learning and supplies the Greedy baseline that motivates the imbalance problem and against which ARM is compared.","marker":"(Wu et al. 2022)"}],"fun_headline_variants":["Conditional MI balances modalities, keeping strong ones strong","ARM: asymmetric MI training lifts benchmarks fairly","Win-win for modalities via conditional mutual information","MI-based reinforcing narrows modal gaps, not just weak boost","Asymmetric MI balancing boosts accuracy without losing strength"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-sample mutual-information scores computed from the model's own outputs genuinely capture how much each modality contributes to getting the label right; if those scores are measuring something else, the fusion weights, losses, and resampling are all tuned to the wrong quantity.","fun_headline_variants_meta":{"raw":{"variants":["Conditional MI balances modalities, keeping strong ones strong","ARM: asymmetric MI training lifts benchmarks fairly","Win-win for modalities via conditional mutual information","MI-based reinforcing narrows modal gaps, not just weak boost","Asymmetric MI balancing boosts accuracy without losing strength"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000301,"raw_usage":{"total_tokens":1843,"prompt_tokens":1163,"completion_tokens":680,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":779,"completion_tokens_details":{"reasoning_tokens":607}},"tokens_in":779,"tokens_out":680,"duration_ms":7429,"temperature":1.0,"reasoning_tokens":607,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:32:04.265950+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the per-sample contribution scores $\\phi_{\\mathrm{CMI}}(x_i)$ that ARM computes against the actual accuracy drop observed when modality $i$ is removed (e.g., zeroed or ablated) at test time on Kinetics Sounds or Food-101; if the scores do not rank-order the ablation-induced drops, the valuation metric is not tracking true modality contribution and the whole reinforcement mechanism is optimizing a proxy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces sample-level modality valuation and inverse contribution-based resampling, which ARM extends from a unimodal to a multimodal perspective with its own valuation metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the log-sum-exp inequality used to make the min operator in Eq. (7) differentiable in Eq. (16), without which the lower-bound loss cannot be optimized by backpropagation."}],"review_version":1}