{"id":"1d7bf37b-8c45-43c8-a6ef-8f12457ca809","arxiv_id":"2412.00833","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AlignMamba fuses audio, video, and language by matching tokens to a language anchor and enforcing distribution similarity, reporting small accuracy gains with large efficiency gains on MOSI and MOSEI.","lead":"A new neural network design for combining audio, video, and text adds a token-matching step and a distribution-matching loss to the fast 'Mamba' model family. On two sentiment-analysis benchmarks it reports slightly better accuracy and much lower memory and inference cost than Transformer-based rivals.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The incomplete-fusion results depend on an unspecified missing-modality input procedure, and the robustness claim in Table 1 cannot be evaluated until that procedure is disclosed and ablated.","rationale":"The reader's weakest_assumption is exactly the load-bearing issue. The complete-fusion accuracy gains are small (about 0.9 points on MOSI) and the efficiency gains are architecture-level and plausible, so the distinguishing empirical contribution is the incomplete-fusion robustness result. That result is not reproducible or interpretable from the paper because the missing-data branch of the model is never specified. The issue is not merely a missing implementation detail: it changes how the alignment modules behave. In Eq. (5), the selected column for each row depends on the cost matrix; if missing positions are inserted as zeros, those zero columns become cheap targets for the cosine cost, so the transport plan can map real tokens to missing positions. If missing tokens are instead dropped, the sequence in Eq. (11) is no longer aligned by timestep, and the MMD loss in Eq. (10) is computed over sequences of different lengths. The language-anchor dependency makes the problem more concrete: when language is missing, the entire local alignment procedure in Eqs. (5)-(6) has no anchor and must be replaced by something. None of these branches appear in Sections 3.1-3.4. The comparison to IMDer and GCNet is therefore uninterpretable until the protocol is disclosed and, ideally, ablated. I am not moving the verdict to REJECT: the complete-data method is well specified, the efficiency measurements are consistent with Mamba's linear-complexity design, and a simple disclosure plus two-condition ablation would settle whether the robustness claim holds. The 'OT' framing in Eq. (4)-(5) is also inflated since the relaxed problem reduces to a per-row nearest-neighbor assignment, but that concern affects the novelty narrative more than the central empirical claim. Thus the appropriate verdict remains conditional, pending the missing-protocol clarification.","tokens_in":12355,"tokens_out":5016,"duration_ms":51430,"concrete_test":"Obtain from the authors the exact missing-modality pipeline used for Table 1: (a) whether missing token positions are zero-filled, replaced with a learned mask token, or dropped before the OT alignment in Eqs. (5)-(6); (b) what anchor is used when language is missing; and (c) whether the model was trained with random modality dropout. Then re-run Table 1 with two implementations, zero-filling missing modalities versus dropping their tokens, while keeping all other hyperparameters fixed. If the average accuracies or the delta values change by more than about one percentage point, the robustness claim is protocol-dependent and the comparison to recovery baselines is not apples-to-apples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim bundles complete and incomplete fusion, but the strongest part of that claim, the robustness result in Table 1, rests on an unstated input-construction rule. Section 3.4 defines the fused sequence Xmm in Eq. (11) as an interleaving of aligned audio, video, and language tokens at every timestep. It does not define how this sequence is formed when a modality is absent, yet Table 1 reports results at missing rates from 10% to 70% and compares against recovery-based baselines such as GCNet and IMDer. If missing tokens are zero-padded, the argmin in Eq. (5) can assign available tokens to arbitrary zero positions, and the MMD loss in Eq. (10) is computed over distributions with different effective support. If missing tokens are dropped, the temporal-priority interleaving in Eq. (11) no longer preserves timestep alignment. The local alignment module also treats language as the anchor in Section 3.2, so an additional fallback is required whenever language is the missing modality; no such fallback is described. Without knowing which protocol was used, the reported average accuracy of 79.9% on MOSI and 81.6% on MOSEI, and especially the smaller performance drop compared with recovery methods, may be an artifact of the protocol choice rather than a genuine robustness advantage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AlignMamba, a Mamba-based multimodal fusion method that adds two alignment mechanisms before feeding unimodal features into a Mamba backbone: an optimal-transport-inspired local alignment module that matches tokens across modalities, and an MMD-based global alignment loss that aligns feature distributions. The method is evaluated on CMU-MOSI and CMU-MOSEI for complete multimodal fusion and for incomplete fusion under 10%–70% missing modality rates, reporting improved accuracy and large reductions in GPU memory and inference time relative to Transformer baselines. The paper also includes ablations of the alignment components and a qualitative transport-plan analysis.","tokens_in":12618,"tokens_out":3922,"duration_ms":37029,"significance":"If the reported results are reproducible, AlignMamba offers a practical way to bring the linear-complexity benefits of Mamba to multimodal fusion while mitigating Mamba's limited cross-modal interaction through explicit token-level and distribution-level alignment. The efficiency gains (e.g., 83.3% inference-time reduction at 6.4k tokens) are substantial and well aligned with the paper's motivation. However, the paper's central empirical claims, especially the incomplete-fusion robustness results in Table 1, rest on an unstated missing-modality handling procedure, and the complete-fusion results lack essential training details. The conceptual contribution is clear, but the experimental verification is currently incomplete. The paper would be strengthened by a precise description of how missing modalities are represented, the exact training protocol, and the values of the key hyperparameters (lambda, sigma, architecture sizes).","major_comments":[{"comment":"The term 'optimal transport' is used loosely. The relaxation in Eq. (4) removes the column-sum constraint, and the closed-form solution in Eq. (5) is a nearest-neighbor assignment of each row to its minimal-cost column, not a coupling between the two marginal distributions. This is a valid algorithmic choice, but the paper's claim to be 'grounded in Optimal Transport' is overstated. The authors should either clarify that they use a one-sided assignment relaxation or discuss the relationship to OT more precisely. This issue does not invalidate the empirical results, but it affects the framing of the method.","section":"Section 3.2"}],"minor_comments":[{"comment":"Equation (8) uses the notation T for the number of samples in both sequences, but the aligned sequences have length T_l while the original audio and video sequences may have different lengths. The notation should be adjusted to avoid ambiguity.","section":"Section 3.3"},{"comment":"In Eq. (4), the relaxed constraints are written with a single row-sum constraint; it would be clearer to also explicitly state the non-negativity constraint and the range of indices, as done in Eq. (2).","section":"Section 3.2"},{"comment":"The modality ablation for 'w/o Language' mentions that 'we only align the audio modality with the video modality,' but the corresponding procedure is not described in Section 3. This is related to the missing-modality protocol issue and should be clarified.","section":"Section 4.4"},{"comment":"Figures 3 and 4 are referenced but the actual plots are not visible in the manuscript text; please ensure the figures are included and labeled with clear axes and legends.","section":"Section 4.3"},{"comment":"The A-distance results in Table 4 are computed on the model's own learned representations. While this is a reasonable diagnostic, the caption or text should note that this measures internal consistency rather than an external property of the data.","section":"Section 4.5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is simple and potentially useful, but the experimental validation is not currently at the level expected for a conference paper. The missing-modality handling must be specified, and full training and hyperparameter details should be added. If the authors provide these, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"If you work on Mamba-based multimodal fusion, this paper is worth your time. The new thing is the specific pipeline: nearest-neighbor token alignment, framed as relaxed optimal transport, plus an MMD distribution loss feeding an interleaved Mamba backbone. The combination, and the explicit alignment-for-Mamba angle, is not in the cited literature. The complete-fusion gains are small but consistent, 0.9% on MOSI and 0.5% on MOSEI, and the efficiency story is the more credible headline.\n\nCredit where it is due. The ablations are honestly constructed and informative: dropping the local alignment costs 2.3%, dropping the global one costs 1.1%, and the modality ablations confirm that language is the anchor. The efficiency measurements (GPU memory, inference time, FLOPs) are consistent with the linear-versus-quadratic complexity story. The math checks out: after removing the incoming-flow constraint, the closed-form argmin in Eq. (5) is the exact solution to the relaxed LP, and the paper discloses the relaxation with a citation to Kusner et al. Calling the result an 'optimal transport plan' oversells what is really a nearest-neighbor assignment, but the derivation is not circular. The citation pattern looks normal.\n\nThe load-bearing problem is the incomplete-fusion evaluation. Section 3.4 defines Xmm as an interleaving of all three modalities at every timestep, and the paper never says what happens when a modality is absent. Table 1 reports numbers at 10% to 70% missing rates against recovery-based baselines like IMDer and GCNet, but the input-construction rule is unstated. Zero-padding lets the argmin in Eq. (5) bind available tokens to arbitrary zero positions and distorts the MMD loss; dropping tokens breaks the timestep interleaving; and if language is missing, the alignment anchor disappears with no fallback. The modality ablation says the model aligns audio with video when text is removed, but that is a different setting and does not resolve the ambiguity. The headline robustness claim, the smaller performance drop, may be an artifact of the protocol choice. This is a fixable reporting gap, but as written it blocks verification.\n\nSecondary issues: no code, no hyperparameters (loss weight, kernel bandwidth, Mamba depth and width), no training protocol. The A-distance table measures the model's own learned representations, so it is internal consistency rather than external validation; fine as a sanity check, not load-bearing.\n\nWho this is for: people building Mamba-based fusion systems who want a cleanly described, well-ablated baseline with believable efficiency numbers. It deserves a serious referee; the method is coherent, the writing is clear, and the efficiency claims are checkable. My verdict is conditional: accept the complete-fusion and efficiency parts, and require the missing-modality protocol to be disclosed and ablated before the robustness claims are taken at face value. If I were the editor, I would send it out.","headline":"Credible Mamba-fusion architecture with believable efficiency numbers and honest ablations, but the incomplete-fusion protocol is never described, leaving the headline robustness claim unverifiable as written.","tokens_in":13135,"tokens_out":9362,"would_cite":true,"duration_ms":74108,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AlignMamba claims that adding optimal-transport-based token matching and a distribution-alignment loss before a Mamba backbone lets a linear-complexity model match or beat Transformer-based multimodal fusion, reporting higher accuracy on…","keywords":["multimodal fusion","Mamba","state space models","optimal transport","maximum mean discrepancy","cross-modal alignment","incomplete multimodal fusion","sentiment analysis"],"falsifier":"Compare AlignMamba against its own two variants (local-only and global-only) on the incomplete-fusion benchmark while varying how missing inputs are constructed — masking with zeros, removing tokens, and learned recovery — and check whether the reported 73.8% accuracy at 70% missing on CMU-MOSI holds under the same input-construction protocol used by GCNet and IMDer.","tokens_in":12156,"feed_emoji":"🔗","tokens_out":5163,"duration_ms":64909,"temperature":0.7,"pith_summary":"AlignMamba sets out to fix a specific weakness of Mamba-based multimodal fusion: Mamba's sequential scanning processes tokens in order, so it misses cross-modal correspondences between unscanned tokens. The paper proposes two complementary alignment mechanisms applied before the Mamba backbone: a local, optimal-transport-based module that matches tokens across modalities, and a global Maximum Mean Discrepancy loss that aligns whole modality distributions. On CMU-MOSI and CMU-MOSEI sentiment benchmarks, the resulting model reports higher accuracy than previous multimodal fusion methods while using less GPU memory and much less inference time. A reader should care because the method points to a way of getting Transformer-level cross-modal modeling at linear-complexity cost.","feed_headline":"Two alignment passes let Mamba beat Transformer multimodal fusion","feed_subtitle":"On CMU-MOSI it gains 0.9% accuracy, cuts GPU memory 20.3% and inference time 83.3%.","key_machinery":"The load-bearing components are the relaxed optimal transport matrix (a nearest-neighbor assignment under cosine distance, following the relaxed OT of Kusner et al. [12]) that projects audio and video tokens onto the language token grid, and the squared Maximum Mean Discrepancy with a Gaussian kernel, used as an auxiliary loss to align the projected distributions with language. The third component is the time-priority interleaving of the aligned unimodal tokens into a single sequence for the Mamba selective scan, which lets the sequential model see cross-modal neighbors at each timestep.","core_discovery":"The central discovery is that explicit cross-modal alignment before a Mamba backbone lets a state space model match or exceed Transformer-based multimodal fusion performance without quadratic attention cost. AlignMamba treats the audio and video feature sequences as discrete distributions and learns a relaxed optimal transport plan that maps each audio and video token to its nearest language token under cosine distance, producing aligned sequences at the language length; an MMD loss then pulls the aligned distributions together in a reproducing kernel Hilbert space. The aligned sequences are interleaved by timestep and fed to Mamba, whose selective scan then processes neighbors from different modalities in sequence. The paper reports that on CMU-MOSI this recipe raises binary classification accuracy to 86.9% (a 0.9% absolute gain over the best prior result it compares against) while reducing GPU memory by 20.3% and inference time by 83.3% relative to Transformer baselines at 6.4k tokens, and that on incomplete fusion with 10-70% modality dropout it retains higher average accuracy and a smaller drop than recovery-based baselines.","pith_inferences":["A natural test the paper leaves open is training with modality dropout so the alignment modules experience missing inputs during training; such a protocol would clarify whether the reported incomplete-fusion robustness comes from the alignment losses or from an unstated input-construction choice.","Because the OT alignment anchors audio and video to the language token grid, the method implicitly assumes language is the most informative modality; for tasks without a dominant modality, an anchor-selection or symmetric alignment rule would be needed, which the paper does not address.","The relaxed OT plan is essentially a row-wise nearest-neighbor assignment, so the local alignment module can be seen as a sparse, interpretable attention that could be transplanted into other backbones or used to initialize fusion without extra parameters."],"forward_implications":["On CMU-MOSI and CMU-MOSEI, AlignMamba reports the highest binary accuracy and F1 among the compared methods for both complete and incomplete fusion, with average incomplete accuracy of 79.9% and 81.6% respectively.","The dual alignment (OT local + MMD global) is the source of the gain: removing either component lowers accuracy on both datasets, and removing the local module costs more (2.3% on MOSI) than removing the global one (1.1%).","Replacing the Mamba fusion with single-stream or multi-stream Mamba variants without explicit alignment hurts performance, meaning naive Mamba scanning does not by itself capture cross-modal correspondences.","At 6.4k tokens, AlignMamba uses 8.53 GB GPU memory versus 10.7 GB for single-stream and 20.3 GB for multi-stream Transformers, and 6.05 s inference time versus 36.13 s and 48.61 s; at 1024 tokens it needs 46.7G FLOPs versus 101.6G and 203.2G."],"supporting_citations":[{"why":"Supplies the Mamba selective state space backbone with linear complexity that AlignMamba builds on.","marker":"[3]"},{"why":"Provides the relaxed optimal transport formulation that makes the token-level alignment tractable.","marker":"[12]"},{"why":"Defines the Transformer baseline whose quadratic complexity motivates the efficiency comparison.","marker":"[31]"},{"why":"A cross-modal Transformer method that AlignMamba compares against and whose attention mechanism it aims to replace.","marker":"[30]"},{"why":"A recovery-based incomplete-multimodal baseline that AlignMamba claims to outperform on missing-modality robustness.","marker":"[34]"},{"why":"Another incomplete-fusion recovery baseline in the comparison table.","marker":"[18]"},{"why":"Provides the CMU-MOSI dataset used for the main accuracy and efficiency evaluations.","marker":"[39]"},{"why":"Provides the CMU-MOSEI dataset used for the second evaluation benchmark.","marker":"[40]"}],"fun_headline_variants":["AlignMamba: local + global alignment boosts Mamba multimodal fusion","AlignMamba: OT alignment makes Mamba beat Transformer fusion","Local+global alignment gives Mamba an edge in multimodal fusion","AlignMamba: two-stage alignment beats Transformer fusion efficiency","AlignMamba: 0.9% accuracy gain, 83% inference speedup over Transformers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The incomplete-fusion experiments assume a specific way of presenting partially missing inputs to AlignMamba, but the paper never states whether missing modalities are zero-padded, dropped, or filled by a recovery module; if the comparison baselines use explicit recovery and AlignMamba simply ignores or zeros the missing stream, the reported robustness could be an artifact of that unevaluated choice.","fun_headline_variants_meta":{"raw":{"variants":["AlignMamba: local + global alignment boosts Mamba multimodal fusion","AlignMamba: OT alignment makes Mamba beat Transformer fusion","Local+global alignment gives Mamba an edge in multimodal fusion","AlignMamba: two-stage alignment beats Transformer fusion efficiency","AlignMamba: 0.9% accuracy gain, 83% inference speedup over Transformers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000735,"raw_usage":{"total_tokens":3282,"prompt_tokens":935,"completion_tokens":2347,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":2252}},"tokens_in":551,"tokens_out":2347,"duration_ms":16904,"temperature":1.0,"reasoning_tokens":2252,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:56:57.035014+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare AlignMamba against its own two variants (local-only and global-only) on the incomplete-fusion benchmark while varying how missing inputs are constructed — masking with zeros, removing tokens, and learned recovery — and check whether the reported 73.8% accuracy at 70% missing on CMU-MOSI holds under the same input-construction protocol used by GCNet and IMDer.","supporting_citations":[{"cited_title":"From word embeddings to document distances","cited_arxiv_id":null,"evidence_quote":"Provides the relaxed optimal transport formulation that makes the token-level alignment tractable."},{"cited_title":"Multimodal transformer for unaligned multimodal language sequences","cited_arxiv_id":null,"evidence_quote":"A cross-modal Transformer method that AlignMamba compares against and whose attention mechanism it aims to replace."},{"cited_title":"Incomplete multimodality-diffused emotion recognition","cited_arxiv_id":null,"evidence_quote":"A recovery-based incomplete-multimodal baseline that AlignMamba claims to outperform on missing-modality robustness."},{"cited_title":"Gcnet: Graph completion network for incomplete mul- timodal learning in conversation","cited_arxiv_id":null,"evidence_quote":"Another incomplete-fusion recovery baseline in the comparison table."},{"cited_title":"Multimodal lan- guage analysis in the wild: Cmu-mosei dataset and inter- pretable dynamic fusion graph","cited_arxiv_id":null,"evidence_quote":"Provides the CMU-MOSEI dataset used for the second evaluation benchmark."}],"review_version":1}