Pith. sign in

REVIEW 3 major objections 5 minor 70 references

Class Similarity-Based Multimodal Classification under Heterogeneous Category Sets

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows a multimodal classifier trained on partially overlapping audio and video label sets can recognize the union of classes via semantic alignment, uncertainty-based modality choice, and class-similarity fusion.

desk verdict A new and practical problem setting with a sensible first solution; the empirical gains are large, but the lack of error bars and the thin validation of the uncertainty-based dominance rule need referee attention. read the letter →

arxiv 2506.09745 v1 pith:27ZGQNJT submitted 2025-06-11 cs.CV

Yangrui Zhu , Junhua Bao , Yipan Wei , Yapeng Li , Bo Du This is my paper
classification cs.CV
keywords multimodalclassificationheterogeneouscategorysetszero-shotlearninguncertaintyestimationclass-similarityfusionaudio-visualrecognitionsemanticalignmentopen-set
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a new task format, Multi-Modal Heterogeneous Category-set Learning (MMHCL), in which each training modality sees only a subset of the class space, yet the resulting model must recognize the union of all classes at test time. It claims this is the first work to treat category-set mismatch across modalities as a learnable problem rather than an obstacle. The paper presents CSCF, which aligns each modality's features to a shared semantic space built from LLM-enriched class descriptions, uses ensemble uncertainty to pick the more reliable modality for each sample, and fuses the weaker modality through class-similarity reweighting. On ActivityNet, UCF, and VGGSound, the method reports mixed-scenario accuracy gains of 10.23, 10.77, and 3.30 percentage points over the strongest competing approach. If this holds, multimodal systems can be assembled from separately collected, separately labeled datasets in one training session.

What carries the argument

The load-bearing machinery is the semantic space plus the uncertainty-decided fusion. Class names are expanded by a large language model into descriptive sentences and embedded with CLIP's text encoder to produce prototypical semantic features; each modality's backbone features are mapped into this space by a learned mapper, and classification scores are cosine similarities between mapped features and all class prototypes, forming the OSRS module. Four mappers per modality form an ensemble whose per-sample cross-entropy spread defines intra-modality inconsistency; combined with the entropy of averaged predictions across modalities, this yields an uncertainty $u^*$ used to declare the dominant modality. The CSMF module then prunes the class-similarity matrix to the top-$k$ neighbors per class and reweights the auxiliary modality's logits before adding them to the dominant logits, so the final decision is driven by the modality that was trained on the true class.

What would settle it

Take a held-out set with at least a few hundred samples per class-side, compute fused top-1 accuracy, and re-compute it after forcing the higher-uncertainty modality to be dominant; if the forced-wrong choice does not lower accuracy substantially on classes where the entropy rule guessed wrong, the uncertainty claim is empty. More sharply: on samples whose true class was seen only by the modality with higher $u$, the fused accuracy should beat that modality alone; if it does not, the rule is falsified.

Watch

Extended reading notes

Core claim

The central claim is that category heterogeneity between modalities is not a failure case to be patched but a learnable setting. A model trained on modality A's subset of classes and modality B's subset can recognize every class across both modalities, provided three ingredients are present: a shared semantic space that carries class meaning, built from LLM-expanded class descriptions embedded by CLIP; a per-sample estimate of which modality is expert for the test class, computed as entropy-based uncertainty over an ensemble of sub-modules; and a fusion rule that lets the non-expert modality contribute only through class-similarity-weighted logits. The paper reports that this combination outperforms seven state-of-the-art methods from generalized zero-shot learning, multimodal classification, and incomplete multi-view learning, with the largest gains on scenarios where one modality must classify classes it never saw.

Load-bearing premise

The method assumes that lower entropy-based uncertainty reliably marks the modality that was trained on the true class of a test sample, so choosing that modality as dominant is usually correct; the paper checks this on only 30 randomly selected samples per comparison, and if the rule fails on many samples the fusion step has no fallback.

Editorial extensions

If this is right

  • A single CSCF training session can ingest audio-only and video-only datasets with different class lists and produce one classifier covering the union of both lists.
  • Semantic alignment through LLM-generated class descriptions gives each modality a path to classes it never saw, so unseen-class accuracy stops being zero for the non-expert modality.
  • Uncertainty-based dominant-modality selection suppresses unseen-class noise from the auxiliary modality, which the paper shows is the main failure of confidence-based selection in unimodal zero-shot baselines.
  • Class-similarity reweighting lets the auxiliary modality add informative logit mass to semantically related classes, improving fusion accuracy beyond simply averaging or picking the more confident model.
  • Existing multimodal, incomplete-view, and generalized zero-shot methods lose performance under MMHCL because they either require full category coverage or cannot fuse cross-modal information; the proposed protocol exposes that gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the uncertainty rule is a fixed statistic rather than a learned module, it should transfer to more than two modalities by taking the argmin over per-modality uncertainties; this follows naturally from the paper's design but is not tested there.
  • The class-similarity top-$k$ fusion suggests a testable extension: learn the per-class $k$ or weight the similarity matrix by dataset-specific confusions, which could recover performance lost when semantically close but visually distinct classes are neighbors.
  • A broader consequence the paper leaves implicit is that if LLM-enriched semantics are a sufficient bridge, datasets collected for different tasks can be combined without re-annotating labels, turning MMHCL into a dataset-engineering strategy.
  • The evaluation splits each dataset's classes evenly and strictly disjointly between modalities; the setting would be stress-tested by partially overlapping label sets, unequal class counts, and three or more modalities, which the paper does not experiment with.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Multi-Modal Heterogeneous Category-set Learning (MMHCL), where each modality is trained on a different subset of categories and the model must classify the union of categories at test time. It proposes CSCF, composed of (i) an Open Set Recognition via Semantics module that maps modality features into a shared CLIP/LLM semantic space and classifies by scaled cosine similarity, (ii) a Dominant Modality Selection Strategy that combines intra-modality ensemble entropy and inter-modality entropy to pick a dominant modality, and (iii) a Class Similarity-guided Multimodal Fusion module that prunes a class-similarity matrix to top-k and adds the auxiliary modality's similarity-weighted logits to the dominant modality's logits. Experiments on ActivityNet, UCF, and VGGSound compare against seven ZSL/MML/IMV baselines and report accmix gains of 10.23, 10.77, and 3.30 percentage points, with ablations showing each module contributes incrementally.

Significance. The MMHCL setting is practically motivated and, among the cited baselines, not directly addressed: standard multimodal methods assume complete category coverage, and incomplete-multi-view methods handle only sample-level missing modalities. The method avoids circularity with the training data by using external CLIP/CLAP features and LLM-generated class descriptions, and the ablation study shows consistent incremental gains from OSRS, DMSS, and CSMF across three datasets. The claimed gains, if robust, would make this a useful baseline for research on combining modality data with heterogeneous label sets. However, the validation of the dominant-modality selection mechanism is thin, and the absence of statistical error bars and full hyperparameter disclosure currently prevents the central claim from being fully supported.

major comments (3)
  1. [§4.4, Eq. (7)–(9), Appendix E] The DMSS decision rule is load-bearing: Eq. (9) hard-switches the fusion on the comparison of uA and uB, where u* = Inc* + dif*. The only validation of this rule is the qualitative inspection of 30 randomly selected samples in Figures 7 and 9; no full-test selection accuracy, per-case confusion matrix, or comparison against an oracle selector is reported. The paper's own Appendix E additionally shows that a modality that never saw a class can produce overconfident predictions on some samples, which is exactly the failure mode that would corrupt the hard switch. Please report the dominant-modality selection accuracy on the full test sets, and provide an ablation that replaces Eq. (9) with an oracle selector to quantify the headroom lost to misselection.
  2. [Table 1] The central claim that CSCF 'significantly outperforms' state-of-the-art methods rests on single numbers with no error bars, no number of seeds, and no significance tests. Since the reported accmix improvements differ across datasets (10.23, 10.77, and 3.30 points) and the top-k hyperparameter is tuned per dataset, the reader cannot distinguish a robust advantage from favorable hyperparameter selection. Please report mean and standard deviation over at least three random seeds per dataset and, if appropriate, paired significance tests or confidence intervals.
  3. [§3.3, Appendix D] Several components that the fusion equations depend on are underspecified. The number K of OSRS modules, the logit scale γ in Eq. (2), and the top-k values in the pruned similarity matrices S* are never given; Appendix D only lists feature dimensions, optimizer, epochs, and batch size. Moreover, Eq. (8)–(9) do not state whether the rows of S* are normalized (e.g., softmax over the retained neighbors) or whether the diagonal is kept, which changes the scale of the added term relative to the dominant logits. Please specify all hyperparameter values per dataset, describe the K OSRS module architectures (Appendix B mentions 'four OSRS modules with different architectures' without details), and give a sensitivity analysis for top-k and γ on all three datasets.
minor comments (5)
  1. [§3.2] The heading 'Consturction of DMSS' should read 'Construction of DMSS'.
  2. [§4.1] The heading 'Implement Details' should read 'Implementation Details', and 'Muti-View' should be 'Multi-View'.
  3. [Table 1 and Appendix C] The notation 'Aall+Ball' appears in Table 1 but is not defined in Appendix C, which defines only As, Bs, Au, Bu, As+Bu, and Bs+Au; please define it explicitly.
  4. [Appendix E, Figure 11] Figure 11 is based on twenty randomly sampled instances and reports no axis label for the horizontal axis beyond 'Sample Index'; please clarify the sampling procedure and consider reporting a larger sample or error bars.
  5. [Algorithm 1] The input line of Algorithm 1 lists only {X A, YA s } before the colon, whereas the problem formulation and surrounding text include modality-B data; please make the notation consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CSCF is an empirical pipeline whose inputs (external CLIP/LLM semantics, model-prediction uncertainty) do not encode the reported accmix gains by construction.

full rationale

The derivation chain is self-contained with respect to the reported claims. OSRS logits are cosine similarities between mapped modality features and externally produced CLIP/LLM class semantics (Eqs. 1-2); the semantic bridge is not derived from the target accmix numbers or from the test labels. DMSS uncertainty (Eqs. 4-7) is computed from the model's own ensemble predictions and is validated as an empirical heuristic on samples, not fitted to maximize Table 1. CSMF fusion (Eqs. 8-9) combines logits using a semantic similarity matrix fixed by the same external class features; the experimental gains are an empirical outcome, not forced by the equations. The only author self-citations are to prior works used as comparison baselines (UVaT [3], DSECN [8]); neither is invoked as a justifying premise for the method's design or as an authority for its claims. No equation reduces to a fitted parameter renamed as a prediction, and no uniqueness or ansatz is imported through a self-citation chain. Even the uncertainty heuristic's limited 30-sample validation and per-dataset top-k selection are empirical robustness concerns, not circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method depends on external semantic representations (CLIP/LLM) and a few hand-set hyperparameters. No new physical entities or ungrounded theoretical constructs are introduced.

free parameters (3)
  • top-k = tuned per dataset (e.g., values analyzed for UCF in Fig. 4)
    Controls the sparsity of the class-similarity matrix in CSMF; the paper shows performance varies with top-k.
  • K (number of OSRS modules) = 4
    The ensemble size for uncertainty estimation; set arbitrarily to 4 without sensitivity analysis.
  • gamma (logit scale in Eq. 2) = not specified
    The temperature scaling factor for cosine similarity logits; affects prediction distributions but its value is not reported.
assumptions (3)
  • domain assumption CLIP text embeddings of LLM-enhanced class names form a reliable semantic space for seen-to-unseen transfer
    OSRS and CSMF rely on these embeddings for all class comparisons; no calibration against a held-out semantic metric is provided.
  • domain assumption Entropy-based uncertainty u* identifies the dominant modality (the one that saw the class in training)
    DMSS uses u* to select the dominant modality; Figures 7 and 9 show this on 30 samples per setting, not on full test sets.
  • domain assumption Training with cross-entropy over all classes (including unseen) does not destroy zero-shot generalization
    Eq. (10) penalizes logits of unseen classes during training; the paper shows this works empirically but provides no analysis of gradient effects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Class Similarity-Based Multimodal Classification under Heterogeneous Category Sets." pith.science (2026). https://pith.science/paper/27ZGQNJT

@misc{pith2026250609745,
  author       = {Pith},
  title        = {Pith review of: Class Similarity-Based Multimodal Classification under Heterogeneous Category Sets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/27ZGQNJT}},
  note         = {Machine review of arXiv:2506.09745}
}
read the original abstract

Existing multimodal methods typically assume that different modalities share the same category set. However, in real-world applications, the category distributions in multimodal data exhibit inconsistencies, which can hinder the model's ability to effectively utilize cross-modal information for recognizing all categories. In this work, we propose the practical setting termed Multi-Modal Heterogeneous Category-set Learning (MMHCL), where models are trained in heterogeneous category sets of multi-modal data and aim to recognize complete classes set of all modalities during test. To effectively address this task, we propose a Class Similarity-based Cross-modal Fusion model (CSCF). Specifically, CSCF aligns modality-specific features to a shared semantic space to enable knowledge transfer between seen and unseen classes. It then selects the most discriminative modality for decision fusion through uncertainty estimation. Finally, it integrates cross-modal information based on class similarity, where the auxiliary modality refines the prediction of the dominant one. Experimental results show that our method significantly outperforms existing state-of-the-art (SOTA) approaches on multiple benchmark datasets, effectively addressing the MMHCL task.

Figures

Figures reproduced from arXiv: 2506.09745 by the authors.

Figure 1
Figure 1. Setting illustration and results: (a) Setup for MMHCL. During the training phase, each modality has [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of Settings. Unimodal Zero-shot learning (UZSL) focuses on seen classes and unseen classes [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overall framework of the proposed CSCF model. In the data processing phase, semantic enhancement is [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Analysis of the hyperparameter top-k on the UCF dataset. "w/o class similarity" means no similarity fusion is used, while "w/ class similarity" indicates that similarity fusion is applied. We gradually increase the top-k to observe its impact on the model’s fusion capa…
Figure 5
Figure 5. Figure 5: Comparison of the model’s ability to recognize unseen classes before and after introducing the OSRS [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: The impact of OSRS(O), DMSS(D), and CSMF(C) on the comprehensive performance of the model. We [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Comparison of prediction uncertainty across modalities. The horizontal axis represents 30 randomly selected [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Comparison of the model(B+O)’s ability to recognize unseen classes before and after incorporating semantic [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: The predictions of each OSRS module within individual modalities are illustrated. The x-axis represents the [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Visualization of the existence states of multimodal data during training and evaluate. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Confidence levels of two zero-shot models trained separately on audio and video modalities during full [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

70 extracted references · 63 canonical work pages

  1. [1]

    Deconstructed generation-based zero-shot model

    Dubing Chen, Yuming Shen, Haofeng Zhang, and Philip HS Torr. Deconstructed generation-based zero-shot model. In AAAI, volume 37, pages 295–303, 2023

    work page 2023

  2. [2]

    Trusted multi-view classification with dynamic evidential fusion

    Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification with dynamic evidential fusion. IEEE TPAMI, 45(2):2551–2566, 2023

    work page 2023

  3. [3]

    Uvat: Uncertainty incorporated view-aware transformer for robust multi-view classification

    Yapeng Li, Yong Luo, and Bo Du. Uvat: Uncertainty incorporated view-aware transformer for robust multi-view classification. IEEE Transactions on Image Processing, 33:5129–5143, 2024

    work page 2024

  4. [4]

    Trusted multi-view learning with label noise, 2024

    Cai Xu, Yilin Zhang, Ziyu Guan, and Wei Zhao. Trusted multi-view learning with label noise, 2024

    work page 2024

  5. [5]

    Incomplete multi-view multi-label learning via label-guided masked view-and category-aware transformers

    Chengliang Liu, Jie Wen, Xiaoling Luo, and Yong Xu. Incomplete multi-view multi-label learning via label-guided masked view-and category-aware transformers. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 8816–8824, 2023

    work page 2023

  6. [6]

    Uncertainty-aware pseudo-labeling and dual graph driven network for incomplete multi-view multi-label classification

    Wulin Xie, Xiaohuan Lu, Yadong Liu, Jiang Long, Bob Zhang, Shuping Zhao, and Jie Wen. Uncertainty-aware pseudo-labeling and dual graph driven network for incomplete multi-view multi-label classification. In ACM MM, MM ’24, page 6656–6665, New York, NY , USA, 2024. Association for Computing Machinery

    work page 2024

  7. [7]

    Deep incomplete multi-view network semi-supervised multi-label learning with unbiased loss

    Quanjiang Li, Tingjin Luo, Mingdie Jiang, Jiahui Liao, and Zhangqi Jiang. Deep incomplete multi-view network semi-supervised multi-label learning with unbiased loss. In ACM MM, MM ’24, page 9048–9056, New York, NY , USA, 2024. Association for Computing Machinery

    work page 2024

  8. [8]

    Improving generalized zero-shot learning by exploring the diverse semantics from external class names

    Yapeng Li, Yong Luo, Zengmao Wang, and Bo Du. Improving generalized zero-shot learning by exploring the diverse semantics from external class names. In CVPR, pages 23344–23353, June 2024

    work page 2024

Show all 70 references
  1. [9]

    Causal visual-semantic correlation for zero-shot learning

    Shuhuang Chen, Dingjie Fu, Shiming Chen, Shuo Ye, Wenjin Hou, and Xinge You. Causal visual-semantic correlation for zero-shot learning. InACM MM, MM ’24, page 4246–4255, New York, NY , USA, 2024. Association for Computing Machinery

    2024

  2. [10]

    Avgzslnet: Audio-visual generalized zero-shot learning by reconstructing label features from multi-modal embeddings

    Pratik Mazumder, Pravendra Singh, Kranti Kumar Parida, and Vinay P Namboodiri. Avgzslnet: Audio-visual generalized zero-shot learning by reconstructing label features from multi-modal embeddings. In WACV, pages 3090–3099, 2021

    2021

  3. [11]

    Audio-visual generalised zero-shot learning with cross-modal attention and language

    Otniel-Bogdan Mercea, Lukas Riesch, A Koepke, and Zeynep Akata. Audio-visual generalised zero-shot learning with cross-modal attention and language. In CVPR, pages 10553–10563, 2022

    2022

  4. [12]

    Hyperbolic audio-visual zero-shot learning

    Jie Hong, Zeeshan Hayder, Junlin Han, Pengfei Fang, Mehrtash Harandi, and Lars Petersson. Hyperbolic audio-visual zero-shot learning. In ICCV, pages 7873–7883, 2023

    2023

  5. [13]

    Sophia Koepke, and Zeynep Akata

    David Kurzendörfer, Otniel-Bogdan Mercea, A. Sophia Koepke, and Zeynep Akata. Audio-visual generalized zero-shot learning using pre-trained large multi-modal models. In CVPR, pages 2627–2638, June 2024

    2024

  6. [14]

    Deep multimodal learning: A survey on recent advances and trends

    Dhanesh Ramachandram and Graham W Taylor. Deep multimodal learning: A survey on recent advances and trends. IEEE signal processing magazine, 34(6):96–108, 2017

    2017

  7. [15]

    Multimodal machine learning: A survey and taxonomy

    Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE TPAMI, 41(2):423–443, 2019

    2019

  8. [16]

    Survey on deep multi-modal data analytics: Collaboration, rivalry, and fusion

    Yang Wang. Survey on deep multi-modal data analytics: Collaboration, rivalry, and fusion. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 17(1s):1–25, 2021

    2021

  9. [17]

    Multimodal classification: Current landscape, taxonomy and future directions

    William C Sleeman IV , Rishabh Kapoor, and Preetam Ghosh. Multimodal classification: Current landscape, taxonomy and future directions. ACM Computing Surveys, 55(7):1–31, 2022

    2022

  10. [18]

    Towards balanced active learning for multimodal classification

    Meng Shen, Yizheng Huang, Jianxiong Yin, Heqing Zou, Deepu Rajan, and Simon See. Towards balanced active learning for multimodal classification. In ACM MM, MM ’23, page 3434–3445, New York, NY , USA, 2023. Association for Computing Machinery

    2023

  11. [19]

    Multimodal classification of urban micro-events

    Maarten Sukel, Stevan Rudinac, and Marcel Worring. Multimodal classification of urban micro-events. In ACM MM, MM ’19, page 1455–1463, New York, NY , USA, 2019. Association for Computing Machinery

    2019

  12. [20]

    Intra- and inter-modal curriculum for multimodal learning

    Yuwei Zhou, Xin Wang, Hong Chen, Xuguang Duan, and Wenwu Zhu. Intra- and inter-modal curriculum for multimodal learning. In ACM MM, MM ’23, page 3724–3735, New York, NY , USA, 2023. Association for Computing Machinery

    2023

  13. [21]

    Multimodal classification of violent online political extremism content with graph convolutional networks

    Stevan Rudinac, Iva Gornishka, and Marcel Worring. Multimodal classification of violent online political extremism content with graph convolutional networks. In ACM MM, Thematic Workshops ’17, page 245–252, New York, NY , USA, 2017. Association for Computing Machinery. 12 A PR...

    2017

  14. [22]

    Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis

    Soujanya Poria, Erik Cambria, and Alexander Gelbukh. Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In Proceedings of the 2015 conference on empirical methods in natural language processing, pa...

    2015

  15. [23]

    Multimodal deep learning and visible-light and hyperspectral imaging for fruit maturity estimation

    Cinmayii A Garillos-Manliguez and John Y Chiang. Multimodal deep learning and visible-light and hyperspectral imaging for fruit maturity estimation. Sensors, 21(4):1288, 2021

    2021

  16. [24]

    Gated multimodal units for information fusion

    John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multimodal units for information fusion. arXiv preprint arXiv:1702.01992, 2017

    2017 arXiv

  17. [25]

    Supervised multimodal bitransformers for classifying images and text

    Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXiv:1909.02950, 2019

    1909 arXiv

  18. [26]

    Multimodal transformer for unaligned multimodal language sequences

    Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhut- dinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume...

    2019

  19. [27]

    More diverse means better: Multimodal deep learning meets remote-sensing imagery classification

    Danfeng Hong, Lianru Gao, Naoto Yokoya, Jing Yao, Jocelyn Chanussot, Qian Du, and Bing Zhang. More diverse means better: Multimodal deep learning meets remote-sensing imagery classification. IEEE Transactions on Geoscience and Remote Sensing, 59(5):4340–4354, 2021

    2021

  20. [28]

    Multimodal transformer fusion for continuous emotion recognition

    Jian Huang, Jianhua Tao, Bin Liu, Zheng Lian, and Mingyue Niu. Multimodal transformer fusion for continuous emotion recognition. In ICASSP, pages 3507–3511. IEEE, 2020

    2020

  21. [29]

    Deep multimodal fusion by channel exchanging

    Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang. Deep multimodal fusion by channel exchanging. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, NeurIPS, volume 33, pages 4835–4845. Curran Associates, Inc., 2020

    2020

  22. [30]

    Unit: Multimodal multitask learning with a unified transformer

    Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified transformer. In ICCV, pages 1439–1449, 2021

    2021

  23. [31]

    What makes multi-modal learning better than single (provably)

    Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). In NeurIPS, volume 34, pages 10944–10956, 2021

    2021

  24. [32]

    A variational information bottleneck approach to multi-omics data integration

    Changhee Lee and Mihaela Van der Schaar. A variational information bottleneck approach to multi-omics data integration. In International Conference on Artificial Intelligence and Statistics, pages 1513–1521. PMLR, 2021

    2021

  25. [33]

    Multi-modal multi-instance learning using weakly correlated histopathological images and tabular clinical information

    Hang Li, Fan Yang, Xiaohan Xing, Yu Zhao, Jun Zhang, Yueping Liu, Mengxue Han, Junzhou Huang, Liansheng Wang, and Jianhua Yao. Multi-modal multi-instance learning using weakly correlated histopathological images and tabular clinical information. In MICCAI, pages 529–539. Sprin...

    2021

  26. [34]

    Multimodal feature fusion for robust event detection in web videos

    Pradeep Natarajan, Shuang Wu, Shiv Vitaladevuni, Xiaodan Zhuang, Stavros Tsakalidis, Unsang Park, Rohit Prasad, and Premkumar Natarajan. Multimodal feature fusion for robust event detection in web videos. In CVPR, pages 1298–1305, 2012

    2012

  27. [35]

    Uncertainty- aware audiovisual activity recognition using deep bayesian variational inference

    Mahesh Subedar, Ranganath Krishnan, Paulo Lopez Meyer, Omesh Tickoo, and Jonathan Huang. Uncertainty- aware audiovisual activity recognition using deep bayesian variational inference. In ICCV, pages 6301–6310, 2019

    2019

  28. [36]

    Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification

    Zongbo Han, Fan Yang, Junzhou Huang, Changqing Zhang, and Jianhua Yao. Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification. In CVPR, pages 20707–20717, June 2022

    2022

  29. [37]

    Trusted unified feature-neighborhood dynamics for multi-view classification

    Haojian Huang, Chuanyu Qin, Zhe Liu, Kaijing Ma, Jin Chen, Han Fang, Chao Ban, Hao Sun, and Zhongjiang He. Trusted unified feature-neighborhood dynamics for multi-view classification. arXiv preprint arXiv:2409.00755, 2024

    2024 arXiv

  30. [38]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of M...

    2016

  31. [39]

    Evidential deep learning to quantify classification uncertainty

    Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In NeurIPS, volume 31, 2018

    2018

  32. [40]

    Greedy policy search: A simple baseline for learnable test-time augmentation

    Alexander Lyzhov, Yuliya Molchanova, Arsenii Ashukha, Dmitry Molchanov, and Dmitry Vetrov. Greedy policy search: A simple baseline for learnable test-time augmentation. In Jonas Peters and David Sontag, editors, Proceedings of the 36th Conference on Uncertainty in Artificial I...

    2020

  33. [41]

    Majority vote of diverse classifiers for late fusion

    Emilie Morvant, Amaury Habrard, and Stéphane Ayache. Majority vote of diverse classifiers for late fusion. In Pasi Fränti, Gavin Brown, Marco Loog, Francisco Escolano, and Marcello Pelillo, editors, Structural, Syntactic, and Statistical Pattern Recognition, pages 153–162, Ber...

    2014

  34. [42]

    Black holes and white rabbits: Metaphor identification with visual features

    Ekaterina Shutova, Douwe Kiela, and Jean Maillard. Black holes and white rabbits: Metaphor identification with visual features. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pa...

    2016

  35. [43]

    Potamianos, C

    G. Potamianos, C. Neti, G. Gravier, A. Garg, and A.W. Senior. Recent advances in the automatic recognition of audiovisual speech. Proceedings of the IEEE, 91(9):1306–1326, 2003

    2003

  36. [44]

    Multiple classifier systems for the classification of audio-visual emotional states

    Michael Glodek, Stephan Tschechne, Georg Layher, Martin Schels, Tobias Brosch, Stefan Scherer, Markus Kächele, Miriam Schmidt, Heiko Neumann, Günther Palm, et al. Multiple classifier systems for the classification of audio-visual emotional states. In Affective Computing and In...

    2011

  37. [45]

    Ramirez, Tadas Baltrušaitis, and Louis-Philippe Morency

    Geovany A. Ramirez, Tadas Baltrušaitis, and Louis-Philippe Morency. Modeling latent discriminative dynamic of multi-dimensional affective signals. In Sidney D’Mello, Arthur Graesser, Björn Schuller, and Jean-Claude Martin, editors, Affective Computing and Intelligent Interacti...

    2011

  38. [46]

    Multi-task, multi-kernel learning for estimating individual wellbeing

    Natasha Jaques, Sara Taylor, Akane Sano, and Rosalind Picard. Multi-task, multi-kernel learning for estimating individual wellbeing. In Proc. NIPS Workshop on Multimodal Machine Learning, Montreal, Quebec, volume 898, page 3, 2015

    2015

  39. [47]

    Missing modalities imputation via cascaded residual autoencoder

    Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin. Missing modalities imputation via cascaded residual autoencoder. In CVPR, July 2017

    2017

  40. [48]

    Vigan: Missing view imputation with generative adversarial networks

    Chao Shang, Aaron Palmer, Jiangwen Sun, Ko-Shin Chen, Jin Lu, and Jinbo Bi. Vigan: Missing view imputation with generative adversarial networks. In 2017 IEEE International Conference on Big Data (Big Data) , pages 766–775, 2017

    2017

  41. [49]

    Incomplete multi-view learning via half-quadratic minimization

    Jiacheng Jiang, Hong Tao, Ruidong Fan, Wenzhang Zhuge, and Chenping Hou. Incomplete multi-view learning via half-quadratic minimization. Neurocomputing, 443:106–116, 2021

    2021

  42. [50]

    Cpm-nets: Cross partial multi-view networks

    Changqing Zhang, Zongbo Han, yajie cui, Huazhu Fu, Joey Tianyi Zhou, and Qinghua Hu. Cpm-nets: Cross partial multi-view networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, v...

    2019

  43. [51]

    A concise yet effective model for non-aligned incomplete multi-view and missing multi-label learning

    Xiang Li and Songcan Chen. A concise yet effective model for non-aligned incomplete multi-view and missing multi-label learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):5918–5932, 2022

    2022

  44. [52]

    Zero-shot learning on semantic class prototype graph

    Zhenyong Fu, Tao Xiang, Elyor Kodirov, and Shaogang Gong. Zero-shot learning on semantic class prototype graph. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(8):2009–2022, 2018

    2009

  45. [53]

    Zero-shot learning via category-specific visual- semantic mapping and label refinement

    Li Niu, Jianfei Cai, Ashok Veeraraghavan, and Liqing Zhang. Zero-shot learning via category-specific visual- semantic mapping and label refinement. IEEE Transactions on Image Processing, 28(2):965–979, 2019

    2019

  46. [54]

    Zero-shot recognition using dual visual-semantic mapping paths

    Yanan Li, Donghui Wang, Huanhang Hu, Yuetan Lin, and Yueting Zhuang. Zero-shot recognition using dual visual-semantic mapping paths. In CVPR, July 2017

    2017

  47. [55]

    Co-representation network for generalized zero-shot learning

    Fei Zhang and Guangming Shi. Co-representation network for generalized zero-shot learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, ICML, volume 97 of Proceedings of Machine Learning Research, pages 7434–7443. PMLR, 09–15 Jun 2019

    2019

  48. [56]

    Class normalization for (continual)? generalized zero-shot learning, 2021

    Ivan Skorokhodov and Mohamed Elhoseiny. Class normalization for (continual)? generalized zero-shot learning, 2021

    2021

  49. [57]

    Msdn: Mutually semantic distillation network for zero-shot learning

    Shiming Chen, Ziming Hong, Guo-Sen Xie, Wenhan Yang, Qinmu Peng, Kai Wang, Jian Zhao, and Xinge You. Msdn: Mutually semantic distillation network for zero-shot learning. In CVPR, pages 7612–7621, June 2022

    2022

  50. [58]

    Zero-shot learning by harnessing adversarial samples

    Zhi Chen, Pengfei Zhang, Jingjing Li, Sen Wang, and Zi Huang. Zero-shot learning by harnessing adversarial samples. In ACM MM, pages 4138–4146, 2023

    2023

  51. [59]

    Zero-vae-gan: Generating unseen features for generalized and transductive zero-shot learning

    Rui Gao, Xingsong Hou, Jie Qin, Jiaxin Chen, Li Liu, Fan Zhu, Zhao Zhang, and Ling Shao. Zero-vae-gan: Generating unseen features for generalized and transductive zero-shot learning. IEEE Transactions on Image Processing, 29:3665–3680, 2020

    2020

  52. [60]

    Krishnan, and Prateek Munjal

    Akanksha Paul, Narayanan C. Krishnan, and Prateek Munjal. Semantically aligned bias reducing zero shot learning. In CVPR, June 2019

    2019

  53. [61]

    Self-supervised domain-aware generative network for generalized zero-shot learning

    Jiamin Wu, Tianzhu Zhang, Zheng-Jun Zha, Jiebo Luo, Yongdong Zhang, and Feng Wu. Self-supervised domain-aware generative network for generalized zero-shot learning. In CVPR, June 2020

    2020

  54. [62]

    Feature generating networks for zero-shot learning

    Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In CVPR, June 2018. 14 A PREPRINT - S EPTEMBER 25, 2025

    2018

  55. [63]

    Generalized zero- and few-shot learning via aligned variational autoencoders

    Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Generalized zero- and few-shot learning via aligned variational autoencoders. In CVPR, June 2019

    2019

  56. [64]

    Latent embedding feedback and discriminative features for zero-shot classification

    Sanath Narayan, Akshita Gupta, Fahad Shahbaz Khan, Cees GM Snoek, and Ling Shao. Latent embedding feedback and discriminative features for zero-shot classification. In ECCV, pages 479–495. Springer, 2020

    2020

  57. [65]

    Free: Feature refinement for generalized zero-shot learning

    Shiming Chen, Wenjie Wang, Beihao Xia, Qinmu Peng, Xinge You, Feng Zheng, and Ling Shao. Free: Feature refinement for generalized zero-shot learning. In ICCV, pages 122–131, October 2021

    2021

  58. [66]

    Dubing Chen, Yuming Shen, Haofeng Zhang, and Philip H.S. Torr. Zero-shot logit adjustment. In Lud De Raedt, editor, IJCAI, pages 813–819, 7 2022. Main Track

    2022

  59. [67]

    Evolving semantic prototype improves generative zero-shot learning

    Shiming Chen, Wenjin Hou, Ziming Hong, Xiaohan Ding, Yibing Song, Xinge You, Tongliang Liu, and Kun Zhang. Evolving semantic prototype improves generative zero-shot learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarl...

    2023

  60. [68]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  61. [69]

    Recent advances in open set recognition: A survey

    Chuanxing Geng, Sheng-Jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE TPAMI, 43(10):3614–3631, 2021

    2021

  62. [70]

    Ensemble learning: A survey

    Omer Sagi and Lior Rokach. Ensemble learning: A survey. Wiley interdisciplinary reviews: data mining and knowledge discovery, 8(4):e1249, 2018. 15 A PREPRINT - S EPTEMBER 25, 2025 Table 3: Examples of class description generated by SA. The blue and purple sentences are audio a...

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.