Pith. sign in

REVIEW 3 major objections 2 minor 1 cited by

Advancing the Foundation Model for Music Understanding

T0 review · 3 major / 2 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Emotion AI over-trusts audio when video disagrees, and a new framework fixes that bias.

desk verdict Unreviewable as submitted: the attached full text is a different paper, so the SoTA claim for MuFun rests on an abstract alone. read the letter →

arxiv 2508.01178 v1 pith:JAIHLSHN submitted 2025-08-02 cs.SD cs.AIcs.IReess.AS

Yi Jiang , Wei Wang , Xianwen Guo , Huiyun Liu , Hanrui Wang , Youri Xu , Haoqi Gu , Zhongqian Xie
show 1 more author
Chuanjiang Luo
This is my paper
classification cs.SDcs.AIcs.IReess.AS
keywords emotionreasoningmultimodallargelanguagemodelmodalitybiasaudio-visualconflictbenchmarkattentionreallocationmodality-specificexpertstokenimbalance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that multimodal emotion models have a systematic bias toward audio cues when video and audio disagree, and that this bias is driven largely by token-count imbalance. To test this, it introduces CA-MER, a three-way benchmark separating video-aligned, audio-aligned, and consistent emotion samples. It then proposes MoSEAR, which combines modality-specific experts with a regularized router during training and an inference-time attention reallocation that selectively rebalances heads that over-focus on audio. On CA-MER and three existing emotion benchmarks, MoSEAR outperforms prior models, including on the consistent subset, indicating the fix does not trade away audio performance.

What carries the argument

The load-bearing mechanism is the attention-reallocation update. It identifies biased layers via a layer-level audio-to-visual attention ratio and biased heads via a head-level ratio, then performs a redistribution that preserves the total attention mass and the intra-modality distribution, with a closed-form solution. This is paired in training with MoSE, a set of three LoRA-based experts (visual, non-visual, and omni) weighted by a regularized gating function that keeps the routing probability within a fixed band.

What would settle it

Duplicate audio tokens in the primary baseline model until they equal video tokens, as the paper's own Section C.2 does, but across multiple random seeds and model initializations; if the bias does not consistently reverse, the token-imbalance mechanism is not sufficient. Separately, have human annotators label a random sample of CA-MER clips after seeing both modalities; if their labels frequently disagree with the benchmark's majority-vote ground truth, the benchmark's validity collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that the failure of multimodal emotion LLMs under emotion conflict is not an inherent reasoning limitation but a modality bias that can be identified in the attention heads and corrected. It shows that audio tokens receive disproportionately high attention in middle layers, and that replicating audio tokens to match video token counts reverses the bias rather than removing it. The proposed two-part mechanism—modality-specific experts with regularized routing during fine-tuning, and a closed-form attention reallocation at inference—reduces audio over-reliance without degrading audio-aligned performance. This yields state-of-the-art accuracy on the introduced CA-MER benchmark and on EMER, MER2023, and DFEW.

Load-bearing premise

The central claim depends on CA-MER being a fair and accurate test of emotion reasoning under conflict, and on the baseline comparisons being matched in training data and compute.

Editorial extensions

If this is right

  • Any multimodal LLM that fuses audio and video for affective tasks should be audited for token-count imbalance and modality attention ratios.
  • The closed-form attention reallocation is a training-free intervention that could be applied at inference to other multimodal reasoning tasks where one modality dominates.
  • The CA-MER benchmark provides a reusable test for whether a model truly integrates conflicting modalities rather than relying on a shortcut.
  • The finding that equalizing token counts reverses bias suggests architectural tokenization choices directly affect emotion reasoning fairness.
  • MoSEAR's performance gains on consistent samples suggest that bias mitigation and overall accuracy are not in tension.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The token-imbalance explanation may generalize beyond emotion reasoning to any audio-visual LLM; testing whether the attention-reallocation ratio is a robust diagnostic across tasks would extend the paper's scope.
  • The use of GPT-based grouping for open-vocabulary evaluation could introduce instability; a human-verified label mapping would strengthen the benchmark's reliability.
  • The inference-time reallocation could be combined with other de-biasing techniques, such as contrastive training or adversarial modality dropout, to see whether gains compound.
  • A direct test of the token-imbalance hypothesis on other base models, varying token ratios independently of architecture, would isolate the mechanism more cleanly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The abstract of arXiv:2508.01178 claims a unified music-understanding foundation model named MuFun, a new benchmark MuCUE, and experimental results showing that MuFun significantly outperforms existing audio large language models across MuCUE tasks. The full text supplied with the review record is, however, a different paper: it describes MoSEAR and CA-MER, a framework and benchmark for multimodal emotion reasoning, with no mention of MuFun, MuCUE, or music information retrieval. The abstract's central empirical claim is therefore not supported by any inspectable evidence in the submitted manuscript.

Significance. If the abstract's claims were substantiated, a single model jointly handling instrumental and lyrical content for genre classification, music tagging, and question answering would be a substantial contribution to MIR, especially if it demonstrated generalization beyond the tasks on which it was trained. The paper also would provide a new multi-faceted evaluation benchmark. However, because the submitted full text is an unrelated paper, none of these contributions can be assessed. The self-proposed nature of MuCUE also creates a structural risk that the claimed superiority is benchmark-specific; external validation on established MIR benchmarks would be needed to support a state-of-the-art claim.

major comments (3)
  1. [Full Text (entire body)] The body of the manuscript does not describe MuFun or MuCUE at all; it presents MoSEAR and CA-MER for multimodal emotion reasoning. Equations (1)-(14) and Tables 1-8 address audio-visual emotion conflict, not music understanding. Consequently, the experiments that the abstract invokes to support 'significantly outperforms existing audio large language models across the MuCUE tasks' are entirely absent. This is a load-bearing defect that cannot be fixed by a local revision.
  2. [Abstract] The central empirical claim is stated without any quantitative result, error bar, dataset size, or comparison protocol. Even if the correct full text were provided, the sentence 'Experiments show our model significantly outperforms existing audio large language models across the MuCUE tasks' would be insufficient to support a state-of-the-art claim without reporting the actual evaluation numbers and the identities and configurations of the baselines.
  3. [Abstract / MuCUE] The evaluation appears to rest entirely on MuCUE, a benchmark introduced by the same authors in this same paper, with no external validation on established MIR benchmarks such as AudioSet, MTG-Jamendo, or MagnaTagATune. This is a structural risk of circularity: the benchmark and the model are co-designed, and the abstract gives no information on how training data, model architecture, and inference protocol are controlled across baselines.
minor comments (2)
  1. [Full Text title/abstract] The full text's title, abstract, CCS concepts, and references all concern emotion reasoning and are inconsistent with the submitted abstract's music-understanding framing; the authors should ensure that the submitted manuscript body matches the claimed paper.
  2. [Full Text author affiliation] The affiliation in the full text lists 'Heifei, China,' which appears to be a typographical error for 'Hefei, China.'

Circularity Check

0 steps flagged · score 0.0 of 10

Unverifiable, not circular: the supplied full text is a different paper, so no derivation chain for MuFun/MuCUE can be inspected.

full rationale

No circularity can be established from the available evidence. The abstract claims that the MuFun foundation model 'significantly outperforms existing audio large language models across the MuCUE tasks,' but the full text supplied with the record is arXiv:2508.01181, 'Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning,' which describes CA-MER and MoSEAR rather than MuFun and MuCUE. Without the actual manuscript describing MuFun's architecture, training data, MuCUE benchmark construction, and evaluation protocol, there is no derivational chain to walk and no equation or fitted parameter that can be shown to reduce to its own inputs. The reviewer's concern that a self-proposed benchmark carrries a structural self-referential risk is a validity concern, not a demonstrated circularity, and the hard rules require quoting the paper and exhibiting a specific reduction before flagging circularity. No such reduction is present in the supplied text. Accordingly, the honest finding is no significant circularity on the available material, with the caveat that the central empirical claim is not verifiable from this record.

Assumptions & free parameters 0 free parameters · 2 assumptions · 2 invented entities

Only the abstract was reviewable. The full text attached to the arXiv record is a different paper (arXiv:2508.01181), so parameters, axioms, and entities beyond those stated in the abstract cannot be audited. The two entities listed are introduced by the abstract itself.

assumptions (2)
  • domain assumption MuCUE's task suite and metrics are a valid operationalization of 'holistic music understanding'.
    The abstract asserts holistic music understanding but does not define its scope. The central outperformance claim is measured exclusively on MuCUE, so the benchmark's validity is a load-bearing premise introduced by the authors.
  • domain assumption Audio large language models are the appropriate comparison class for the state-of-the-art claim.
    The abstract compares only against audio large language models. If specialized MIR models still beat MuFun on individual tasks, the 'holistic' generalization claim would be weakened, so this comparison choice is an unstated assumption.
invented entities (2)
  • MuFun
    purpose: Unified foundation model for holistic music understanding that jointly processes instrumental and lyrical content.
    A new model introduced in the abstract. No external validation, released weights, or API are mentioned.
  • MuCUE
    purpose: Benchmark for multi-faceted music understanding evaluation, used to compare MuFun against audio LLMs.
    A new benchmark introduced in the abstract, constructed and used by the same authors. No independent application of the benchmark outside this paper is mentioned.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Advancing the Foundation Model for Music Understanding." pith.science (2026). https://pith.science/paper/JAIHLSHN

@misc{pith2026250801178,
  author       = {Pith},
  title        = {Pith review of: Advancing the Foundation Model for Music Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JAIHLSHN}},
  note         = {Machine review of arXiv:2508.01178}
}
read the original abstract

The field of Music Information Retrieval (MIR) is fragmented, with specialized models excelling at isolated tasks. In this work, we challenge this paradigm by introducing a unified foundation model named MuFun for holistic music understanding. Our model features a novel architecture that jointly processes instrumental and lyrical content, and is trained on a large-scale dataset covering diverse tasks such as genre classification, music tagging, and question answering. To facilitate robust evaluation, we also propose a new benchmark for multi-faceted music understanding called MuCUE (Music Comprehensive Understanding Evaluation). Experiments show our model significantly outperforms existing audio large language models across the MuCUE tasks, demonstrating its state-of-the-art effectiveness and generalization ability.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. RSPO: Risk-Seeking Policy Optimization for Pass@k and Max@k Metrics in Large Language Models

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    RSPO trains LLMs directly against Pass@k and Max@k using unbiased gradient estimators derived from the probability that a response is the maximum among k samples.

Reference graph

Works this paper leans on

108 extracted references · 51 canonical work pages · cited by 1 Pith paper

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  2. [2]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Frontier Large Vision- Language Model with Versatile Abilities.arXiv preprint arXiv:2308.12966(2023)

    arXiv 2023

  3. [3]

    John A Bargh and Lawrence E Williams. 2007. The nonconscious regulation of emotion.Handbook of emotion regulation1 (2007), 429–445

    2007

  4. [4]

    Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. 2014. Crema-d: Crowd-sourced emotional multimodal actors dataset.IEEE Transactions on Affective Computing5, 4 (2014), 377–390

    2014

  5. [5]

    Haifeng Chen, Chujia Guo, Yan Li, Peng Zhang, and Dongmei Jiang. 2023. Semi-Supervised Multimodal Emotion Recognition with Class-Balanced Pseudo- labeling. InACM MM

    2023

  6. [6]

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning.arXiv preprint arXiv:2310.09478(2023)

    arXiv 2023

  7. [7]

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao

  8. [8]

    Yin Chen, Jia Li, Shiguang Shan, Meng Wang, and Richang Hong. 2024. From static to dynamic: Adapting landmark-aware image models for facial expression recognition in videos.IEEE Transactions on Affective Computing(2024)

    2024

Show all 108 references
  1. [9]

    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. 2024. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning.NeurIPS(2024)

    2024

  2. [10]

    Zebang Cheng, Yuxiang Lin, Zhaoru Chen, Xiang Li, Shuyi Mao, Fan Zhang, Dai- jun Ding, Bowen Zhang, and Xiaojiang Peng. 2023. Semi-Supervised Multimodal Emotion Recognition with Expression MAE. InACM MM

    2023

  3. [11]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https://vicuna.lmsys.org

    2023

  4. [12]

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919(2023)

    2023 arXiv

  5. [13]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human...

    2019

  6. [14]

    Chaoyue Ding, Daoming Zong, Baoxiang Li, Ken Zheng, Dinghao Zhou, Jiakui Li, and Qunyan Zhou. 2023. Learning Aligned Audiovisual Representations for Multimodal Sentiment Analysis. InProceedings of the 1st International Workshop on Multimodal and Responsible Affective Computing

    2023

  7. [15]

    Weiquan Fan, Xiangmin Xu, Xiaofen Xing, Weidong Chen, and Dongyan Huang

  8. [16]

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, et al. 2025. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction.arXiv preprint arXiv:2501.01957 (2025)

    2025 arXiv

  9. [17]

    James J Gross et al. 2014. Emotion regulation: Conceptual and empirical founda- tions.Handbook of emotion regulation2 (2014), 3–20

    2014

  10. [18]

    Qingpei Guo, Kaiyou Song, Zipeng Feng, Ziping Ma, Qinglong Zhang, Sirui Gao, Xuzheng Yu, Yunxiao Sun, Jingdong Chen, Ming Yang, et al . 2025. M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance.arXiv preprint arXiv:2502.18778(2025)

    2025 arXiv

  11. [19]

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. 2024. Onellm: One framework to align all modalities with language. InCVPR

    2024

  12. [20]

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick

  13. [21]

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Rus- lan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM Transactions on Audio, Speech, and Language Processi...

    2021

  14. [22]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. InICLR

    2022

  15. [23]

    Jinpeng Hu, Tengteng Dong, Luo Gang, Hui Ma, Peng Zou, Xiao Sun, Dan Guo, Xun Yang, and Meng Wang. 2024. Psycollm: Enhancing llm for psychological understanding and evaluation.IEEE Transactions on Computational Social Systems (2024)

    2024

  16. [24]

    Masked autoencoders are scalable vision learners. InCVPR

  17. [25]

    Ashley Hutchison and Larry Gerstein. 2017. Emotion recognition, emotion expression, and cultural display rules: Implications for counseling.Journal of Asia Pacific Counseling7, 1 (2017)

    2017

  18. [26]

    Maryam Imani and Gholam Ali Montazer. 2019. A survey of emotion recognition methods with emphasis on E-Learning environments.Journal of Network and Computer Applications147 (2019), 102423

    2019

  19. [27]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation.ACM computing surveys55, 12 (2023), 1–38

    2023

  20. [28]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)

    2024 arXiv

  21. [29]

    Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang

  22. [30]

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. 2024. Chat-univi: Unified visual representation empowers large language models with image and video understanding. InCVPR

    2024

  23. [31]

    Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. 2025. See What You Are Told: Visual Attention Sink in Large Multimodal Models. InICLR

    2025

  24. [32]

    Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. 2020. Dfew: A large-scale database for recognizing dynamic facial expressions in the wild. InACM MM

    2020

  25. [33]

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. 2023. Mimic-it: Multi-modal in-context instruction tuning.arXiv preprint arXiv:2306.05425(2023)

    2023 arXiv

  26. [34]

    Hanting Li, Hongjing Niu, Zhaoqing Zhu, and Feng Zhao. 2023. Intensity-aware loss for dynamic facial expression recognition in the wild. InAAAI

    2023

  27. [35]

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355(2023)

    2023 arXiv

  28. [36]

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. 2024. Mvbench: A comprehensive multi-modal video understanding benchmark. InCVPR

    2024

  29. [37]

    Shanglin Lei, Guanting Dong, Xiaoping Wang, Keheng Wang, and Sirui Wang

  30. [38]

    Instructerc: Reforming emotion recognition in conversation with a retrieval multi-task llms framework.CoRR(2023)

    2023

  31. [39]

    Zheng Lian, Haiyang Sun, Licai Sun, Zhuofan Wen, Siyuan Zhang, Shun Chen, Hao Gu, Jinming Zhao, Ziyang Ma, Xie Chen, et al . 2024. Mer 2024: Semi- supervised learning, noise robustness, and open-vocabulary multimodal emotion recognition. InProceedings of the 2nd International ...

    2024

  32. [40]

    Zheng Lian, Haiyang Sun, Licai Sun, Jiangyan Yi, Bin Liu, and Jianhua Tao

  33. [41]

    Zheng Lian, Licai Sun, Haiyang Sun, Kang Chen, Zhuofan Wen, Hao Gu, Bin Liu, and Jianhua Tao. 2024. GPT-4V with emotion: A zero-shot benchmark for Generalized Emotion Recognition.Information Fusion(2024), 102367

    2024

  34. [42]

    Zheng Lian, Licai Sun, Mingyu Xu, Haiyang Sun, Ke Xu, Zhuofan Wen, Shun Chen, Bin Liu, and Jianhua Tao. 2023. Explainable multimodal emotion reasoning. CoRR(2023)

    2023

  35. [43]

    Yanwei Li, Chengyao Wang, and Jiaya Jia. 2024. Llama-vid: An image is worth 2 tokens in large language models. InECCV

    2024

  36. [44]

    Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mngyu Xu, Kexin Wang, Ke Xu, Yu He, Ying Li, Jinming Zhao, et al . 2023. Mer 2023: Multi-label learning, modality robustness, and semi-supervised learning. InACM MM

    2023

  37. [45]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    2024 arXiv

  38. [46]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. InCVPR

    2024

  39. [47]

    AffectGPT: Dataset and framework for explainable multimodal emotion recognition.arXiv preprint arXiv:2407.07653(2024)

    2024 arXiv

  40. [48]

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A survey on hallucination in large vision-language models.arXiv preprint arXiv:2402.00253(2024)

    2024 arXiv

  41. [49]

    Shi Liu, Kecheng Zheng, and Wei Chen. 2024. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. InECCV

    2024

  42. [50]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 204. Video-llava: Learning united visual representation by alignment before projection. InEMNLP

  43. [51]

    Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Ghosh, Luke Zettlemoyer, and Armen Aghajanyan. 2024. Moma: Efficient early-fusion pre-training with mixture of modality-aware experts.arXiv preprint arXiv:2407.21770(2024)

    2024 arXiv

  44. [52]

    Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jiawen Liu, Jifeng Dai, Yu Qiao, and Xizhou Zhu. 2025. Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training. InCVPR

    2025

  45. [53]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan

  46. [54]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  47. [55]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al

  48. [56]

    Haowen Pan, Yixin Cao, Xiaozhi Wang, Xun Yang, and Meng Wang. 2023. Finding and editing multi-modal neurons in pre-trained transformers.Findings of ACL (2023)

    2023

  49. [57]

    Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan. 2022. Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild. InACM MM. MM ’25, October 27–31, 2025, Dublin, Ireland Zhi...

    2022

  50. [58]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. InICLR

    2019

  51. [59]

    Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models.arXiv preprint arXiv:2309.05922(2023)

    2023 arXiv

  52. [60]

    Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. 2023. Scaling vision-language models with sparse mixture of experts. In Findings of EMNLP

    2023

  53. [61]

    Video-chatgpt: Towards detailed video understanding via large vision and language models. InACL

  54. [62]

    OpenAI. 2023. GPT-4V(ision) system card. https://openai.com/research/gpt-4v- system-card

    2023

  55. [63]

    Peipei Song, Dan Guo, Xun Yang, Shengeng Tang, Erkun Yang, and Meng Wang

  56. [64]

    In NeurIPS

    Training language models to follow instructions with human feedback. In NeurIPS

  57. [65]

    Licai Sun, Zheng Lian, Bin Liu, and Jianhua Tao. 2023. Mae-dfer: Efficient masked autoencoder for self-supervised dynamic facial expression recognition. InACM MM

    2023

  58. [66]

    Haowen Pan, Xiaozhi Wang, Yixin Cao, Zenglin Shi, Xun Yang, Juanzi Li, and Meng Wang. 2025. Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMs.ICLR(2025)

    2025

  59. [67]

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2024. Kosmos-2: Grounding Multimodal Large Language Models to the World. InICLR

    2024

  60. [68]

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS

    2022

  61. [69]

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al

  62. [70]

    Peipei Song, Dan Guo, Jun Cheng, and Meng Wang. 2023. Contextual Attention Network for Emotional Video Captioning.IEEE Transactions on Multimedia25 (2023), 1858–1867

    2023

  63. [71]

    Peipei Song, Dan Guo, Xun Yang, Shengeng Tang, and Meng Wang. 2024. Emo- tional Video Captioning With Vision-Based Emotion Interpretation Network. IEEE Transactions on Image Processing33 (2024), 1122–1135

    2024

  64. [72]

    Hongxia Xie, Chu-Jun Peng, Yu-Wen Tseng, Hung-Jen Chen, Chan-Feng Hsu, Hong-Han Shuai, and Wen-Huang Cheng. 2024. EmoVIT: Revolutionizing Emo- tion Insights with Visual Instruction Tuning. InCVPR

    2024

  65. [73]

    In Proceedings of the 31st ACM International Conference on Multimedia

    Emotion-Prior Awareness Network for Emotional Video Captioning. In Proceedings of the 31st ACM International Conference on Multimedia. 589–600

  66. [74]

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai

  67. [75]

    Pandagpt: One model to instruction-follow them all.arXiv preprint arXiv:2305.16355(2023)

    2023 arXiv

  68. [76]

    Xun Yang, Shanshan Wang, Jian Dong, Jianfeng Dong, Meng Wang, and Tat-Seng Chua. 2022. Video moment retrieval with cross-modal neural architecture search. IEEE Transactions on Image Processing31 (2022), 1204–1216

    2022

  69. [77]

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389 (2023)

    2023 arXiv

  70. [78]

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. Salmonn: Towards generic hearing abilities for large language models. InICLR

    2024

  71. [79]

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023. mplug-owl: Modular- ization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178(2023)

    2023 arXiv

  72. [80]

    Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang. 2020. Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. InACL

    2020

  73. [81]

    Image as a foreign language: Beit pretraining for vision and vision-language tasks. InCVPR

  74. [82]

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al . 2023. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. InNeurIPS

    2023

  75. [83]

    Jialin Wu, Xia Hu, Yaqing Wang, Bo Pang, and Radu Soricut. 2024. Omni-smola: Boosting generalist multimodal models with soft mixture of low-rank experts. In CVPR

    2024

  76. [84]

    Kesen Zhao, Beier Zhu, Qianru Sun, and Hanwang Zhang. 2025. Unsupervised visual chain-of-thought reasoning via preference optimization. InICCV

    2025

  77. [85]

    Qize Yang, Detao Bai, Yi-Xing Peng, and Xihan Wei. 2025. Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis.arXiv preprint arXiv:2501.09502(2025)

    2025 arXiv

  78. [86]

    Xun Yang, Tianyu Chang, Tianzhu Zhang, Shanshan Wang, Richang Hong, and Meng Wang. 2024. Learning hierarchical visual transformation for domain generalizable visual matching and recognition.International Journal of Computer Vision132, 11 (2024), 4823–4849

    2024

  79. [87]

    Xun Yang, Fuli Feng, Wei Ji, Meng Wang, and Tat-Seng Chua. 2021. Deconfounded video moment retrieval with causal intervention. InSIGIR. 1–10

    2021

  80. [88]

    Beier Zhu, Yulei Niu, Xian-Sheng Hua, and Hanwang Zhang. 2022. Cross-domain empirical risk minimization for unbiased long-tailed classification. InAAAI

    2022

  81. [89]

    Xun Yang, Jianming Zeng, Dan Guo, Shanshan Wang, Jianfeng Dong, and Meng Wang. 2024. Robust video question answering via contrastive cross-modality representation learning.Science China Information Sciences67, 10 (2024), 202104

    2024

  82. [90]

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. InICLR

    2024

  83. [93]

    Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-llama: An instruction-tuned audio-visual language model for video understanding. InEMNLP

    2023

  84. [94]

    Jiaxing Zhao, Xihan Wei, and Liefeng Bo. 2025. R1-Omni: Explainable Omni- Multimodal Emotion Recognition with Reinforcing Learning.arXiv preprint arXiv:2503.05379(2025)

    2025 arXiv

  85. [95]

    Jiaxing Zhao, Qize Yang, Yixing Peng, Detao Bai, Shimin Yao, Boyuan Sun, Xiang Chen, Shenghao Fu, Xihan Wei, Liefeng Bo, et al. 2025. HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding.arXiv preprint arXiv:2501.15111(2025)

    2025 arXiv

  86. [97]

    Zengqun Zhao and Qingshan Liu. 2021. Former-dfer: Dynamic facial expression recognition transformer. InACM MM

    2021

  87. [98]

    Sheng Zhou, Junbin Xiao, Qingyun Li, Yicong Li, Xun Yang, Dan Guo, Meng Wang, Tat-Seng Chua, and Angela Yao. 2025. Egotextvqa: Towards egocentric scene-text aware video question answering. InCVPR. 3363–3373

    2025

  88. [99]

    Beier Zhu, Jiequan Cui, and Hanwang Zhang. 2024. Robust Fine-tuning of Zero- shot Models via Variance Reduction. InNeurIPS

    2024

  89. [101]

    gpt-4o-audio- preview

    Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. 2023. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. InNeurIPS. Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning MM ’25, October 27–31, ...

    2023

  90. [102]

    Video Analysis: {video analysis}

  91. [103]

    Audio Analysis: {audio analysis}

  92. [104]

    Subtitle: {subtitle}

  93. [105]

    Your final explanation should align with the emotional label: {final label}

    Carefully analyze the emotional tone from each clue. Your final explanation should align with the emotional label: {final label}. Please ensure that your explanation covers the following:

  94. [106]

    Key emotional expressions in the visual, audio, and textual clues

  95. [107]

    How these clues either support or conflict with each other

  96. [108]

    gpt-3.5-turbo-16k-0613

    A final, coherent emotional inference that aligns with the final label. Answer in English. Your response should be concise and flow naturally, no more than 150 words. C Implementation Details C.1 Open Vocabulary Evaluation Metric For our CA-MER benchmark, we adopt the same eva...

    2025

  97. [2021]

    InICASSP

    LSSED: a large-scale dataset and benchmark for speech emotion recognition. InICASSP

  98. [2022]

    InNeurIPS

    Flamingo: a visual language model for few-shot learning. InNeurIPS

  99. [2023]

    Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic.arXiv preprint arXiv:2306.15195(2023)

    2023 arXiv

  100. [2024]

    Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. InCVPR

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.