Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmark

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that empathetic response generation should move from text-only to a three-modality task — text, speech, and talking-face video — and that its end-to-end system, Empatheia, generates all three consistently better than a…

desk verdict AvaMERG is a real benchmark contribution, but the Empatheia system's headline gains rest on an unfair text-only baseline and no uncertainty quantification. read the letter →

arxiv 2502.04976 v1 pith:ASD43CSN submitted 2025-02-07 cs.MM

Han Zhang , Zixiang Meng , Meng Luo , Hong Han , Lizi Liao , Erik Cambria , Hao Fei This is my paper · ORCID
classification cs.MM
keywords empatheticresponsegenerationmultimodaldialogueavatar-basedbenchmarklargelanguagemodeltalking-facechain-of-empathyreasoningspeechemotionaffectivecomputing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Empathy in conversation is usually studied as a text-only problem, but humans read emotion from tone of voice and facial expression as much as from words. This paper tries to establish a new task — avatar-based multimodal empathetic response generation (MERG) — in which a system must take in text, speech, and video and reply in all three modalities at once. To make that possible it contributes AvaMERG, a dataset of 33,048 dialogues and 152,021 utterances, each aligned with human-read speech and talking-face video across diverse avatar profiles, and Empatheia, an end-to-end multimodal LLM that reasons about the user's emotion step by step before generating text, audio, and avatar. If the paper is right, empathetic AI can be built as one coherent system rather than as separate text, voice, and avatar modules glued together, and the community gains a benchmark to measure that progress.

What carries the argument

The machine that carries the argument is the continuous-embedding message-passing loop: the LLM core emits text tokens plus speech-signal and video-signal tokens, and those embeddings — not discrete commands — are what drive the two generators. Two purpose-built modules sit in the path: the Content Synchronizer, a Transformer variational autoencoder that reconstructs the response's content for speech and video from the text signal, and the Style Disentangler, a VAE that separates emotion features from profile features (age, gender, tone) and feeds them to the generators. The Chain-of-Empathy five-step instruction is the reasoning scaffold that makes the empathy explicit and interpretable. These are bound together by a four-stage training schedule: supervised fine-tuning on CoE examples with LoRA, content consistency learning, style alignment and consistency learning with emotion and profile classification losses, and a final joint tuning with the combined loss $L_{oal} = L_{emp} + \alpha L_{ccl} + \beta L_{sac}$.

What would settle it

Rebuild the pipeline baseline with the same HuBERT and CLIP encoders so it consumes the user's speech and video, train it on the same AvaMERG data, and compare the six MERG metrics; if the gap to Empatheia largely closes, the end-to-end advantage is not established. A second check is to re-annotate a random sample of AvaMERG independently and confirm the reported Cohen's kappa of 0.78 and unanimous three-annotator acceptance, since dataset quality is load-bearing for every downstream comparison.

Watch

Extended reading notes

Core claim

The paper's central claim is that multimodal empathy is not just a harder version of text empathy but a task with its own failure modes — content mismatches (speech says one thing, avatar expresses another), style inconsistencies (a female voice with a male face), and error propagation in pipelines — and that these can be addressed by design. AvaMERG provides the testbed: it extends the Empathetic Dialogues text corpus with identity labels (age, gender, tone, race) and authentic recordings, producing 152,021 utterances in which text, speech, and talking-face video are synchronized in content, emotion, and profile. Empatheia is the proposed solution: a Vicuna-based multimodal LLM that ingests speech and video through HuBERT and CLIP encoders, reasons via a Chain-of-Empathy prompt (event scenario, user emotion, emotion cause, response goal, response), and sends continuous embeddings through a Content Synchronizer and a Style Disentangler to control StyleTTS2 and DreamTalk generators. The experiments report that Empatheia outperforms the text ERG baselines KEMP, CEM, and CASE on emotion accuracy and diversity, and outperforms a pipeline baseline on every speech, video, and consistency metric, including six newly defined human-evaluated measures.

Load-bearing premise

The load-bearing premise is that a text-only pipeline is a fair opponent for a task defined by multimodal input: the pipeline baseline never sees the user's speech or face, so the headline MERG gains partly depend on that asymmetry holding.

Editorial extensions

If this is right

  • If Empatheia's results hold, empathetic response generation can be framed end-to-end: one model produces text, speech, and a talking-face avatar in a single pass, with no module-boundary error propagation.
  • Multimodal input pays off even for text output: removing speech and video from the input lowers textual empathy accuracy, so future ERG systems should listen to and look at users rather than read only their typed words.
  • Each Chain-of-Empathy step contributes measurable gains, so stepwise emotion reasoning is a useful component for any model that must infer emotion causes before responding.
  • AvaMERG gives the community a 33,048-dialogue, 152,021-utterance benchmark with synchronized text, speech, and video, making MERG a trainable and comparable task.
  • The six human evaluation metrics proposed for MERG — content accuracy, style accuracy, and cross-modal consistency for speech and video — provide a template for judging multimodal empathy beyond automatic scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment is to give the pipeline baseline the same multimodal input the task defines; the paper's ablations show multimodal input helps, so the reported margin over a text-only pipeline may narrow under that fairer setup.
  • The dataset's strong emotion imbalance — 56.7 percent of samples are sad — means the system's strengths may be partly tied to the data distribution; testing on a balanced emotion split would show how general the empathy gains are.
  • Because the generated voice and face are synthetic but feel human, the ethical concerns listed in the paper's appendix (deepfake misuse, emotional dependence, avatar-profile bias) become deployment blockers rather than side notes.
  • The 32-to-7 mapping from fine-grained text emotions to coarse multimodal emotions suggests coarse labels may be the right granularity for cross-modal consistency; breaking the seven coarse classes into finer ones would test that ceiling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes a new task, avatar-based Multimodal Empathetic Response Generation (MERG), and contributes AvaMERG, a benchmark of 33,048 dialogues and 152,021 utterances with text, speech, and talking-face video extended from EmpatheticDialogues. It also presents Empatheia, an end-to-end MLLM with a Chain-of-Empathy reasoning mechanism, a content synchronizer, a style disentangler, and a four-stage training procedure, and reports automatic and human evaluations against a pipeline baseline and text-only ERG baselines. The central claim is that Empatheia consistently outperforms baselines on both textual ERG and MERG.

Significance. The dataset is a potentially significant community resource: it is large, openly released, covers diverse avatar profiles, topics, and emotions, and includes manual verification with reported Cohen's kappa of 0.78 for dataset annotation. The model design is coherent and the ablations consistently show degradation when CS, SD, CoE, or training losses are removed, which provides internal support for the proposed components. However, the headline comparative claim is not yet robustly established because the only MERG baseline is text-only and the newly introduced human evaluation metrics are reported without reliability or significance information.

major comments (4)
  1. [§6.1 and Appendix E.2] The MERG comparison does not establish the claimed superiority because the pipeline baseline is trained and evaluated on text only. The task definition in §3.1 makes the user's speech and video part of the multimodal query, but Appendix E.2 states that the pipeline 'exclusively utilize[s] text data' and 'will not be equipped with the CoE strategy,' while the LLM receives only response text and profile attributes before invoking StyleTTS2 and DreamTalk. Consequently, the gains in Table 5 (e.g., SCA 3.92 vs 3.23, MSC 3.91 vs 3.19) may reflect the input-modality handicap rather than the benefit of end-to-end multimodal generation. I would like to see either a multimodal pipeline variant that consumes the user's audio/video (e.g., through the same encoders or through transcribed/described signals), or comparisons with existing multimodal empathetic systems such as EmpathyEar [11], StickerConv [51], and Talk With Human-like Agents [48]. The w/o SPC&VID ablation in Table 4 is not an alternative multimodal baseline.
  2. [§6.3, Table 5, and Appendix E.3.4] The six newly defined human metrics for MERG (SCA, VCA, SSA, VSA, MCC, MSC) lack reliability and significance evidence. Appendix E.3.4 reports that three evaluators scored 200 test cases on six 1–5 scales, but no inter-annotator agreement, confidence intervals, or significance tests are reported for any automatic or human metric in Tables 2–5. Differences of 0.1–0.5 on a 5-point scale, as in several rows of Table 5, can easily be within rater noise. The manuscript's own Appendix B acknowledges that multimodal evaluation 'relies solely on human evaluations, which introduces significant uncertainty.' Reporting Cohen's kappa or ICC for each of the six metrics and paired significance tests (e.g., bootstrap or Wilcoxon) is necessary before the claim of consistent superiority can be accepted.
  3. [§6.1, Tables 2 and 4] The textual ERG comparison uses outdated non-LLM baselines (KEMP, CEM, CASE from 2021–2022) against a fine-tuned Vicuna-7B, so the large margins in Table 2 (e.g., Empathy Accuracy 48.51 vs 40.96, Dist-2 14.76 vs 2.14) are expected and do not demonstrate a methodological advance in empathetic text generation. Since the abstract and conclusion claim superiority 'on both textual ERG and MERG,' the text-only comparison needs at least one strong LLM-based ERG baseline (e.g., an instruction-tuned LLM with Chain-of-Thought prompting, or a recent open LLM) under the same training/evaluation protocol. The w/o CoE and w/o SPC&VID ablations are useful, but they only measure internal contributions.
  4. [Appendix E.3.4, Eqs. (31)–(36)] The definitions of the six human MERG metrics are not operational enough to be reproducible. Equations (31)–(36) define SCA, VCA, SSA, VSA, MCC, and MSC in terms of abstract 'Consistency' and 'Similarity' functions without specifying the rating rubric, the comparison target (gold response vs. holistic judgment), or the scale anchors. For example, MSC in Eq. (36) compares generated speech style with gold video style, which seems to conflate content and style and is inconsistent with the prose definition 'consistency of style across the three modalities.' Please provide the exact instructions given to annotators and clarify each metric's comparison target.
minor comments (5)
  1. [§6.2] The sentence 'we present the results of speech generation and avatar generation in Table 3 and Table ??, respectively' contains an unresolved 'Table ??'; please insert the correct table reference.
  2. [Table 5] The column headers SEA and VEA do not match the metric names SSA and VSA defined in §6.1 and Appendix E.3.4; please make these consistent.
  3. [Figure 2 caption] The caption contains the typo 'Eemotion distribution'; also, the percentages in the pie chart should sum to 100%.
  4. [Appendix C.2, Figure 11] The topic labels are rendered as an unreadable codepoint sequence, making the emotion-topic heatmap impossible to interpret; please provide readable labels.
  5. [§4.3] The CS and SD modules are called 'VAE' modules, but no KL-divergence term or variational sampling is described in Eqs. (1)–(6) or the training losses; either justify the terminology or rename them.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular step found: Empatheia's reported gains are empirical evaluations on held-out multimodal outputs, and the self-referential feature-alignment losses do not by construction determine the measured metrics.

full rationale

Empatheia's central claim is empirical: it outperforms KEMP, CEM and CASE on textual ERG, and outperforms a constructed pipeline baseline on MERG, with results reported on the AvaMERG test split. The only mechanism that could look circular is the training loop in Sections 5.2-5.4, where 'gold' content and style features are pre-extracted from the same StyleTTS2 and DreamTalk encoders used at inference, and L_ccl (Eq. 9) and L_sal (Eq. 10) train the CS and SD modules to reproduce those embeddings. This is a self-referential alignment objective, but it is not a prediction masquerading as a result: the reported MOS, SMOS, CPBD, SSIM, Sync_conf and the human SCA/VCA/SSA/VSA/MCC/MSC scores are computed on the final generated speech and talking-head video on held-out data, and matching a frozen encoder's embedding does not by construction force any specific value of those externally rated or standard metrics on unseen samples. No equation in the paper reduces a reported result to a fitted parameter or to the training targets themselves. The paper's self-citations (e.g., [7], [11], [46], [47]) are used for contextual related work and architectural inspiration such as Chain-of-Thought and general MLLMs, not as the sole justification for the empirical superiority claim; the CoE ablation provides independent evidence of that component's contribution. There are legitimate evaluation concerns: the only MERG baseline in Appendix E.2 is text-only and does not consume the user's speech or video; the authors' own EmpathyEar [11] is cited but not compared; and Appendix B concedes that multimodal evaluation 'relies solely on human evaluations, which introduces significant uncertainty.' These are external-validity and robustness limitations, not circular reductions. Accordingly, no circular step is established.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central experimental claim depends on the two loss weights (alpha, beta), the number of transformer blocks, and the number of special tokens, all tuned on the validation set. The benchmark construction relies on several domain assumptions about the quality of GPT-4 annotations, volunteer performances, and the use of pre-trained generator encoders as training targets. No new physical or ontological entities are introduced.

free parameters (4)
  • alpha (loss weight for content consistency learning) = 0.2
    Chosen by grid search over {0.1, 0.2, 0.3, 0.4} as shown in Figure 14; affects the overall loss in Eq. 13.
  • beta (loss weight for style alignment and consistency learning) = 0.3
    Chosen by grid search over {0.1, 0.2, 0.3, 0.4} as shown in Figure 14; affects the overall loss in Eq. 13.
  • Number of transformer blocks in CS and SD = 4
    Selected based on Figure 12, which reports average human-eval score peaking at 4 layers.
  • Number of audio/video special tokens = 16
    Selected based on Figure 13, which reports peak performance at 16 tokens.
assumptions (4)
  • domain assumption The Empathetic Dialogue (ED) dataset provides a valid base for building AvaMERG.
    Section 3.2 extends ED without questioning whether its dialogues are representative enough for the new multimodal task.
  • domain assumption GPT-4 annotations for emotions, profiles, topics, and generated dialogues are sufficiently accurate.
    Section 3.2 and Appendix C.1.1 rely on GPT-4 for labels and data expansion, with human review but no quantitative measure of GPT-4 accuracy.
  • domain assumption Volunteers can authentically portray the required emotions in speech and video after instruction.
    Section 3.2 describes instructed recordings with directors; the paper assumes these acted performances match real user emotional expressions.
  • domain assumption The gold content and style features extracted from StyleTTS2 and DreamTalk encoders are valid targets for cross-modal alignment.
    Sections 5.2 and 5.3 use these encoders as ground truth for content and style consistency losses, assuming the encoders reliably represent semantics and style.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmark." pith.science (2026). https://pith.science/paper/ASD43CSN

@misc{pith2026250204976,
  author       = {Pith},
  title        = {Pith review of: Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ASD43CSN}},
  note         = {Machine review of arXiv:2502.04976}
}
read the original abstract

Empathetic Response Generation (ERG) is one of the key tasks of the affective computing area, which aims to produce emotionally nuanced and compassionate responses to user's queries. However, existing ERG research is predominantly confined to the singleton text modality, limiting its effectiveness since human emotions are inherently conveyed through multiple modalities. To combat this, we introduce an avatar-based Multimodal ERG (MERG) task, entailing rich text, speech, and facial vision information. We first present a large-scale high-quality benchmark dataset, \textbf{AvaMERG}, which extends traditional text ERG by incorporating authentic human speech audio and dynamic talking-face avatar videos, encompassing a diverse range of avatar profiles and broadly covering various topics of real-world scenarios. Further, we deliberately tailor a system, named \textbf{Empatheia}, for MERG. Built upon a Multimodal Large Language Model (MLLM) with multimodal encoder, speech and avatar generators, Empatheia performs end-to-end MERG, with Chain-of-Empathetic reasoning mechanism integrated for enhanced empathy understanding and reasoning. Finally, we devise a list of empathetic-enhanced tuning strategies, strengthening the capabilities of emotional accuracy and content, avatar-profile consistency across modalities. Experimental results on AvaMERG data demonstrate that Empatheia consistently shows superior performance than baseline methods on both textual ERG and MERG. Overall, this work is expected to pioneer the MERG research by introducing a novel benchmark and an end-to-end model, laying a solid foundation for future advancements in multimodal empathetic response generation.

Figures

Figures reproduced from arXiv: 2502.04976 by the authors.

Figure 1
Figure 1. A snippet of avatar-based Multimodal Empathetic [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Visualized statistics of AvaMERG dataset. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Architecture of our Empatheia MLLM for MERG. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Illustration of the Content Synchronizer and Style [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Illustrations of the proposed training strategies. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Results on various emotions, ages, and genders. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: T-SNE visualization of emotion and profile features. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Qualitative results of two testing instances. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: The distribution of tone. applicability. AvaMERG sets itself apart with a well-balanced [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: The mapping of fine-grained textual emotions to [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Emotion-topic heatmap for dialogue utterances. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Impact of the transformer layer number. meticulous assessments of 200 test cases based on the aforemen￾tioned six aspects, grading them from 1 to 5 according to the model’s performance (where 1 represents extremely poor performance and 5 represents excellence). Prior …
Figure 14
Figure 14. Figure 14: Impact of loss weights. 0% 20% 50% 80% 100% Percentage 10 20 30 40 50 Acc. Pipeline Empatheia 0% 20% 50% 80% 100% Percentage 1 2 3 4 5 Avg.Score Pipeline Empatheia [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Performance comparison under different quanti [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: In-Depth Analysis of the Effectiveness of CoE. [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: Qualitative results A of MERG. xxxxxx [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: Qualitative results B of MERG. xxxxxx Current user query That sounds really tough. It's hard when your body doesn't cooperate like it used to. Have you found any little things that help you cope? × Response by Pipeline model Honestly, it sounds like you're just not tr…
Figure 21
Figure 21. Figure 21: Qualitative results A of CoE mechanism. xxxxxx [PITH_FULL_IMAGE:figures/full_fig_p023_21.png]
Figure 23
Figure 23. Figure 23: Qualitative results C of CoE mechanism. xxxxxx [PITH_FULL_IMAGE:figures/full_fig_p024_23.png]
Figure 24
Figure 24. Figure 24: Qualitative results D of CoE mechanism. xxxxxx [PITH_FULL_IMAGE:figures/full_fig_p025_24.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EmpaAva: An Open-source Agentic 3D-Avatar Empathetic Live Chatbot

    cs.CL 2026-08 conditional novelty 6.0 of 10

    EmpaAva is an open-source, LLM-orchestrated 3D avatar chatbot that perceives user affect from speech and video, plans empathetic replies, and delivers them with synchronized emotional speech and facial motion.

Reference graph

Works this paper leans on

81 extracted references · 50 canonical work pages · cited by 1 Pith paper

  1. [11]

    Hao Fei, Han Zhang, Bin Wang, Lizi Liao, Qian Liu, and Erik Cambria. 2024. EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbot. arXiv preprint arXiv:2406.15177 (2024)

    work page Pith review arXiv 2024

  2. [51]

    arXiv preprint arXiv:2405.18357 (2024)

    Faithful Logical Reasoning via Symbolic Chain-of-Thought. arXiv preprint arXiv:2405.18357 (2024)

    arXiv 2024

  3. [48]

    Haoqiu Yan, Yongxin Zhu, Kai Zheng, Bing Liu, Haoyu Cao, Deqiang Jiang, and Linli Xu. 2024. Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 15009–15022

    work page 2024

  4. [1]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    arXiv 2023

  5. [2]

    Changyu Chen, Yanran Li, Chen Wei, Jianwei Cui, Bin Wang, and Rui Yan. 2024. Empathetic Response Generation with Relation-aware Commonsense Knowledge. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining. 87–95

    2024

  6. [3]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https://lmsys.org/blog/2023-03-30-vicuna/

    2023

  7. [4]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research 25, 70 (2024), 1–53

    2024

  8. [5]

    Joon Son Chung and Andrew Zisserman. 2017. Out of time: automated lip sync in the wild. In Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II

    2017

Show all 81 references
  1. [6]

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. 2023. Dreamllm: Syn- ergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499 (2023)

    2023 arXiv

  2. [7]

    Hao Fei, Bobo Li, Qian Liu, Lidong Bing, Fei Li, and Tat-Seng Chua. 2023. Rea- soning Implicit Sentiment with Chain-of-Thought Prompting. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 1171–1182

    2023

  3. [8]

    Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan

  4. [9]

    Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua, and Shuicheng Yan. 2024. Enhancing video-language representations with structural spatio-temporal alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  5. [10]

    Hao Fei, Yuan Yao, Zhuosheng Zhang, Fuxiao Liu, Ao Zhang, and Tat-Seng Chua

  6. [12]

    In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024): Tutorial Summaries

    From Multimodal LLM to Human-level AI: Modality, Instruction, Reasoning, Efficiency and Beyond. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024): Tutorial Summaries. 1–8

    2024

  7. [13]

    Jun Gao, Yuhan Liu, Haolin Deng, Wei Wang, Yu Cao, Jiachen Du, and Ruifeng Xu. 2021. Improving empathetic response generation by recognizing emotion cause in conversations. In Findings of the association for computational linguistics: EMNLP 2021. 807–819

    2021

  8. [14]

    Hao Fei, Yue Zhang, Yafeng Ren, and Donghong Ji. 2020. Latent emotion memory for multi-label emotion classification. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34. 7692–7699

    2020

  9. [15]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  10. [16]

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Rus- lan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language process...

    2021

  11. [17]

    Bobo Li, Hao Fei, Fei Li, Yuhan Wu, Jinsong Zhang, Shengqiong Wu, Jingye Li, Yijiang Liu, Lizi Liao, Tat-Seng Chua, et al . 2022. Diaasq: A benchmark of conversational aspect-based sentiment quadruple analysis. arXiv preprint arXiv:2211.05705 (2022)

    2022 arXiv

  12. [18]

    Diederik P Kingma. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

    2013 arXiv

  13. [19]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning . 19730–19742

    2023

  14. [20]

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055 (2015)

    2015 arXiv

  15. [21]

    Qintong Li, Piji Li, Zhaochun Ren, Pengjie Ren, and Zhumin Chen. 2022. Knowl- edge bridging for empathetic dialogue generation. In Proceedings of the AAAI conference on artificial intelligence . 10993–11001

    2022

  16. [22]

    Jian Li and Weiheng Lu. 2024. A Survey on Benchmarks of Multimodal Large Language Models. arXiv preprint arXiv:2408.08632 (2024)

    2024 arXiv

  17. [23]

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 (2023)

    2023 arXiv

  18. [24]

    Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mes- garani. 2024. Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. Advances in Neural Information Processing Systems 36 (2024)

    2024

  19. [25]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruc- tion tuning. Advances in neural information processing systems 36 (2024)

    2024

  20. [26]

    Zhaojiang Lin, Andrea Madotto, Jamin Shin, Peng Xu, and Pascale Fung. 2019. Moel: Mixture of empathetic listeners. arXiv preprint arXiv:1908.07687 (2019)

    2019 arXiv

  21. [27]

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. 2024. Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision Language Audio and Action. In Proceedings of the IEEE/CVF Conference on Computer Vis...

    2024

  22. [28]

    Jaime Lorenzo-Trueba, Junichi Yamagishi, Tomoki Toda, Daisuke Saito, Fernando Villavicencio, Tomi Kinnunen, and Zhenhua Ling. 2018. The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods. arXiv preprint arXiv:1804.04262 (2018)

    2018 arXiv

  23. [29]

    Meng Luo, Han Zhang, Shengqiong Wu, Bobo Li, Hong Han, and Hao Fei. 2024. NUS-Emo at SemEval-2024 Task 3: Instruction-Tuning LLM for Multimodal Emotion-Cause Analysis in Conversations. InProceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024) . 1599–1606

    2024

  24. [30]

    Meng Luo, Hao Fei, Bobo Li, Shengqiong Wu, Qian Liu, Soujanya Poria, Erik Cambria, Mong-Li Lee, and Wynne Hsu. 2024. PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis. arXiv preprint arXiv:2408.09481 (2024)

    2024 arXiv

  25. [31]

    Navonil Majumder, Pengfei Hong, Shanshan Peng, Jiankun Lu, Deepanway Ghosal, Alexander Gelbukh, Rada Mihalcea, and Soujanya Poria. 2020. MIME: MIMicking emotions for empathetic response generation. arXiv preprint arXiv:2010.01454 (2020)

    2020 arXiv

  26. [32]

    Yifeng Ma, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yingya Zhang, and Zhidong Deng. 2023. Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:2312.09767 (2023)

    2023 arXiv

  27. [33]

    Yushan Qian, Wei-Nan Zhang, and Ting Liu. 2023. Harnessing the power of large language models for empathetic response generation: Empirical investigations and improvements. arXiv preprint arXiv:2310.05140 (2023)

    2023 arXiv

  28. [34]

    Niranjan D Narvekar and Lina J Karam. 2011. A no-reference image blur metric based on the cumulative probability of blur detection (CPBD). IEEE Transactions on Image Processing 20, 9 (2011), 2678–2683

    2011

  29. [35]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  30. [36]

    Aravind Sesagiri Raamkumar and Yinping Yang. 2022. Empathetic conversational systems: A review of current advances, gaps, and opportunities.IEEE Transactions on Affective Computing (2022), 2722–2739

    2022

  31. [37]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deep- speed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining ...

    2020

  32. [38]

    Hannah Rashkin. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207 (2018)

    2018 arXiv

  33. [39]

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai

  34. [40]

    Sahand Sabour, Chujie Zheng, and Minlie Huang. 2022. Cem: Commonsense- aware empathetic response generation. In Proceedings of the AAAI Conference on Artificial Intelligence. 11229–11237

    2022

  35. [41]

    Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008)

    2008

  36. [42]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  37. [43]

    GLM Team, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv e-prints (2024), arXiv–2406

    2024

  38. [44]

    Zhou Wang and Alan C Bovik. 2002. A universal image quality index.IEEE signal processing letters 9, 3 (2002), 81–84

    2002

  39. [45]

    Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. 2024. Towards Semantic Equivalence of Tokenization in Multimodal LLM. arXiv preprint arXiv:2406.05127 (2024)

    2024 arXiv

  40. [46]

    Mahesh Viswanathan and Madhubalan Viswanathan. 2005. Measuring speech quality for text-to-speech systems: development and assessment of a modified mean opinion score (MOS) scale.Computer speech & language 19, 1 (2005), 55–83

    2005

  41. [47]

    Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu

  42. [49]

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2024. NExT- GPT: Any-to-Any Multimodal LLM. In Proceedings of the International Conference on Machine Learning. 53366–53397. WWW ’25, April 28–May 2, 2025, Sydney, NSW, Australia. Han Zhang et al

    2024

  43. [50]

    Zhou Yang, Zhaochun Ren, Wang Yufeng, Shizhong Peng, Haizhou Sun, Xi- aofei Zhu, and Xiangwen Liao. 2024. Enhancing Empathetic Response Genera- tion by Augmenting LLMs with Small-scale Empathetic Models. arXiv preprint arXiv:2402.11801 (2024)

    2024 arXiv

  44. [52]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)

    2023 arXiv

  45. [53]

    Zhou Yang, Zhaochun Ren, Yufeng Wang, Xiaofei Zhu, Zhihao Chen, Tiecheng Cai, Yunbing Wu, Yisong Su, Sibo Ju, and Xiangwen Liao. 2024. Exploiting emotion-semantic correlations for empathetic response generation.arXiv preprint arXiv:2402.17437 (2024)

    2024 arXiv

  46. [54]

    Jinfeng Zhou, Chujie Zheng, Bo Wang, Zheng Zhang, and Minlie Huang. 2022. Case: Aligning coarse-to-fine cognition and affection for empathetic response generation. arXiv preprint arXiv:2208.08845 (2022)

    2022 arXiv

  47. [55]

    Yiqun Zhang, Fanheng Kong, Peidong Wang, Shuang Sun, SWangLing SWan- gLing, Shi Feng, Daling Wang, Yifei Zhang, and Kaisong Song. 2024. STICKER- CONV: Generating Multimodal Empathetic Responses from Scratch. In Proceed- ings of the 62nd Annual Meeting of the Association for Co...

    2024

  48. [57]

    Li Zheng, Donghong Ji, Fei Li, Hao Fei, Shengqiong Wu, Jingye Li, Bobo Li, and Chong Teng. 2023. ECQED: emotion-cause quadruple extraction in dialogs.arXiv preprint arXiv:2306.03969 (2023)

    2023 arXiv

  49. [59]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592 (2023). A Ethic Considerations We can identify the following potential ethical c...

    2023 arXiv

  50. [60]

    dia_id":

    Emotion Cause: Based on the conversation context, sentiment, and dialogue history, analyze and identify the underlying emotional cause or trigger for the speaker. 3. Goal to Response: Analyze the last response from the lis- tener and identify the intended goal behind that resp...

    2025

  51. [61]

    Speaker: When I left the bathroom in high school once I had toilet paper stuck to my shoe

  52. [62]

    Listener: I bet it was very embarrassing?

  53. [63]

    It’s like they have no empathy or think about what if it was them

    Speaker: Yeah it sure was, you know how mean teenagers can be. It’s like they have no empathy or think about what if it was them

  54. [64]

    Achievements and Self-Realization

    Listener: It’s alright, we’ve all been there many times. Speaker Emotion Embarrassed Event Scenario The speaker experienced embarrassment after accidentally leaving the bathroom with toilet paper stuck to their shoe. Emotion Cause Embarrassing public incident involving toilet ...

    2025

  55. [65]

    Empathy (Emp): whether the response demonstrates an under- standing of the user’s emotions and experiences, and responds appropriately

  56. [66]

    Coherence (Coh): whether the response is logically consistent and contextually relevant

  57. [67]

    Informativity (Inf): whether the response provides useful and meaningful information

  58. [68]

    Avg. Score

    Fluency (Flu): whether the response is grammatically well- formed and easy to read. E.3.2 Speech Generation Evaluation Metrics. For the speech generation component of MERG, we use subjective and objective metrics to assess the quality and emotional expres- siveness of the gene...

    2025

  59. [69]

    As shown, the model’s performance peaks when the number of tokens reaches 16

    Figure 13 illustrates the impact of different numbers of audio- visual special tokens on model performance. As shown, the model’s performance peaks when the number of tokens reaches 16. We scale the content and style learning losses using 𝛼 and 𝛽 to bring them closer to the ma...

    2025

  60. [70]

    Event Scenario: A breakup occurred during a heavy metal festival in Las Vegas while sharing a room with the ex-girlfriend

  61. [71]

    User Emotion: Lonely

  62. [72]

    Emotion Cause: The emotional turmoil was caused by the breakup during an otherwise enjoyable music festival, combined with the discomfort of being stuck in the same room with the ex-girlfriend. 4.Goal to Response: Acknowledge the conflicting emotions, validate the difficulty o...

    2025

  63. [73]

    Event Scenario: An emergency kit was packed for winter weather, which proved essential during a blizzard

  64. [74]

    User Emotion: Prepared

  65. [75]

    4.Goal to Response: Validate the speaker’s sense of relief and preparedness, acknowledging the stressful situation they avoided

    Emotion Cause: Relief and appreciation come from the foresight of packing the kit, which turned out to be essential when their tire went flat in the middle of a blizzard. 4.Goal to Response: Validate the speaker’s sense of relief and preparedness, acknowledging the stressful s...

  66. [76]

    Event Scenario: Eagerly counting down the days, anxiety built for the upcoming summer cruise

  67. [77]

    User Emotion: Excited

  68. [78]

    4.Goal to Response: To provide empathy and acknowledge the speaker’s excitement and enjoyment of the trip

    Emotion Cause: The anticipation and anxiety surrounding the upcoming trip. 4.Goal to Response: To provide empathy and acknowledge the speaker’s excitement and enjoyment of the trip. CoE reasoning in Empatheia A week before my cruise this summer, I was counting down the days wi...

    2025

  69. [79]

    Event Scenario: Reflecting on moments of losing temper with children without justification

  70. [80]

    User Emotion: Guilty

  71. [81]

    4.Goal to Response: Offer empathy and validation for the speaker’s feelings, acknowledging the challenge of parenting and the importance of communication

    Emotion Cause: Irritation and frustration accumulated throughout the day led to these outbursts. 4.Goal to Response: Offer empathy and validation for the speaker’s feelings, acknowledging the challenge of parenting and the importance of communication. CoE reasoning in Empathei...

  72. [2023]

    arXiv preprint arXiv:2305.16355 (2023)

    Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355 (2023)

    2023 arXiv

  73. [2024]

    Proceedings of the Advances in neural information processing systems

    VITRON: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing. Proceedings of the Advances in neural information processing systems

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.