REVIEW 5 cited by
AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The emergence of multimodal large language models (MLLMs) advances multimodal emotion recognition (MER) to the next level, from naive discriminative tasks to complex emotion understanding with advanced video understanding abilities and natural language description. However, the current community suffers from a lack of large-scale datasets with intensive, descriptive emotion annotations, as well as a multimodal-centric framework to maximize the potential of MLLMs for emotion understanding. To address this, we establish a new benchmark for MLLM-based emotion understanding with a novel dataset (MER-Caption) and a new model (AffectGPT). Utilizing our model-based crowd-sourcing data collection strategy, we construct the largest descriptive emotion dataset to date (by far), featuring over 2K fine-grained emotion categories across 115K samples. We also introduce the AffectGPT model, designed with pre-fusion operations to enhance multimodal integration. Finally, we present MER-UniBench, a unified benchmark with evaluation metrics tailored for typical MER tasks and the free-form, natural language output style of MLLMs. Extensive experimental results show AffectGPT's robust performance across various MER tasks. We have released both the code and the dataset to advance research and development in emotion understanding: https://github.com/zeroQiaoba/AffectGPT.
Forward citations
Cited by 5 Pith papers
-
COSI-Lab: Conference Living Lab for Modeling Multi-Perspective Multimodal Social Intention
COSI-Lab is a weakly scripted conference-workshop dataset with multi-perspective apparent-intent annotations, self-reported goals, and benchmarks for social intention inference and conversation group detection.
-
DeceptionX: From Multimodal Evidence to Explainable Deception Detection
DeceptionX trains a multimodal LLM to explain lie judgments with visual and audio evidence, but its benchmark gains rest on overlapping train/test data and label-conditioned annotations.
-
Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captions
A new large-scale facial emotion caption dataset and a global-local contrastive training framework with positive mining improve zero-shot facial expression recognition.
-
Toward Annotation-Efficient Continuous Emotion Arousal Quantification via Group-Level EEG Dynamic Neural Synchrony
Group-level EEG dynamic neural synchrony (CorrCA) preferentially tracks the rate of change of continuous arousal and shows valence-dependent structure across four datasets.
-
EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models
EMO-R3, which combines a three-step emotional reasoning prompt with a reward for the model agreeing with its own image–emotion judgments, raises visual emotion-recognition accuracy by about one point over plain GRPO.
Discussion (0). Sign in to comment.