Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

FEALLM: Advancing Facial Emotion Analysis in Multimodal Large Language Models with Emotional Synergy and Reasoning

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FEALLM claims that a multimodal LLM trained on aligned FE and AU instruction data with explicit reasoning, plus local-detail and shallow-feature fusion, outperforms existing MLLMs on facial emotion analysis and generalizes zero-shot.

desk verdict Useful FE+AU instruction dataset, joint benchmark, and architecture with modest but consistent gains; the causal reasoning claim outruns the evidence. read the letter →

arxiv 2505.13419 v1 pith:4NJZJVON submitted 2025-05-19 cs.CV

Zhuozhao Hu , Kaishen Yuan , Xin Liu , Zitong Yu , Yuan Zong , Jingang Shi , Huanjing Yue , Jingyu Yang This is my paper · ORCID
classification cs.CV
keywords facialemotionanalysismultimodallargelanguagemodelactionunitdetectionexpressionrecognitioninstructiontuningreasoningzero-shotgeneralizationFEABench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a multimodal large language model can perform facial emotion analysis, recognizing both coarse expressions and fine-grained action units, when trained on instruction data that explicitly connects the two. It builds a facial emotion analysis instruction dataset from expert-annotated images, adds generated reasoning text that derives an emotion from its action units, and introduces FEABench to evaluate both tasks together. It then proposes a model architecture that feeds local face crops and shallow visual features into the language model, arguing that these details let the model link muscle movements to emotions. A sympathetic reader would care because the claim, if true, moves MLLMs from black-box expression classification toward interpretable, causal emotion reasoning that transfers to new datasets.

What carries the argument

The central machinery is the pairing of an instruction dataset with a detail-capturing model. The Local Clue Aggregator (LCA) crops each face into sixteen local regions (eight directions at half and three-quarters of the side length), runs each through a convolutional block, applies self-attention over the region tokens, and projects them into the LLM embedding space. The Multi-perspective Projector (MPP) takes feature maps from five layers of the visual encoder, uses the deepest as query and the shallower ones as key and value in a cross-attention step, fuses those with the LCA's local features through a second cross-attention with a learnable residual, and aligns the result with a two-layer MLP. The claim carried by these modules is that low-level texture and local muscle regions, which global visual features suppress, are precisely the evidence needed to reason from action units to emotions.

What would settle it

Take a fixed set of face images and feed the text generator deliberately wrong AU labels while keeping the image unchanged; if the generated reasoning text follows the wrong labels fluently and plausibly, then the reasoning is label-conditioned rather than image-grounded, and a model trained on it will inherit that spurious link. A second check is to have human raters mark whether each AU mentioned in the reasoning text is actually visible in the image; systematic mismatches would falsify the grounding claim.

Watch

Extended reading notes

Core claim

On the proposed FEABench, the model reaches 67.36% accuracy on facial expression recognition and 59.60% average F1 across twelve action units, outperforming the same base model fine-tuned on the same instruction data without the new modules (59.97% and 55.79%). In zero-shot settings it reaches 69.95% on RAF-DB and 41.91% on AffectNet, and average AU F1 of 45.98% on BP4D and 42.61% on DISFA. The paper attributes these gains to two mechanisms: instruction data that includes emotion-reasoning descriptions bridging AUs to emotions, and an architecture that adds local facial details and low-level encoder features to the global visual tokens. It also reports that the reasoning instruction component alone lifts FER accuracy from 63.65% to 67.36% and AU F1 from 58.79% to 59.60%, and that removing any cropping direction or shallow/local feature in the modules hurts both tasks.

Load-bearing premise

The load-bearing premise is that the text generated by GPT-4o, given only the image and its correct labels, is genuinely grounded in what is visible in the face and causally faithful to how AUs produce emotions; if the reasoning is fluent but merely restates the labels, the claimed emotional synergy and reasoning would not be real.

Editorial extensions

If this is right

  • Joint instruction tuning with aligned FE and AU descriptions lets one model handle both expression recognition and action-unit detection, so separate task-specific heads are not required.
  • Emotion-reasoning text contributes beyond the underlying labels alone: adding it improves both FER and AUD, suggesting that explicit muscle-to-emotion chains are a useful training signal.
  • Local crops and shallow visual features each add independent gains, so detail injection is a viable way to improve MLLMs on fine-grained visual tasks.
  • Zero-shot results on four external datasets indicate that this style of instruction tuning transfers to new subjects and new annotation schemes without task-specific fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test is to swap the AU labels supplied to the text generator while keeping the image fixed; if generated reasoning follows the wrong labels fluently, the model is learning label-conditioned scripts rather than image-grounded causes.
  • The reasoning text may also be replaceable by deterministic templates that map each AU set to canned emotion chains; matching FEALLM's gains with templates would show the content, not the generation model, is what matters.
  • The LCA/MPP recipe suggests a general design principle for MLLMs on fine-grained recognition: explicitly crop salient regions and re-inject shallow features instead of relying on global attention to discover small visual clues.
  • FEABench could be extended to score reasoning consistency, checking that the emotion the model states is logically entailed by the AUs it reports, so that emotional reasoning is measured rather than just fluency.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes FEALLM, a multimodal large language model for joint facial expression recognition (FER) and action unit detection (AUD). The authors construct an FEA instruction dataset from Aff-Wild2 (14,892 training images, 1,335 test images from disjoint subjects), using GPT-4o prompted with both the face image and ground-truth FE/AU labels to generate three types of instruction text: emotion summaries, facial movement descriptions, and emotion reasoning descriptions. They also introduce FEABench, a benchmark that evaluates FER and AUD jointly on the held-out portion of this data. FEALLM augments LLaVA-1.5-7B with a Local Clue Aggregator (LCA) that processes 16 cropped facial regions and a Multi-perspective Projector (MPP) that fuses shallow visual-encoder features with local features via cross-attention. Fine-tuning only the LCA, MPP, and LoRA modules, the model achieves 67.36% FER accuracy and 59.60% average AU F1 on FEABench, and is evaluated zero-shot on RAF-DB, AffectNet, BP4D, and DISFA. Ablations show positive contributions from the instruction types and from the LCA and MPP modules.

Significance. If the results are validated under a fairer comparison protocol, the paper would make a useful contribution by providing a joint FER/AUD instruction dataset and benchmark, and by demonstrating that MLLMs can be adapted to facial detail perception through local crops and shallow-feature fusion. The dataset and code release are positive; the claim to be the first to establish causal reasoning between AUs and FEs in instruction data, if properly supported, would be of interest to the affective computing and MLLM communities. However, the central claims of genuine emotional reasoning and impressive zero-shot generalization are currently not fully supported by the experimental design, and several load-bearing evaluation choices need to be addressed before the paper's main conclusions can be accepted.

major comments (5)
  1. [Section 3.2, Figure 3] The FEA instruction data generation pipeline uses GPT-4o with both the image and the ground-truth FE/AU labels as input, and the only validation step is checking consistency between the generated text and the labels. Section 3.1 itself warns that GPT-generated annotations are unreliable (e.g., EmoLA's AU F1 of 76% and AU10 F1 of 58.1%), yet no manual review or visual-grounding check is applied to GPT-4o's free-text facial-movement and reasoning descriptions. If the generated descriptions contain plausible but image-inconsistent statements (e.g., 'raised cheeks' when AU6 is annotated but not visibly present), fine-tuning on this text can teach the model to produce label-derived rationales rather than perceptually grounded reasoning. Because the FEABench evaluation set is sampled from the same Aff-Wild2 source as the training set, shared FE-AU co-occurrence statistics could inflate AUD gains without genuine AU perception. This directly threatens the paper's central claims of emotional synergy and causal reasoning, and the authors should provide evidence that the generated text is visually grounded, for example by human evaluation on a sampled subset or by an image-only generation setting without label prompting.
  2. [Section 5.2, Table 1] The FEABench comparison is mostly against zero-shot MLLMs, which is not an apples-to-apples baseline for a model fine-tuned on the proposed FEA instruction dataset. The only fine-tuned baseline is LLaVA-1.5+LoRA. To support the claim that FEALLM is the best method on FEABench and that the LCA and MPP modules are responsible for the gains, the authors should fine-tune other MLLMs (e.g., Qwen-VL-Chat, InternVL2, MiniCPM-o) on the same FEA instruction data under the same training protocol. Without such comparisons, the margins in Table 1 may largely reflect the presence or absence of instruction tuning rather than the proposed architecture.
  3. [Section 5.2, Tables 1, 3, 4, 5] No error bars, multiple seeds, or significance tests are reported for any experiment. This is particularly important because training runs for only one epoch, the fine-tuned baseline margins are small (e.g., FEABench average AU F1 59.60 vs. 55.79; BP4D average 45.98 vs. 45.51), and the zero-shot AUD results show the fine-tuned LLaVA+LoRA baseline outperforming FEALLM on several individual AUs (e.g., BP4D AU6 74.83 vs. 72.90, AU12 85.58 vs. 84.88; DISFA AU12 60.20 vs. 54.50). The authors should report mean and standard deviation over at least three independent fine-tuning runs, or provide a clear justification for why a single run is sufficient.
  4. [Section 5.3, Table 5] The zero-shot AUD evaluation on BP4D and DISFA uses 2% uniform frame sampling and only evaluates the AUs shared with the FEA instruction dataset. This protocol deviates from standard practice in the AUD literature, which typically reports frame-level F1 on the full test set, and it makes the numbers in Table 5 not directly comparable with established AUD baselines. The authors should either adopt the standard evaluation protocol or clearly state the deviation and its implications in the main text. In addition, given that the fine-tuned LLaVA+LoRA baseline beats FEALLM on several AUs, the paper's phrase 'impressive generalization capability' is not sufficiently supported
  5. [Section 5.4, Table 3] The claim that adding emotion reasoning instructions yields 'substantial progress' in both FER and AUD is overstated: the gain is +3.71 accuracy points for FER (63.65 to 67.36) but only +0.81 F1 points for AUD (58.79 to 59.60). Moreover, the ablation adds reasoning text as an additional training signal, so it does not isolate the effect of reasoning supervision from the effect of simply having more instruction examples. A control condition that adds an equal amount of non-reasoning instruction text (e.g., more emotion-summary or facial-movement templates) is needed to support the claim that reasoning instructions, not just data quantity, drive the improvement.
minor comments (5)
  1. [Section 3.2] The paper states that 'more than ten question templates' are used per instruction type and refers to the Appendix for details, but the submitted version does not include an appendix with these templates or the cropping illustration, which limits reproducibility of the dataset construction.
  2. [Section 5.1] The pre-training dataset is referred to as 'LLaVA-LCS-558K'; this appears to be a typo for 'LLaVA-558K' or 'LLaVA-LCS-558K' depending on the actual split, and should be corrected for clarity.
  3. [Section 3.3] The evaluation metric for AUD is described as F1, but the paper does not specify how per-AU F1 scores are aggregated into the reported average (macro-average over AUs is implied but should be stated explicitly), nor how the regular-expression extraction handles AUs that are mentioned in negation or absent.
  4. [Figure 7] There is a grammatical error in the caption and example text: 'a open mouth' should be 'an open mouth'.
  5. [Section 4.3, Eq. (6)-(7)] The learnable scaling factors gamma_1 and gamma_2 are introduced but their initialization is not specified; please state the initial values and whether they are scalar or per-channel parameters.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central FEA results are supervised learning on expert-validated labels with external zero-shot checks; GPT-4o text generation is a data-quality caveat, not a circular derivation.

full rationale

The derivation chain is self-contained. FEALLM's LCA and MPP modules are justified by ablations on held-out FEABench subjects (Tables 3-4, Figs. 5-6), not by fitting to the evaluation labels. The zero-shot results on RAF-DB, AffectNet, BP4D, and DISFA (Tables 2 and 5) use external annotations independent of the training data, so the generalization claim has independent grounding. FEABench evaluation uses a subject-disjoint split of the same Aff-Wild2 source as training, which is a standard supervised benchmark split; the evaluation labels are expert annotations, not outputs of the model or of a parameter fitted to the test set. The FEA instruction data are generated by prompting GPT-4o with both the image and the ground-truth FE/AU labels (Section 3.2), so the resulting reasoning text is label-conditioned; this is a legitimate concern about whether the model's fluent reasoning is visually grounded, and Section 3.1 itself warns that GPT-generated annotations can be unreliable. However, this is a data-quality and validity limitation, not circularity by construction: the paper does not evaluate reasoning quality using the generated text as ground truth, and the quantitative FE/AU claims are measured against expert labels. Multiple self-citations appear (e.g., Refs. [31,32,33,44,45,51]), but none is load-bearing; they support related-work context and metric conventions, not the paper's central derivation. No uniqueness theorem, ansatz, or fitted prediction is imported from the authors' prior work. Therefore no circular step meets the quoted-evidence bar.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. The central empirical claims rest on hand-selected architecture hyperparameters and on assumptions about the reliability of Aff-Wild2 labels, GPT-4o's generated descriptions, and the representativeness of the evaluation protocol.

free parameters (5)
  • LoRA rank = 128
    Set in Section 5.1 for LLM fine-tuning; no search reported.
  • Number of local crops = 16 (8 directions x 2 sizes)
    Section 4.2: crops at half and three-quarters side length; ablation in Fig. 5 shows each cropping type matters, but the specific count is hand-picked.
  • Sampled visual encoder layers = L=5, layers 3,8,13,18 (shallow) and 23 (deep)
    Section 5.1: uniformly sampled; the choice of layer indices is a design decision.
  • Conv channels and kernel = d=64, 3x3
    Section 5.1: architecture hyperparameter for the Local Clue Aggregator.
  • Training schedule = learning rates 1e-3, 2e-5, 2e-4; one epoch; batch sizes 64/16
    Section 5.1: chosen by hand; no sensitivity analysis reported.
assumptions (5)
  • domain assumption Aff-Wild2 FE and AU annotations are reliable and validated by multiple domain experts.
    Section 3.1: the entire dataset construction rests on these labels; no independent verification by the authors.
  • domain assumption GPT-4o descriptions, generated from image plus ground-truth labels, are accurate and consistent with the annotations after validation.
    Section 3.2: the validation step is described but its criteria and failure rate are not quantified.
  • domain assumption The seven basic emotions and 12 AUs are a sufficient representation of facial emotion for training and evaluation.
    Section 3.1: follows Aff-Wild2 and FACS; limits generalization to finer-grained or compound emotions.
  • ad hoc to paper The generated 'emotion reasoning' text captures causal relationships between AUs and FEs.
    Section 3.2 and Figure 3: this is the load-bearing assumption behind the reasoning claim; no causal validation is provided.
  • domain assumption Zero-shot evaluation on 2% uniform frame samples and shared AUs is representative of model performance on BP4D and DISFA.
    Section 5.3: sampling and shared-AU selection affect comparability across methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FEALLM: Advancing Facial Emotion Analysis in Multimodal Large Language Models with Emotional Synergy and Reasoning." pith.science (2026). https://pith.science/paper/4NJZJVON

@misc{pith2026250513419,
  author       = {Pith},
  title        = {Pith review of: FEALLM: Advancing Facial Emotion Analysis in Multimodal Large Language Models with Emotional Synergy and Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4NJZJVON}},
  note         = {Machine review of arXiv:2505.13419}
}
read the original abstract

Facial Emotion Analysis (FEA) plays a crucial role in visual affective computing, aiming to infer a person's emotional state based on facial data. Scientifically, facial expressions (FEs) result from the coordinated movement of facial muscles, which can be decomposed into specific action units (AUs) that provide detailed emotional insights. However, traditional methods often struggle with limited interpretability, constrained generalization and reasoning abilities. Recently, Multimodal Large Language Models (MLLMs) have shown exceptional performance in various visual tasks, while they still face significant challenges in FEA due to the lack of specialized datasets and their inability to capture the intricate relationships between FEs and AUs. To address these issues, we introduce a novel FEA Instruction Dataset that provides accurate and aligned FE and AU descriptions and establishes causal reasoning relationships between them, followed by constructing a new benchmark, FEABench. Moreover, we propose FEALLM, a novel MLLM architecture designed to capture more detailed facial information, enhancing its capability in FEA tasks. Our model demonstrates strong performance on FEABench and impressive generalization capability through zero-shot evaluation on various datasets, including RAF-DB, AffectNet, BP4D, and DISFA, showcasing its robustness and effectiveness in FEA tasks. The dataset and code will be available at https://github.com/953206211/FEALLM.

Figures

Figures reproduced from arXiv: 2505.13419 by the authors.

Figure 1
Figure 1. Illustration of the innovativeness of the proposed [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) The pipeline for generating FEA instruction data, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An example of the FEA Instruction Dataset. The [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: (a) The overview of the FEALLM architecture. (b) The Multi-perspective Projector integrates shallow features from [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Ablation study on shallow and local features in the [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Visualization examples of FEALLM in FEA tasks. The face image (a) is from the FER dataset RAF-DB, (b) is from the [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CoEmoGen: Towards Semantically-Coherent and Scalable Emotional Image Content Generation

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    CoEmoGen generates emotionally faithful images from emotion categories using MLLM-crafted captions and a hierarchical LoRA module, validated on a new EmoArt dataset.

Reference graph

Works this paper leans on

60 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report.arXiv preprint arXiv:2309.16609(2023)

    arXiv 2023

  3. [3]

    Simone Bianco, Luigi Celona, Marco Donzella, and Paolo Napoletano. 2023. Im- proving image captioning descriptiveness by ranking and llm-based fusion.arXiv preprint arXiv:2306.11593(2023)

    arXiv 2023

  4. [4]

    Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2024. Wiki-LLaVA: Hierarchical Retrieval- Augmented Generation for Multimodal LLMs. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1818–1826

    work page 2024

  5. [5]

    Yue Cao, Yangzhou Liu, Zhe Chen, Guangchen Shi, Wenhai Wang, Danhuai Zhao, and Tong Lu. 2024. MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understanding.arXiv preprint arXiv:2410.11829 (2024)

    arXiv 2024

  6. [6]

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning.arXiv preprint arXiv:2310.09478(2023)

    arXiv 2023

  7. [7]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al . 2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271(2024)

    arXiv 2024

  8. [8]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024. How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open- Source Suites.arXiv preprint arXiv:2404.16821(2024)

    arXiv 2024

Show all 60 references
  1. [9]

    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. 2024. Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning.arXiv preprint arXiv:2406.11161(2024)

    2024 arXiv

  2. [10]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https://lmsys.org/blog/2023-...

    2023

  3. [11]

    Wenliang Dai, Junnan Li, DONGXU LI, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023. InstructBLIP: To- wards General-purpose Vision-Language Models with Instruction Tuning. In Advances in Neural Information Processing Systems, A. Oh, T...

    2023

  4. [12]

    Paul Ekman. 2003. Darwin, deception, and facial expression.Annals of the new York Academy of sciences1000, 1 (2003), 205–221

    2003

  5. [13]

    Paul Ekman and Wallace V Friesen. 1978. Facial action coding system.Environ- mental Psychology & Nonverbal Behavior(1978)

    1978

  6. [14]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685(2021)

    2021 arXiv

  7. [15]

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. 2024. Minicpm: Unveiling the potential of small language models with scalable training strategies.arXiv preprint arXiv:2404.06395(2024)

    2024 arXiv

  8. [16]

    Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. 2024. Bliva: A simple multimodal llm for better handling of text-rich visual questions. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 2256–2264

    2024

  9. [17]

    László A Jeni, Jeffrey F Cohn, and Fernando De La Torre. 2013. Facing imbalanced data–recommendations for the use of performance metrics. In2013 Humaine association conference on affective computing and intelligent interaction. IEEE, 245–251

    2013

  10. [18]

    D Kollias and S Zafeiriou. 2018. Aff-wild2: Extending the aff-wild database for affect recognition. arXiv 2018.arXiv preprint arXiv:1811.077702 (2018)

    2018 arXiv

  11. [19]

    Xing Lan, Jian Xue, Ji Qi, Dongmei Jiang, Ke Lu, and Tat-Seng Chua. 2024. Ex- pllm: Towards chain of thought for facial expression recognition.arXiv preprint arXiv:2409.02828(2024)

    2024 arXiv

  12. [20]

    Shih-Chieh Lee, Gong-Hong Lin, Chen-Chung Liu, En-Chi Chiu, and Ching-Lin Hsieh. 2021. Development of the CAT–FER: A computerized adaptive test of facial emotion recognition for adults with schizophrenia.The American Journal of Occupational Therapy75, 1 (2021), 7501205140p1–75...

    2021

  13. [21]

    Guanbin Li, Xin Zhu, Yirui Zeng, Qing Wang, and Liang Lin. 2019. Semantic relationships guided representation learning for facial action unit recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 8594–8601

    2019

  14. [22]

    Hanting Li, Mingzhe Sui, Feng Zhao, Zhengjun Zha, and Feng Wu. 2021. MVT: mask vision transformer for facial expression recognition in the wild.arXiv preprint arXiv:2106.04520(2021)

    2021 arXiv

  15. [23]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning. PMLR, 19730–19742

    2023

  16. [24]

    Shan Li and Weihong Deng. 2020. Deep facial expression recognition: A survey. IEEE transactions on affective computing13, 3 (2020), 1195–1215

    2020

  17. [25]

    Shan Li, Weihong Deng, and JunPing Du. 2017. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. InProceedings of the IEEE conference on computer vision and pattern recognition. 2852–2861

    2017

  18. [26]

    Yifan Li, Anh Dao, Wentao Bao, Zhen Tan, Tianlong Chen, Huan Liu, and Yu Kong

  19. [27]

    Yifan Li, Zhixin Lai, Wentao Bao, Zhen Tan, Anh Dao, Kewei Sui, Jiayi Shen, Dong Liu, Huan Liu, and Yu Kong. 2025. Visual Large Language Models for Generalized and Specialized Applications.arXiv preprint arXiv:2501.02765(2025)

    2025 arXiv

  20. [28]

    Zheng Lian, Licai Sun, Haiyang Sun, Kang Chen, Zhuofan Wen, Hao Gu, Bin Liu, and Jianhua Tao. 2024. Gpt-4v with emotion: A zero-shot benchmark for generalized emotion recognition.Information Fusion108 (2024), 102367

    2024

  21. [29]

    Fangyu Liu, Guy Emerson, and Nigel Collier. 2023. Visual spatial reasoning. Transactions of the Association for Computational Linguistics11 (2023), 635–651

    2023

  22. [30]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26296–26306

    2024

  23. [31]

    Xin Liu, Henglin Shi, Haoyu Chen, Zitong Yu, Xiaobai Li, and Guoying Zhao

  24. [32]

    Xin Liu, Kaishen Yuan, Xuesong Niu, Jingang Shi, Zitong Yu, Huanjing Yue, and Jingyu Yang. 2024. Multi-scale promoted self-adjusting correlation learning for facial action unit detection.IEEE Transactions on Affective Computing(2024)

    2024

  25. [33]

    Hao Lu, Xuesong Niu, Jiyao Wang, Yin Wang, Qingyong Hu, Jiaqi Tang, Yuting Zhang, Kaishen Yuan, Bin Huang, Zitong Yu, et al. 2024. Gpt as psychologist? preliminary evaluations for gpt-4v on visual affective computing. InProceedings of the IEEE/CVF Conference on Computer Vision...

    2024

  26. [34]

    Cheng Luo, Siyang Song, Weicheng Xie, Linlin Shen, and Hatice Gunes. 2022. Learning multi-dimensional edge feature-based au relation graph for facial action unit recognition.arXiv preprint arXiv:2205.01782(2022)

    2022 arXiv

  27. [35]

    Fuyan Ma, Bin Sun, and Shutao Li. 2021. Facial expression recognition with visual transformers and attentional selective fusion.IEEE Transactions on Affective Computing14, 2 (2021), 1236–1248

    2021

  28. [36]

    Brais Martinez, Michel F Valstar, Bihan Jiang, and Maja Pantic. 2017. Automatic analysis of facial actions: A survey.IEEE transactions on affective computing10, 3 (2017), 325–347

    2017

  29. [37]

    S Mohammad Mavadati, Mohammad H Mahoor, Kevin Bartlett, Philip Trinh, and Jeffrey F Cohn. 2013. Disfa: A spontaneous facial action intensity database.IEEE Transactions on Affective Computing4, 2 (2013), 151–160

    2013

  30. [38]

    Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. 2017. Affectnet: A database for facial expression, valence, and arousal computing in the wild.IEEE Transactions on Affective Computing10, 1 (2017), 18–31

    2017

  31. [39]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. InInternational conference on machine learnin...

    2021

  32. [40]

    Yan Shi, Zijun Zhang, Kaining Huang, Wudi Ma, and Shanshan Tu. 2020. Human- computer interaction based on face feature localization.Journal of Visual Com- munication and Image Representation70 (2020), 102740

    2020

  33. [41]

    Igor Sterner, Weizhe Lin, Jinghong Chen, and Bill Byrne. 2024. Few-Shot VQA with Frozen LLMs: A Tale of Two Approaches.arXiv preprint arXiv:2403.11317 (2024)

    2024 arXiv

  34. [42]

    Zhengyao Wen, Wenzhong Lin, Tao Wang, and Ge Xu. 2023. Distract your attention: Multi-head cross attention network for facial expression recognition. Biomimetics8, 2 (2023), 199

    2023

  35. [43]

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, et al. 2024. Q-instruct: Improving low-level visual abilities for multi-modality foundation models. In Proceedings of the IEEE/CVF Conference on Comp...

    2024

  36. [44]

    Bohao Xing, Zitong Yu, Xin Liu, Kaishen Yuan, Qilang Ye, Weicheng Xie, Huanjing Yue, Jingyu Yang, and Heikki Kälviäinen. 2024. Emo-llama: Enhancing facial emotion understanding with instruction tuning.arXiv preprint arXiv:2408.11424 (2024)

    2024 arXiv

  37. [45]

    Bohao Xing, Kaishen Yuan, Zitong Yu, Xin Liu, and Heikki Kälviäinen. 2025. AU-TTT: Vision Test-Time Training model for Facial Action Unit Detection. arXiv preprint arXiv:2503.23450(2025)

    2025 arXiv

  38. [46]

    Jing Yang, Jie Shen, Yiming Lin, Yordan Hristov, and Maja Pantic. 2023. FAN-Trans: Online Knowledge Distillation for Facial Action Unit Detection. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 6019–6027

    2023

  39. [47]

    Qu Yang, Mang Ye, and Bo Du. 2024. Emollm: Multimodal emotional understand- ing meets large language models.arXiv preprint arXiv:2406.16442(2024)

    2024 arXiv

  40. [48]

    Huanjin Yao, Wenhao Wu, Taojiannan Yang, YuXin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li, Wanli Ouyang, and Jingdong Wang

  41. [49]

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. 2024. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2024

  42. [50]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models.arXiv preprint arXiv:2306.13549(2023)

    2023 arXiv

  43. [51]

    Kaishen Yuan, Zitong Yu, Xin Liu, Weicheng Xie, Huanjing Yue, and Jingyu Yang

  44. [52]

    Dan Zeng, Zhiyuan Lin, Xiao Yan, Yuting Liu, Fei Wang, and Bo Tang. 2022. Face2exp: Combating data biases for facial expression recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20291– 20300

    2022

  45. [53]

    Xing Zhang, Lijun Yin, Jeffrey F Cohn, Shaun Canavan, Michael Reale, Andy Horowitz, Peng Liu, and Jeffrey M Girard. 2014. Bp4d-spontaneous: a high- resolution spontaneous 3d dynamic facial expression database.Image and Vision Computing32, 10 (2014), 692–706

    2014

  46. [54]

    Yuhang Zhang, Chengrui Wang, and Weihong Deng. 2021. Relative uncertainty learning for facial expression recognition.Advances in Neural Information Pro- cessing Systems34 (2021), 17616–17627

    2021

  47. [55]

    InEuropean Conference on Computer Vision

    Auformer: Vision transformers are parameter-efficient facial action unit detectors. InEuropean Conference on Computer Vision. Springer, 427–445

  48. [56]

    Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2020. Deformable detr: Deformable transformers for end-to-end object detection.arXiv preprint arXiv:2010.04159(2020)

    2020 arXiv

  49. [59]

    Yuhang Zhang, Chengrui Wang, Xu Ling, and Weihong Deng. 2022. Learn from all: Erasing attention consistency for noisy label facial expression recognition. In European Conference on Computer Vision. Springer, 418–434

    2022

  50. [2021]

    InProceedings of the IEEE/CVF conference on computer vision and pattern recognition

    imigue: An identity-free video dataset for micro-gesture understanding and emotion analysis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10631–10642

  51. [2024]

    Dense Connector for MLLMs.arXiv preprint arXiv:2405.13800(2024)

    2024 arXiv

  52. [2025]

    InEuropean Conference on Computer Vision

    Facial affective behavior analysis with instruction tuning. InEuropean Conference on Computer Vision. Springer, 165–186

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.