Pith. sign in

REVIEW 5 major objections 5 minor 58 references

Emotion Knowledge Enhancement for Vision Large Language Models: A Self-Verification Approach for High-Quality Emotion Instruction Data Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning a vision LLM on self-verified three-level emotion instruction data beats GPT-4o on all three facial emotion tasks.

desk verdict A useful recipe for generating multi-grained emotion instruction data, but the main empirical claim is undermined by an unvalidated LLM extraction step that likely favors the fine-tuned model. read the letter →

arxiv 2505.18168 v1 pith:PQADGMTQ submitted 2025-05-14 cs.LG cs.GR

Feifan Wang , Tengfei Song , Minggui He , Chang Su , Zhanglin Wu , Hao Yang , Wenming Zheng , Osamu Yoshie This is my paper · ORCID
classification cs.LGcs.GR
keywords facialemotionanalysisvisionlargelanguagemodelinstructiondatagenerationactionunitdetectionvalence-arousalestimationepistemicuncertaintyMonteCarlosamplingself-verification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a vision large language model can acquire fine-grained facial emotion perception without expensive expert annotation. Its pipeline, SEKE, uses a closed-source VLLM to fill in missing emotion labels, guided by existing human annotations and by the correlations among three description levels: discrete expression, valence-arousal, and action units. An uncertainty-aware Monte Carlo sampling step filters the most reliable predictions and feeds them back into a summary prompt, producing the FEID instruction dataset. Fine-tuning LLaVA-OneVision on FEID is reported to surpass GPT-4o and six open-source models on the FEAB benchmark across expression classification, action-unit detection, and valence/arousal estimation. If these results hold, they indicate that data completeness and correlation reasoning, rather than model scale, are the deciding factors for VLLM emotion understanding.

What carries the argument

The load-bearing mechanism is the inherent correlation among the three emotion description levels, used as a generative constraint. Given partial manual labels, SEKE prompts GPT-4o to predict the missing labels while reasoning through these correlations, and each FEID sample records all three descriptions plus correlation reasoning. Reliability is enforced by SV-UAMC: the model samples repeatedly, starting at two and capped at five, estimates epistemic uncertainty per task from sample variance, and probabilistically continues sampling when the maximum normalized uncertainty is high; the collected predictions and uncertainty statistics are then summarized into a final instruction. Fine-tuning LLaVA-OneVision on the 26,238 generated samples is the training mechanism that turns this data into a perception model.

What would settle it

Re-run the SEKE model on a facial emotion test set drawn from sources with no subject or image overlap with the six datasets used to build FEID; if the margin over GPT-4o collapses, the gains are dataset-specific. Separately, parse the same model outputs with two independent extraction methods, a deterministic regex parser and a different LLM, and compare the metrics; material discrepancies would show that the evaluation pipeline, not the model, produced part of the reported advantage.

Watch

Extended reading notes

Core claim

The central claim is that comprehensive, correlated instruction data is what unlocks emotion perception in VLLMs. On the FEAB benchmark, the SEKE fine-tuned model reaches 69.4% expression accuracy, 54.9% average action-unit F1, and valence and arousal MAE of 0.295 and 0.231, against GPT-4o's 61.1%, 29.0%, 0.342, and 0.291. Ablations show that dropping either the human prior-knowledge guidance or the uncertainty-aware self-verification step lowers performance, and that training on any single description type degrades all three tasks. The paper reads this as evidence that a model learns to triangulate coarse and fine cues when every training sample carries all three descriptions plus a reasoning sentence connecting them.

Load-bearing premise

The evaluation's fairness rests on the assumption that holding out one tenth of subjects from the same six datasets that supplied FEID training images removes dataset-specific transfer, and that the LLM used to extract numeric answers from free-form model text is unbiased; if either fails, the reported superiority could be an artifact of the test setup rather than of the generated instructions.

Editorial extensions

If this is right

  • On FEAB, the SEKE-trained LLaVA-OneVision beats GPT-4o by 8.3 points in expression accuracy, 25.9 points in average action-unit F1, and lowers valence/arousal MAE by 0.047/0.060.
  • Ablation against alternative instruction datasets (EmoVIT, MAFW, MERR-Fine) shows FEID produces the strongest downstream model, implying existing emotion instruction data lacks enough fine-grained cues.
  • Removing either prior-knowledge guidance or SV-UAMC from the generation pipeline measurably hurts the fine-tuned model, so both components carry the reported gain.
  • Training on only one of the three description types declines on all tasks, implying the correlation knowledge, not any single label type, drives the improvement.
  • On Aff-Wild2, SEKE-generated missing labels align with manual labels better than direct GPT queries, with a 16.9% action-unit F1 improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same fill-in-the-missing-labels loop should transfer to other domains where multiple correlated annotation levels exist, such as structured medical image findings, provided one level is cheap to obtain.
  • Because GPT-4o both generated the data and is beaten by the fine-tuned model, the pipeline suggests that diversity and correlation structure in synthetic instruction data can elicit capabilities the generator cannot reliably exhibit zero-shot; that is a testable generalization.
  • The subject-holdout split still shares image provenance with training data, so evaluating the SEKE model on an entirely unrelated emotion dataset would clarify whether the gains are genuine perception or dataset statistics.
  • The uncertainty-aware acceptance rule is a general-purpose quality filter: using a different closed-source generator or a larger sampling cap would show whether the filter's benefit scales.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SEKE, a pipeline that uses GPT-4o to generate facial emotion instruction data by combining manual annotations from existing datasets (expression, valence/arousal, action units) with a knowledge-guided prompt and an uncertainty-aware Monte Carlo sampling scheme (SV-UAMC). The generated dataset, FEID, is used to fine-tune LLaVA-OneVision, yielding the SEKE model. The authors also create a benchmark, FEAB, with subject-independent splits from the same six source datasets, and report that SEKE outperforms closed- and open-source VLLMs on expression accuracy, average AU F1, and valence/arousal MAE on FEAB. Ablations compare FEID with prior instruction datasets and test the contributions of prior knowledge and SV-UAMC.

Significance. If the central claim holds, the paper makes a useful practical contribution: it provides a relatively low-cost method for generating multi-grained emotion instruction data and releases FEID and FEAB, which could support future work in affective VLLMs. The paper also ships a systematic ablation design (Tables 3-4 and Figure 4) that isolates the contribution of the generated data, and the idea of using epistemic-uncertainty-aware self-verification for data generation is interesting. However, the empirical support for the headline claim currently depends on an unvalidated LLM-based answer-extraction step whose details are deferred to a missing appendix, and on a single-run comparison without error bars or significance tests. The significance of the resource is real, but the reported performance gaps need stronger evaluation evidence.

major comments (5)
  1. [§4.2, Evaluation Metrics] The paper defines MAE as "taking the square root of the average squared differences between predicted and actual values," which is the definition of RMSE, not MAE. Since all valence/arousal results (Tables 2-5 and Figure 5) are labeled MAE, the numbers may actually be RMSE values. This must be corrected, and the experiments should be rerun or re-reported with the true metric, or the text should explicitly state that RMSE is used.
  2. [§4.2 and Appendix C] The evaluation pipeline uses an LLM to convert free-form model outputs into a universal template before computing metrics, and details are deferred to Appendix C, which is missing from the submission. This step is load-bearing: SEKE was fine-tuned on FEID answers generated with the same 11 rewrite templates and output format, so its outputs are expected to match the extractor's schema, while baselines such as GPT-4o, Qwen2.5-VL, and LLaVA-OneVision were not. If the extractor normalizes, omits, or misparses baseline answers, the reported gaps (e.g., 61.1 to 69.4 expression accuracy, 29.0 to 54.9 AU F1, 0.342 to 0.295 valence MAE) would be inflated by format alignment. The authors need to validate the extractor with manual inspection, report inter-annotator agreement or an error analysis, and release the extraction code so the comparison is neutral.
  3. [Table 2 and §4.3] All results in Table 2 appear to come from a single evaluation run, yet the abstract and Section 4.3 state that SEKE "significantly outperforms" state-of-the-art methods. Without standard deviations over multiple runs or a significance test, the word "significantly" is not supported. At minimum, the authors should report variance across seeds for the SEKE model and for the strongest baselines.
  4. [§4.1 and §4.5.2] FEAB is constructed by holding out 1/10 of subjects from the same six datasets whose remaining images form FEID. Because FEID training images come from the same datasets, the model may learn dataset-specific statistics (lighting, camera, label distribution) rather than general emotion perception. The paper should provide cross-dataset evidence or an analysis controlling for this. In addition, the reliability experiment in §4.5.2 uses Aff-Wild2 samples with manual labels for all three descriptions; the text does not state whether these samples are disjoint from FEID training, and if they overlap, the reported accuracy of generated missing descriptions would be circular.
  5. [§3.2, Algorithm 1 and Eq. (4)] The UAMC sampling procedure depends on several hand-set parameters: N=5, an initial sampling count of 2, and the acceptance probability p_acc = 1/2 + (1/2) U_bar. These choices are not justified by a sensitivity analysis or an ablation. Since the efficiency and reliability claims of SV-UAMC rest on these parameters, the authors should vary N and the acceptance rule to show that the reported gains are not tied to a particular setting.
minor comments (5)
  1. [Section 4.2] There is a typo: "for metics computing" should read "for metrics computing."
  2. [Table 2] Several entries in Table 2 appear to be missing spaces or contain OCR artifacts (e.g., "31.716.632.4", "69.969.38.2", "41 33"). The table should be reformatted so each numeric column is unambiguous.
  3. [References [24] and [25]] References [24] and [25] cite the same paper (LLaVA-NeXT-Interleave) twice; one duplicate should be removed.
  4. [Appendices] The text references Appendices A-D (templates, sample distributions, extraction details, visualization examples), but none of these appendices are present in the submitted version. All referenced material must be included for the claims to be checkable.
  5. [Abstract and Section 4.3] The phrase "significantly outperforms" in the abstract is not backed by any statistical test in the body; consider softening the wording until such tests are added.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central performance claim is evaluated against manual labels on subject-disjoint data, and no fitted parameter or self-citation forces the reported result.

full rationale

The paper's central claim is that fine-tuning LLaVA-OneVision on the SEKE-generated FEID improves facial emotion analysis. The benchmark FEAB uses manual labels from held-out subjects drawn from the same six source datasets; evaluation is not against SEKE's generated labels, so the main result is external to the generation procedure. The data-generation pipeline contains hand-set constants (N=5, two initial samples, acceptance rule p_acc), but these are not fit to FEAB, and no quantity reported on FEAB is a re-labeling of these constants. The self-citations (e.g., Cui et al. [11], Song et al. [44,45]) appear as related-work background on AU detection and emotion analysis and do not carry the derivation. The LLM-based answer extraction described in Section 4.2 and deferred to the missing Appendix C is a genuine evaluation risk: SEKE's outputs were trained on the 11 rewrite templates, while baselines produce heterogeneous text, so extraction asymmetry could inflate the reported gap; however, this is a measurement-fairness concern rather than a construction-level circularity. No equation in the paper defines a predicted quantity as the fit input by construction, and no load-bearing claim rests on an unverified self-citation. Under the hard rules, without an exhibited reduction, the appropriate finding is no significant circularity, score 0.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The paper does not postulate theoretical entities. Its contributions rest on hand-set sampling parameters, the emotion-correlation assumption, and the reliability of manual source labels. No external artifact (code, dataset URL, commit hash) is provided, so the auxiliary parameters cannot currently be audited.

free parameters (7)
  • Maximum sampling number N = 5
    Set by hand in Section 4.2; caps the UAMC loop and affects how many GPT-4o answers contribute to each generated instruction, with no sensitivity analysis.
  • Initial sampling count = 2
    Algorithm 1 starts each sample with two GPT-4o responses; chosen without analysis and directly shapes the variance estimate.
  • Acceptance probability rule p_acc = 0.5 + 0.5 * normalized max uncertainty
    Ad hoc accept/reject schedule in Equation 4; controls how many extra samples are taken and is not derived from a decision-theoretic criterion.
  • FEAB subject split ratio = 1/10
    Section 4.1 selects one tenth of subjects from each source dataset for the benchmark; this split ratio is arbitrary and changes FEID/FEAB sizes and difficulty.
  • FEID per-dataset sampling quotas = not specified
    Section 4.1 randomly extracts partial samples from CK+, RAF-DB, AffectNet, Aff-Wild2, DISFA, and BP4D while balancing expression categories; the exact quotas are not given and affect task balance.
  • DISFA AU occurrence threshold = intensity > 2
    Section 4.1 converts DISFA AU intensities to binary labels with a threshold of 2; a standard but arbitrary binarization that influences AU training labels.
  • Fine-tuning learning rate = 1e-5
    Set in Section 4.2 for SFT; needed to reproduce the tuned model, though not central to the data-generation claim.
assumptions (4)
  • domain assumption Inherent correlations between discrete expression, valence-arousal, and action units allow missing annotations to be inferred from available ones.
    Used throughout Section 3.1; the data-generation method fills missing labels via these assumed correlations.
  • domain assumption The variance of GPT-4o outputs over repeated stochastic sampling is a valid estimator of epistemic uncertainty and correlates with output reliability.
    Section 3.2, Equations 1 to 3; SV-UAMC depends on this assumption to filter annotations.
  • domain assumption Manual annotations in CK+, RAF-DB, AffectNet, DISFA, BP4D, and Aff-Wild2 are reliable enough to serve as prior knowledge and as FEAB ground truth.
    Section 4.1; the benchmark and the prior-knowledge prompts both treat these labels as trustworthy.
  • standard math Standard Monte Carlo identities approximate the prediction probability and variance via finite samples.
    Equations 1 to 3 use the standard MC estimator for expectation and variance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Emotion Knowledge Enhancement for Vision Large Language Models: A Self-Verification Approach for High-Quality Emotion Instruction Data Generation." pith.science (2026). https://pith.science/paper/PQADGMTQ

@misc{pith2026250518168,
  author       = {Pith},
  title        = {Pith review of: Emotion Knowledge Enhancement for Vision Large Language Models: A Self-Verification Approach for High-Quality Emotion Instruction Data Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PQADGMTQ}},
  note         = {Machine review of arXiv:2505.18168}
}
read the original abstract

Facial emotion perception in the vision large language model (VLLM) is crucial for achieving natural human-machine interaction. However, creating high-quality annotations for both coarse- and fine-grained facial emotion analysis demands costly expertise. The lack of such high-quality instruction data limits the performance of VLLMs in facial emotion perception. To address this, we propose a self-verification approach with emotion knowledge enhancement (SEKE), which generates high-quality instruction data for multi-grained emotion analysis cost-effectively using closed-source VLLM. This approach integrates prior human knowledge to VLLM inference, guided by the inherent correlations between three grained levels of emotion descriptions, i.e., discrete expression, valence-arousal, and action unit, to reliably generate comprehensive annotations. A self-verification strategy with Uncertainty-Aware Monte Carlo sampling (SV-UAMC) is further embedded to efficiently extract more accurate VLLM predictions, further improving annotation reliability. Consequently, we construct a facial emotion instruction dataset (FEID) containing three comprehensive descriptions, which provides coarse- and fine-grained emotional information for effective model training. Additionally, we introduce a facial emotion analysis benchmark (FEAB) to measure the VLLM's corresponding ability. Our method significantly outperforms state-of-the-art methods on three downstream facial emotion analysis tasks.

Figures

Figures reproduced from arXiv: 2505.18168 by the authors.

Figure 1
Figure 1. Benchmark performance of SEKE model. Abstract Facial emotion perception in the vision large language model (VLLM) is crucial for achieving natural human-machine interaction. How￾ever, creating high-quality annotations for both coarse- and fine￾grained facial emotion analysis demands costly expertise. The lack of such high-quality instruction data limits the performance of VLLMs in facial emotion perception. To addre… view at source ↗
Figure 2
Figure 2. The overall architecture of the proposed self-verification approach with emotion knowledge enhancement (SEKE) to [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The VLLM architecture used to train our model on [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of performance when emotion descrip [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comparison of the reliability of annotated missing [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Two examples comparing the comprehensive emotional reasoning of the SEKE model with that of models fine-tuned [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 27 canonical work pages

  1. [1]

    A comprehensive review of facial expression recognition techniques.Multimedia Systems, 29(1):73–103, 2023

    R Rashmi Adyapady and B Annappa. A comprehensive review of facial expression recognition techniques.Multimedia Systems, 29(1):73–103, 2023

    work page 2023

  2. [2]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024

    2024

  3. [3]

    Claude-3.5

    Anthropic. Claude-3.5. https://www.anthropic.com/news/claude-3-5-sonnet, 2024

    2024

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  5. [5]

    Video-based facial micro-expression analysis: A survey of datasets, features and algorithms.IEEE transactions on pattern analysis and machine intelligence, 44(9):5826–5846, 2021

    Xianye Ben, Yi Ren, Junping Zhang, Su-Jing Wang, Kidiyo Kpalma, Weixiao Meng, and Yong-Jin Liu. Video-based facial micro-expression analysis: A survey of datasets, features and algorithms.IEEE transactions on pattern analysis and machine intelligence, 44(9):5826–5846, 2021

    work page 2021

  6. [6]

    Knowledge-driven self-supervised representa- tion learning for facial action unit recognition

    Yanan Chang and Shangfei Wang. Knowledge-driven self-supervised representa- tion learning for facial action unit recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20417– 20426, June 2022

    work page 2022

  7. [7]

    Minigpt-v2: large language model as a unified interface for vision- language multi-task learning.arXiv preprint arXiv:2310.09478, 2023

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision- language multi-task learning.arXiv preprint arXiv:2310.09478, 2023

    arXiv

  8. [8]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to Emotion Knowledge Enhancement for Vision Large Language Models: A Self-Verification Approach for High-Quality Emotion Instruction Data Generation Conference’17, July 2017, Washington, DC, USA (a) (b) Figure 6: Tw...

    arXiv 2017

Show all 58 references
  1. [9]

    Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning.arXiv preprint arXiv:2406.11161, 2024

    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning.arXiv preprint arXiv:2406.11161, 2024

    2024 arXiv

  2. [10]

    Knowledge augmented deep neural networks for joint facial expression and action unit recognition

    Zijun Cui, Tengfei Song, Yuru Wang, and Qiang Ji. Knowledge augmented deep neural networks for joint facial expression and action unit recognition. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Information Processing Systems, vol...

    2020

  3. [11]

    Knowledge augmented deep neural networks for joint facial expression and action unit recognition.Advances in Neural Information Processing Systems, 33:14338–14349, 2020

    Zijun Cui, Tengfei Song, Yuru Wang, and Qiang Ji. Knowledge augmented deep neural networks for joint facial expression and action unit recognition.Advances in Neural Information Processing Systems, 33:14338–14349, 2020

    2020

  4. [12]

    Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning

    Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. InInternational conference on machine learning, pages 1184–1193. PMLR, 2018

    2018

  5. [13]

    Consulting Psychologists Press, 1978

    Paul Ekman and Wallace V Friesen.Manual for the facial action coding system. Consulting Psychologists Press, 1978

    1978

  6. [14]

    Universals and cultural differences in the judgments of facial expressions of emotion.Journal of personality and social psychology, 53(4):712, 1987

    Paul Ekman, Wallace V Friesen, Maureen O’sullivan, Anthony Chan, Irene Diacoyanni-Tarlatzis, Karl Heider, Rainer Krause, William Ayhan LeCompte, Tom Pitcairn, Pio E Ricci-Bitti, et al. Universals and cultural differences in the judgments of facial expressions of emotion.Journa...

    1987

  7. [15]

    Oxford University Press, USA, 1997

    Rosenberg Ekman.What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA, 1997

    1997

  8. [16]

    Selfme: Self-supervised motion learning for micro-expression recognition

    Xinqi Fan, Xueli Chen, Mingjie Jiang, Ali Raza Shahid, and Hong Yan. Selfme: Self-supervised motion learning for micro-expression recognition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13834–13843, 2023

    2023

  9. [17]

    Stimuvar: Spatiotemporal stimuli-aware video affective reasoning with multi- modal large language models.arXiv preprint arXiv:2409.00304, 2024

    Yuxiang Guo, Faizan Siddiqui, Yang Zhao, Rama Chellappa, and Shao-Yuan Lo. Stimuvar: Spatiotemporal stimuli-aware video affective reasoning with multi- modal large language models.arXiv preprint arXiv:2409.00304, 2024

    2024 arXiv

  10. [18]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection- allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection- allocation. InProceedings of the IEEE/CVF Conferenc...

    2024

  11. [19]

    Disentangling identity and pose for facial ex- pression recognition.IEEE Transactions on Affective Computing, 13(4):1868–1878, 2022

    Jing Jiang and Weihong Deng. Disentangling identity and pose for facial ex- pression recognition.IEEE Transactions on Affective Computing, 13(4):1868–1878, 2022

    2022

  12. [20]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  13. [21]

    What uncertainties do we need in bayesian deep learning for computer vision?Advances in neural information processing systems, 30, 2017

    Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision?Advances in neural information processing systems, 30, 2017

    2017

  14. [22]

    Expression, affect, action unit recogni- tion: Aff-wild2, multi-task learning and arcface.arXiv preprint arXiv:1910.04855, 2019

    Dimitrios Kollias and Stefanos Zafeiriou. Expression, affect, action unit recogni- tion: Aff-wild2, multi-task learning and arcface.arXiv preprint arXiv:1910.04855, 2019

    1910 arXiv

  15. [23]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  16. [25]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024

    2024 arXiv

  17. [26]

    Deep facial expression recognition: A survey.IEEE transactions on affective computing, 13(3):1195–1215, 2020

    Shan Li and Weihong Deng. Deep facial expression recognition: A survey.IEEE transactions on affective computing, 13(3):1195–1215, 2020

    2020

  18. [27]

    Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild

    Shan Li, Weihong Deng, and JunPing Du. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 2852–2861, 2017

    2017

  19. [28]

    Af- fectgpt: Dataset and framework for explainable multimodal emotion recognition

    Zheng Lian, Haiyang Sun, Licai Sun, Jiangyan Yi, Bin Liu, and Jianhua Tao. Af- fectgpt: Dataset and framework for explainable multimodal emotion recognition. arXiv preprint arXiv:2407.07653, 2024

    2024 arXiv

  20. [29]

    Gpt-4v with emotion: A zero-shot benchmark for generalized emotion recognition.Information Fusion, 108:102367, 2024

    Zheng Lian, Licai Sun, Haiyang Sun, Kang Chen, Zhuofan Wen, Hao Gu, Bin Liu, and Jianhua Tao. Gpt-4v with emotion: A zero-shot benchmark for generalized emotion recognition.Information Fusion, 108:102367, 2024

    2024

  21. [30]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  22. [31]

    Donald B Lindsley. Emotion. 1951

    1951

  23. [32]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  24. [33]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  25. [34]

    Facial expressions elicit multiplexed perceptions of emotion categories and dimensions.Current Biology, 32(1):200–209, 2022

    Meng Liu, Yaocong Duan, Robin AA Ince, Chaona Chen, Oliver GB Garrod, Philippe G Schyns, and Rachael E Jack. Facial expressions elicit multiplexed perceptions of emotion categories and dimensions.Current Biology, 32(1):200–209, 2022

    2022

  26. [35]

    Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild

    Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan. Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild. InProceedings of the 30th ACM international conference on multim...

    2022

  27. [36]

    Patrick Lucey, Jeffrey F Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. The extended cohn-kanade dataset (ck+): A complete dataset Conference’17, July 2017, Washington, DC, USA Feifan Wang, Tengfei Song, Minggui He, Chang Su, Zhanglin Wu, Hao Yang, Wenming...

    2017

  28. [37]

    Disfa: A spontaneous facial action intensity database.IEEE Transactions on Affective Computing, 4(2):151–160, 2013

    S Mohammad Mavadati, Mohammad H Mahoor, Kevin Bartlett, Philip Trinh, and Jeffrey F Cohn. Disfa: A spontaneous facial action intensity database.IEEE Transactions on Affective Computing, 4(2):151–160, 2013

    2013

  29. [38]

    Affectnet: A database for facial expression, valence, and arousal computing in the wild.IEEE Transactions on Affective Computing, 10(1):18–31, 2017

    Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild.IEEE Transactions on Affective Computing, 10(1):18–31, 2017

    2017

  30. [39]

    Multi-label co- regularization for semi-supervised facial action unit recognition

    Xuesong Niu, Hu Han, Shiguang Shan, and Xilin Chen. Multi-label co- regularization for semi-supervised facial action unit recognition. InAdvances in Neural Information Processing Systems, pages 909–919, 2019

    2019

  31. [40]

    OpenAI. Gpt-4v. https://openai.com/index/gpt-4v-system-card/, 2023

    2023

  32. [41]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. https://openai.com/index/hello-gpt-4o, 2024

    2024

  33. [42]

    A unified and interpretable emotion representation and expression generation

    Reni Paskaleva, Mykyta Holubakha, Andela Ilic, Saman Motamed, Luc Van Gool, and Danda Paudel. A unified and interpretable emotion representation and expression generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2447–2456, 2024

    2024

  34. [43]

    A circumplex model of affect.Journal of personality and social psychology, 39(6):1161, 1980

    James A Russell. A circumplex model of affect.Journal of personality and social psychology, 39(6):1161, 1980

    1980

  35. [44]

    Uncertain graph neural networks for facial action unit detection.Proceedings of the AAAI Conference on Artificial Intelligence, 35(7):5993–6001, 2021

    Tengfei Song, Lisha Chen, Wenming Zheng, and Qiang Ji. Uncertain graph neural networks for facial action unit detection.Proceedings of the AAAI Conference on Artificial Intelligence, 35(7):5993–6001, 2021

    2021

  36. [45]

    Hybrid message passing with performance-driven structures for facial action unit detection

    Tengfei Song, Zijun Cui, Wenming Zheng, and Qiang Ji. Hybrid message passing with performance-driven structures for facial action unit detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6267–6276, June 2021

    2021

  37. [46]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  38. [47]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  39. [48]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  40. [49]

    Rethinking the learning paradigm for dynamic facial expression recognition

    Hanyang Wang, Bo Li, Shuang Wu, Siyuan Shen, Feng Liu, Shouhong Ding, and Aimin Zhou. Rethinking the learning paradigm for dynamic facial expression recognition. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17958–17968, 2023

    2023

  41. [50]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s pe...

    2024 arXiv

  42. [51]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442, 2024

    2024 arXiv

  43. [52]

    Emovit: Revolutionizing emotion insights with visual instruction tuning

    Hongxia Xie, Chu-Jun Peng, Yu-Wen Tseng, Hung-Jen Chen, Chan-Feng Hsu, Hong-Han Shuai, and Wen-Huang Cheng. Emovit: Revolutionizing emotion insights with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26596...

    2024

  44. [53]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  45. [54]

    Robust emotion recognition in context debiasing

    Dingkang Yang, Kun Yang, Mingcheng Li, Shunli Wang, Shuaibing Wang, and Lihua Zhang. Robust emotion recognition in context debiasing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12447–12457, June 2024

    2024

  46. [55]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  47. [56]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023

    2023

  48. [57]

    A high-resolution spontaneous 3d dynamic facial expression database

    Xing Zhang, Lijun Yin, Jeffrey F Cohn, Shaun Canavan, Michael Reale, Andy Horowitz, and Peng Liu. A high-resolution spontaneous 3d dynamic facial expression database. In2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), pages ...

    2013

  49. [58]

    Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database.Image and Vision Computing, 32(10):692–706, 2014

    Xing Zhang, Lijun Yin, Jeffrey F Cohn, Shaun Canavan, Michael Reale, Andy Horowitz, Peng Liu, and Jeffrey M Girard. Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database.Image and Vision Computing, 32(10):692–706, 2014

    2014

  50. [59]

    Khfa: Knowledge-driven hierarchical feature alignment framework for subject- invariant facial action unit detection.IEEE Transactions on Instrumentation and Measurement, 2024

    Huijuan Zhao, Shuangjiang He, Congju Du, Linyun Liu, and Li Yu. Khfa: Knowledge-driven hierarchical feature alignment framework for subject- invariant facial action unit detection.IEEE Transactions on Instrumentation and Measurement, 2024

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.