Pith. sign in

REVIEW 4 major objections 6 minor 62 references

Interpretable Concept-based Deep Learning Framework for Multimodal Human Behavior Modeling

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Concept-based emotion modeling can match opaque black-box accuracy while producing human-readable explanations.

desk verdict A genuinely additive concept-based XAI architecture, but the 'human-interpretable' claim needs validation of the OpenFace-supervised concepts before it holds up. read the letter →

arxiv 2502.10145 v1 pith:Z2HI4B3P submitted 2025-02-14 cs.CV cs.MM

Xinyu Li , Marwa Mahmoud This is my paper · ORCID
classification cs.CVcs.MM
keywords explainableAIconcept-basedlearningfacialexpressionrecognitionmultimodalaffectivecomputingattentionmechanismactionunitsengagementestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that an emotion-recognition system can be fully interpretable and still match or beat opaque black-box models. It introduces the Attention-Guided Concept Model (AGCM), which predicts emotion labels through intermediate concepts—facial action units (AUs), gaze, head pose, and acoustic cues—and reports both the score of each concept and the facial region where it was observed. On RAF-DB, AffectNet-7, AffectNet-8, Aff-Wild2, and NOXI, the reported numbers are competitive with or above several black-box baselines while the model emits human-readable explanations. If the claim is right, regulated high-stakes affective computing can satisfy transparency requirements without sacrificing accuracy.

What carries the argument

The load-bearing object is the Attention-Guided Concept Generator (ACG), a transformer-based module that produces one probability score and one spatial attention map per concept. It combines Multi-scale Spatial Attention (MSA) heads, whose weighted sum forms the concept attention map, and Channel Attended Concept Mapping (CACM), which selects informative feature channels. A three-part loss—cross-entropy task loss, binary cross-entropy concept-probability loss, and cosine-similarity concept-map loss—trains the concept representations and the task prediction together, which the ablation attributes with most of the accuracy gain. In the multimodal stage, frozen visual concept predictions are fused with acoustic concept embeddings through a sequential bottleneck and a transformer label predictor.

What would settle it

Compare AGCM's concept probabilities and attention maps against human AU annotations and human-marked facial regions on a held-out set; if confident predictions often cite concepts that human annotators judge absent, the explanation is unfaithful even if task accuracy stays high.

Watch

Extended reading notes

Core claim

AGCM learns a mapping from input to label through trainable concept embeddings: for each concept, an activated and an inactivated neural embedding are weighted by a learned probability score, then concatenated and passed to a task predictor. The spatial side of the model is supervised by landmark-derived region-of-interest maps, so attention is trained to land where the concept actually appears. The paper's central claim is that this joint design delivers both faithfulness and performance, reporting 94.40% on RAF-DB, 69.45% on AffectNet-7, 65.62% on AffectNet-8, 47.52 F1 on audio-visual Aff-Wild2, and 0.80 CCC on NOXI engagement estimation. The authors interpret these numbers as showing that AGCM matches or exceeds black-box model performance while maintaining conceptual explainability.

Load-bearing premise

The explainability claim rests on the premise that the automatically extracted facial action units and landmark-derived attention maps used as supervision are accurate enough to be trusted as the concepts they claim to explain.

Editorial extensions

If this is right

  • If AGCM's claim is right, an emotion-recognition system can give regulators and clinicians the what and where behind each prediction without an accuracy penalty.
  • AGCM's concept supervision can extend to any spatial or temporal signal with concept annotations, so the same architecture transfers from facial expressions to engagement estimation, and in principle to ECG, EEG, or EDA.
  • The authors' end-to-end joint training is necessary for their result: by-step training, where concept learning and task learning are separated, drops accuracy by roughly 4-12 points on every benchmark they report.
  • Multimodal fusion consistently beats the visual-only branch (Aff-Wild2 F1 47.52 versus 44.95; NOXI CCC 0.80 versus 0.59), so co-learning across modalities is part of the claimed performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension the paper does not run: corrupt the automatic AU labels by flipping a fraction of them and re-measure task accuracy and Concept Alignment Score; if accuracy holds while concept scores drift away from ground truth, the explanations are not faithful even though the framework still appears to work.
  • A consequence the authors leave implicit: because the concept supervision comes from an external AU detector, the interpretability guarantee is bounded by that detector's accuracy on faces outside its training distribution, so deployment to new demographics should include an audit of concept scores against human judgment.
  • The what-where explanation design could transfer to medical imaging: replacing AUs with clinician-supplied lesion ROIs would give the same kind of explainable classification in a high-stakes domain where transparency regulations also apply.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes the Attention-Guided Concept Model (AGCM), a concept-based neural framework for interpretable affective computing. The visual branch uses a Vision Transformer backbone and per-concept attention maps and activation probabilities, supervised by automatically extracted Action Units, gaze, and head-pose concepts; a second stage freezes the visual branch, adds an acoustic concept generator, and performs sequence-to-sequence fusion for multimodal prediction. The authors evaluate AGCM on RAF-DB, AffectNet-7/8, Aff-Wild2, and the NOXI engagement dataset, reporting accuracy, F1, and CCC scores that match or exceed several black-box baselines while providing 'what' and 'where' explanations as concept probabilities and attention maps.

Significance. AGCM addresses a genuine need in affective computing for inherently interpretable models: it couples concept supervision with spatial attention and demonstrates the combination across multiple benchmarks, with an ablation (Table VI) showing that each component contributes to task accuracy. If the concept explanations are faithful, this would be a practically useful framework for high-stakes applications. However, the core interpretability claim is not yet established because the concept supervision is generated automatically and is never validated against human annotation, and the quantitative concept-alignment metric (CAS, Table V) is computed against those same automatic labels. The reported performance gains over black-box baselines are also presented without variance or significance testing. The architectural contribution is moderate relative to prior concept-embedding models, but the empirical scope is substantial.

major comments (4)
  1. [§III.A, §IV.C, Eq. (2)] The 'what' and 'where' explanations are supervised entirely by automatically generated concept annotations: Section IV.C states that OpenFace 2.0 was used to extract 18 AUs for RAF-DB and AffectNet, and Section III.A says ROI maps are generated from an open-source landmark detector, without specifying how each AU's ROI is constructed or applying any confidence threshold. Since Eq. (2) trains the concept probabilities with BCE and the concept attention maps with cosine similarity against these targets, any systematic OpenFace error is inherited by both the concept scores and the attention maps, while task accuracy can remain high because the task loss does not reward correct concept semantics. The paper never validates the automatic AU labels against human-coded AUs, so the central claim that AGCM provides human-interpretable conceptual explanations is not yet supported. I recommend reporting AU detection accuracy (e.g., against BP4D or DISFA annotations) for the same 18 AUs, describing the ROI construction, and/or adding a user study on explanation plausibility.
  2. [§IV.F, Table V] The Concept Alignment Score is computed against the same OpenFace-derived labels used to train the model, so Table V primarily demonstrates that AGCM can reproduce OpenFace outputs rather than that the predicted concepts align with human semantic judgments. The 'No Concept' baseline already attains 63.6–67.5% CAS, suggesting a high automatic floor for the metric. The assertions in Section IV.F that AGCM has a 'superior ability to learn meaningful and aligned concepts' and in Section V.A that it delivers 'accurate conceptual explanations' are therefore not supported by the reported evidence. I recommend evaluating CAS against human-annotated concept labels, or at least reporting OpenFace's own accuracy on the concept labels used.
  3. [§IV.D, Tables II and VI] All reported task results are means over four random seeds with no standard deviations, confidence intervals, or significance tests, and the black-box baselines are drawn from published numbers with different backbones and training protocols. For example, the RAF-DB margin over S2D is only 1.83%, and the Table VI ablation steps from 93.31% to 94.40% with the addition of Concept Map Loss; without variance information it is impossible to judge whether these differences are real. I recommend reporting per-seed results with standard deviations and appropriate statistical tests, and clearly flagging which baseline numbers are copied from other papers rather than reproduced.
  4. [§III.B, §IV.G.1, Eq. (3)] The paper claims that the learned probability scores 'quantify the contribution of individual concepts to the predicted label' and that the weighted attention map in Eq. (3) summarizes the explanation. However, the concept probabilities are trained to match the external AU labels, not to equal a measured contribution of each concept to the final prediction, and no causal or perturbation analysis is provided to verify that changing a concept's probability changes the output in the expected direction. The occlusion experiments in Section IV.H demonstrate prediction robustness, but they do not establish that the reported concept scores faithfully reflect the model's decision pathway. A simple intervention test (e.g., zeroing one concept's probability and measuring the change in prediction) would substantially strengthen this claim.
minor comments (6)
  1. [§IV.B] The acoustic concept construction is underspecified: the text says pitch/loudness variations are 'calculated using their first-order derivatives' and Jitter 'is inherently a percentage,' but no window length, sampling rate, or normalization formula is given, making the audio concepts difficult to reproduce.
  2. [§IV.C] The early-stopping criterion, the exact Vision Transformer variant, and the patch size are not specified, which limits reproducibility of the reported numbers.
  3. [Table IV] The row labeled 'Feature FC FC V 25.27' appears to have a formatting error: the architecture column contains 'FC FC' and the entry is not aligned with the other rows.
  4. [References] Reference [57] (Zarlenga et al.) has a garbled author list, with given names such as 'B. Pietro' and 'C. Gabriele' treated as family names; the citation should be corrected.
  5. [§III.C] The assumption that 'acoustic concepts are shared across all frames' of a clip is stated without discussion; this is a strong simplification for frame-level explanations, since vocal cues can change within a one-second clip.
  6. [Figure 11] The caption uses both 'Noxi' and 'NOXI' for the same dataset; the dataset name should be 'NOXI' consistently throughout.

Circularity Check

2 steps flagged · score 4.0 of 10

Concept Alignment Scores are computed against the same OpenFace-extracted AUs used as supervision, and attention maps are trained to match landmark-derived ROI maps, so the 'what' and 'where' explanations partly reduce to the framework's own training targets; the accuracy claims remain externally benchmarked.

  1. fitted input called prediction [Section IV.C, Eq. (2), Section IV.F, Table V]
    "For datasets lacking AU annotations, we utilized OpenFace 2.0 [8] to automatically extract 18 Action Units (AUs), which served as intermediary concepts in our proposed framework. ... To evaluate the reliability of learned concept representations, we employ the Concept Alignment Score (CAS) [57], which measures how well the predicted concepts align with their corresponding ground truth labels. L = CE(ˆt, t)+ sum_i BCE(p(ˆc_i^+), c_i)+ sum_i (1−sim(ˆa_i, a_i))."

    The concept labels c_i in the BCE term of Eq. (2) are exactly the automatically extracted OpenFace AUs, and the CAS reported in Table V is computed against the same 'ground truth labels' used for supervision. High CAS therefore measures how well AGCM has reproduced the OpenFace labeling signal it was optimized to fit, not whether the concepts are semantically valid or human-interpretable. The paper's claim of a 'superior ability to learn meaningful and aligned concepts' is supported by a metric whose ground truth is the very labels the model was trained to output, so the 'what' explanation is a supervised readout of OpenFace rather than an independently verified conceptual explanation.

  2. self definitional [Section III.A, Eq. (2), Section IV.G.1]
    "Since manually annotated attention maps are not always available for large-scale datasets, the spatial maps are localized based on facial landmarks [11, 37]. In this paper, we utilize an open-source landmark detector [49] for automatic landmark detection. According to the landmark locations, Regions of Interest (ROI) maps are generated for all AUs, which are subsequently used to supervise the spatial concept attention map throughout model training. L = ... + sum_i (1−sim(ˆa_i, a_i))."

    The concept map loss in Eq. (2) trains each predicted attention map â_i to match the landmark-derived ROI map a_i. The paper then presents these maps as the spatial component of its explanation, e.g., Fig. 6 says AGCM 'generates attention maps for all concepts and assigns probability scores based on the highlighted regions.' Because a_i is the supervision target, the 'where' explanation is by construction a learned version of the input ROI map; the model is not independently discovering where concepts are observed, only reproducing the landmark-defined ROI prior. The spatial explanation therefore reduces to the training target renamed as a model explanation.

full rationale

AGCM's headline accuracy results (Tables II, IV, VII) are comparisons against external black-box baselines on standard benchmark splits, and no circularity is present in those numerical claims. The explainability evidence, however, has two self-referential components. First, the concept labels c_i in the BCE term of Eq. (2) are automatically extracted by OpenFace 2.0, and the Concept Alignment Score used to argue that concepts are 'meaningful and aligned' is measured against those same labels; high CAS therefore attests to fitting the OpenFace labeling pipeline rather than to independent semantic validity. Second, the spatial attention maps â_i are trained in Eq. (2) to match landmark-derived ROI maps a_i, then presented as the 'where' explanation; the localization is supervised regression to the ROI prior, not an independently discovered fact. These are partial circularities in the explanatory claim, not in the accuracy derivation. The reliance on [35] (same authors) for backbone pretraining and as a baseline is not load-bearing, since the core performance comparisons are external. The unvalidated accuracy of OpenFace and landmark labels is additionally a correctness risk, but the CAS and ROI-map evidence would be circular even if those labels were perfectly accurate, because the same labels serve as both training target and evaluation ground truth.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on externally generated concept labels (OpenFace AUs, landmark ROIs) whose quality is never validated, and on hand-chosen hyperparameters (embedding size, loss weights, clip window) that are not ablated. No physical entities are invented: the concepts are neural embeddings of existing attributes (FACS AUs, gaze, head pose, acoustic prosody), so the invented-entities ledger is empty.

free parameters (5)
  • Concept embedding size = 16 for FER; 32 for engagement
    Chosen by hand after the sweep in Fig. 14; directly tunes task performance everywhere and is reported in Appendix B, with no principled selection rule.
  • Concept probability and map loss weights = 1.0 and 1.0
    Section IV.C: 'The concept probability and map loss weights were set to 1.' No ablation is reported for these weights, although they set the balance between explanation alignment and task accuracy.
  • Audio clip window and stride = 1 second, 33 ms stride
    Section IV.B: the video is split into one-second clips with a 33 ms stride; this window determines the validity of the shared-temporal-concepts assumption and is not ablated.
  • Visualization threshold rho = 0.5
    Eq. (3) uses rho = 0.5 to decide which concepts appear in the weighted attention map; affects the displayed explanation, not the loss.
  • Number of MSA heads = 3
    Section III.B: 'three MSA heads are employed to capture diverse spatial patterns'; chosen by hand with no ablation reported.
assumptions (4)
  • domain assumption OpenFace 2.0 AU detections are treated as ground truth for the 18 concept labels supervising RAF-DB and AffectNet.
    Section IV.C: 'we utilized OpenFace 2.0 [8] to automatically extract 18 Action Units (AUs), which served as intermediary concepts.' The explanation scores inherit errors in these automatic labels; accuracy against human AU annotation is never measured.
  • domain assumption Landmark-derived ROI maps correctly localize each AU's spatial region.
    Section III.A: 'the spatial maps are localized based on facial landmarks' and 'ROI maps are generated for all AUs.' The per-AU construction (shape, size, landmark subset) is unspecified, and the concept map loss in Eq. 2 trains attention to match these maps.
  • domain assumption Temporal concepts are shared across all frames within a one-second clip.
    Section III.C: 'a key assumption is made: for a given video clip, temporal concepts are shared across all frames.' Acoustic concept scores are constant across the frames of a clip; the paper flags this itself but does not test its effect on by-frame predictions.
  • domain assumption A ViT pretrained on VGGFace2 transfers to FER and engagement tasks.
    Section IV.C: the backbone 'was pre-trained on VGGFace2 [16] for the facial recognition task.' Standard transfer practice, but the stripped ablation model (90.47% on RAF-DB) already exceeds most published baselines, so the backbone and recipe carry a large share of the reported performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Interpretable Concept-based Deep Learning Framework for Multimodal Human Behavior Modeling." pith.science (2026). https://pith.science/paper/Z2HI4B3P

@misc{pith2026250210145,
  author       = {Pith},
  title        = {Pith review of: Interpretable Concept-based Deep Learning Framework for Multimodal Human Behavior Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z2HI4B3P}},
  note         = {Machine review of arXiv:2502.10145}
}
read the original abstract

In the contemporary era of intelligent connectivity, Affective Computing (AC), which enables systems to recognize, interpret, and respond to human behavior states, has become an integrated part of many AI systems. As one of the most critical components of responsible AI and trustworthiness in all human-centered systems, explainability has been a major concern in AC. Particularly, the recently released EU General Data Protection Regulation requires any high-risk AI systems to be sufficiently interpretable, including biometric-based systems and emotion recognition systems widely used in the affective computing field. Existing explainable methods often compromise between interpretability and performance. Most of them focus only on highlighting key network parameters without offering meaningful, domain-specific explanations to the stakeholders. Additionally, they also face challenges in effectively co-learning and explaining insights from multimodal data sources. To address these limitations, we propose a novel and generalizable framework, namely the Attention-Guided Concept Model (AGCM), which provides learnable conceptual explanations by identifying what concepts that lead to the predictions and where they are observed. AGCM is extendable to any spatial and temporal signals through multimodal concept alignment and co-learning, empowering stakeholders with deeper insights into the model's decision-making process. We validate the efficiency of AGCM on well-established Facial Expression Recognition benchmark datasets while also demonstrating its generalizability on more complex real-world human behavior understanding applications.

Figures

Figures reproduced from arXiv: 2502.10145 by the authors.

Figure 1
Figure 1. Difference between the black-box models, current eXplainable AI [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Feature-based approaches offer inherent interpretability and are [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Example of patch-level AU map generated using landmark detection [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: The architecture of our proposed Attention-Guided Concept Model (AGCM) for the spatial visual modality. The model uses a transformer backbone [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: In the multimodal fusion stage, the pre-learned visual branch functions as a Visual Attention-Guided Concept Generator. The parameters of the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: AGCM offers human-interpretable and intuitive explanations by [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: AGCM facilitates both the co-learning and interpretation of mul [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Example of the face occlusion samples, with the expression label [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Example of hand-over-face occlusion, with the predicted facial [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Example of the engagement estimation, gaze, head pose direction [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Example of the engagement estimation for distraction and cognitive [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: (a) Feature-based models rely on manual feature preprocessing using external automatic toolkits, such as OpenFace, which operate outside the [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Task performance evaluation (%) with different embedding sizes. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Explanation examples of map-based TS-CAM [21], attention map-based FER (Att-Map) [11], and the proposed AGCM framework. In addition to all [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 60 canonical work pages

  1. [1]

    AbaeiKoupaei and H

    N. AbaeiKoupaei and H. Al Osman. A multi-modal stacked ensemble model for bipolar disorder classification. IEEE Transactions on Affective Computing , 14(1):236–244, 2020

    work page 2020

  2. [2]

    A. G. Adami. Modeling prosodic differences for speaker recognition. Speech Commun., 49:277–291, 2007. 14

    work page 2007

  3. [3]

    R. Adolphs. Recognizing emotion from facial expressions: psychological and neurological mechanisms. Behavioral and Cognitive Neuroscience Reviews, 1(1):21–62, 2002

    work page 2002

  4. [4]

    X. An, J. Deng, J. Guo, Z. Feng, X. Zhu, J. Yang, and T. Liu. Killing two birds with one stone: Efficient and robust training of face recognition cnns by partial fc. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4042–4051, 2022

    work page 2022

  5. [5]

    Avola, M

    D. Avola, M. Cascio, L. Cinque, A. Fagioli, and G. L. Foresti. Affective action and interaction recognition by multi-view rep- resentation learning from handcrafted low-level skeleton fea- tures. International journal of neural systems , 32(10):2250040, 2022

    work page 2022

  6. [6]

    Bachorowski and M

    J.-A. Bachorowski and M. J. Owren. V ocal expression of emotion: Acoustic properties of speech are associated with emo- tional intensity and context. Psychological science , 6(4):219– 224, 1995

    work page 1995

  7. [7]

    Baltru ˇsaitis, C

    T. Baltru ˇsaitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence , 41(2):423–443, 2018

    work page 2018

  8. [8]

    Baltrusaitis, A

    T. Baltrusaitis, A. Zadeh, Y . C. Lim, and L.-P. Morency. Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) , pages 59–66. IEEE, 2018

    work page 2018

Show all 62 references
  1. [9]

    A. J. Barnett, F. R. Schwartz, C. Tao, C. Chen, Y . Ren, J. Y . Lo, and C. Rudin. A case-based interpretable deep learning model for classification of mass lesions in digital mammography. Nature Machine Intelligence , 3(12):1061–1070, 2021

    2021

  2. [10]

    Bayliss, A

    A. Bayliss, A. Frischen, M. Fenske, and S. Tipper. Affective evaluations of objects are influenced by observed gaze direction and emotional expression. Cognition, 104:644–653, 2007

    2007

  3. [11]

    Belharbi, M

    S. Belharbi, M. Pedersoli, A. L. Koerich, S. Bacon, and E. Granger. Guided interpretable facial expression recognition via spatial action unit cues. arXiv preprint arXiv:2402.00281 , 2024

    2024 arXiv

  4. [12]

    Bengio, A

    Y . Bengio, A. Courville, and P. Vincent. Representation learn- ing: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798–1828, 2013

    2013

  5. [13]

    Bento, J

    N. Bento, J. Rebelo, M. Barandas, A. V . Carreiro, A. Campag- ner, F. Cabitza, and H. Gamboa. Comparing handcrafted fea- tures and deep neural representations for domain generalization in human activity recognition. Sensors, 22(19):7324, 2022

    2022

  6. [14]

    Bisogni, L

    C. Bisogni, L. Cimmino, M. De Marsico, F. Hao, and F. Nar- ducci. Emotion recognition at a distance: The robustness of machine learning based on hand-crafted facial features vs deep learning models. Image and Vision Computing , 136:104724, 2023

    2023

  7. [15]

    Cafaro, J

    A. Cafaro, J. Wagner, T. Baur, S. Dermouche, M. Torres Torres, C. Pelachaud, E. Andr ´e, and M. Valstar. The noxi database: multimodal recordings of mediated novice-expert interactions. In Proceedings of the 19th ACM International Conference on Multimodal Interaction, pages 35...

    2017

  8. [16]

    Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) , pages 67–74. IEEE, 2018

    2018

  9. [17]

    Y . Chen, J. Li, S. Shan, M. Wang, and R. Hong. From static to dynamic: Adapting landmark-aware image models for facial expression recognition in videos. IEEE Transactions on Affective Computing, 2024

    2024

  10. [18]

    Corti ˜nas-Lorenzo and G

    K. Corti ˜nas-Lorenzo and G. Lacey. Toward explainable affective computing: A review. IEEE Transactions on Neural Networks and Learning Systems , 2023

    2023

  11. [19]

    Dosovitskiy

    A. Dosovitskiy. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  12. [20]

    Regulation (eu) 2024/1689 on eu artificial intelli- gence act, 2024

    EUR-Lex. Regulation (eu) 2024/1689 on eu artificial intelli- gence act, 2024. Accessed: Oct. 12, 2024

    2024

  13. [21]

    W. Gao, F. Wan, X. Pan, Z. Peng, Q. Tian, Z. Han, B. Zhou, and Q. Ye. Ts-cam: Token semantic coupled attention map for weakly supervised object localization. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2886–2895, 2021

    2021

  14. [22]

    Grau and D

    J. Grau and D. K. Nelson. The distinction between integral and separable dimensions: evidence for the integrality of pitch and loudness. Journal of experimental psychology. General , 117 4:347–70, 1988

    1988

  15. [23]

    Greipl, K

    S. Greipl, K. Bernecker, and M. Ninaus. Facial and bodily expressions of emotional engagement. Proceedings of the ACM on Human-Computer Interaction , 5:1 – 25, 2021

    2021

  16. [24]

    M. Gund, A. R. Bharadwaj, and I. Nwogu. Interpretable emotion classification using temporal convolutional models. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 6367–6374. IEEE, 2021

    2020

  17. [25]

    H. He, D. Wang, M. R. Hasan, T. Gedeon, and M. Z. Hos- sain. Tca-net: Triplet concatenated-attentional network for multimodal engagement estimation. In 2024 IEEE International Conference on Image Processing (ICIP) , pages 2062–2068. IEEE, 2024

    2024

  18. [26]

    Heimerl, K

    A. Heimerl, K. Weitz, T. Baur, and E. Andr ´e. Unraveling ml models of emotion with nova: Multi-level explainable ai for non-experts. IEEE Transactions on Affective Computing , 13(3):1155–1167, 2020

    2020

  19. [27]

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhut- dinov, and A. Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language pro- cessing, 29:3451–3460, 2021

    2021

  20. [28]

    Hupont, S

    I. Hupont, S. Tolan, H. Gunes, and E. G ´omez. The landscape of facial processing applications in the context of the european ai act and the development of trustworthy systems. Scientific Reports, 12(1):10688, 2022

    2022

  21. [29]

    R. G. Kamilo ˘glu, A. H. Fischer, and D. A. Sauter. Good vibrations: A review of vocal expressions of positive emotions. Psychonomic bulletin & review , 27:237–265, 2020

    2020

  22. [30]

    Kollias and S

    D. Kollias and S. Zafeiriou. Expression, affect, action unit recognition: Aff-wild2, multi-task learning and arcface. arXiv preprint arXiv:1910.04855, 2019

    1910 arXiv

  23. [31]

    Krasich, R

    K. Krasich, R. McManus, S. Hutt, M. Faber, S. D’Mello, and J. Brockmole. Gaze-based signatures of mind wandering during real-world scene processing. Journal of Experimental Psychology: General, 147:1111–1124, 2018

    2018

  24. [32]

    Kumar, A

    M. Kumar, A. Aijaz, O. Chattar, J. Shukla, and R. Mutharaju. Opacity, transparency, and the ethics of affective computing. IEEE Transactions on Affective Computing , 15(1):4–17, 2023

    2023

  25. [33]

    Li and W

    S. Li and W. Deng. Deep facial expression recognition: A survey. IEEE Transactions on Affective Computing, 13(3):1195– 1215, 2020

    2020

  26. [34]

    S. Li, W. Deng, and J. Du. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2852–2861, 2017

    2017

  27. [35]

    Li and M

    X. Li and M. Mahmoud. Unlocking the black box: Concept- based modeling for interpretable affective computing appli- cations. In 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG) , pages 1–10, 2024

    2024

  28. [36]

    B. Ma, W. Zhang, F. Qiu, and Y . Ding. A unified approach to facial affect analysis: the mae-face visual representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5924–5933, 2023

    2023

  29. [37]

    J. Ma, X. Li, Y . Ren, R. Yang, and Q. Zhao. Landmark- based facial feature construction and action unit intensity predic- tion. Mathematical Problems in Engineering, 2021(1):6623239, 2021

    2021

  30. [38]

    Malik, P

    S. Malik, P. Kumar, and B. Raman. Towards interpretable facial emotion recognition. In Proceedings of the Twelfth Indian Conference on Computer Vision, Graphics and Image Processing, pages 1–9, 2021

    2021

  31. [39]

    J. Mao, R. Xu, X. Yin, Y . Chang, B. Nie, A. Huang, and 15 Y . Wang. Poster++: A simpler and stronger facial expression recognition network. Pattern Recognition, page 110951, 2024

    2024

  32. [40]

    Mollahosseini, B

    A. Mollahosseini, B. Hasani, and M. H. Mahoor. Affectnet: A database for facial expression, valence, and arousal comput- ing in the wild. IEEE Transactions on Affective Computing , 10(1):18–31, 2017

    2017

  33. [41]

    Polzehl, A

    T. Polzehl, A. Schmitt, F. Metze, and M. Wagner. Anger recognition in speech using acoustic and linguistic cues. Speech Commun., 53:1198–1209, 2011

    2011

  34. [42]

    why should i trust you?

    M. T. Ribeiro, S. Singh, and C. Guestrin. “why should i trust you?” explaining the predictions of any classifier. In Pro- ceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1135–1144, 2016

    2016

  35. [43]

    C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence , 1(5):206–215, 2019

    2019

  36. [44]

    Savchenko, L

    A. Savchenko, L. V . Savchenko, and I. Makarov. Classifying emotions and engagement in online learning based on a single facial expression recognition neural network. IEEE Transac- tions on Affective Computing , 13:2132–2143, 2022

    2022

  37. [45]

    A. V . Savchenko. Facial expression recognition with adaptive frame rate based on multiple testing correction. In Proceedings of the 40th International Conference on Machine Learning , ICML’23. JMLR.org, 2023

    2023

  38. [46]

    Tao and Q

    H. Tao and Q. Duan. Hierarchical attention network with progressive feature fusion for facial expression recognition. Neural Networks, 170:337–348, 2024

    2024

  39. [47]

    Tsalera, A

    E. Tsalera, A. Papadakis, M. Samarakou, and I. V oyiatzis. Feature extraction with handcrafted methods and convolutional neural networks for facial emotion recognition. Applied Sci- ences, 12(17):8455, 2022

    2022

  40. [48]

    V . N. Tu, V . T. Huynh, H.-J. Yang, S.-H. Kim, S. Nawaz, K. Nandakumar, and M. Z. Zaheer. Dctm: Dilated convolutional transformer model for multimodal engagement estimation in conversation. In Proceedings of the 31st ACM International Conference on Multimedia , pages 9521–9525, 2023

    2023

  41. [49]

    J. Wang, K. Sun, T. Cheng, B. Jiang, C. Deng, Y . Zhao, D. Liu, Y . Mu, M. Tan, X. Wang, et al. Deep high-resolution repre- sentation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence , 43(10):3349–3364, 2020

    2020

  42. [50]

    Z. Wen, W. Lin, T. Wang, and G. Xu. Distract your attention: Multi-head cross attention network for facial expression recog- nition. Biomimetics, 8(2):199, 2023

    2023

  43. [51]

    Wu and T

    Y . Wu and T. Lee. Enhancing sound texture in cnn-based acoustic scene classification. ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 815–819, 2019

    2019

  44. [52]

    Y . Yang, L. Hu, C. Zu, J. Zhang, Y . Hou, Y . Chen, J. Zhou, L. Zhou, and Y . Wang. Cl-transfer: Collaborative learning based transformer for facial expression recognition with masked reconstruction. Pattern Recognition, 156:110741, 2024

    2024

  45. [53]

    J. Yoon, C. Kang, S. Kim, and J. Han. D-vlog: Multimodal vlog dataset for depression detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 12226–12234, 2022

    2022

  46. [54]

    C. Yu, P. M. Aoki, and A. Woodruff. Detecting user engagement in everyday conversations. INTERSPEECH 2004, pages 1329– 1332, 2004

    2004

  47. [55]

    J. Yu, K. Lu, M. Jing, Z. Liang, B. Zhang, J. Sun, and J. Liang. Sliding window seq2seq modeling for engagement estimation. In Proceedings of the 31st ACM International Conference on Multimedia, pages 9496–9500, 2023

    2023

  48. [56]

    S. Yu, A. Androsov, H. Yan, and Y . Chen. Bridging computer and education sciences: a systematic review of automated emo- tion recognition in online learning environments. Computers & Education, page 105111, 2024

    2024

  49. [57]

    M. E. Zarlenga, B. Pietro, C. Gabriele, M. Giuseppe, F. Gian- nini, M. Diligenti, S. Zohreh, P. Frederic, S. Melacci, W. Adrian, et al. Concept embedding models: Beyond the accuracy- explainability trade-off. In Advances in Neural Information Processing Systems , volume 35, pa...

    2022

  50. [58]

    Zhang, B

    W. Zhang, B. Ma, F. Qiu, and Y . Ding. Multi-modal facial affective analysis based on masked autoencoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5793–5802, 2023

    2023

  51. [59]

    Z. Zhang. Microsoft kinect sensor and its effect. IEEE multimedia, 19(2):4–10, 2012

    2012

  52. [60]

    S. Zhao, S. Wang, M. Soleymani, D. Joshi, and Q. Ji. Affective computing for large-scale heterogeneous multimedia data: A survey. ACM Transactions on Multimedia Computing, Com- munications, and Applications (TOMM) , 15(3s):1–32, 2019

    2019

  53. [61]

    H. Zhou, S. Huang, F. Zhang, and C. Xu. Ceprompt: Cross- modal emotion-aware prompting for facial expression recog- nition. IEEE Transactions on Circuits and Systems for Video Technology, 2024

    2024

  54. [62]

    happiness,

    W. Zhou, J. Lu, Z. Xiong, and W. Wang. Leveraging tcn and transformer for effective visual-audio fusion in continuous emo- tion recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5756–5763, 2023. 16 Activated AU12 Neural Em...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.