Pith. sign in

REVIEW 4 major objections 5 minor 82 references

A Feature-level Bias Evaluation Framework for Facial Expression Recognition Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Demographic bias in facial expression recognition can be measured directly in a model's feature space, without test-set demographic labels, and this measurement tracks human-annotated ground truth more closely than pseudo-label pipelines.

desk verdict Clever adaptation of iEAT to FER, but the test's marginal averaging over the test set means it can flag dataset composition as model bias; the central validation is shakier than the low L1 distances suggest. read the letter →

arxiv 2505.20512 v1 pith:QFJH5L42 submitted 2025-05-26 cs.CV

Tangzheng Lian , Oya Celiktutan This is my paper · ORCID
classification cs.CV
keywords facialexpressionrecognitiondemographicbiasevaluationalgorithmicfairnessfeature-spacedifferentialassociationpermutationtestpseudo-demographiclabelsAffectNet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tackles a practical obstacle in fairness auditing of facial expression recognition (FER) models: public FER datasets rarely carry demographic labels, so previous bias studies either restricted their scope or invented pseudo-labels by training attribute classifiers, which can distort the very bias they aim to measure. The authors propose to read bias instead from the feature space of the frozen FER model, comparing how strongly embeddings of each expression associate with face images from a separate probe dataset labelled by perceived gender, age, and race. Against human-annotated demographic labels on the AffectNet test set, they claim this feature-level differential association reproduces true classification-level bias more closely than pseudo-label pipelines, especially for age and race, and it never misidentifies the reference group where pseudo-label methods fail. They also argue that many reported bias numbers may be statistical noise, and supply a permutation-test module that zeros out any disparity not significant at $\alpha = 0.05$. If right, the paper gives auditors a practical way to check FER models for demographic bias using only a face-attribute dataset and the model's own embeddings.

What carries the argument

The load-bearing object is the differential association $\mathrm{DiA}^e_{(j,j')} = A(e, s_j) - A(e, s_{j'})$, where $A(e, s_j)$ is the average of $\cos(\cdot)+1$ between embeddings of test images showing expression $e$ and embeddings of probe images labelled with demographic group $s_j$. This is a reformulation of the Image Embedding Association Test (iEAT), an association test originally built for binary target–attribute pairs in self-supervised models, reworked here for multi-class classification with multi-group sensitive attributes. The machinery does its work under a bridging assumption: differential association in the feature space is taken to mirror the true positive rate disparity $\mathrm{DEO}^e_{(\max,k)}$ that the model actually exhibits between demographic groups. A plug-and-play statistical module—a permutation test with $B = 10000$ shuffles and one-sided $p$-values—decides which observed values survive; anything failing the $\alpha = 0.05$ threshold is set to zero, and the reference group is always the one with the highest performance or strongest association.

What would settle it

Construct or find a test set where expression content and demographic appearance are deliberately anti-correlated (for instance, anger shown mostly on female faces and happiness mostly on male faces) with reliable human demographic labels; if the differential-association ranking then disagrees with the TPR-disparity ranking, the bridging assumption fails. A lighter check: recompute the differential associations after cropping probe images to the face region or filtering probe sets to neutral expressions only, and see whether the L1 distances to ground truth degrade; if they shift sharply, the probe-content confound is doing real work.

Watch

Extended reading notes

Core claim

The central claim is that demographic bias in an FER model can be evaluated without demographic labels on the test set by computing differential associations in the model's own feature space. For each expression $e$ and each demographic group $s_j$, the mean cosine similarity (shifted by $+1$) between embeddings of test images showing $e$ and embeddings of probe images labelled $s_j$ defines an association $A(e, s_j)$; the difference between two groups, $\mathrm{DiA}^e_{(j,j')}$, is the feature-level bias estimate. Tables I–VI show that, across seven expressions and three attributes, both UTKFace and FairFace probe versions land closer in average L1 distance to the human-annotated ground-truth bias (true positive rate disparities on AffectNet) than do two pseudo-demographic-label pipelines, and the feature-level method identifies the correct reference group in every case where the pseudo-label methods return NaN. The claim extends across architectures: run on eight network backbones, the feature-level method tracks the ground-truth average bias for both CNNs and Transformers. Alongside this, the permutation-test statistical module is the paper's second contribution: with 10,000 label shuffles and a one-sided threshold $\alpha = 0.05$, each disparity or association is either retained as statistically significant or set to zero, filtering out noise before any bias claim is made.

Load-bearing premise

Everything rests on the bridging assumption that the feature-space differential association of Equations (4)–(5) tracks the classification-level bias of Equation (6) — a link the paper validates empirically but never derives — and on the probe datasets representing perceived demographic groups without their own expression content contaminating the similarity comparison.

Editorial extensions

If this is right

  • FER models can be audited for demographic bias without collecting demographic labels on the test set, and a modest probe dataset (UTKFace, about 24,000 images) is sufficient for the audit.
  • Pseudo-demographic-label pipelines misidentify the best-performing demographic group for age and race, producing NaN results in the comparison tables, while the feature-level method identifies the correct reference group across all seven expressions.
  • Bias evaluations that skip statistical testing can report noise as bias; the permutation module makes significance checking plug-and-play for both the new framework and existing label-based pipelines.
  • On AffectNet, age and race biases are markedly larger than gender biases, transformer backbones show higher average bias than CNNs, and within each architecture family bias declines as model size grows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not control for emotional content inside the probe images: if a demographic group's probe set happens to display the expression under evaluation, the cosine-similarity association could be inflated, so a natural extension is to restrict probe sets to neutral expressions and re-check whether the L1 rankings survive.
  • Because the audit operates on embeddings, it could transfer to settings with even scarcer demographic labels, such as fine-tuned downstream tasks or multimodal emotion-recognition systems; the paper names this as future work rather than a demonstrated result.
  • The ground-truth comparison rests on a single annotation source, so an independent human-annotated FER test set would be the sharpest check on whether the feature-level advantage generalises across annotation protocols.
  • The permutation test is applied expression by expression without multiple-comparison correction, so a stricter analysis with a family-wise correction across the seven expressions would likely shrink the set of biases reported as significant.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a feature-level bias evaluation framework for FER models when demographic labels are unavailable in the test set. Instead of relying on pseudo-demographic labels, the method uses a probe dataset (UTKFace or FairFace) and computes differential associations (DiA) between expression-specific test embeddings and demographic probe embeddings in the feature space of the frozen FER model (Eqs. 4-5). A permutation-based statistical module is introduced to test the significance of both DiA and the conventional performance disparity (DEO, Eq. 6). The framework is validated on the AffectNet test set using human-annotated demographic labels from Hu et al. as ground truth, comparing L1 distances against pseudo-label baselines across gender, race, and age (Tables II, IV, VI), and is further applied to eight network architectures and a sensitivity analysis of the significance threshold.

Significance. If the central claim holds, the framework would fill a practical gap: evaluating demographic bias in FER models without demographic test labels, while avoiding the distortions of pseudo-label classifiers. The paper also contributes a plug-and-play statistical module and a large-scale bias analysis. Strengths include released code, a reproducibility-focused supplementary with per-sample exclusion lists, validation against human-annotated ground truth, and experiments across multiple architectures and probe datasets. However, the central statistic is marginal over the test set and may be confounded by test-set composition and probe expression content; the validation evidence is therefore not yet conclusive.

major comments (4)
  1. [IV-A, Eqs. (4)-(5)] The differential association DiA is a marginal statistic: A(e,s_j) averages cosine similarity over all test embeddings Z_e and all probe embeddings Z_{s_j}. On a test set in which expression e is demographically imbalanced, any model whose features encode demographic attributes—including a classifier that is perfectly fair in the TPR sense—will produce a nonzero DiA. The ground truth DEO in Eq. (6) is conditional (TPR is computed within each demographic group), so the agreement in Tables II, IV, and VI may be driven by the shared dependence of both quantities on the same composition imbalance rather than by DiA capturing classification bias. No derivation or control experiment (e.g., a random-initialized encoder or a balanced subset of the test set) rules out this confound; this is the load-bearing weakness of the validation.
  2. [IV-B, Eqs. (7)-(9)] The reference group s^e_max is selected from the test data before permutation testing. Under the null hypothesis, the maximum over groups is a data-dependent statistic, so the one-sided p-values for DiP and DiA computed against that selected maximum are anti-conservative; the selection step is not part of the null distribution. No multiple-comparison correction is applied across the n-1 remaining groups, seven expressions, three attributes, or multiple architectures, so the claim that the statistical module 'ensures statistical significance' (abstract, Section IV-B) is overstated.
  3. [V-A and Eq. (4)] The probe datasets UTKFace and FairFace contain natural facial expressions, but A(e,s_j) averages over all embeddings in the probe group regardless of expression content. Expression-related differences between demographic probe subsets (e.g., female faces smiling more often, infant faces showing particular expressions) can inflate the association for the corresponding target expression independently of FER model bias. The paper does not report expression distributions in the probe groups or any control for this confound; this threat is especially acute for the age results in Table V, where very young faces may systematically display fear or surprise.
  4. [V-D, Eq. (11)] The L1 evaluation metric is computed on statistically zeroed values eV from Eq. (10). Because both the candidate method and the ground truth are zeroed by the same permutation module, the L1 comparison partly measures agreement in significance decisions rather than the magnitude of bias. The paper should also report distances on raw DEO/DiA values and include confidence intervals or effect sizes to ensure that the proposed methods are not favored merely by having larger p-values.
minor comments (5)
  1. [IV-B, Eq. (9)] The indicator function is written as 'l(·)'; this appears to be a typographical remnant (likely 1{·}) and should be corrected.
  2. [Figure 4] The curves for UTKFace (ours) and FairFace (ours) are nearly indistinguishable from the ground truth in the figure; adding distinct markers or a zoomed inset would make the sensitivity comparison readable.
  3. [V-C] The pilot study demonstrates qualitative clustering via t-SNE; adding a quantitative measure (e.g., linear-probe accuracy or silhouette score per attribute) would make the claim 'FER models are good attribute encoders' more rigorous.
  4. [Tables V and VI] The symbol '↓' in the bias-direction rows is not defined in the captions; please define it at first use, for example in a footnote to Table V.
  5. [Abstract and IV-A] The phrase 'without demographic labels in the test set' is not strictly accurate for the validation setting, which uses the human-annotated test set from Hu et al.; consider phrasing 'for deployment settings where such labels are unavailable' to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the DiA statistic (Eqs. 4–5) is parameter-free, uses frozen encoders and external probe datasets, and is validated against an external human-annotated DEO ground truth, so no prediction reduces to its input by construction.

full rationale

The paper's derivation chain is not circular. The feature-level differential association DiA (Eq. 5) is computed from the frozen FER encoder's embeddings of the AffectNet test set and the external probe datasets UTKFace/FairFace (Eq. 4); it contains no fitted parameters. The ground truth it is validated against, DEO (Eq. 6), is the TPR disparity computed from Hu et al.'s human-annotated demographic labels, an external benchmark that is not derived from DiA and is not produced by the authors. The closeness measure (Eq. 11) compares two independently computed quantities, and agreement is empirical rather than enforced. There is no load-bearing self-citation: the only self-citation in the paper ([70], Lian et al., BMVC 2023) supports the choice of Swin-B as a representative architecture and plays no role in the central claim. The pilot study in Section V-C is in-paper empirical evidence that FER encoders preserve demographic structure, not a self-referential premise that forces the main result. The skeptical concern that DiA marginalizes over the test set's demographic composition within each expression (so a fair model could produce nonzero DiA on an imbalanced test set, and both DiA and DEO share the same test set) is a construct-validity/confounding concern about whether DiA isolates model behavior; it does not exhibit the specific reductions the circularity test requires, since DEO is group-conditional while DiA is marginal and the two statistics are not equal by construction. Under the review rules, that concern belongs under correctness risk, not circularity. The finding is therefore a non-finding on circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method has no learned parameters and no invented entities. The free parameters are standard hyperparameters for the statistical module and data cleaning. The load-bearing assumptions are the feature-space to classification-level bridging assumption and the appropriateness of the probe datasets. The ground-truth reliance on Hu et al. is an external dependency.

free parameters (4)
  • significance threshold alpha = 0.05
    Chosen following prior iEAT work, not fitted to data. The sensitivity analysis in Section V-F shows results vary with alpha.
  • permutation count B = 10000
    Chosen following prior iEAT work, not fitted to data.
  • scale factor s in centroid filtering = not reported
    Hyperparameter in Algorithm 1 of the supplementary for data cleaning; chosen by hand.
  • closeness threshold t and probability p in closeness filtering = not reported
    Hyperparameters in Algorithm 2 of the supplementary for data cleaning; chosen by hand.
assumptions (5)
  • domain assumption FER models encode demographic attributes in their feature space even when trained only on expression labels.
    Supported by the pilot study in Section V-C, which shows t-SNE clusters by gender, race, and age. The paper assumes this holds for all evaluated FER models.
  • domain assumption The differential association in feature space (Equation 5) reflects classification-level bias measured by TPR disparity (Equation 6).
    This is the central bridging assumption. It is validated empirically against ground truth but has no theoretical derivation.
  • domain assumption Probe datasets (UTKFace, FairFace) provide representative embeddings of visually perceived demographic groups without confounding expression content.
    The probe images may contain facial expressions that overlap with the seven tested expressions, potentially inflating association scores. The paper does not control for this.
  • domain assumption The permutation test with the reference group selected as the observed maximum is a valid null model.
    The reference group is chosen from the same data, and no multiple-comparison correction is applied across the n-1 groups, which can bias the p-value.
  • domain assumption The human-annotated demographic labels on the AffectNet test set from Hu et al. [42] are reliable ground truth.
    The entire method comparison depends on this external annotation being correct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Feature-level Bias Evaluation Framework for Facial Expression Recognition Models." pith.science (2026). https://pith.science/paper/QFJH5L42

@misc{pith2026250520512,
  author       = {Pith},
  title        = {Pith review of: A Feature-level Bias Evaluation Framework for Facial Expression Recognition Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QFJH5L42}},
  note         = {Machine review of arXiv:2505.20512}
}
read the original abstract

Recent studies on fairness have shown that Facial Expression Recognition (FER) models exhibit biases toward certain visually perceived demographic groups. However, the limited availability of human-annotated demographic labels in public FER datasets has constrained the scope of such bias analysis. To overcome this limitation, some prior works have resorted to pseudo-demographic labels, which may distort bias evaluation results. Alternatively, in this paper, we propose a feature-level bias evaluation framework for evaluating demographic biases in FER models under the setting where demographic labels are unavailable in the test set. Extensive experiments demonstrate that our method more effectively evaluates demographic biases compared to existing approaches that rely on pseudo-demographic labels. Furthermore, we observe that many existing studies do not include statistical testing in their bias evaluations, raising concerns that some reported biases may not be statistically significant but rather due to randomness. To address this issue, we introduce a plug-and-play statistical module to ensure the statistical significance of biased evaluation results. A comprehensive bias analysis based on the proposed module is then conducted across three sensitive attributes (age, gender, and race), seven facial expressions, and multiple network architectures on a large-scale dataset, revealing the prominent demographic biases in FER and providing insights on selecting a fairer network architecture.

Figures

Figures reproduced from arXiv: 2505.20512 by the authors.

Figure 1
Figure 1. Race (left column) and age (right column) distribution for each facial [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the previous bias evaluation pipeline in FER (green) and our proposed framework (orange), which evaluates biases in the feature [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. t-SNE visualization from our pilot study: A FER model trained solely on expression labels effectively encodes unseen face images into well-defined [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Sensitivity analysis of the threshold α across four bias evaluation methods. The closer the curves align with the ground truth, the better the evaluation method. Similar patterns of biases are observed across studies. As discussed in previous sections, several of our f…
Figure 5
Figure 5. Figure 5: Experimental results of our proposed methods across multiple network [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 1
Figure 1. Figure 1: Selected potentially misannotated samples in the four datasets. [PITH_FULL_IMAGE:figures/full_fig_p014_1.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 65 canonical work pages

  1. [1]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255

    2009

  2. [2]

    Towards fairer datasets: Filtering and balancing the distribution of the people sub- tree in the imagenet hierarchy,

    K. Yang, K. Qinami, L. Fei-Fei, J. Deng, and O. Russakovsky, “Towards fairer datasets: Filtering and balancing the distribution of the people sub- tree in the imagenet hierarchy,” inProceedings of the 2020 conference on fairness, accountability, and transparency, 2020, pp. 547–558

    2020

  3. [3]

    Distributionally robust neural networks for group shifts: On the importance of regular- ization for worst-case generalization,

    S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang, “Distributionally robust neural networks for group shifts: On the importance of regular- ization for worst-case generalization,” inThe International Conference on Learning Representations (ICLR), 2020

    work page 2020

  4. [4]

    Directional bias amplification,

    A. Wang and O. Russakovsky, “Directional bias amplification,” in International Conference on Machine Learning. PMLR, 2021, pp. 10 882–10 893

    work page 2021

  5. [5]

    Domino: Discovering systematic errors with cross-modal embeddings,

    S. Eyuboglu, M. Varma, K. Saab, J.-B. Delbrouck, C. Lee-Messer, J. Dunnmon, J. Zou, and C. R ´e, “Domino: Discovering systematic errors with cross-modal embeddings,” inThe International Conference on Learning Representations (ICLR), 2022

    work page 2022

  6. [6]

    No subclass left behind: Fine-grained robustness in coarse-grained classification problems,

    N. Sohoni, J. Dunnmon, G. Angus, A. Gu, and C. R ´e, “No subclass left behind: Fine-grained robustness in coarse-grained classification problems,”Advances in Neural Information Processing Systems, vol. 33, pp. 19 339–19 352, 2020

    work page 2020

  7. [7]

    Trustworthy artificial intelligence,

    S. Thiebes, S. Lins, and A. Sunyaev, “Trustworthy artificial intelligence,” Electronic Markets, vol. 31, pp. 447–464, 2021

    work page 2021

  8. [8]

    The global landscape of ai ethics guidelines,

    A. Jobin, M. Ienca, and E. Vayena, “The global landscape of ai ethics guidelines,”Nature machine intelligence, vol. 1, no. 9, pp. 389–399, 2019

    work page 2019

Show all 82 references
  1. [9]

    Explainable artificial intelligence (xai): Concepts, taxonomies, opportu- nities and challenges toward responsible ai,

    A. B. Arrieta, N. D ´ıaz-Rodr´ıguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. Garc ´ıa, S. Gil-L ´opez, D. Molina, R. Benjaminset al., “Explainable artificial intelligence (xai): Concepts, taxonomies, opportu- nities and challenges toward responsible ai,”Information f...

    2020

  2. [10]

    Reliable crowdsourcing and deep locality- preserving learning for expression recognition in the wild,

    S. Li, W. Deng, and J. Du, “Reliable crowdsourcing and deep locality- preserving learning for expression recognition in the wild,” inProceed- ings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2852–2861

    2017

  3. [11]

    Anatomizing bias in facial analysis,

    R. Singh, P. Majumdar, S. Mittal, and M. Vatsa, “Anatomizing bias in facial analysis,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 11, 2022, pp. 12 351–12 358

    2022

  4. [12]

    Fairgrape: Fairness-aware gradient pruning method for face attribute classification,

    X. Lin, S. Kim, and J. Joo, “Fairgrape: Fairness-aware gradient pruning method for face attribute classification,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 414–432

    2022

  5. [13]

    Fair contrastive learning for facial attribute classification,

    S. Park, J. Lee, P. Lee, S. Hwang, D. Kim, and H. Byun, “Fair contrastive learning for facial attribute classification,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10 389–10 398

    2022

  6. [14]

    Fair attribute classification through latent space de-biasing,

    V . V . Ramaswamy, S. S. Kim, and O. Russakovsky, “Fair attribute classification through latent space de-biasing,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 9301–9310

    2021

  7. [15]

    Bias and fairness in face detection,

    H. F. Menezes, A. S. Ferreira, E. T. Pereira, and H. M. Gomes, “Bias and fairness in face detection,” in2021 34th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). IEEE, 2021, pp. 247–254

    2021

  8. [16]

    Enhancing fairness in face detection in computer vision systems by demographic bias mitigation,

    Y . Yang, A. Gupta, J. Feng, P. Singhal, V . Yadav, Y . Wu, P. Natarajan, V . Hedau, and J. Joo, “Enhancing fairness in face detection in computer vision systems by demographic bias mitigation,” inProceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, 2022, pp...

    2022

  9. [17]

    Gender shades: Intersectional accuracy disparities in commercial gender classification,

    J. Buolamwini and T. Gebru, “Gender shades: Intersectional accuracy disparities in commercial gender classification,” inConference on fair- ness, accountability and transparency. PMLR, 2018, pp. 77–91

    2018

  10. [18]

    Mitigating bias in face recognition us- ing skewness-aware reinforcement learning,

    M. Wang and W. Deng, “Mitigating bias in face recognition us- ing skewness-aware reinforcement learning,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9322–9331

    2020

  11. [19]

    Racial faces in the wild: Reducing racial bias by information maximization adaptation network,

    M. Wang, W. Deng, J. Hu, X. Tao, and Y . Huang, “Racial faces in the wild: Reducing racial bias by information maximization adaptation network,” inProceedings of the ieee/cvf international conference on computer vision, 2019, pp. 692–702

    2019

  12. [20]

    Mitigating face recognition bias via group adaptive classifier,

    S. Gong, X. Liu, and A. K. Jain, “Mitigating face recognition bias via group adaptive classifier,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 3414–3424

    2021

  13. [21]

    Consistent instance false positive improves fairness in face recognition,

    X. Xu, Y . Huang, P. Shen, S. Li, J. Li, F. Huang, Y . Li, and Z. Cui, “Consistent instance false positive improves fairness in face recognition,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 578–586

    2021

  14. [22]

    Farapy: An augmented reality feed- back system for facial paralysis using action unit intensity estimation,

    G. Barrios Dell’Olio and M. Sra, “Farapy: An augmented reality feed- back system for facial paralysis using action unit intensity estimation,” inThe 34th Annual ACM Symposium on User Interface Software and Technology, 2021, pp. 1027–1038

    2021

  15. [23]

    Partially occluded facial action recognition and interaction in virtual reality applications,

    U. Ciftci, X. Zhang, and L. Tin, “Partially occluded facial action recognition and interaction in virtual reality applications,” in2017 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2017, pp. 715–720

    2017

  16. [24]

    Real time face detection and facial expression recognition: Development and applications to human computer interaction

    M. S. Bartlett, G. Littlewort, I. Fasel, and J. R. Movellan, “Real time face detection and facial expression recognition: Development and applications to human computer interaction.” in2003 Conference on computer vision and pattern recognition workshop, vol. 5. IEEE, 2003, pp. 53–53

    2003

  17. [25]

    Emotion recognition in human-computer interaction,

    R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. V otsis, S. Kollias, W. Fellenz, and J. G. Taylor, “Emotion recognition in human-computer interaction,”IEEE Signal processing magazine, vol. 18, no. 1, pp. 32–80, 2001

    2001

  18. [26]

    Automated detection of facial expressions during computer-assisted instruction in individuals on the autism spectrum,

    A. A. Ahmed and M. S. Goodwin, “Automated detection of facial expressions during computer-assisted instruction in individuals on the autism spectrum,” inProceedings of the 2017 CHI Conference on Human Factors in Computing Systems, 2017, pp. 6050–6055

    2017

  19. [27]

    Toward fair facial expression recogni- tion with improved distribution alignment,

    M. Kolahdouzi and A. Etemad, “Toward fair facial expression recogni- tion with improved distribution alignment,” inProceedings of the 25th International Conference on Multimodal Interaction, 2023, pp. 574–583

    2023

  20. [28]

    Investigating bias and fairness in facial expression recognition,

    T. Xu, J. White, S. Kalkan, and H. Gunes, “Investigating bias and fairness in facial expression recognition,” inComputer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16. Springer, 2020, pp. 506–523

    2020

  21. [29]

    Counterfactual fairness for facial expression recognition,

    J. Cheong, S. Kalkan, and H. Gunes, “Counterfactual fairness for facial expression recognition,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 245–261

    2022

  22. [30]

    Domain-incremental continual learning for mitigating bias in facial expression and action unit recog- nition,

    N. Churamani, O. Kara, and H. Gunes, “Domain-incremental continual learning for mitigating bias in facial expression and action unit recog- nition,”IEEE Transactions on Affective Computing, 2022

    2022

  23. [31]

    Using positive matching contrastive loss with facial action units to mitigate bias in facial expression recognition,

    V . Suresh and D. C. Ong, “Using positive matching contrastive loss with facial action units to mitigate bias in facial expression recognition,” in2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 2022, pp. 1–8

    2022

  24. [32]

    Facial emotion recog- nition analysis based on age-biased data,

    H. Park, Y . Shin, K. Song, C. Yun, and D. Jang, “Facial emotion recog- nition analysis based on age-biased data,”Applied Sciences, vol. 12, no. 16, p. 7992, 2022

    2022

  25. [33]

    Responsible ai: Gender bias assess- ment in emotion recognition,

    A. Domnich and G. Anbarjafari, “Responsible ai: Gender bias assess- ment in emotion recognition,”arXiv preprint arXiv:2103.11436, 2021

    2021 arXiv

  26. [34]

    Facial expression recognition: Impact of gender on fairness and expressions,

    C. Manresa-Yee, S. Ramis Guarinos, and J. M. Buades Rubio, “Facial expression recognition: Impact of gender on fairness and expressions,” in Proceedings of the XXII International Conference on Human Computer Interaction, 2022, pp. 1–8

    2022

  27. [35]

    Gender stereotyping impact in facial expression recognition,

    I. Dominguez-Catena, D. Paternain, and M. Galar, “Gender stereotyping impact in facial expression recognition,” inJoint European Confer- ence on Machine Learning and Knowledge Discovery in Databases. Springer, 2022, pp. 9–22

    2022

  28. [36]

    Less can be more: representational vs. stereotypical gender bias in facial expression recognition,

    I. Dominguez-Catena, D. Paternain, A. Jurio, and M. Galar, “Less can be more: representational vs. stereotypical gender bias in facial expression recognition,”Progress in Artificial Intelligence, pp. 1–21, 2024

    2024

  29. [37]

    Ethical ai in facial expres- sion analysis: Racial bias,

    A. H. Sham, K. Aktas, D. Rizhinashvili, D. Kuklianov, F. Alisinanoglu, I. Ofodile, C. Ozcinar, and G. Anbarjafari, “Ethical ai in facial expres- sion analysis: Racial bias,”Signal, Image and Video Processing, vol. 17, no. 2, pp. 399–406, 2023

    2023

  30. [38]

    Demographic effects on facial emotion expression: an interdisciplinary investigation of the facial action units of happiness,

    Y . Fan, J. C. Lam, and V . O. Li, “Demographic effects on facial emotion expression: an interdisciplinary investigation of the facial action units of happiness,”Scientific reports, vol. 11, no. 1, p. 5214, 2021

    2021

  31. [39]

    Understanding and mitigating annotation bias in facial expression recognition,

    Y . Chen and J. Joo, “Understanding and mitigating annotation bias in facial expression recognition,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14 980–14 991

    2021

  32. [40]

    Causal structure learning of bias for fair affect recognition,

    J. Cheong, S. Kalkan, and H. Gunes, “Causal structure learning of bias for fair affect recognition,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 340–349

    2023

  33. [41]

    Affectnet: A database for facial expression, valence, and arousal computing in the wild,

    A. Mollahosseini, B. Hasani, and M. H. Mahoor, “Affectnet: A database for facial expression, valence, and arousal computing in the wild,”IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 18–31, 2017

    2017

  34. [42]

    Rethinking affect analysis: A protocol for ensuring fairness and consis- tency,

    G. Hu, D. Kollias, E. Papadopoulou, P. Tzouveli, J. Wei, and X. Yang, “Rethinking affect analysis: A protocol for ensuring fairness and consis- tency,”IEEE Transactions on Biometrics, Behavior, and Identity Science, 2025. 12

    2025

  35. [43]

    Metrics for dataset demographic bias: A case study on facial expression recognition,

    I. Dominguez-Catena, D. Paternain, and M. Galar, “Metrics for dataset demographic bias: A case study on facial expression recognition,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  36. [44]

    Discovering fair representations in the data domain,

    N. Quadrianto, V . Sharmanska, and O. Thomas, “Discovering fair representations in the data domain,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 8227– 8236

    2019

  37. [45]

    A study on the distribution of social biases in self-supervised learning visual models,

    K. Sirotkin, P. Carballeira, and M. Escudero-Vi ˜nolo, “A study on the distribution of social biases in self-supervised learning visual models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10 442–10 451

    2022

  38. [46]

    A multidimensional analysis of social biases in vision transformers,

    J. Brinkmann, P. Swoboda, and C. Bartelt, “A multidimensional analysis of social biases in vision transformers,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4914–4923

    2023

  39. [47]

    Image representations learned with unsu- pervised pre-training contain human-like biases,

    R. Steed and A. Caliskan, “Image representations learned with unsu- pervised pre-training contain human-like biases,” inProceedings of the 2021 ACM conference on fairness, accountability, and transparency, 2021, pp. 701–713

    2021

  40. [48]

    Fairness with overlapping groups; a probabilistic perspective,

    F. Yang, M. Cisse, and S. Koyejo, “Fairness with overlapping groups; a probabilistic perspective,”Advances in neural information processing systems, vol. 33, pp. 4067–4078, 2020

    2020

  41. [49]

    Facial age affects emotional expression decoding,

    M. F ¨olster, U. Hess, and K. Werheid, “Facial age affects emotional expression decoding,”Frontiers in psychology, vol. 5, p. 30, 2014

    2014

  42. [50]

    Do we expect women to look happier than they are? a test of gender-dependent perceptual correction,

    J. E. Steephen, S. R. Mehta, and R. S. Bapi, “Do we expect women to look happier than they are? a test of gender-dependent perceptual correction,”Perception, vol. 47, no. 2, pp. 232–235, 2018

    2018

  43. [51]

    Ambiguity in social catego- rization: The role of prejudice and facial affect in race categorization,

    K. Hugenberg and G. V . Bodenhausen, “Ambiguity in social catego- rization: The role of prejudice and facial affect in race categorization,” Psychological Science, vol. 15, no. 5, pp. 342–345, 2004

    2004

  44. [52]

    Female, white, 27? bias evaluation on data and algorithms for affect recognition in faces,

    J. Pahl, I. Rieger, A. M ¨oller, T. Wittenberg, and U. Schmid, “Female, white, 27? bias evaluation on data and algorithms for affect recognition in faces,” inProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022, pp. 973–987

    2022

  45. [53]

    Measuring individual differences in implicit cognition: the implicit association test

    A. G. Greenwald, D. E. McGhee, and J. L. Schwartz, “Measuring individual differences in implicit cognition: the implicit association test.” Journal of personality and social psychology, vol. 74, no. 6, p. 1464, 1998

    1998

  46. [54]

    Understanding and using the implicit association test: I. an improved scoring algorithm

    A. G. Greenwald, B. A. Nosek, and M. R. Banaji, “Understanding and using the implicit association test: I. an improved scoring algorithm.” Journal of personality and social psychology, vol. 85, no. 2, p. 197, 2003

    2003

  47. [55]

    Why do fearful facial expressions elicit behavioral approach? evidence from a combined approach-avoidance implicit association test

    J. L. Hammer and A. A. Marsh, “Why do fearful facial expressions elicit behavioral approach? evidence from a combined approach-avoidance implicit association test.”Emotion, vol. 15, no. 2, p. 223, 2015

    2015

  48. [56]

    Facial make-up elicits positive attitudes at the implicit level: Evidence from the implicit association test,

    J. Richetin, J.-C. Croizet, P. Huguetet al., “Facial make-up elicits positive attitudes at the implicit level: Evidence from the implicit association test,”Current Research in Social Psychology, vol. 9, no. 11, pp. 145–164, 2004

    2004

  49. [57]

    Implicit racial attitudes influence perceived emotional intensity on other-race faces,

    Q. Wang, G. Chen, Z. Wang, C. S. Hu, X. Hu, and G. Fu, “Implicit racial attitudes influence perceived emotional intensity on other-race faces,” PLoS One, vol. 9, no. 8, p. e105946, 2014

    2014

  50. [58]

    Not always black and white: The effect of race and emotional expression on implicit attitudes,

    J. R. Steele, M. George, M. K. Cease, T. L. Fabri, and J. Schlosser, “Not always black and white: The effect of race and emotional expression on implicit attitudes,”Social Cognition, vol. 36, no. 5, pp. 534–558, 2018

    2018

  51. [59]

    T. L. Lesick,Not at Face Value: Exploring Whether Whites Exhibit Bias in Categorizing Angry and Happy Black Versus White Expressions. The Florida State University, 2023

    2023

  52. [60]

    Semantics derived au- tomatically from language corpora contain human-like biases,

    A. Caliskan, J. J. Bryson, and A. Narayanan, “Semantics derived au- tomatically from language corpora contain human-like biases,”Science, vol. 356, no. 6334, pp. 183–186, 2017

    2017

  53. [61]

    American== white in multimodal language- and-image ai,

    R. Wolfe and A. Caliskan, “American== white in multimodal language- and-image ai,” inProceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, 2022, pp. 800–812

    2022

  54. [62]

    Contrastive language- vision ai models pretrained on web-scraped multimodal data exhibit sexual objectification bias,

    R. Wolfe, Y . Yang, B. Howe, and A. Caliskan, “Contrastive language- vision ai models pretrained on web-scraped multimodal data exhibit sexual objectification bias,” inProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 2023, pp. 1174–1185

    2023

  55. [63]

    Certifying and removing disparate impact,

    M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, and S. Venkata- subramanian, “Certifying and removing disparate impact,” inproceed- ings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, 2015, pp. 259–268

    2015

  56. [64]

    Age progression/regression by condi- tional adversarial autoencoder,

    S. Y . Zhang, Zhifei and H. Qi, “Age progression/regression by condi- tional adversarial autoencoder,” inIEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017

    2017

  57. [65]

    Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation,

    K. Karkkainen and J. Joo, “Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation,” inProceed- ings of the IEEE/CVF winter conference on applications of computer vision, 2021, pp. 1548–1558

    2021

  58. [66]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  59. [67]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022

    2021

  60. [68]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  61. [69]

    Learning multi- dimensional edge feature-based au relation graph for facial action unit recognition,

    C. Luo, S. Song, W. Xie, L. Shen, and H. Gunes, “Learning multi- dimensional edge feature-based au relation graph for facial action unit recognition,” inProceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, 2022, pp. 1239–1246

    2022

  62. [70]

    Su- pervised contrastive learning with identity-label embeddings for facial action unit recognition,

    T. Lian, D. A. Adama, P. Machado, and D. E. D. Vinkemeier, “Su- pervised contrastive learning with identity-label embeddings for facial action unit recognition,” inBMVC, 2023

    2023

  63. [71]

    Cohen,Statistical power analysis for the behavioral sciences

    J. Cohen,Statistical power analysis for the behavioral sciences. Rout- ledge, 2013

    2013

  64. [72]

    Racial influence on automated perceptions of emotions,

    L. Rhue, “Racial influence on automated perceptions of emotions,” Available at SSRN 3281765, 2018

    2018

  65. [73]

    Biased attention: Do vision trans- formers amplify gender bias more than convolutional neural networks?

    A. Mandal, S. Leavy, and S. Little, “Biased attention: Do vision trans- formers amplify gender bias more than convolutional neural networks?” inBritish Machine Vision Conference (BMVC), 2023

    2023

  66. [74]

    Overwriting pretrained bias with fine- tuning data,

    A. Wang and O. Russakovsky, “Overwriting pretrained bias with fine- tuning data,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 3957–3968

    2023

  67. [75]

    Transformer-based multimodal emotional perception for dynamic facial expression recogni- tion in the wild,

    X. Zhang, M. Li, S. Lin, H. Xu, and G. Xiao, “Transformer-based multimodal emotional perception for dynamic facial expression recogni- tion in the wild,”IEEE Transactions on Circuits and Systems for Video Technology, 2023

    2023

  68. [76]

    The misgendering machines: Trans/hci implications of auto- matic gender recognition,

    O. Keyes, “The misgendering machines: Trans/hci implications of auto- matic gender recognition,”Proceedings of the ACM on human-computer interaction, vol. 2, no. CSCW, pp. 1–22, 2018

    2018

  69. [77]

    Gender recognition or gender reductionism? the social implications of embedded gender recognition systems,

    F. Hamidi, M. K. Scheuerman, and S. M. Branham, “Gender recognition or gender reductionism? the social implications of embedded gender recognition systems,” inProceedings of the 2018 chi conference on human factors in computing systems, 2018, pp. 1–13

    2018

  70. [78]

    Gender as a variable in natural-language processing: Ethical considerations

    B. N. Larson, “Gender as a variable in natural-language processing: Ethical considerations.” Association for Computational Linguistics, 2017. 1 Supplementary Material: A Feature-level Bias Evaluation Framework for Facial Expression Recognition Models Supplementary Material Overview

    2017

  71. [79]

    Section I : Description of the data cleaning process

  72. [80]

    Section II: Training details and performance of the deep learning models used in this study

  73. [81]

    Section III : Detailed values of ~V e,M k and all corre- sponding p-values across facial expressions and sensitive attributes for each method in our main experiments based on Swin-B

  74. [82]

    Section IV : Experimental results for the additional network architectures. I. D ATA CLEANING Both the facial expression and facial attribute datasets in- herently involve a certain level of subjectivity and uncertainty in their annotations. As a result, ambiguous annotations ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.