Pith. sign in

REVIEW 4 major objections 5 minor 107 references

Face4FairShifts: A Large Image Benchmark for Fairness and Robust Learning across Visual Domains

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces Face4FairShifts, a 100,000-image face benchmark with natural domain shifts and sensitive-attribute correlations, and reports that it is harder and more stable than existing fairness benchmarks.

desk verdict A potentially useful fairness benchmark that needs its label-quality evidence and internal consistency fixed before the 'harder and more stable' claims can be trusted. read the letter →

arxiv 2509.00658 v1 pith:NAG5DB7A submitted 2025-08-31 cs.CV cs.CYcs.LG

Yumeng Lin , Dong Li , Xintao Wu , Minglai Shao , Xujiang Zhao , Zhong Chen , Chen Zhao This is my paper
classification cs.CVcs.CYcs.LG
keywords fairnessbenchmarkfacialattributedatasetdomaingeneralizationout-of-distributiondetectioncorrelationshiftdistributionvisualdomains
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Face4FairShifts is a proposed benchmark of 100,000 facial images in four visually distinct domains—Photo, Art, Cartoon, and Sketch—each labeled with 42 human annotations across 15 attributes. The authors aim to give fairness and domain-generalization research a dataset where domain shifts and correlations between sensitive attributes and labels arise naturally, not from artificially splitting one dataset. Across four research areas, they report that state-of-the-art baselines perform worse on Face4FairShifts than on existing benchmarks, while baseline rankings stay more consistent. If the central claim holds, the benchmark provides a common, harder testbed for deciding whether fair models survive real visual-style changes and for developing methods that do.

What carries the argument

The load-bearing object is the benchmark itself: 100,000 face images across Photo, Art, Cartoon, and Sketch, each image carrying 42 binary labels in 15 attribute groups, majority-voted from at least three annotators. The mechanism is the domain construction: different rendering styles create natural covariate shifts, while the disparate-impact heatmaps show that correlations between sensitive attributes and class labels change across domains, so a model that is fair in one style can become unfair in another. This is what makes the benchmark a testbed for fairness-aware domain adaptation rather than just another attribute dataset.

What would settle it

Re-annotate a random sample of, say, 1,000 images per domain with an independent annotation team and measure per-attribute agreement against the released labels. If agreement is much lower in Art and Sketch, or near chance for subjective attributes like Attractive, then the reported disparate-impact heatmaps and the claim that the dataset is harder would reflect annotation noise, not domain shift.

Watch

Extended reading notes

Core claim

Face4FairShifts is claimed to be the first facial benchmark with both genuine covariate shifts—the same attributes rendered as photos, artwork, cartoons, and sketches—and measurable shifts in sensitive-attribute/label correlations from one domain to the next. Because the domains come from separate sources and web crawls rather than from splitting one dataset, the covariate shift is larger, quantified by Jensen-Shannon divergence in feature space. In fairness learning, OOD generalization, OOD detection, and fairness-aware OOD generalization, the paper reports that baselines score lower on Face4FairShifts while their relative ordering stays more stable, which it interprets as a harder and more

Load-bearing premise

The benchmark's validity rests on majority-voted human annotations being accurate and equally reliable in all four domains, but no inter-annotator agreement, label-noise rate, or quality-control outcome is reported, and attributes like Attractive are admitted to be subjective.

Editorial extensions

If this is right

  • Fairness-learning baselines (LFR, GSR, AD, CSAD, FNF) all show larger demographic-parity and equalized-odds gaps on Face4FairShifts than on CelebA, UTKFace, FairFace, or UTK-FairFace.
  • Domain-generalization baselines (ERM, IRM, GDRO, Mixup, MMD, MBDG) score lower accuracy and F1 on the new benchmark, with smaller variance across methods, so fair-generalization results on older datasets should be rechecked here.
  • OOD detectors separate the four visual domains well in inter-domain sensory detection, meaning the benchmark's domain shift is genuinely visible in feature space rather than being an artificial partition.
  • The dataset gives fairness-aware OOD generalization methods a common testbed where correlations between sensitive attributes and labels occur naturally and vary across domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the benchmark could also serve as a general stress test for face attribute classifiers, not just fairness-specific methods, since it varies rendering style while keeping identity-related attributes roughly constant.
  • Beyond the paper: an independent re-annotation study on a sample of images would clarify how much of the reported hardness comes from annotation noise rather than domain shift, because the paper reports no inter-annotator agreement statistics.
  • Beyond the paper: stratifying model performance by age or race within each domain could reveal which demographic groups lose the most accuracy when style changes, a comparison the paper does not report.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Face4FairShifts, a large facial-image benchmark for fairness and robustness under visual domain shifts. It contains 100K images across four domains (Photo, Art, Cartoon, Sketch) and is claimed to carry 42 annotations across 15 attributes covering demographic and facial features. The authors describe data collection from existing datasets and web crawling, manual annotation by 66 paid annotators with majority voting, and a 5-person QC team. They then present extensive experiments across four research areas: fairness learning, OOD generalization, OOD detection, and fairness-aware OOD generalization, comparing Face4FairShifts with FairFace and UTK-FairFace on covariate-shift magnitude, task difficulty, and cross-baseline stability. The headline empirical claims are that Face4FairShifts poses greater challenges to current baselines while yielding more stable performance across methods.

Significance. If the label quality and statistical claims hold, this is a potentially valuable resource for the FairOG community: it provides a large, multi-domain facial dataset with rich attribute annotations, and the t-SNE/JS-divergence analysis suggests larger natural covariate shifts than the commonly used FairFace/UTK-FairFace partitions. The breadth of the experimental evaluation—covering fairness, OOD generalization/detection, and FairOG—is a strength, and the public release of code and dataset increases usability. However, the benchmark's validity and its headline 'harder and more stable' conclusions rest on the quality of human annotations, which are not quantified, and on statistical stability indicators that lack error bars. Several internal inconsistencies (e.g., 39 vs. 42 annotations and 14 vs. 15 attributes) and a broken Figure 4 (all DI cells show NaN) further reduce confidence in the paper as written.

major comments (4)
  1. The paper reports majority voting and a 5-person QC random review but gives no inter-annotator agreement (e.g., Fleiss' kappa), no disagreement rates, and no per-domain or per-attribute label-noise analysis. This is load-bearing because all downstream comparisons—accuracy, fairness metrics, and the 'harder/more stable' conclusions—use these labels as ground truth. In particular, systematically noisier labels in crawled Art/Sketch domains (which Appendix D itself concedes may be subjective) could produce exactly the observed pattern: lower accuracy and compressed cross-baseline variance due to a common noise ceiling. The LLM comparison in Figure 2(right) cannot validate human labels because it treats them as ground truth and is aggregated across domains. Please report agreement statistics and, ideally, robustness checks such as training on high-agreement subsets or estimating label noise
  2. The lower heatmaps in Figure 4 display the string 'NaN' in every cell of every domain. If these are the actual DI values, the paper's claimed fairness disparities are not reported and the statement that 'Figure 4 highlights substantial correlation shifts' is unsupported. If this is a rendering artifact, the figure must be regenerated. Either way, the current figure cannot be used to verify the DI-based fairness characterization, and the formula's k/1/k transformation needs a clear definition of which group is unprivileged and how zero or undefined ratios are handled.
  3. The number of annotations and attributes is inconsistent throughout the manuscript: the abstract says '39 annotations within 14 attributes', while the Introduction, Figure 1, and Figure 3 describe 42 annotations across 15 attributes; Table 2 and Table 9 list 39 annotations. Since the annotation count is a core dataset property, this factual inconsistency must be resolved in the final version and all occurrences checked.
  4. The stability claim rests on the ϵ and ± indicators computed from per-baseline metric values. However, only Table 10 reports standard deviations; Tables 4–8 are averages over three runs without any variance or significance information. The differences in ϵ and ± across datasets may be within run-to-run noise, especially given the small number of runs. Please report per-run spread or confidence intervals for the stability indicators, or otherwise justify that the observed stability differences are statistically meaningful.
minor comments (5)
  1. The term 'sensory OOD detection' is used without definition. Clarify that it refers to covariate/sensory-level shifts, and how it differs from semantic OOD detection.
  2. The 'minimum acceptable image size' is said to be manually determined but its value is never reported, and the manual Photoshop cropping step is described only qualitatively. For reproducibility, provide the exact size threshold and the number of images discarded at each filtering stage.
  3. Figure 2(right) compares LLM and human annotations only via accuracy/F1 against human labels. This demonstrates disagreement but not human correctness; report human inter-annotator agreement instead, and consider domain-wise breakdowns rather than the current aggregate.
  4. The checklist answers 'NA' for 'New assets' and 'Safeguards' despite the paper introducing a new, web-crawled dataset. Provide a data sheet, license information, terms of use for crawled images, and a broader-impacts discussion for this facial dataset.
  5. Typographical issue: 'no-bread' should be 'no-beard'. Also, the text says 'all 14 attributes' while the figure and the rest of the paper use 15 attributes.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark claims rest on external empirical evaluation, not on self-citation or definitional reduction.

full rationale

This is a dataset/benchmark contribution whose central claims are empirical measurements: the dataset is large, spans four domains, and poses difficulty/stability characteristics as measured by standard baselines on held-out splits. The dataset construction (Sections 2.1-2.3) is independent of the evaluation results; the evaluations themselves use external methods (ERM, IRM, GDRO, Mixup, MMD, MBDG, LFR, GSR, AD, CSAD, FNF, etc.) under standard protocols. The self-cited baselines FEDORA [89] and FLAIR [45] are included as comparison methods, not as premises of the difficulty claim; their presence does not make the benchmark conclusion circular. The JS-divergence/t-SNE analysis (Section 3.1) is an independent feature-space comparison. The LLM comparison (Figure 2, right) uses human annotations as ground truth and therefore cannot independently validate label quality, but this is a limitation explicitly acknowledged in Appendix D, not a circular derivation. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no ansatz is smuggled via citation. The claims reduce to measured performance comparisons, which is standard benchmark practice, so there is no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The dataset's composition depends on chosen filtering thresholds and binary groupings, but no new physical or formal entities are introduced.

free parameters (5)
  • YOLOv5 person detection confidence threshold = 0.7
    Chosen threshold to filter crawled images; affects domain composition.
  • Minimum acceptable image size = not reported
    Manually determined threshold; impacts which cropped faces are retained.
  • Age binarization threshold for UTKFace/FairFace = 50 years
    Used to define elderly vs non-elderly; arbitrary and affects fairness metric results.
  • Race binarization = Black vs non-Black
    Ad hoc grouping for fairness metrics; collapses seven FairFace races into binary.
  • Disparate impact k definition = k = P(Y=1|Z=-1)/P(Y=1|Z=1); DI = k if k<=1 else 1/k
    Non-standard definition; unclear which class label Y is used in Section 2.4.
assumptions (5)
  • domain assumption The four domains (Photo, Art, Cartoon, Sketch) constitute meaningful covariate shifts for faces.
    Central to the benchmark claim; supported only by t-SNE and JS divergence on two backbones.
  • domain assumption Majority-voted human annotations are accurate ground truth.
    No inter-annotator agreement or label-noise analysis is reported in Section 2.3.
  • standard math Standard mathematical definitions of KL/JS divergence and fairness metrics are used.
    Used in Sections 2.4 and C.1.
  • domain assumption The 80% rule for disparate impact is a valid fairness criterion.
    Cited from [11]; applied to interpret DI values.
  • domain assumption Pretrained ImageNet features (ResNet50/ViT-B) are suitable for measuring covariate shift.
    Used for t-SNE and JS divergence in Figure 5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Face4FairShifts: A Large Image Benchmark for Fairness and Robust Learning across Visual Domains." pith.science (2026). https://pith.science/paper/NAG5DB7A

@misc{pith2026250900658,
  author       = {Pith},
  title        = {Pith review of: Face4FairShifts: A Large Image Benchmark for Fairness and Robust Learning across Visual Domains},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NAG5DB7A}},
  note         = {Machine review of arXiv:2509.00658}
}
read the original abstract

Ensuring fairness and robustness in machine learning models remains a challenge, particularly under domain shifts. We present Face4FairShifts, a large-scale facial image benchmark designed to systematically evaluate fairness-aware learning and domain generalization. The dataset includes 100,000 images across four visually distinct domains with 39 annotations within 14 attributes covering demographic and facial features. Through extensive experiments, we analyze model performance under distribution shifts and identify significant gaps. Our findings emphasize the limitations of existing related datasets and the need for more effective fairness-aware domain adaptation techniques. Face4FairShifts provides a comprehensive testbed for advancing equitable and reliable AI systems. The dataset is available online at https://meviuslab.github.io/Face4FairShifts/.

Figures

Figures reproduced from arXiv: 2509.00658 by the authors.

Figure 1
Figure 1. An example of the FACE4FAIRSHIFTS dataset, showcasing four domains (Photo, Art, Cartoon, and Sketch) with 42 annotations across 15 attributes. are obtained through web crawling due to the limited availability of public datasets. This approach ensures a diverse and comprehensive dataset while maintaining consistency across domains. Collection from Existing Facial Datasets. We systematically collect existing datasets … view at source ↗
Figure 2
Figure 2. (Left) An overview of the FACE4FAIRSHIFTS dataset after data processing, comprising images collected from existing datasets and supplemented by web-crawled data across all domains. (Right) Comparison of four large language models (GPT-4o, Claude 3.7 Sonnet, Gemini 2.5, and Kimi Latest) on 15 facial attribute categories in terms of accuracy and F1-score, using human annotations as ground truth. 2.2 Data Processing In… view at source ↗
Figure 3
Figure 3. Statistical overview of annotations in the Face4FairShifts dataset. The number at the top of [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Visualization of fairness statistics in FACE4FAIRSHIFTS using heatmaps across all domains. Numbers on the x-axis and y-axis represent attributes Gender (1), Race (2), Age (3), Beard (4), Glasses (5), Eyes (6), Mouth (7), Smile (8), Nose (9), Chin (10), Hair (11), Braid…
Figure 5
Figure 5. Figure 5: t-SNE visualization of datasets for covariate shifts across different domains, pre-trained [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: An example of data annotation. Each image in [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: The prompt template used to generate 15 attribute annotations with LLMs. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Experiment settings for OOD detection. In inter-domain sensory OOD detection, test [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

107 extracted references · 69 canonical work pages

  1. [1]

    In UCI Irvine Machine Learning Repository

    Adult income dataset. In UCI Irvine Machine Learning Repository . https://archive.ics.uci.edu/dataset/2/adult

  2. [2]

    In UCI Irvine Machine Learning Repository

    Bank marketing dataset. In UCI Irvine Machine Learning Repository . https://archive.ics.uci.edu/dataset/222/bank+marketing

  3. [3]

    In Kaggle

    Compas dataset. In Kaggle. https://www.kaggle.com/datasets/danofer/compass

  4. [4]

    In UCI Irvine Machine Learning Repository

    German credit dataset. In UCI Irvine Machine Learning Repository . https://archive.ics.uci.edu/dataset/144/statlog+german+credit+data

  5. [5]

    https://metmuseum.github.io/

    The metropolitan museum of art collection api. https://metmuseum.github.io/

  6. [6]

    Agarwal, A

    A. Agarwal, A. Beygelzimer, M. Dudík, J. Langford, and H. Wallach. A reductions approach to fair classification. In International conference on machine learning , pages 60–69. PMLR, 2018

    2018

  7. [7]

    B. An, Z. Che, M. Ding, and F. Huang. Transferring fairness under distribution shifts via fair consistency regularization. Advances in Neural Information Processing Systems , 35:32582– 32597, 2022

    2022

  8. [8]

    Arjovsky, L

    M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019

    arXiv 1907

Show all 107 references
  1. [9]

    H. Bai, G. Canal, X. Du, J. Kwon, R. D. Nowak, and Y . Li. Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection. In International Conference on Machine Learning, pages 1454–1471. PMLR, 2023

    2023

  2. [10]

    Balunovi ´c, A

    M. Balunovi ´c, A. Ruoss, and M. Vechev. Fair normalizing flows. arXiv preprint arXiv:2106.05937, 2021

    2021 arXiv

  3. [11]

    D. Biddle. Adverse impact and test validation: A practitioner’s guide to valid and defensible employment testing. Routledge, 2017

    2017

  4. [12]

    Buolamwini and T

    J. Buolamwini and T. Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency , pages 77–91. PMLR, 2018

    2018

  5. [13]

    Chen, C.-S

    B.-C. Chen, C.-S. Chen, and W. H. Hsu. Face recognition and retrieval using cross-age reference coding with cross-age celebrity dataset. IEEE Transactions on Multimedia , 17(6):804–815, 2015

    2015

  6. [14]

    M. J. Choi, J. J. Lim, A. Torralba, and A. S. Willsky. Exploiting hierarchical context on a large database of object categories. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 129–136. IEEE, 2010

    2010

  7. [15]

    Creager, J.-H

    E. Creager, J.-H. Jacobsen, and R. Zemel. Environment inference for invariant learning. In International Conference on Machine Learning , pages 2189–2200. PMLR, 2021

    2021

  8. [16]

    Davis and M

    J. Davis and M. Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd international conference on Machine learning , pages 233–240, 2006

    2006

  9. [17]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248–255. Ieee, 2009

    2009

  10. [18]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  11. [19]

    Dwork, M

    C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference , pages 214–226, 2012. 10

    2012

  12. [20]

    M. Eitz, J. Hays, and M. Alexa. How do humans sketch objects? ACM Transactions on graphics (TOG), 31(4):1–10, 2012

    2012

  13. [21]

    D. M. Endres and J. E. Schindelin. A new metric for probability distributions.IEEE Transactions on Information theory, 49(7):1858–1860, 2003

    2003

  14. [22]

    Escalera, M

    S. Escalera, M. Torres Torres, B. Martinez, X. Baró, H. Jair Escalante, I. Guyon, G. Tzimiropou- los, C. Corneou, M. Oliu, M. Ali Bagheri, et al. Chalearn looking at people and faces of the world: Face analysis workshop and challenge 2016. In Proceedings of the IEEE Conference...

    2016

  15. [23]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88:303–338, 2010

    2010

  16. [24]

    D.-P. Fan, Z. Huang, P. Zheng, H. Liu, X. Qin, and L. Van Gool. Facial-sketch synthesis: a new challenge. Machine Intelligence Research, 19(4):257–287, 2022

    2022

  17. [25]

    Fei-Fei, R

    L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training ex- amples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop , pages 178–178. IEEE, 2004

    2004

  18. [26]

    Giguere, B

    S. Giguere, B. Metevier, Y . Brun, B. C. da Silva, P. S. Thomas, and S. Niekum. Fairness guarantees under demographic shift. In Proceedings of the 10th International Conference on Learning Representations (ICLR), 2022

    2022

  19. [27]

    Griffin, A

    G. Griffin, A. Holub, P. Perona, et al. Caltech-256 object category dataset. Technical report, Technical Report 7694, California Institute of Technology Pasadena, 2007

    2007

  20. [28]

    Gulrajani and D

    I. Gulrajani and D. Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020

    2007 arXiv

  21. [29]

    H. Han, A. K. Jain, F. Wang, S. Shan, and X. Chen. Heterogeneous face attribute estimation: A deep multi-task learning approach. IEEE transactions on pattern analysis and machine intelligence, 40(11):2597–2609, 2017

    2017

  22. [30]

    Hardt, E

    M. Hardt, E. Price, and N. Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016

    2016

  23. [31]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770– 778, 2016

    2016

  24. [32]

    Hendrycks and K

    D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016

    2016 arXiv

  25. [33]

    G. B. Huang, M. Mattar, T. Berg, and E. Learned-Miller. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Workshop on faces in’Real- Life’Images: detection, alignment, and recognition , 2008

    2008

  26. [34]

    Jiang, Z

    Y . Jiang, Z. Huang, X. Pan, C. C. Loy, and Z. Liu. Talk-to-edit: Fine-grained facial editing via dialog. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13799–13808, 2021

    2021

  27. [35]

    Jiang, Z

    Y . Jiang, Z. Huang, T. Wu, X. Pan, C. C. Loy, and Z. Liu. Talk-to-edit: Fine-grained 2d and 3d facial editing via dialog. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023

    2023

  28. [36]

    Jocher, A

    G. Jocher, A. Stoken, J. Borovec, NanoCode012, ChristopherSTAN, L. Changyu, Laughing, A. Hogan, lorenzomammana, tkianai, yxNONG, AlexWang1900, L. Diaconu, Marc, wang- haoyang0106, ml5ah, Doug, Hatovix, J. Poznanski, L. Yu, changyu98, P. Rai, R. Ferriday, T. Sullivan, W. Xinyu,...

    2020

  29. [37]

    X. Ju, A. Zeng, J. Wang, Q. Xu, and L. Zhang. Human-art: A versatile human-centric dataset bridging natural and artificial scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 618–629, 2023

    2023

  30. [38]

    Karkkainen and J

    K. Karkkainen and J. Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 1548–1558, 2021

    2021

  31. [39]

    Karras, T

    T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017

    2017 arXiv

  32. [40]

    Karras, M

    T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems , 33:12104–12114, 2020

    2020

  33. [41]

    P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. L. Phillips, I. Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning , pages 5637–5664. PMLR, 2021

    2021

  34. [42]

    Kumar, A

    N. Kumar, A. Berg, P. N. Belhumeur, and S. Nayar. Describable visual attributes for face verification and image search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(10):1962–1977, 2011

    1962

  35. [43]

    C.-H. Lee, Z. Liu, L. Wu, and P. Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020

    2020

  36. [44]

    D. Li, Y . Yang, Y .-Z. Song, and T. M. Hospedales. Deeper, broader and artier domain gen- eralization. In Proceedings of the IEEE international conference on computer vision , pages 5542–5550, 2017

    2017

  37. [45]

    D. Li, C. Zhao, M. Shao, and W. Wang. Learning fair invariant representations under covariate and correlation shifts simultaneously. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management , pages 1174–1183, 2024

    2024

  38. [46]

    H. Li, S. J. Pan, S. Wang, and A. C. Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5400–5409, 2018

    2018

  39. [47]

    W. Liu, X. Wang, J. Owens, and Y . Li. Energy-based out-of-distribution detection.Advances in neural information processing systems, 33:21464–21475, 2020

    2020

  40. [48]

    Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015

    2015

  41. [49]

    Manning and H

    C. Manning and H. Schutze. F oundations of statistical natural language processing. MIT press, 1999

    1999

  42. [50]

    Merler, N

    M. Merler, N. Ratha, R. S. Feris, and J. R. Smith. Diversity in faces. arXiv preprint arXiv:1901.10436, 2019

    1901 arXiv

  43. [51]

    Mishra, S

    A. Mishra, S. N. Rai, A. Mishra, and C. Jawahar. Iiit-cfw: A benchmark database of cartoon faces in the wild. In Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part I 14 , pages 35–47. Springer, 2016

    2016

  44. [52]

    Mukhoti, A

    J. Mukhoti, A. Kirsch, J. van Amersfoort, P. H. Torr, and Y . Gal. Deep deterministic uncertainty: A new simple baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24384–24394, 2023

    2023

  45. [53]

    A. H. Murphy. The finley affair: A signal event in the history of forecast verification. Weather and forecasting, 11(1):3–20, 1996

    1996

  46. [54]

    Noguchi and S

    M. Noguchi and S. Shirakawa. Simple domain generalization methods are strong baselines for open domain generalization. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2024. 12

    2024

  47. [55]

    C. Oh, H. Won, J. So, T. Kim, Y . Kim, H. Choi, and K. Song. Learning fair representation via distributional contrastive disentanglement. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1295–1305, 2022

    2022

  48. [56]

    X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406–1415, 2019

    2019

  49. [57]

    T.-H. Pham, X. Zhang, and P. Zhang. Fairness and accuracy under domain generalization.ArXiv, 2023

    2023

  50. [58]

    Ricanek and T

    K. Ricanek and T. Tesafaye. Morph: A longitudinal image database of normal adult age- progression. In 7th international conference on automatic face and gesture recognition (FGR06), pages 341–345. IEEE, 2006

    2006

  51. [59]

    Robey, G

    A. Robey, G. J. Pappas, and H. Hassani. Model-based domain generalization. Advances in Neural Information Processing Systems, 34:20210–20229, 2021

    2021

  52. [60]

    Y . Roh, K. Lee, S. E. Whang, and C. Suh. Improving fair training under correlation shifts. ICML, 2023

    2023

  53. [61]

    Rothe, R

    R. Rothe, R. Timofte, and L. Van Gool. Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision, 126(2):144–157, 2018

    2018

  54. [62]

    L. Ruff, R. Vandermeulen, N. Goernitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International conference on machine learning, pages 4393–4402. PMLR, 2018

    2018

  55. [63]

    Saenko, B

    K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11 , pages 213–226. Springer, 2010

    2010

  56. [65]

    Sagawa, P

    S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731, 2019

    1911 arXiv

  57. [66]

    Sangkloy, N

    P. Sangkloy, N. Burnell, C. Ham, and J. Hays. The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions on Graphics (TOG), 35(4):1–12, 2016

    2016

  58. [67]

    M. Shao, D. Li, C. Zhao, X. Wu, Y . Lin, and Q. Tian. Supervised algorithmic fairness in distribution shifts: A survey. arXiv preprint arXiv:2402.01327, 2024

    2024 arXiv

  59. [68]

    Shimodaira

    H. Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference , 90(2):227–244, 2000

    2000

  60. [69]

    Y . Shu, Z. Cao, C. Wang, J. Wang, and M. Long. Open domain generalization with domain- augmented meta-learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9624–9633, 2021

    2021

  61. [70]

    Y . Sun, X. Wang, and X. Tang. Hybrid deep learning for face verification. InProceedings of the IEEE international conference on computer vision , pages 1489–1496, 2013

    2013

  62. [71]

    Y . Sun, X. Wang, and X. Tang. Deep learning face representation from predicting 10,000 classes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1891–1898, 2014

    2014

  63. [72]

    W. R. Tan, C. S. Chan, H. Aguirre, and K. Tanaka. Improved artgan for conditional synthesis of natural image and artwork. IEEE Transactions on Image Processing, 28(1):394–409, 2019. 13

    2019

  64. [73]

    Thomee, D

    B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016

    2016

  65. [74]

    Torralba and A

    A. Torralba and A. A. Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521–1528. IEEE, 2011

    2011

  66. [75]

    Torralba, B

    A. Torralba, B. C. Russell, and J. Yuen. Labelme: Online image annotation and applications. Proceedings of the IEEE, 98(8):1467–1484, 2010

    2010

  67. [76]

    Vapnik.The nature of statistical learning theory

    V . Vapnik.The nature of statistical learning theory . Springer science & business media, 1999

    1999

  68. [77]

    A. Vyas, N. Jammalamadaka, X. Zhu, D. Das, B. Kaul, and T. L. Willke. Out-of-distribution detection using an ensemble of self supervised leave-out classifiers. In Proceedings of the European conference on computer vision (ECCV) , pages 550–564, 2018

    2018

  69. [78]

    C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. 2011

    2011

  70. [79]

    H. Wang, C. Zhao, and F. Chen. Madod: Generalizing ood detection to unseen domains via g-invariance meta-learning. In 2024 IEEE International Conference on Big Data (BigData) , pages 1134–1143. IEEE, 2024

    2024

  71. [80]

    K. Wang, S. Zhou, C. A. Fu, and J. X. Yu. Mining changes of classification by correspondence tracing. In Proceedings of the 2003 SIAM International Conference on Data Mining , pages 95–106. SIAM, 2003

    2003

  72. [81]

    X. Wang, J. Zhang, L. Qi, and Y . Shi. Generalizable decision boundaries: Dualistic meta- learning for open set domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11564–11573, 2023

    2023

  73. [82]

    Widmer and M

    G. Widmer and M. Kubat. Learning in the presence of concept drift and hidden contexts. Machine learning, 23:69–101, 1996

    1996

  74. [83]

    S. Yan, H. Song, N. Li, L. Zou, and L. Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677, 2020

    2001 arXiv

  75. [84]

    Zemel, Y

    R. Zemel, Y . Wu, K. Swersky, T. Pitassi, and C. Dwork. Learning fair representations. In International conference on machine learning , pages 325–333. PMLR, 2013

    2013

  76. [85]

    B. H. Zhang, B. Lemoine, and M. Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , pages 335–340, 2018

    2018

  77. [86]

    Zhang, X

    W. Zhang, X. Wang, and X. Tang. Coupled information-theoretic encoding for face photo-sketch recognition. In CVPR 2011, pages 513–520. IEEE, 2011

    2011

  78. [87]

    Zhang, Z

    Y . Zhang, Z. Yin, Y . Li, G. Yin, J. Yan, J. Shao, and Z. Liu. Celeba-spoof: Large-scale face anti-spoofing dataset with rich annotations. In European Conference on Computer Vision (ECCV), 2020

    2020

  79. [88]

    Zhang, Y

    Z. Zhang, Y . Song, and H. Qi. Age progression/regression by conditional adversarial autoen- coder. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5810–5818, 2017

    2017

  80. [89]

    C. Zhao, K. Jiang, X. Wu, H. Wang, L. Khan, C. Grant, and F. Chen. Algorithmic fairness generalization under covariate and dependence shifts simultaneously. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 4419–4430, 2024

    2024

  81. [90]

    Zheng, Y

    Y . Zheng, Y . Zhao, M. Ren, H. Yan, X. Lu, J. Liu, and J. Li. Cartoon face recognition: A benchmark dataset. In Proceedings of the 28th ACM international conference on multimedia , pages 2264–2272, 2020. 14

    2020

  82. [91]

    B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence , 40(6):1452–1464, 2017

    2017

  83. [92]

    elderly" and

    W. Zhu, H. Zheng, H. Liao, W. Li, and J. Luo. Learning bias-invariant representation by cross-sample mutual information minimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15002–15012, 2021. 15 A Related Work Table 9 presents a list ...

    2021

  84. [93]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and introduction clearly present the motivation, contribution, and limitations of this study. Guidelines:...

  85. [94]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper discusses the limitations in Appendix D. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that ...

  86. [95]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] 23 Justification: This paper is an experimental work without theoretical results. Guidelines: • The answer N...

  87. [96]

    The code is attached, and we are committed to releasing it publicly upon acceptance

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  88. [97]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code 24 Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Our paper provides open ...

  89. [98]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: See Experiments and Appendix....

  90. [99]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: All experiments are conducted multiple times, and ...

  91. [100]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Experiments compute res...

  92. [101]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: The research conducted in the paper conforms, in every respect, with the NeurIP...

  93. [102]

    • If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: Only Technical reports Guidelines: • The answer NA means that there is no societal impact of the work perf...

  94. [103]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: There is no suc...

  95. [104]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All datasets and...

  96. [105]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not release new assets. Guidelines: • The answer NA means that the paper does not release new assets. ...

  97. [106]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  98. [107]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  99. [108]

    Answer: [NA] Justification: The core method development in this research does not involve LLMs

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.