Pith. sign in

REVIEW 3 major objections 6 minor 32 references

Fairness of Deep Ensembles: On the interplay between per-group task difficulty and under-representation

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Averaging several independently trained deep networks narrows accuracy gaps between protected groups while raising accuracy for every group, and when one group's task is harder, a perfectly balanced dataset is no longer optimal.

desk verdict A clean synthetic demonstration that homogeneous ensembles can give positive-sum fairness and that optimal balance shifts with task difficulty; the real-data support is thinner and needs error bars and a clear selection protocol. read the letter →

arxiv 2501.14551 v1 pith:NXWALO7Q submitted 2025-01-24 cs.LG

Estanislao Claucich , Sara Hooker , Diego H. Milone , Enzo Ferrante , Rodrigo Echeveste This is my paper · ORCID
classification cs.LG
keywords deepensemblesalgorithmicfairnessgroupprotectedattributestaskdifficultyunder-representationpositive-sumdatasetbalancing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish two things. First, a plain homogeneous ensemble of deep networks—same architecture, same training data, different random initializations—narrows the accuracy gap between protected subgroups while improving accuracy for every subgroup, an outcome the authors call positive-sum fairness. Second, when subgroups differ in intrinsic task difficulty, a perfectly balanced training set is not the fairness-optimal choice: over-representing the harder group simultaneously improves overall accuracy and reduces the gap. The authors test this in controlled synthetic settings and in two real image datasets. If right, it means a simple, model-agnostic technique can act as a fairness intervention with no accuracy trade-off, and that standard rebalancing advice should be revised to account for per-group difficulty.

What carries the argument

The central object is a homogeneous deep ensemble: several models with identical architecture and hyperparameters, trained independently on the same samples, differing only in initialization and training stochasticity, then averaged. The paper's second load-bearing idea is a proxy: when a balanced dataset yields a performance gap, they read that gap as relative task difficulty. The experiments manipulate difficulty in controlled synthetic ways (flipping labels in one subgroup, rotating one subgroup's optimal decision boundary) and vary the male/female sample ratio, then measure gap and accuracy as ensemble size grows. The ensemble does the fairness work because averaging reduces variance and disproportionately lifts the group with higher per-sample error.

What would settle it

Find a real or synthetic classification task where a group underperforms on balanced training data, then show that the underperformance vanishes when label noise is removed while balancing is unchanged, or that over-representing that group lowers overall accuracy and widens the gap; such a case would show the balanced-performance proxy confounds difficulty with data quality and break the recommended over-representation rule.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that homogeneous ensembling is a positive-sum fairness intervention: it consistently reduces the accuracy gap between protected subgroups while raising the accuracy of every subgroup, with no leveling-down trade-off. A second discovery is that under-representation and per-group task difficulty interact so that balance is not the fairness optimum: when one group's task is intrinsically harder, the sample ratio that minimizes the gap and maximizes overall accuracy over-represents the harder group, and a perfectly balanced dataset can hurt both.

Load-bearing premise

The whole argument rests on treating the performance gap seen when training on perfectly balanced data as a true measure of task difficulty per group, rather than an artifact of sampling, label noise, or model choice.

Editorial extensions

If this is right

  • For every synthetic and real setting tested, adding models to a homogeneous ensemble increased accuracy for both protected groups and shrank the gap, so fairness gains do not come at the cost of overall performance.
  • The representation ratio that minimizes the male-female gap also maximizes overall accuracy; as the harder group's difficulty grows, the optimal ratio shifts away from 50-50 toward over-representing the harder group.
  • A perfectly balanced dataset can be strictly worse than an imbalanced one on both overall accuracy and group gap when per-group task difficulty differs.
  • In the real datasets, non-balanced ratios (roughly 40-60 for one and 80-20 for the other) achieved positive-sum fairness, with the underperforming group improving the most as ensemble size grew.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: if ensemble gains are largest for the harder group and grow with difficulty, then using an ensemble may reduce the need for aggressive rebalancing; in some regimes ensembles could substitute for oversampling rather than complement it.
  • Our inference: the label-noise and rotated-boundary manipulations produce the same qualitative conclusion, which hints the effect is about irreducible per-group error rather than the specific mechanism; a direct test would vary per-group error structure while holding balanced accuracy fixed.
  • Our inference: the recommended over-representation level should depend on an estimate of per-group difficulty, so a practical workflow is to run a balanced pilot, estimate the gap, and oversample the underperforming group until gap and overall accuracy are jointly optimized; this follows from the paper's ideal-balance-ratio plots but is not stated by the authors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper investigates whether homogeneous deep ensembles improve group fairness (defined by protected attributes) without sacrificing overall performance, and how the interplay between subgroup under-representation and per-group task difficulty affects the optimal data balance. Using synthetic Gaussian binary classification tasks with controlled label noise or rotated decision boundaries, and two real image datasets (CelebA, CheXpert), the authors report that ensembles consistently increase accuracy for all groups while reducing the accuracy gap between groups ('positive-sum fairness'). They further report that when one subgroup's task is harder, the balance ratio that minimizes the gap also maximizes overall accuracy, and can be far from 50-50; a perfectly balanced dataset can be suboptimal. The conclusions are derived from accuracy comparisons across ensemble sizes and balance ratios.

Significance. If the findings hold, they offer a practical, architecture-agnostic fairness intervention (ensembling) that avoids the usual leveling-down trade-off, and they challenge the default assumption that balanced training data is always optimal for fairness. The synthetic experimental design is careful: 5 folds, 500 random ensemble draws, no fitted constants, and the ensemble-versus-single-model comparison is an independent measurement. The paper also explicitly discusses limitations of the difficulty proxy. However, the real-data evidence for the suboptimality of balanced training currently lacks statistical grounding, and the selection of 'ideal' balance ratios is under-specified, so the strength of the practical claims exceeds the current evidence.

major comments (3)
  1. [Section 4.2, Figure 4] The procedure by which the 'ideal balance ratio' (40-60 for CheXpert, 80-20 for CelebA) was chosen is not stated; if the ratio was selected by minimizing gap or maximizing accuracy on the same test set used for evaluation, the reported 3.4-point gap reduction and 0.45% overall gain are likely optimistic due to selection over multiple candidate ratios. Please specify the selection protocol (e.g., a validation split), and report the full balance-ratio sweeps with confidence intervals or bootstrap error bars for the headline numbers.
  2. [Sections 3.1 and 6] The conclusion that harder groups need over-representation relies on the assumption that accuracy differences under balanced training measure intrinsic task difficulty. The paper acknowledges this in Section 6, but the Discussion (Section 5) states the conclusion without this caveat. Since the real-data demonstrations cannot separate intrinsic difficulty from other causes of performance gaps (e.g., sampling variability, label quality, model choice), the real-data claims should be explicitly conditional on the proxy, or supplemented with a more direct difficulty measure.
  3. [Section 4.1 and Section 5] The statement that ensembles reduce the gap 'to be insignificant' (end of Section 1 and repeated in Section 5) is a statistical claim unsupported by any test or confidence interval. Please add a significance test (e.g., a bootstrap confidence interval for the gap difference) or rephrase to 'small' or 'near zero' to avoid the statistical connotation.
minor comments (6)
  1. [Title page] 'Equal contributtion' should be 'Equal contribution'.
  2. [Section 3.1] 'We begun' should be 'We began'; 'raging from' should be 'ranging from'.
  3. [Section 1] The sentence 'the intrinsic difficulty of the task for different subgroups is not trivial to determine and.' has a dangling 'and.' and should be completed or removed.
  4. [Figures 2 and 3 captions] 'forlabel noise scenario' is missing a space; it should read 'for label noise scenario'.
  5. [Section 4.2] In the CheXpert paragraph, 'the performance difference between males and females was not very significant' uses 'significant' informally; consider 'large' to avoid confusion with statistical significance.
  6. [Section 4.2] The phrase 'close to the balanced scenario (40%)' is ambiguous; clarify that 40% refers to the male proportion in a 40-60 M-F split.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: ensemble benefits and ideal-balance findings are direct measurements; the difficulty proxy is an acknowledged assumption, not a fitted input.

full rationale

The paper's central claims are measured quantities rather than outputs of fitted models. In the synthetic scenarios, per-group difficulty is manipulated externally (label-flip percentage, decision-boundary rotation), and the relationship between these manipulations, balance ratio, ensemble size, and accuracy/gap is read directly from training and evaluation; no parameter is fitted to the target result and then re-predicted. The real-data experiments likewise compare directly measured single-model versus ensemble accuracies across balance ratios. The only definitional element is the Section 3.1/6 proxy that identifies task difficulty with balanced-dataset accuracy; this could give a tautological flavor to the 'harder group needs over-representation' conclusion, but the paper explicitly acknowledges in Section 6 that the proxy may fail (unequal sampling variability, model choice), and the synthetic results do not depend on that proxy because difficulty is externally controlled. No load-bearing conclusions rest on self-citations: Ko et al. and Schweighofer et al. are related-work anchors, not premises. The selection of the ideal balance ratio in Section 4.2 is a search over test-set outcomes without confidence intervals, which is a statistical-robustness limitation, not circularity. Overall, no circular step rises above the 0-2 band; score 1 reflects the minor definitional echo in the difficulty proxy.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This ledger is light because the paper is an empirical study rather than a derivation: there are no fitted constants in the central argument. The key assumptions are the balanced-accuracy proxy for task difficulty, the use of absolute accuracy gap as the fairness metric, and the representativeness of the synthetic and two real datasets. These are domain assumptions, not mathematical axioms, and the first is explicitly acknowledged as fragile.

assumptions (3)
  • domain assumption Performance differences on a balanced training set are a valid proxy for intrinsic per-group task difficulty.
    Introduced in Section 3.1 as the operational definition and explicitly flagged in Section 6 as possibly invalid when sampling is unequal or when model choice changes.
  • domain assumption The absolute accuracy gap between two protected groups is an adequate fairness measure for this study.
    Used throughout Figures 2-4 to define the gap and to select the 'ideal' balance ratio; no alternative fairness metrics are analyzed.
  • domain assumption Results from the synthetic Gaussian binary task and two image datasets are representative of fairness dynamics in other domains.
    The authors rely on these settings to support general statements in Sections 1 and 5; acknowledged as an open question in Section 6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fairness of Deep Ensembles: On the interplay between per-group task difficulty and under-representation." pith.science (2026). https://pith.science/paper/NXWALO7Q

@misc{pith2026250114551,
  author       = {Pith},
  title        = {Pith review of: Fairness of Deep Ensembles: On the interplay between per-group task difficulty and under-representation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NXWALO7Q}},
  note         = {Machine review of arXiv:2501.14551}
}
read the original abstract

Ensembling is commonly regarded as an effective way to improve the general performance of models in machine learning, while also increasing the robustness of predictions. When it comes to algorithmic fairness, heterogeneous ensembles, composed of multiple model types, have been employed to mitigate biases in terms of demographic attributes such as sex, age or ethnicity. Moreover, recent work has shown how in multi-class problems even simple homogeneous ensembles may favor performance of the worst-performing target classes. While homogeneous ensembles are simpler to implement in practice, it is not yet clear whether their benefits translate to groups defined not in terms of their target class, but in terms of demographic or protected attributes, hence improving fairness. In this work we show how this simple and straightforward method is indeed able to mitigate disparities, particularly benefiting under-performing subgroups. Interestingly, this can be achieved without sacrificing overall performance, which is a common trade-off observed in bias mitigation strategies. Moreover, we analyzed the interplay between two factors which may result in biases: sub-group under-representation and the inherent difficulty of the task for each group. These results revealed that, contrary to popular assumptions, having balanced datasets may be suboptimal if the task difficulty varies between subgroups. Indeed, we found that a perfectly balanced dataset may hurt both the overall performance and the gap between groups. This highlights the importance of considering the interaction between multiple forces at play in fairness.

Figures

Figures reproduced from arXiv: 2501.14551 by the authors.

Figure 1
Figure 1. Impact of ensembling on an imbalanced scenario (20% male and 80% female) and with a label noise applied to 40% of female [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Results for the label noise synthetic scenario. Each column represents different noise levels in the female subgroup while keeping a constant 20% under-representation of males (20-80 M-F). (a) Samples of both distributions under different noise-balance configurations. The following rows correspond to (b) model accuracy for male (orange), female (light blue), and overall (green); (c) absolute performance gap (line co… view at source ↗
Figure 3
Figure 3. Results for the label noise synthetic scenario. (Left) Overall accuracy and gap for different noise levels by the balance ratio between male and female subgroups for label noise scenario. (Right) Ideal male-female balance ratio for the different noise percentages. As the discrepancy in difficulty increases for both subgroups, more samples are needed for the most difficult one to achieve the best overall performance … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: (Top) For CheXpert, female was identified as the most difficult group, and we found an ideal balance ratio of 40-60 M-F (more [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Results for the rotating decision boundary synthetic scenario. Each column represents a different angle on which the decision boundary is rotated for the female subgroup. The higher the angle, the higher the difficulty for this subgroup. Every case was trained with a s…
Figure 6
Figure 6. Figure 6: (Left) Overall accuracy and gap for different noise percentages by the balance ratio between male and female subgroups [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 19 canonical work pages

  1. [1]

    Chirag Agarwal, Daniel D’souza, and Sara Hooker. 2022. Estimating Example Difficulty Using Variance of Gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 10368–10378

    work page 2022

  2. [2]

    Tom L Beauchamp. 2003. Methods and principles in biomedical ethics. Journal of Medical ethics 29, 5 (2003), 269–274

    work page 2003

  3. [3]

    Samia Belhadj, Sanguk Park, Ambika Seth, Hesham Dar, and Thijs Kooi. 2024. Positive-Sum Fairness: Leveraging Demographic Attributes to Achieve Fair AI Outcomes Without Sacrificing Group Gains. In MICCAI Workshop on Fairness of AI in Medical Imaging . Springer, 56–66

    work page 2024

  4. [4]

    Leo Breiman. 2001. Random Forests. Machine Learning 45, 1 (Oct 2001), 5–32. doi:10.1023/A:1010933404324

    doi:10.1023/a:1010933404324 2001

  5. [5]

    Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency . PMLR, 77–91

    work page 2018

  6. [6]

    Simon Caton and Christian Haas. 2024. Fairness in machine learning: A survey. Comput. Surveys 56, 7 (2024), 1–38

    2024

  7. [7]

    Hugh Chen, Scott Lundberg, and Su-In Lee. 2017. Checkpoint Ensembles: Ensemble Methods from a Single Training Process. arXiv:1710.03282 [cs.LG] https://arxiv.org/abs/1710.03282

    arXiv 2017

  8. [8]

    Dietterich

    Thomas G. Dietterich. 2000. Ensemble Methods in Machine Learning. In Multiple Classifier Systems. Springer Berlin Heidelberg, Berlin, Heidelberg, 1–15

    work page 2000

Show all 32 references
  1. [9]

    Enzo Ferrante and Rodrigo Echeveste. 2024. Open Challenges on Fairness of Artificial Intelligence in Medical Imaging Applications. arXiv preprint arXiv:2407.16953 (2024)

    2024 arXiv

  2. [10]

    Usman Gohar, Sumon Biswas, and Hridesh Rajan. 2023. Towards Understanding Fairness and its Composition in Ensemble Machine Learning. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1533–1545. doi:10.1109/icse48619.2023.00133

    2023

  3. [11]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. arXiv:1512.03385 [cs.CV] https://arxiv.org/abs/1512.03385

    2015 arXiv

  4. [12]

    Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. 2020. Characterising Bias in Compressed Models. arXiv:2010.03058 [cs.LG] https://arxiv.org/abs/2010.03058

    2020 arXiv

  5. [13]

    Mong, Safwan S

    Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik ...

    2019 arXiv

  6. [14]

    Kazmi, Rasheed Hussain, Alma Oracevic, and Asad Khattak

    Patrik Kenfack, Adil Khan, S.M. Kazmi, Rasheed Hussain, Alma Oracevic, and Asad Khattak. 2021. Impact of Model Ensemble On the Fairness of Classifiers in Machine Learning. 1–6. doi:10.1109/ICAPAI49758.2021.9462068

    2021

  7. [15]

    Wei-Yin Ko, Daniel D’souza, Karina Nguyen, Randall Balestriero, and Sara Hooker. 2023. Fair-ensemble: When fairness naturally emerges from deep ensembling. arXiv preprint arXiv:2303.00586 (2023)

    2023 arXiv

  8. [16]

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems 30 (2017). 10 Estanislao Claucich, Sara Hooker, Diego H. Milone, Enzo Ferrante,...

    2017

  9. [17]

    Agostina J Larrazabal, César Martínez, Jose Dolz, and Enzo Ferrante. 2021. Orthogonal ensemble networks for biomedical image segmentation. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27...

    2021

  10. [18]

    Larrazabal, Nicolás Nieto, Victoria Peterson, Diego H

    Agostina J. Larrazabal, Nicolás Nieto, Victoria Peterson, Diego H. Milone, and Enzo Ferrante. 2020. Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proceedings of the National Academy of Sciences of the United States of Am...

    2020 doi

  11. [19]

    Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. 2015. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314 (2015)

    2015 arXiv

  12. [20]

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep Learning Face Attributes in the Wild. In Proceedings of International Conference on Computer Vision (ICCV)

    2015

  13. [21]

    Lucas Mansilla, Estanislao Claucich, Rodrigo Echeveste, Diego H Milone, and Enzo Ferrante. [n. d.]. Demographically-Informed Prediction Discrepancy Index: Early Warnings of Demographic Biases for Unlabeled Populations. Transactions on Machine Learning Research ([n. d.])

  14. [22]

    Giung Nam, Jongmin Yoon, Yoonho Lee, and Juho Lee. 2021. Diversity matters when learning from ensembles. Advances in neural information processing systems 34 (2021), 8367–8377

    2021

  15. [23]

    Milton Pividori, Georgina Stegmayer, and Diego H. Milone. 2016. Diversity control for improving the analysis of consensus clustering. Information Sciences 361-362 (2016), 120–134. doi:10.1016/j.ins.2016.04.027

    2016 doi

  16. [24]

    María Agustina Ricci Lara, Rodrigo Echeveste, and Enzo Ferrante. 2022. Addressing fairness in artificial intelligence for medical imaging. nature communications 13, 1 (2022), 4581

    2022

  17. [25]

    Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. 2020. An Investigation of Why Overparameterization Exacerbates Spurious Correlations. arXiv:2005.04345 [cs.LG] https://arxiv.org/abs/2005.04345

    2020 arXiv

  18. [26]

    Kajetan Schweighofer, Adrian Arnaiz-Rodriguez, Sepp Hochreiter, and Nuria Oliver. 2024. The Disparate Benefits of Deep Ensembles.arXiv preprint arXiv:2410.13831 (2024)

    2024 arXiv

  19. [27]

    Chen, and Marzyeh Ghassemi

    Laleh Seyyed-Kalantari, Guanxiong Liu, Matthew McDermott, Irene Y. Chen, and Marzyeh Ghassemi. 2020. CheXclusion: Fairness gaps in deep chest X-ray classifiers. arXiv:2003.00827 [cs.CV] https://arxiv.org/abs/2003.00827

    2020 arXiv

  20. [28]

    Laleh Seyyed-Kalantari, Haoran Zhang, Matthew B. A. McDermott, Irene Y. Chen, and Marzyeh Ghassemi. 2021. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature Medicine 27, 12 (2021), 2176–2182. doi:1...

    2021

  21. [29]

    Shoaib Ahmed Siddiqui, Nitarshan Rajkumar, Tegan Maharaj, David Krueger, and Sara Hooker. 2023. Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics. InThe Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=P...

    2023

  22. [30]

    Nina Weng, Siavash Bigdeli, Eike Petersen, and Aasa Feragen. 2023. Are Sex-based Physiological Differences the Cause of Gender Bias for Chest X-ray Diagnosis? arXiv:2308.05129 [eess.IV] https://arxiv.org/abs/2308.05129

    2023 arXiv

  23. [31]

    Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. 2020. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Proceedings of the 2020 conference on fairness, accountability, and transparency . 547–558

    2020

  24. [32]

    Dominik Zietlow, Michael Lohaus, Guha Balakrishnan, Matthäus Kleindessner, Francesco Locatello, Bernhard Schölkopf, and Chris Russell. 2022. Leveling Down in Computer Vision: Pareto Inefficiencies in Fair Deep Classifiers. arXiv:2203.04913 [cs.CV] https://arxiv.org/abs/2203.04...

    2022 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.