A Primer on Causal and Statistical Dataset Biases for Fair and Robust Image Analysis

Ben Glocker; Charles Jones

REVIEW 3 major objections 5 minor 2 references

A Primer on Causal and Statistical Dataset Biases for Fair and Robust Image Analysis

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that fair representation learning cannot be both effective and harmless when training and test data are identically distributed; its value depends entirely on an assumed distribution shift.

desk verdict A solid but derivative primer; the futility theorem is an idealization you'd need the original papers to verify. read the letter →

arxiv 2509.04295 v1 pith:HW2KBLFI submitted 2025-09-04 cs.LG cs.CYstat.ML

Charles Jones , Ben Glocker This is my paper · ORCID

classification cs.LGcs.CYstat.ML

keywords datasetbiasfairrepresentationlearningsubgroupseparabilitycausalinferencedistributionshiftmedicalimagingnolunchlabel

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the failures of fairness methods in medical image analysis are not implementation bugs but consequences of the causal and statistical structure of biased datasets. It unifies dataset bias and distribution shift under one structural causal model, introducing two named problems: the 'no fair lunch' problem, meaning no model or fairness metric can be universally correct because multiple causal models fit the same data; and 'subgroup separability,' the degree to which images encode which subgroup a person belongs to. Its central theoretical result is Proposition 4.8: fair representation learning cannot simultaneously be effective (remove sensitive information that plain ERM would encode) and harmless (retain all task-relevant information) when training and test data are identically distributed. If true, iid benchmarks cannot show the value of fair representations; the only setting where they can help is one where the training bias disappears at test time, and that validity hinges on causal assumptions and on subgroup separability.

What carries the argument

The load-bearing device is a structural causal model of the imaging pipeline that decomposes each input into XZ (pathological structures caused by the true condition Z) and XA (features encoding the sensitive attribute A). Unbiasedness is defined as the conditional independence Y ⊥ XA | XZ, and applying d-separation yields three basic bias mechanisms: presentation, prevalence, and annotation disparities. On top of this, the paper defines subgroup separability as p(a | x), the probability with which an image identifies its owner's subgroup, and the futility result is obtained by translating the two FRL goals—effectiveness and harmlessness—into mutual-information equalities whose combined cons

What would settle it

Train an ERM model and a fair representation model on a biased dataset whose training and test distributions are identical and where the Markov-blanket idealization holds; measure whether the fair representation removes sensitive information that ERM uses while still matching ERM's target accuracy. Proposition 4.8 predicts this cannot happen; observing it would refute the theorem.

Watch

Extended reading notes

Core claim

The paper's core claim, stated as Proposition 4.8 ('Futility'), is that under the causal decomposition of images into disease-relevant features XZ and sensitive features XA, a fair representation RFRL that is marginally independent of the sensitive attribute A can satisfy two natural desiderata—effectiveness (it drops sensitive information that an unconstrained ERM representation would keep) and harmlessness (it retains all information needed to predict the target Y at test time)—only if the training and test structural causal models differ. Because the two desiderata force the training distribution to be biased (Y not independent of XA given XZ) and the test distribution to be unbiased (Y i

Load-bearing premise

The proof assumes that an unconstrained model trained with enough data encodes exactly the inputs that form a Markov blanket around the target—no extra sensitive information and no omitted relevant information—so real finite-sample models may not meet that idealization.

Editorial extensions

If this is right

Fair representation learning evaluated on iid test sets cannot be expected to outperform ERM; the widespread absence of gains is predicted by Proposition 4.8, not a defect in implementations.
Any claim that a fair representation improves deployment performance is implicitly a claim that a distribution shift exists and that the sensitive pathway is spurious; researchers should state that shift explicitly.
Subgroup separability mediates how label bias harms groups: high separability isolates a mislabelled subgroup and degrades its accuracy, while low separability lets the correct mapping of one group rescue the other.
Bias mitigation needs to be matched to the causal mechanism—presentation, prevalence, or annotation disparities—because the three mechanisms imply different preserved and removed pathways.
Datasets and benchmarks that report subgroup separability and the assumed deployment shift would make fairness results interpretable, whereas aggregate iid accuracy cannot.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that the same conditional-independence proof should apply to any debiasing objective that removes a spurious pathway while trying to keep target information, not just fair representation learning.
If Proposition 4.8 is right, 'fairness' is not a property of a learned representation alone; the same representation can be fair for one deployment and unfair for another, so evaluation should be framed as a property of the train-to-test shift.
A practical extension would be to select FRL only when subgroup separability is high and a shift that deletes the sensitive pathway is plausible; in low-separability settings the analysis predicts FRL mainly harms performance.
Benchmark designers could simulate deployment shifts, such as changing subgroup prevalence or annotation policy, and then test FRL; under such shifts the paper's theory predicts FRL should beat ERM, especially at high separability.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a well-written condensation of three already-published papers, not a new contribution. If you want the proof of Proposition 4.8, you'll need to pull the ICLR paper; the primer gives only a sketch and is upfront about that.

What it does well: the causal decomposition of images into X_Z and X_A with the three bias mechanisms (presentation, prevalence, annotation) is a genuinely useful framing, and the worked medical examples in Fig.2 are clarifying. Subgroup separability is a real empirical phenomenon, and Table 1 and Fig.3 are some of the clearest evidence I've seen that how easily a model can identify your subgroup determines how much label bias hurts your performance. The futility theorem, even as a sketch, captures an intuition people have been circling: under iid evaluation, removing information that currently helps predict the target will only help if that information is spurious. That's worth stating explicitly.

The soft spots: the stress-test is right that Assumption 4.1 is load-bearing and not justified for real deep networks. The theorem says that if ERM encodes exactly the Markov blanket, then effectiveness and harmlessness can't both hold under iid. But deep networks routinely encode sensitive information that isn't part of a Markov blanket—accidental correlations, optimization artifacts—and they don't converge to minimal sufficient statistics. In that regime, effective and harmless representations could coexist without distribution shift. The paper appeals to information bottleneck, but that's a principle, not a training guarantee, and the practical implications are left open. Also, 'no fair lunch' is the classic NFL theorem with a new name; the framing is nice but the content is not new. And since the empirical numbers are recycled from prior papers, this document doesn't add independent evidence.

Upshot: if you're new to this area, this is a good 20-page entry point. I'd hand it to a student before the original papers. But a researcher should cite the underlying papers, not this primer. I wouldn't send this to a research venue as a standalone paper; the omitted proofs make it unverifiable on its own. If a venue explicitly wants expository syntheses, then perhaps, but I'd want a paragraph on the limits of Assumption 4.1.

Referee Report

3 major / 5 minor

Summary. This primer develops a causal taxonomy of dataset bias in medical image analysis and uses it to analyze the limits of fair representation learning (FRL). The authors decompose the input image into latent factors X_Z (disease-related) and X_A (sensitive-attribute-related), define an 'unbiased' distribution via Y ⊥ X_A | X_Z, and identify three bias mechanisms (presentation, prevalence, annotation disparities). They introduce subgroup separability as the degree to which sensitive attributes are predictable from the input, measure it across eleven medical imaging dataset-attribute pairs (Table 1), and show that label-bias degradation tracks separability (Fig. 3). The central theoretical claim is Proposition 4.8: under definitions of Effectiveness and Harmlessness, a fair representation cannot be both effective and harmless if train and test distributions are i.i.d. The paper sketches the proof via Lemmas 4.2, 4.3, 4.6, and 4.7, but explicitly omits proofs. It also presents empirical evidence (Fig. 5) that FRL's performance gap relative to ERM correlates with subgroup separability under distribution shift, supporting two hypotheses about the conditions under which FRL can help.

Significance. If Proposition 4.8 and its supporting lemmas are correct, the paper offers a clean formal explanation for the widely observed failure of FRL methods to beat ERM on i.i.d. benchmarks: any method that removes sensitive information while preserving task-relevant information implicitly assumes a train/test distribution shift. The causal unification of fairness and distribution shift (Section 2) is a useful conceptual contribution, and subgroup separability (Section 3) is an empirically grounded, practically relevant quantity that appears to predict how much damage label bias causes and when FRL might help. The empirical results, while not the main theoretical contribution, are suggestive and consistent with the proposed framework. The paper is clearly written and makes its assumptions explicit, which is a strength. However, the theoretical core is presented in abridged form: the proofs are deferred to the authors' prior ICLR paper, and the key assumption (Assumption 4.1) is an idealization that is not empirically validated. The significance of the result therefore hinges on material that is not fully contained in this manuscript.

major comments (3)

[§4, Proposition 4.8 and Lemmas 4.2–4.7] The central theoretical claim is stated without proof: the paper says 'We omit proofs in this abridged version' (p. 6), and Lemmas 4.2, 4.3, 4.6, and 4.7 are asserted with no derivations. Proposition 4.8 is the main novelty of the paper, and its validity cannot be checked from the text. The manuscript should include complete proofs (or at least a full proof of Proposition 4.8 and its lemmas in an appendix), or state explicitly that this is an expository summary of a separately published result and give a precise theorem-by-theorem pointer. As it stands, the theoretical contribution is not self-contained.
[§4, Assumption 4.1 (p. 6)] Assumption 4.1 equates unconstrained ERM representations with the Markov blanket of Y: R_ERM = f_θ(X*) iff Y ⊥ (X \ X*) | X*. This requires infinite training data, convergence to a minimal sufficient statistic, and an optimization procedure that does not encode non-predictive features. In finite-sample or overparameterized regimes, ERM can encode the sensitive attribute A even when Y ⊥ X_A | X_Z (e.g. through spurious correlations), which would allow Effectiveness and Harmlessness to coexist in i.i.d. settings. The appeal to the information bottleneck principle is descriptive, not a training guarantee. The authors should either prove a finite-sample analogue, provide empirical evidence that real trained models approximate Assumption 4.1, or explicitly restrict the futility claim to the infinite-data idealization and discuss its limits for real models.
[§4, Lemma 4.2 (p. 7)] The statement 'Fair representations must depend on X_Z only: R_FRL ⊥ A ⇒ R_FRL = f_θ(X_Z)' is not generally valid. A representation can depend on X_A and still be marginally independent of A if X_A contains variation not caused by A or if the function f_θ is non-injective in a way that cancels the A-dependence. Establishing this lemma requires a stronger assumption about the relationship between X_A and A (e.g. that X_A is a deterministic function of A, or that the representation is constrained to be conditionally independent of A given X_Z). Without such an assumption, Lemma 4.2, and hence the chain leading to Proposition 4.8, is not justified.

minor comments (5)

[§2] Typo 'conventinoal' (p. 2).
[§4, Fig. 5] The legend entry 'ERM = FRL' is unclear: does it mean the plotted quantity is the ERM-to-FRL gap, or that the two coincide on some points? Please clarify.
[§4, Fig. 5 and caption] The sentence 'we aggregate results for each method over nine runs ... and repeat for each dataset-attribute combination in from our experiments in §3' contains a grammatical error ('in from') and is vague about which FRL methods are included. Since the figure is used as evidence for Hypotheses 4.9 and 4.10, please specify the FRL methods, data splits, and the exact definition of Δ Acc.
[§3, Table 1] The use of test-set AUC as a proxy for subgroup separability is reasonable, but the paper does not report the baseline prevalence or the classifiers' architecture/hyperparameters. A sentence on the training setup would improve reproducibility.
[§2, 'No fair lunch'] The no-free-lunch argument is informal and relies on a citation to Wolpert & Macready. For a primer this is acceptable, but the connection to the later formal results could be made tighter, e.g. by stating explicitly which causal assumptions are needed for each bias mechanism.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: Proposition 4.8 is a formal consequence of explicitly stated assumptions and definitions, not a disguised restatement of its inputs.

full rationale

The central futility result (Proposition 4.8) is derived from an explicit chain: Assumption 4.1 (ERM learns a Markov blanket), Lemma 4.2 (fair representations depend only on XZ), Lemma 4.3 (ERM fairness iff unbiased training), and Definitions 4.4/4.5 of effectiveness and harmlessness. The derivations of Lemmas 4.6 and 4.7 connect these definitions to distributional bias properties, and the conclusion that train and test distributions must differ is a logical consequence, not an input. The definitions of E and H are intuitive information-theoretic goals; the substantive content is in the lemmas using the Markov blanket assumption. The paper's self-citations (adaptations of Jones et al. 2023/2024/2025) are disclosed, and the cited works are peer-reviewed; the paper provides a proof sketch rather than a bare citation. The weakest point, Assumption 4.1, is an idealization about ERM's representations, but an unproven assumption is a correctness/validity concern, not circularity. The empirical analyses in Sections 3 and 4 are post-hoc measurements and correlations, not predictions from fitted parameters. No load-bearing step reduces to a self-citation or to a definitional restatement of the target claim.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The central claim rests on a causal decomposition of images into disease and sensitive features, an idealization of ERM as learning a Markov blanket, and a set of definitions of effectiveness/harmlessness that the authors choose. These are all assumptions from the authors' own prior framework, not externally benchmarked. There are no fitted free parameters in the theoretical derivation.

assumptions (5)

domain assumption Causal decomposition X = (X_Z, X_A) with Z → X_Z and A → X_A, and unbiasedness defined as Y ⊥ X_A | X_Z (Definition 2.1)
This factorization of images into disease-related and sensitive-attribute features is assumed at the outset and is not derived. It is the framework that makes the later definitions meaningful.
domain assumption Assumption 4.1: Unconstrained representations depend on inputs X* iff they form a Markov blanket over Y at train-time
Load-bearing for Lemmas 4.2-4.3 and the futility result; it presumes ERM learns the full Markov blanket and nothing else.
standard math No Free Lunch / causal underdetermination (Wolpert & Macready 1997; Holland 1986): for any training dataset, opposing causal models are equally compatible with observed data
Used to argue no method can be perfect over all deployment settings; accepted result from the literature.
standard math D-separation criterion applied to the assumed causal diagram yields the three bias mechanisms
Standard graphical causal inference; assumes the diagrams in Fig.1 correctly encode the generative process.
ad hoc to paper Definitions 4.4 and 4.5: Effectiveness and Harmlessness as mutual information equalities
These definitions are chosen by the authors to formalize the goals of FRL; the impossibility result follows from them, so it is a result about these particular definitions.

invented entities (1)

Latent causal factor decomposition X = (X_Z, X_A)
purpose: To define unbiasedness and the three bias mechanisms in Fig.1
This factorization is a modeling assumption imposed on images; no evidence is provided that such clean separation exists in real images. It is the conceptual entity on which the causal analysis rests.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Primer on Causal and Statistical Dataset Biases for Fair and Robust Image Analysis." pith.science (2026). https://pith.science/paper/HW2KBLFI

@misc{pith2026250904295,
  author       = {Pith},
  title        = {Pith review of: A Primer on Causal and Statistical Dataset Biases for Fair and Robust Image Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HW2KBLFI}},
  note         = {Machine review of arXiv:2509.04295}
}

read the original abstract

Machine learning methods often fail when deployed in the real world. Worse still, they fail in high-stakes situations and across socially sensitive lines. These issues have a chilling effect on the adoption of machine learning methods in settings such as medical diagnosis, where they are arguably best-placed to provide benefits if safely deployed. In this primer, we introduce the causal and statistical structures which induce failure in machine learning methods for image analysis. We highlight two previously overlooked problems, which we call the \textit{no fair lunch} problem and the \textit{subgroup separability} problem. We elucidate why today's fair representation learning methods fail to adequately solve them and propose potential paths forward for the field.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages

[1]

Potential Sources of Dataset Bias Complicate Investigation of Underdiagnosis by Machine Learning Algorithms

Bernhardt, M., Jones, C., & Glocker, B. ( 2022). “Potential Sources of Dataset Bias Complicate Investigation of Underdiagnosis by Machine Learning Algorithms”. In: Nature Medicine 28.6, pp. 1157–1158 (cited on p. 3). Chen, R. J. et al. ( 2023). “Algorithmic Fairness in Artificial Intelligence for Medicine and Healthcare”. In: Nature Biomedical Engineering...

arXiv 2022
[28]

Understanding Deep Learning Requires Rethinking General- ization

3, pp. 325–333 (cited on pp. 3, 6). Zhang, C. et al. ( 2016). “Understanding Deep Learning Requires Rethinking General- ization”. In: International Conference on Learning Representations (cited on p. 2). Zhang, H. et al. ( 2022). “Improving the Fairness of Chest X-ray Classifiers”. In: Proceedings of the Conference on Health, Inference, and Learning , pp....

work page 2016

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Potential Sources of Dataset Bias Complicate Investigation of Underdiagnosis by Machine Learning Algorithms

Bernhardt, M., Jones, C., & Glocker, B. ( 2022). “Potential Sources of Dataset Bias Complicate Investigation of Underdiagnosis by Machine Learning Algorithms”. In: Nature Medicine 28.6, pp. 1157–1158 (cited on p. 3). Chen, R. J. et al. ( 2023). “Algorithmic Fairness in Artificial Intelligence for Medicine and Healthcare”. In: Nature Biomedical Engineering...

arXiv 2022

[2] [28]

Understanding Deep Learning Requires Rethinking General- ization

3, pp. 325–333 (cited on pp. 3, 6). Zhang, C. et al. ( 2016). “Understanding Deep Learning Requires Rethinking General- ization”. In: International Conference on Learning Representations (cited on p. 2). Zhang, H. et al. ( 2022). “Improving the Fairness of Chest X-ray Classifiers”. In: Proceedings of the Conference on Health, Inference, and Learning , pp....

work page 2016