REVIEW 3 major objections 6 minor 3 references
Falsifying Discriminant Validity of Predictive Algorithms
T0 review · 3 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read The paper claims that a statistical comparison of calibrated prediction losses can falsify whether an algorithm predicts an impermissible proxy like race at least as well as its intended outcomes; in the LSAC admissions model it flags race,
desk verdict The framing is useful but the calibrated-loss test is confounded by base rates; the LSAC race finding is a base-rate artifact rather than evidence of predicting race. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the pair of definitions loss discriminance (the algorithm's loss on the impermissible proxy exceeds its loss on all permissible proxies) and loss indiscriminance (loss on the impermissible proxy is no larger for at least one permissible proxy). The procedure rests on separate Platt scaling of the algorithm's predictions for each outcome — the paper argues this calibration is crucial so that losses are compared on a common calibrated scale — followed by nonparametric tests: a one-sided t-test or Wilcoxon signed-rank test on paired loss differences for a single permissible proxy, and, for multiple proxies, a conditional rank test whose statistic is the mean within-samp
What would settle it
Take the LSAC model, randomly permute its predictions (or replace them with a constant), and run Algorithm 1 with race as the impermissible proxy and GPA as permissible. If the test returns INDISCRIMINANT with race ranked first for roughly 93% of observations, the flag is driven by the white/93% base rate rather than by the model encoding race. Equivalently, compute the calibrated log loss of the constant majority-class predictor for each outcome: if the test declares race 'better predicted' than GPA solely from these floor losses, the comparison scale is base-rate-determined.
Extended reading notes
Core claim
The paper claims that discriminant validity of a predictive algorithm can be operationalized as a falsification procedure: compare per-outcome Platt-calibrated prediction losses and test whether the impermissible proxy's loss is higher than every permissible proxy's loss (loss discriminance). When the test fails to reject, the result is labelled INDISCRIMINANT — an inconclusive safety flag that should prompt re-examination of problem formulation, not model tuning. In the LSAC admissions analysis the model passes with respect to gender (p≈0) and fails with respect to race (p≈1, with race best-predicted for 91% of applicants); in COMPAS the decile scores fail with respect to age (p≈1) and give
Load-bearing premise
The test assumes that after Platt scaling, losses for outcomes with very different base rates are directly comparable, so that a lower calibrated loss genuinely means better prediction; for heavily skewed outcomes the loss floor itself is much lower, which can make an uninformative model appear to 'predict' the skewed impermissible proxy well.
Editorial extensions
If this is right
- An INDISCRIMINANT result is an actionable safety flag: developers should revisit problem formulation, proxy choice, and task suitability rather than tune the model further.
- The method works without access to the true target outcome, using only permissible proxy outcomes and the measured impermissible variable, and makes minimal distributional assumptions.
- When several impermissible proxies are evaluated, standard multiple-testing corrections (Bonferroni, Holm) apply; in COMPAS the race result is borderline once corrected.
- Standard performance metrics like AUC can be ambiguous about discriminant validity (in COMPAS, AUC for re-arrest and age are nearly equal); the loss-comparison test resolves that ambiguity statistically.
- Passing the test is necessary but not sufficient for validity: convergent and external validity still require complementary analysis.
Reading between the lines
- The calibration step may make the test sensitive to outcome base rates: for a Bernoulli outcome the achievable loss floor is its entropy, so a proxy with a 93% majority class (like 'white' in LSAC) can show artificially low calibrated loss even for an uninformative predictor. The race finding may partly reflect this base-rate asymmetry rather than the model genuinely encoding race; the appendix's
- A concrete testable extension would be to run the same procedure on a null model (e.g., predictions permuted or constant majority-class predictions). If the null model also fails discriminant validity for skewed impermissible proxies, a base-rate-normalized loss, or a calibration that maps scores to a common reference distribution, would be needed before the test can support causal claims about wh
- The framework maps naturally onto negative controls in causal inference, with the impermissible proxy playing the role of a negative control outcome; combining it with selection-bias adjustments (as needed in COMPAS, where only released defendants are observed) would strengthen its external-validity claims.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a falsification framework for assessing the discriminant validity of predictive algorithms. The claim is that by comparing Platt-calibrated prediction losses on permissible proxies (e.g., GPA) against losses on an impermissible proxy (e.g., race, gender), a nonparametric hypothesis test can determine whether the algorithm predicts the intended outcome better than the impermissible one. A single-proxy version uses a one-sided t-test or Wilcoxon signed-rank test on paired loss differences, while a multiple-proxy version uses a conditional rank/permutation test. The method is illustrated on LSAC admissions data (race fails the test, gender passes) and on COMPAS recidivism data (age fails), with additional negative-control and ablation analyses in the appendix.
Significance. If the proposed test were valid, it would provide a concrete, operational link between psychometric discriminant validity and algorithmic evaluation, which is a valuable and timely contribution. The paper usefully synthesizes placebo tests, negative controls, and discriminant-validity practices from other fields, and its explicit discussion of the inverse fallacy and the inconclusiveness of failing to reject is a strength. The negative control in Appendix B is good empirical practice. However, the central methodological claim — that separate Platt scaling makes calibrated losses comparable across outcomes with different marginal distributions — is incorrect, and the headline race finding is an artifact of base-rate entropy. This invalidates the core contribution as presented.
major comments (3)
- [§5.1, Algorithm 1, Table 3/A.1] The claim that separate Platt scaling 'ensures that the loss comparisons between different proxies are made on a common, calibrated scale' is incorrect. For a binary outcome with base rate p, the minimum achievable expected log loss is the entropy H(p), attained by a perfectly calibrated constant predictor. After Platt scaling, a model with zero information about race outputs ~0.93 probabilities, giving log loss ≈ H(0.93) ≈ 0.25 nats, whereas a balanced GPA proxy has a floor of H(0.5) ≈ 0.69 nats. The race loss is therefore lower for nearly every observation regardless of whether the model encodes any race information. This is exactly what Table 3 shows: the race result flips from mean difference −0.03 (DISCRIMINANT) to +0.50 (INDISCRIMINANT) solely because calibration is applied. The appendix dismisses the uncalibrated comparison as an artifact, but the calibrated comparison is equally
- [§6.1.2, Fig. 1 (right)] The headline finding — race is best-predicted (rank 1) for 91% of observations — is reproduced by a null model with no race information. Under Platt scaling, the calibrated constant predictor for race achieves loss H(0.93) for every row, while the GPA proxies have floors around H(0.5); thus race attains rank 1 for roughly 93% of rows. The p≈1 result is therefore not evidence that the algorithm 'predicts race better than permissible outcomes'; it is a base-rate artifact. The test as implemented cannot distinguish 'impermissible predicted better' from 'impermissible outcome has a lower entropy floor,' and the paper's interpretation of this p-value is not supported.
- [§5.2 and Definition 4.2] The conditional rank test's null hypothesis of loss exchangeability is not aligned with the paper's notion of loss indiscriminance. Definition 4.2 declares indiscriminance when l(Ŷ, Ỹ) ≤ l(Ŷ, Yj) for some j. But with log loss, that inequality is satisfied by a constant predictor whenever H(p̃) ≤ H(p_j), independent of any property of the algorithm. The exchangeability assumption used in Algorithm 2 therefore does not correspond to 'the algorithm fails to predict the impermissible outcome'; it corresponds to equality of the complete loss distributions, which is false at baseline whenever outcome marginals differ. The permutation p-values are not calibrated for the intended null.
minor comments (6)
- [Terminology] Inconsistent terminology: Definition 4.2 uses INDISCRIMINANT, while Algorithm 1's Ensure and step 17 use INDISCRIMINATE (inconclusive). Please harmonize.
- [Appendix A.1] Appendix A.1 states the race base rate is 94%, while the main text §6.1 states 93%. Please reconcile the discrepancy.
- [Abstract] The phrase 'the algorithm is predicting another outcome than what was intended' is ungrammatical; suggest 'an outcome other than the intended one.'
- [Figure 1] The caption says the dashed line shows the expected 25% under the null, but the null expected rank for M=3 is 2.5, not a 25% line. Please clarify what the dashed line represents.
- [Title] The full-text title is 'Falsifying Predictive Algorithms' while the arXiv title is 'Falsifying Discriminant Validity of Predictive Algorithms'; ensure consistency.
- [Tables 3 and 4] P-values are reported as ≈1 and ≈0. Reporting exact values or confidence intervals would be more informative.
Circularity Check
No load-bearing circularity: the falsification test is held-out and self-contained; the base-rate critique is a validity concern, not a circular reduction.
full rationale
The paper's derivation chain is a standard statistical testing construction: Def. 4.1 defines loss discriminance via loss comparisons; Def. 4.3 defines a falsification procedure as control of the probability of returning INDISCRIMINANT under indiscriminance; Alg. 1/2 instantiate this with paired t/Wilcoxon and conditional-rank/permutation tests. The only near-tautological element is Def. 4.3, which restates Type I error control; this is a definition rather than a claim derived from the target, so it does not constitute a circular step. The empirical results in §6 are held-out applications: the model is trained on a training split, Platt scaling is fit on a calibration split, and losses are computed on an evaluation split (§6.1), so the race/gender findings are not fitted parameters renamed as predictions. The self-citations (Coston et al. 2023; Guerdan et al. 2023; Coston et al. 2020; Rambachan et al. 2022) motivate framing and limitations but are not load-bearing: no uniqueness theorem or ansatz is imported from them. The serious base-rate/entropy problem with comparing calibrated losses across outcomes with different marginal distributions is a substantive validity threat, not circularity; the paper itself notes in Appendix A.1 that the raw result without calibration is 'an artifact of comparing losses computed from uncalibrated predictions against labels with different base rates (race: 94%, GPA: 50%)', and it flags that INDISCRIMINANT is inconclusive (§5.1, §6.2.1) and that the COMPAS sample has external-validity limits (§6.2). Those acknowledgments support an honest non-finding on circularity.
Assumptions & free parameters
free parameters (3)
- Platt scaling coefficients A_y, B_y per outcome =
not reported
- Proxy binarization thresholds =
race = white vs non-white; age < 25; high income = category 5; GPA above median
- Loss function =
log loss vs Brier score
assumptions (4)
- domain assumption Permissible proxies are valid measures of the intended construct
- domain assumption Platt scaling produces well-calibrated probabilities for each outcome
- ad hoc to paper Losses are comparable across outcomes after calibration
- standard math Standard paired-test and permutation-test validity conditions
Cite this review
Pith. "Pith review of Falsifying Discriminant Validity of Predictive Algorithms." pith.science (2026). https://pith.science/paper/TKQYEAVU
@misc{pith2026260117146,
author = {Pith},
title = {Pith review of: Falsifying Discriminant Validity of Predictive Algorithms},
year = {2026},
howpublished = {\url{https://pith.science/paper/TKQYEAVU}},
note = {Machine review of arXiv:2601.17146}
}
read the original abstract
Empirical investigations into unintended model behavior often show that the algorithm is predicting another outcome than what was intended. These expos\'es highlight the need to identify when algorithms predict unintended quantities - ideally before deploying them into consequential settings. We propose a falsification framework that provides a principled statistical test for discriminant validity: the requirement that an algorithm predict intended outcomes better than impermissible ones. Drawing on falsification practices from causal inference, econometrics, and psychometrics, our framework compares calibrated prediction losses across outcomes to assess whether the algorithm exhibits discriminant validity with respect to a specified impermissible proxy. In settings where the target outcome is difficult to observe, multiple permissible proxy outcomes may be available; our framework accommodates both this setting and the case with a single permissible proxy. Throughout we use nonparametric hypothesis testing methods that make minimal assumptions on the data-generating process. We illustrate the method in an admissions setting, where the framework establishes discriminant validity with respect to gender but fails to establish discriminant validity with respect to race. This demonstrates how falsification can serve as an early validity check. We also provide analysis in a criminal justice setting, where we highlight the limitations of our framework and emphasize the need for complementary approaches to assess other aspects of construct validity and external validity.
Figures
Reference graph
Works this paper leans on
-
[1991]
doi: 10.1007/BF00960446. Sally Ho and Garance Burke. Oregon dropping AI tool used in child abuse cases. Associated Press, June 2 2022. URLhttps://apnews.com/article/ politics-technology-pennsylvania-child-abuse-1ea160dc5c2c203fdab456e3c2d97930. Accessed: 2026-01-08. Sally Ho and Garance Burke. Justice department scrutinizes Pittsburgh child welfare AI too...
arXiv 2022
-
[2020]
doi: 10.1056/NEJMms2004740. Angelina Wang, Sayash Kapoor, Solon Barocas, and Arvind Narayanan. Against predictive op- timization: On the legitimacy of decision-making algorithms that optimize predictive accuracy. ACM Journal on Responsible Computing, 1(1):1–45, 2024. Jamelle Watson-Daniels, Solon Barocas, Jake M Hofman, and Alexandra Chouldechova. Multi- ...
-
[2021]
Vic Barnett, Toby Lewis, et al.Outliers in statistical data, volume 3
URLhttps://arxiv.org/abs/2106.05498. Vic Barnett, Toby Lewis, et al.Outliers in statistical data, volume 3. Wiley New York, 1994. Solon Barocas and Andrew D Selbst. Big data’s disparate impact.Calif. L. Rev., 104:671, 2016. Solon Barocas, Moritz Hardt, and Arvind Narayanan.Fairness and Machine Learning: Limitations and Opportunities. MIT Press, Cambridge,...
arXiv 1994
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.