Pith. sign in

REVIEW 4 major objections 5 minor 14 references

humancompatible.detect: a Python Toolkit for Detecting Bias in AI Models

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces humancompatible.detect, an open-source Python toolkit that claims to detect bias in AI models by finding the maximally discrepant intersectional subgroup with linear sample complexity and provable guarantees.

desk verdict Useful, readable fairness-toolkit release, but the advertised guarantees are imported from self-cited prior work and the demos skip the uncertainty quantification the paper itself preaches; worth serious review, not a methods breakthrough. read the letter →

arxiv 2509.24340 v2 pith:JRSKB3VT submitted 2025-09-29 cs.AI

German M. Matilla , Jiri Nemecek , Illia Kryvoviaz , Jakub Marecek This is my paper · ORCID
classification cs.AI
keywords biasdetectionintersectionalfairnessmaximumsubgroupdiscrepancysubsampledl-infinitydistancestoolkitsamplecomplexityAIauditmixed-integeroptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

humancompatible.detect is an open-source Python toolkit for bias detection in AI systems, built around two methods: Maximum Subgroup Discrepancy (MSD) and subsampled ℓ∞ distances. MSD searches over all intersections of protected attributes and returns the subgroup with the largest difference in outcome probability between two distributions, expressed as a human-readable conjunctive rule. The paper's central claim is that this search is tractable because the required number of samples grows linearly in the number of protected attributes—not exponentially in the number of subgroups—and that the returned subgroup is provably optimal when found via mixed-integer optimization. The subsampled ℓ∞ test checks a given subgroup against a tolerance with PAC-style guarantees on the estimation error. If these guarantees hold, the toolkit would make intersectional bias auditing feasible in practice, including for compliance with regulations such as the EU AI Act.

What carries the argument

Maximum Subgroup Discrepancy (MSD): the statistical distance sup_S |µ(S) − ν(S)|, computed via mixed-integer optimization, which searches all subgroup intersections and returns a conjunctive rule for the subgroup with the highest probability difference; its linear-in-attributes sample complexity is what makes exhaustive intersectional search tractable. The companion method, subsampled ℓ∞ distances, is a pass/fail test of whether a subgroup's histogram of outcomes differs from the overall population by more than a threshold, with PAC-style bounds on the estimation error when the test is subsampled.

What would settle it

Run the toolkit on a synthetic dataset with a single planted intersectional subgroup of known discrepancy, such as a 0.1 gap for 'Race = Blue AND Age = 0-18,' and check whether MSD returns exactly that subgroup and reports an MSD value matching the true gap within the stated error bound. A second check: for a small set of protected attributes, brute-force enumerate all subgroups and compare the true maximum discrepancy with MSD's output on the same data.

Watch

Extended reading notes

Core claim

The core claim is that intersectional bias can be detected by maximizing a statistical distance over all subgroup intersections rather than measuring each subgroup separately. MSD is defined as sup_{S∈S} |µ(S) − ν(S)|, and the paper reports that its implementation uses mixed-integer optimization to obtain the provably optimal subgroup, with sample complexity linear in the number of protected attributes and error controlled through uncertainty quantification. On the demonstration datasets, MSD recovers rules such as 'Race = Blue AND Age = 0-18' and, on ACS income data, 'Race = White AND Sex = Male,' while the subsampled ℓ∞ test gives a pass/fail verdict against a user-specified tolerance. The

Load-bearing premise

The central claim rests on the correctness of the theoretical guarantees imported from companion papers, and on the mixed-integer optimizer actually certifying optimality on finite real-world datasets; if either fails, the 'provably optimal subgroup' and 'linear sample complexity' claims do not transfer to the toolkit as shipped.

Editorial extensions

If this is right

  • A user can ask for the single most biased intersectional subgroup instead of testing all exponentially many intersections, and the answer is a simple conjunction of attribute-value pairs, e.g., 'AGE between 18-25 AND SEX = Male.'
  • For a given subgroup, the subsampled ℓ∞ test yields a clear pass/fail against a tolerance, with quantified uncertainty, so the result can be used directly in audit documentation.
  • The toolkit supports DataFrames, CSVs, and two-sample comparisons, covering both data-quality checks (training distribution vs. population) and output-bias checks (positive outcomes across groups).
  • If the linear sample-complexity claim holds, audits with many protected attributes remain feasible; at the scale of billions of subgroup intersections, enumeration is impossible, but an optimization-based search is not.
  • Because outputs are human-readable rules, the tool can feed directly into mitigation strategies targeting the discovered subgroup.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The practical bottleneck may shift from data collection to solver time: if the linear sample-complexity claim is correct, the cost of an audit is dominated by the mixed-integer optimization, so benchmarking solver scaling on large attribute sets is the natural next stress test.
  • The returned conjunctive rules could be used to design targeted mitigation—e.g., reweighting or post-processing only the flagged subgroup—and a testable extension would compare that targeted approach against global fairness algorithms in reducing intersectional bias.
  • The two-sample mode invites temporal and geographic fairness monitoring: comparing the same model's outcomes across states or across quarters could detect emerging bias before it becomes entrenched, an extension the paper only hints at in its single cross-state example.
  • A robust deployment could include a verification mode that exposes the mixed-integer optimization gap on each call, so users can distinguish a certified optimum from the best rule found; this is an extension, not a claim in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents humancompatible.detect, an open-source Python toolkit for bias detection in AI models. The toolkit implements two methods: Maximum Subgroup Discrepancy (MSD), which searches for the most biased subgroup over an exponential set of protected-attribute intersections via mixed-integer optimization, and a subsampled l_inf distance test for pass/fail evaluation of a given subgroup against a tolerance. The paper claims that MSD has linear sample complexity in the number of protected attributes and returns the provably optimal (maximally discrepant) subgroup, and that the l_inf test comes with PAC-style guarantees. Demonstrations include a synthetic example where an intersectional subgroup is biased despite marginal fairness, and a real-data experiment on ACS Income data for Florida and New Hampshire, where MSD returns rules such as 'Race = White AND Sex = Male.' The conclusion emphasizes the toolkit's ease of use and guarantees for smaller datasets.

Significance. If the underlying guarantees hold, the toolkit addresses a practically important gap: scalable detection of intersectional bias without enumerating an exponential number of subgroups. The open-source release and reproducible notebooks are strengths, and the two methods are well motivated by real regulatory needs. However, the paper's demonstrations do not verify that the advertised guarantees actually transfer to the implementation. The central claims rest on two self-cited papers whose proofs are not reproduced, and the reported experiments provide only point estimates with no uncertainty quantification, no solver certificates, and no comparison against existing baselines. The paper is therefore a promising software announcement, but the evidence presented is insufficient to substantiate the performance and guarantee claims.

major comments (4)
  1. [Section 3, MSD paragraph] The claim that MSD 'obtains the provably optimal solution' is load-bearing, but the paper does not explain how a user or reader can verify that the optimizer actually proved global optimality. The real-data demonstration in Section 4.2 reports only MSD values (0.193, 0.217, 0.226) with no MIP gap, solver status, or time limit. If the mixed-integer solver stops at a time limit, the returned subgroup is not certified as maximally discrepant. Please either report solver certificates in the demonstrations or state explicitly in the paper how the implementation certifies optimality and how the user can access that certificate.
  2. [Section 2.4 and Section 4] Section 2.4 correctly states that 'any of the estimates of bias is meaningless without knowing the uncertainty in the estimate,' and Section 3 says the sample complexity can be controlled by controlling the error. Yet the demonstrations in Sections 4.1 and 4.2 report only point estimates (e.g., 'MSD = 0.193' and a bare 'fail' for the l_inf test) with no confidence intervals, PAC bounds, or sampling error. This directly contradicts the paper's own UQ principle and undermines the claimed advantage of 'bias evaluations with guarantees' in Section 5. Add uncertainty quantification to the demonstrated outputs, or clarify that the tool does not currently provide it.
  3. [Section 3, subsampled l_inf test] The description of the subsampled l_inf test is too vague to support the guarantee claim. The paper only says that M. Matilla and Marecek (2025) 'developed guaranties on the size of the error with a certain probability,' without specifying how the implementation chooses the subsample size, how the tolerance delta relates to the PAC bound, or what confidence level is provided. The example in Section 4.1 reports a pass/fail outcome for delta=0.125 with no mention of sample size or confidence. Since this test is one of the toolkit's two core methods, the paper should state the implemented bound and demonstrate it concretely.
  4. [Section 4, Demonstration] The demonstrations do not substantiate the abstract's claims that the toolkit addresses MMD scalability and Wasserstein-1 computability issues. No runtime measurements, sample-complexity comparisons, or baseline comparisons (e.g., against MMD, Wasserstein, or simple subgroup scanning) are provided. A toolkit paper whose main selling point is 'addresses these challenges' should include at least a small benchmark showing that the implementation scales and is more sample-efficient than standard alternatives. Without this, the performance claims are unsupported.
minor comments (5)
  1. [Section 2.1] Typo: 'estimate data quality qualtiy' should be 'estimate data quality.'
  2. [Section 2.2] Typo: 'fourty million' should be 'forty million.'
  3. [Section 3 and passim] The word 'guaranties' is used repeatedly; it should be 'guarantees.' Also, the l_inf symbol is typeset as 'l inf' in the text; please use proper math notation.
  4. [Section 4.1] The phrase 'For a police threshold delta = 0.125' should likely be 'For a prespecified threshold delta = 0.125.' Also, the ellipsis in 'MSD value of 0.111. . .' is a typo.
  5. [General] The figure captions refer to modifications of figures from Nemecek et al. (2025). If the figures are reproduced, the paper should indicate whether permission was obtained where required.

Circularity Check

1 steps flagged · score 3.0 of 10

Partial circularity: the toolkit's headline guarantees (provably optimal MSD, controllable sample complexity) are imported from two self-cited prior works and are not verified in this implementation; the toolkit itself is a distinct artifact with illustrative demonstrations.

  1. self citation load bearing [Section 3, 'Solution' (Maximum Subgroup Discrepancy; Subsampled Distances on Measure Spaces)]
    "The MSD algorithm has linear sample complexity in the number of protected attributes, returns naturally defined subgroups (as conjunctions of feature value pairs), and the group that is guaranteed to have the highest discrepancy for practical dataset sizes with many protected attributes. Using mixed-integer optimization, it obtains the provably optimal solution (i.e. the maximally discrepant subgroup). ... M. Matilla and Marecek (2025) have developed guaranties on the size of the error with a certain probability."

    The paper advertises 'provably optimal' MSD and controllable sample complexity as the toolkit's central advantages, but neither guarantee is derived or demonstrated in this manuscript. Both are attributed to two prior works by the same authors (Nemecek et al., 2025; Matilla and Marecek, 2025). The demonstrations report only point estimates (e.g., MSD = 0.193) with no solver certificate, MIP gap, or uncertainty interval, so nothing in this paper tests whether the cited theorems' assumptions hold for the released code. The load-bearing argument for 'guaranteed' bias detection therefore reduces to an unverified self-citation chain. This is not a full definitional equivalence, and the toolkit has independent content, so circularity is partial.

full rationale

This is a software/toolkit paper rather than a new theoretical derivation. MSD is defined directly as a supremum over subgroups, and the subsampled l-infinity test is described operationally. The challenge is that the claims that make the toolkit distinctive — linear sample complexity, provable optimality of the returned subgroup, and controllable error — are not re-derived here. They are cited from Nemecek et al. (2025) and Matilla and Marecek (2025), both involving the present authors. The cited KDD paper is peer-reviewed and the arXiv preprint may contain proofs, so the self-citation is not automatically circular. However, the current paper never connects the assumptions of those proofs to the actual implementation: no solver status, no MIP gap, no confidence interval, and no uncertainty quantification are reported, even though Section 2.4 states that bias estimates are meaningless without uncertainty. Thus, as presented, the guarantee chain reduces to an unverified self-citation. The tool's demonstrations on synthetic data and ACS Income do provide some independent evidence that the implementation runs and finds plausible subgroups, which keeps the circularity partial rather than total. There is no fitted parameter renamed as a prediction, no known result merely relabeled, and no ansatz smuggled in via citation. The score of 3 reflects the load-bearing but not fully definitional nature of the self-citation dependency.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; the tolerance δ=0.125 and the protected attribute choices are user inputs, not fitted values. The central claims rest on three unproven-in-this-paper assumptions, two of which point directly at self-cited theoretical papers.

assumptions (3)
  • domain assumption MSD has linear sample complexity in the number of protected attributes and an exact maximum over subgroups can be obtained via mixed-integer optimization.
    Invoked in Section 3 without proof; authority is the self-cited Nemecek et al. (2025).
  • domain assumption Subsampled ℓ∞ distance test has PAC-style error guarantees with controllable sample complexity.
    Invoked in Section 3; authority is the self-cited M. Matilla and Marecek (2025).
  • domain assumption The synthetic and ACS datasets adequately represent real deployment settings for fairness auditing.
    Demonstrations in Section 4 assume these two data sources exercise the toolkit's guarantees; no formal justification is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of humancompatible.detect: a Python Toolkit for Detecting Bias in AI Models." pith.science (2026). https://pith.science/paper/JRSKB3VT

@misc{pith2026250924340,
  author       = {Pith},
  title        = {Pith review of: humancompatible.detect: a Python Toolkit for Detecting Bias in AI Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JRSKB3VT}},
  note         = {Machine review of arXiv:2509.24340}
}
abstract

There is a strong recent emphasis on trustworthy AI. In particular, international regulations, such as the AI Act, demand that AI practitioners measure data quality on the input and estimate bias on the output of high-risk AI systems. However, there are many challenges involved, including scalability (MMD) and computability (Wasserstein-1) issues of traditional methods for estimating distances on measure spaces. Here, we present humancompatible.detect, a toolkit for bias detection that addresses these challenges. It incorporates two newly developed methods to detect and evaluate bias: maximum subgroup discrepancy (MSD) and subsampled $\ell_\infty$ distances. It has an easy-to-use API documented with multiple examples. humancompatible.detect is licensed under the Apache License, Version 2.0.

Figures

Figures reproduced from arXiv: 2509.24340 by the authors.

Figure 1
Figure 1. An illustration of why evaluating marginal bias does not suffice. A classifier [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. humancompatible.detect Workflow for bias evaluation Subsampled Distances on Measure Spaces Our second approach addresses the same issue regarding protected group intersections in a different manner. In particular, we sub￾sample a test whether the distance between the two distributions is less than a given value (∆). For a given subgroup, we can check whether the given subgroup faces a bias within the threshold ∆. In… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 1 canonical work pages

  1. [3]

    Julia Angwin, Jeff Larson, Lauren Kirchner, and Surya Mattu

    doi: 10.1109/IEEESTD.2025.10851955. Julia Angwin, Jeff Larson, Lauren Kirchner, and Surya Mattu. Ma- chine Bias, May

    arXiv 2025

  2. [6]

    Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan

    URLhttps://arxiv.org/abs/2502.02623v1. Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A Survey on Bias and Fairness in Machine Learning.ACM Comput. Surv., 54(6):115:1– 115:35, July

    arXiv

  3. [8]

    ISBN 979-8-4007-1454-2

    ACM. ISBN 979-8-4007-1454-2. doi: 10.1145/3711896.3736857. URLhttps://dl.acm.org/doi/10.1145/3711896.3736857. Evaggelia Pitoura, Kostas Stefanidis, and Georgia Koutrika. Fairness in rankings and recommenders: Models, methods and research directions. In2021 IEEE 37th In- ternational Conference on Data Engineering (ICDE), pages 2358–2361,

    arXiv

  4. [9]

    Regulation (EU) 2024/1689

    doi: 10.1109/ICDE51399.2021.00265. Regulation (EU) 2024/1689. Regulation (EU) 2024/1689 of the European Parliament and of the Council. of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90...

    arXiv 2021

  5. [10]

    Dimitris Sacharidis, Giorgos Giannopoulos, George Papastefanatos, and Kostas Stefanidis

    Place: OJ L, 2024/1689, 12.7.2024. Dimitris Sacharidis, Giorgos Giannopoulos, George Papastefanatos, and Kostas Stefanidis. Auditing for spatial fairness,

    2024

  6. [13]

    ISBN 979-8-4007-0330-0

    Association for Computing Machinery. ISBN 979-8-4007-0330-0. doi: 10.1145/3613904.3642761. URLhttps://dl.acm.org/doi/10.1145/3613904.3642761. Leonid Nisonovich Vaserstein. Markov processes over denumerable products of spaces, describing large systems of automata.Problemy Peredachi Informatsii, 5(3):64–72,

    arXiv

  7. [14]

    9 Matilla, Nˇemeˇcek, Kryvoviaz, and Mareˇcek Quan Zhou, Ramen Ghosh, Robert Shorten, and Jakub Mareˇ cek

    doi: 10.1613/jair.1.14050. 9 Matilla, Nˇemeˇcek, Kryvoviaz, and Mareˇcek Quan Zhou, Ramen Ghosh, Robert Shorten, and Jakub Mareˇ cek. Closed-loop view of the regulation of ai: Equal impact across repeated interactions. In2024 IEEE 40th Inter- national Conference on Data Engineering Workshops (ICDEW), page 176–181. IEEE, May

    work page doi:10.1613/jair.1.14050

  8. [15]

    URLhttp://dx.doi.org/10.1109/ ICDEW61823.2024.00029

    doi: 10.1109/icdew61823.2024.00029. URLhttp://dx.doi.org/10.1109/ ICDEW61823.2024.00029. 10

    arXiv 2024

Show all 14 references
  1. [2012]

    IEEE standard for algorithmic bias considerations.IEEE Std 7003-2024, pages 1–59,

    IEEE Standards Association. IEEE standard for algorithmic bias considerations.IEEE Std 7003-2024, pages 1–59,

    2024

  2. [2016]

    Andrii Kliachkin, Eleni Psaroudaki, Jakub Marecek, and Dimitris Fotakis

    URLhttps://www.propublica.org/article/ machine-bias-risk-assessments-in-criminal-sentencing. Andrii Kliachkin, Eleni Psaroudaki, Jakub Marecek, and Dimitris Fotakis. Fairness in rank- ing: Robustness through randomization without the protected attribute. In40th Interna- tional...

    2024

  3. [2021]

    doi: 10.1145/3457607

    ISSN 0360-0300. doi: 10.1145/3457607. URLhttps://dl.acm.org/ doi/10.1145/3457607. Jiˇ r ´ ı Nˇ emeˇ cek, Mark Kozdoba, Illia Kryvoviaz, Tom´ aˇ s Pevn´ y, and Jakub Mareˇ cek. Bias Detection via Maximum Subgroup Discrepancy. InProceedings of the 31st ACM SIGKDD Conference on K...

    doi

  4. [2022]

    Ashish Sharma, Kevin Rushton, Inna Wanyin Lin, Theresa Nguyen, and Tim Althoff

    URLhttps://tsapps.nist.gov/publication/get_pdf.cfm? pub_id=934464. Ashish Sharma, Kevin Rushton, Inna Wanyin Lin, Theresa Nguyen, and Tim Althoff. Fa- cilitating Self-Guided Mental Health Interventions Through Human-Language Model In- teraction: A Case Study of Cognitive Restr...

    2024

  5. [2023]

    Reva Schwartz, Apostol Vassilev, Kristen K

    URLhttps://arxiv.org/abs/2302.12333. Reva Schwartz, Apostol Vassilev, Kristen K. Greene, Lori Perine, Andrew Burt, and Patrick Hall. Towards a Standard for Identifying and Managing Bias in Artificial In- telligence, March

    arXiv

  6. [2025]

    Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Sch¨ olkopf, and Alexander Smola

    arXiv preprint arXiv:2506.08616. Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Sch¨ olkopf, and Alexander Smola. A kernel two-sample test.The Journal of Machine Learning Research, 13(1):723– 773,

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.