Pith. sign in

REVIEW 5 major objections 5 minor 74 references

Fairness Testing through Extreme Value Theory

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that worst-case AI discrimination can be measured by fitting extreme value distributions to counterfactual outcomes, and that doing so reveals blind spots in average-based fairness evaluation and mitigation.

desk verdict The ECD metric is not comparable across groups as defined, so the paper's central empirical claims rest on a threshold artifact, though the underlying idea is worth a serious revision. read the letter →

arxiv 2501.11597 v1 pith:RFXKVL7A submitted 2025-01-20 cs.SE cs.AIcs.CYcs.LG

Verya Monjezi , Ashutosh Trivedi , Vladik Kreinovich , Saeid Tizpaz-Niari This is my paper · ORCID
classification cs.SEcs.AIcs.CYcs.LG
keywords extremevaluetheorycounterfactualdiscriminationfairnesstestingworst-casebiasmitigationgenerativeAIstatisticalreturnlevels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Average-case fairness metrics can create a false sense of safety, because discrimination that affects only the extreme tail of outcomes stays invisible when the mean looks fair. The paper proposes extreme counterfactual discrimination (ECD), defined as the difference $\mu_u - \mu_p$ between the location parameters of generalized extreme value (GEV) distributions fitted to the upper tail of each group's counterfactual discrimination scores, and claims it quantifies the worst-case disadvantage attributable to group membership. Across four classifiers and nine datasets, the paper reports that the GEV fit is statistically valid in 95% of scenarios, that ECD finds significant tail discrimination in 48% of cases while average metrics find only 25%, and that average-based bias mitigators increase worst-case discrimination in 35% of cases. It also claims that an ECD-guided mitigation strategy improves tail fairness in 90% of cases without degrading average fairness, which would make tail awareness a practical complement to standard fairness tooling.

What carries the argument

The load-bearing machinery is the generalized extreme value (GEV) distribution fitted to the upper tail of counterfactual discrimination scores via the threshold approach, where the location parameter $\mu$ of each group's fitted GEV is interpreted as the expected extreme discrimination for that group and $\mu_u - \mu_p$ defines ECD. To gather enough extreme samples, the paper runs a randomized test-case generation loop that measures counterfactual discrimination, applies an exponentiality test based on the coefficient of variation to a growing number of top-$k$ samples to certify that the tail is exponential or light-tailed, and augments under-represented groups with synthetic samples drawn from a learned generative model. The fitted GEV then supplies return levels, i.e., the expected worst-case discrimination over $m$ future interactions, which is how the paper translates the tail fit into an actionable fairness guarantee.

What would settle it

Recompute ECD for the paper's 40 model-task scenarios using a common threshold rule and sample-size-corrected location estimates, and check whether the reported finding that average-based mitigators increase worst-case discrimination in 35% of cases still holds; if the effect reverses under this re-analysis, the central empirical claim is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the worst-case counterfactual discrimination of a model can be estimated and compared across protected groups by fitting a generalized extreme value (GEV) distribution to the upper tail of the counterfactual discrimination values for each group and taking the difference of the location parameters, $\mathrm{ECD} = \mu_u - \mu_p$. When the fitted shape parameter is zero or negative, the paper treats the fit as valid for extrapolation, yielding return levels that state the expected maximum discrimination within $m$ future interactions with the model. Empirically, the claim is that this procedure works in 95% of the 80 (model, dataset, group) scenarios, that tail discrimination appears in 48% of the 40 task-scenarios whereas average discrimination appears in 25%, and that ECD catches tail discrimination that the conditional value at risk measure misses in 18 of 40 experiments. The paper further claims that average-based bias mitigators increase worst-case discrimination in 35% of cases, that the tail-aware MiniMax-Fairness method increases it in 30% of cases, and that a new ECD-optimizing in-process mitigator improves tail fairness in 90% of its 20 DNN and logistic regression configurations while preserving average fairness.

Load-bearing premise

The load-bearing premise is that the GEV location parameters $\mu_u$ and $\mu_p$ can be compared directly across groups even though each group is fitted at its own threshold and from a very different number of tail samples; if the apparent location shift is an artifact of those fitting choices, the ECD metric and the empirical claims built on it do not follow.

Editorial extensions

If this is right

  • Fairness evaluation should report a tail metric such as ECD alongside average metrics; the paper's data suggest that average-only evaluation misses substantial tail discrimination in about a third of scenarios.
  • Bias mitigators that optimize average fairness cannot be assumed to help worst-case outcomes; the paper reports that average-based mitigators increase ECD in 35% of its scenarios.
  • A mitigation objective that directly minimizes ECD, with accuracy as a constraint, can reduce tail discrimination without worsening average fairness, according to the paper's results across 20 DNN and logistic regression configurations.
  • EVT-based return levels turn worst-case fairness into a concrete operational statement, such as the maximum expected discrimination within the next $m$ queries to a deployed model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One natural extension the paper leaves implicit is to check whether ECD rankings are stable when a single shared threshold and tail size rule is applied to both groups, since the paper fits each group at its own threshold and with its own sample count.
  • The same extreme-value machinery could be adapted to continuous protected attributes and regression outcomes, where counterfactual differences are not binary flips and tail events must be redefined.
  • A direct validation experiment would compare ECD's extrapolated return levels against the maximum counterfactual discrimination observed in a large held-out sample, which the paper itself notes as a lower-bound ground truth.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a fairness criterion called extreme counterfactual discrimination (ECD), defined as the difference between the GEV location parameters mu_u - mu_p fitted to the upper tail of each protected group's counterfactual discrimination (CD) distribution, and argues that this quantifies worst-case discrimination. The authors present a randomized test-case generation algorithm that combines generative AI (CTGAN, TVAE) with a coefficient-of-variation based exponentiality test, and report experiments over 9 datasets and 4 model classes. Their headline claims are that EVT fits the tail distribution in 95% of cases, that average-based bias mitigators increase worst-case discrimination in 35% of cases, that MiniMax-Fairness increases it in 30% of cases, and that their proposed ECD-Fair mitigator improves tail fairness in 90% of cases without degrading average-case fairness.

Significance. The paper addresses a real and underexplored problem: fairness at the extreme tail of outcome distributions, rather than only on average. If the ECD metric were valid, the paper would be a useful addition to fairness testing, providing both an evaluation lens and a mitigation objective. The authors ship a replication package, use standard EVT software, and compare against CVaR and Minimax-Fairness, which are relevant baselines. However, as presented, the metric's definition and fitting procedure make the central quantitative claims (the 35%, 30%, and 90% figures, and the 95% EVT fit rate) not supportable without substantial revision.

major comments (5)
  1. [Definition III.2 and Section IV, 'Inferring the tail distributions of counterfactual discrimination'] The ECD metric is defined as mu_u - mu_p, where mu is the GEV location parameter fitted to each group's threshold exceedances. Because the threshold tau is chosen separately per group so that only k_max = 50 samples exceed it (Section II; Table IV), and the group sample sizes differ by up to two orders of magnitude (#N ranges from roughly 0.1k to 25.7k in Table IV), the same exceedance count corresponds to very different tail quantiles (e.g., roughly the 0.2% vs 1.6% extremes for the white vs black groups in the running example). The location parameter in a threshold-exceedance model depends on the threshold itself, so ECD = mu_u - mu_p conflates threshold and sample-size differences with genuine tail differences. The paper never standardizes thresholds or compares return levels at a common exceedance probability; therefore the headline empirical findings (RQ2, RQ3, RQ4) are not interpretable as stated.
  2. [Algorithm 1 and Section V-B (Answer RQ2)] The claim that EVT fits in 95% of cases is partly built into the method: Algorithm 1 regenerates samples until the coefficient-of-variation test passes or a timeout occurs, and cases that do not pass are later reported as 'NV' in Tables V and VI. The CV test selects for exponential or light tails, so the 95% figure measures the success of the sampling loop under a permissive selection criterion rather than an independent validation of the GEV fit. The paper should report the number of timeouts, the number of NV exclusions, and evaluate the fitted distributions on held-out tail data (e.g., Q-Q plots and goodness-of-fit tests for all scenarios, not only the selected ones).
  3. [Section IV and Table IV] The statistical model is ambiguous. The text states that 'we use the threshold approach where extreme events that exceed some high threshold u are extreme values' and then says 'we fit the GEV distribution,' but the threshold approach for exceedances yields a generalized Pareto distribution, not a GEV. Moreover, the reported 'location' mu values (e.g., 0.15 and 0.28 for white and black in Section II) are larger than the thresholds tau (0.12 and 0.20), which is not the usual parameterization of a threshold-exceedance model. The authors must specify exactly which distribution is fitted, to what data (block maxima of exceedances or all exceedances), and how the location parameter is defined in that fit.
  4. [Section V-B, paragraph classifying ECD differences] The statement 'We classify any ECD difference exceeding 0.05 as discrimination' is used to derive the 48% detection rate and the comparisons in RQ2-RQ4, but no justification, statistical test, or confidence interval is provided for this 0.05 cutoff. The abstract and the RQ answers repeatedly use the word 'significantly' without a corresponding hypothesis test. The paper should report the uncertainty in ECD (e.g., bootstrap or profile-likelihood intervals) and state how the 0.05 threshold was chosen.
  5. [Table IV] Several fitted shape parameters are degenerate or implausible: for example, Credit male (DNN) has xi = -141.17, Students male (DNN) has xi = -347.33, and several LR/SVM rows report xi near -1 with mu near 0 and sigma near 0 (e.g., Census White LR). For a bounded outcome (CD is in [-1,1]), such extreme negative shape values imply the fitted distribution has an upper endpoint essentially at the observed maximum, and the reported standard errors of 0.00 (epsilon) are not credible. These cases are nonetheless included in the ECD comparisons; the paper should either exclude or specially treat degenerate fits and report the sensitivity of the RQ3/RQ4 results to their inclusion.
minor comments (5)
  1. [Abstract vs. Section V-C] The abstract says average-based mitigators 'increase the worst-case discrimination significantly in 5% of cases,' while Answer RQ3 and the introduction state 35%; the correct number should be reconciled.
  2. [Answer RQ1] The text 'TA VE' in the answer to RQ1 is a typo and should read 'TVAE.'
  3. [Section IV, 'Inferring the tail distributions of counterfactual discrimination'] The phrase 'we initially set the threshold of extreme values to Mk max' is unclear; it should say that the threshold is set to the k_max-th largest observed CD value.
  4. [Table II] The table lists protected attributes for each dataset, but the running example in Section II uses Adult with Race, while the table lists both Sex and Race for Adult; the text should clarify which protected attribute is used in each reported experiment.
  5. [Figures 1 and 2] The figures would benefit from a legend or explicit color labels for the threshold lines and group densities; the captions currently refer to 'red lines' and 'left/mid-left' without a visual legend.

Circularity Check

1 steps flagged · score 5.0 of 10

The 95% EVT-fit claim is partly self-fulfilling: the same CV/exponentiality test both decides when to stop generating tail samples and certifies the fit; ECD itself is not definitionally circular.

  1. fitted input called prediction [Section IV (Approach), 'Collecting tail samples with statistical guarantees'; Section V-B, Answer RQ2]
    "we perform the exponential test, adopted from [16], [17] on the current samples D* ... If for all values of k ∈ [k min, kmax], the CV is less than (1.0+ 1/(4∗k)), then we are statistically confident that we have enough samples from the tail to infer valid extreme value distribution ... Otherwise ... we may not be able to fit an EVT distribution in the tail under the current samples D* ... We repeat the search until we pass the CV or a timeout occurs. ..."

    The feasibility claim is evaluated by the same criterion that controls data collection. Algorithm 1 keeps generating synthetic samples until the coefficient-of-variation/exponentiality test passes or a timeout occurs; the 'valid EVT' determination then accepts the type I/III shapes selected by that procedure, and scenarios that time out or fail are labeled NV and omitted from the reported rates. Hence the 95% EVT-fit rate is not an independent empirical discovery about CD tails; it is largely the pass rate of the paper's own acceptance test after synthetic augmentation. The fit is not defined as the test result, so the circularity is partial rather than definitional.

full rationale

The only defensible circularity is the 95% EVT-fit rate. The paper's sampling loop stops when the CV test passes, and the same exponentiality/shape criteria are used to declare a fit 'valid'; failures are excluded as NV. This makes the headline feasibility claim partly an artifact of the algorithm's acceptance rule, not an independent validation against external tail data. The paper itself acknowledges there is 'lack of ground truth regarding the tail of ML outcome distributions,' which reinforces that the fit rate is internally defined. I do not score the threshold/sample-size comparability of ECD = mu_u - mu_p as circularity: it is a serious validity threat to the interpretability of the metric, but it does not make an equation reduce to its own input by definition. Self-citations (PARFAIT-ML [42], worst-case EVT for ML [70]) are not load-bearing; they provide implementation bases and related-work context. ECD-Fair is self-referential in the sense that it optimizes ECD and is then evaluated on ECD, but the test-set evaluation across 20 benchmarks is not definitionally forced. Overall this is partial circularity, not definitional equivalence.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several domain assumptions about EVT applicability and synthetic data fidelity, plus arbitrary free parameters for threshold and significance. The most concerning is the implicit comparability of GEV locations across groups.

free parameters (4)
  • k_min and k_max (tail sample counts) = k_min=10, k_max=50
    Set in Algorithm 1 implementation to define the range of exceedances for the CV test and the GEV threshold; the choice fixes the threshold at the 50th largest CD and affects all ECD estimates.
  • ECD significance threshold = 0.05
    Used in Section V-B to classify ECD differences as discrimination (grayscale in Table IV); no theoretical or statistical justification is given.
  • GAN sample count m = 1
    Algorithm 1 adds m generated samples per iteration; the paper sets m=1, which is ambiguous (one sample or one batch) and affects the search behavior.
  • Timeout T = 1200 s
    Stops the tail-sampling loop; cases that time out before passing the CV test are marked NV, which affects the feasibility rate.
assumptions (6)
  • domain assumption Counterfactual discrimination values are independent and identically distributed for individuals within a group.
    Invoked in Section IV: 'Since these values for different individuals are independent of each other, we can consider the estimation over a large number of independent and identically distributed random variables.' This is needed for EVT applicability.
  • standard math The maxima of i.i.d. random variables converge to a GEV distribution under the assumed conditions.
    The asymptotic basis of EVT, cited from Leadbetter et al. [25]; the paper relies on this to justify GEV fitting.
  • domain assumption The coefficient of variation test with threshold 1 + 1/(4k) correctly identifies exponential/light tails and provides statistical guarantees.
    Adopted from Abella et al. [17] and used in Algorithm 1 to decide whether to generate more samples; the paper does not independently validate it for CD distributions.
  • domain assumption CTGAN and TVAE generate samples from the target group's feature distribution, and these samples are representative for the tail of the ML outcome distribution.
    Used in Section IV to augment scarce tail samples; only overall feature-distribution similarity (FID, KL) is evaluated, not the tail of CD, so this is a load-bearing assumption.
  • domain assumption Flipping the protected attribute yields a valid counterfactual individual.
    Acknowledged in Limitations: 'our approach assumes that flipping the sensitive values leads to valid representations'.
  • ad hoc to paper A GEV location parameter mu fitted from the top 50 CD values can be compared across groups with different thresholds and sample sizes.
    This comparability is asserted implicitly by defining ECD = mu_u - mu_p; no adjustment for differing thresholds or sample sizes is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fairness Testing through Extreme Value Theory." pith.science (2026). https://pith.science/paper/RFXKVL7A

@misc{pith2026250111597,
  author       = {Pith},
  title        = {Pith review of: Fairness Testing through Extreme Value Theory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RFXKVL7A}},
  note         = {Machine review of arXiv:2501.11597}
}
read the original abstract

Data-driven software is increasingly being used as a critical component of automated decision-support systems. Since this class of software learns its logic from historical data, it can encode or amplify discriminatory practices. Previous research on algorithmic fairness has focused on improving average-case fairness. On the other hand, fairness at the extreme ends of the spectrum, which often signifies lasting and impactful shifts in societal attitudes, has received significantly less emphasis. Leveraging the statistics of extreme value theory (EVT), we propose a novel fairness criterion called extreme counterfactual discrimination (ECD). This criterion estimates the worst-case amounts of disadvantage in outcomes for individuals solely based on their memberships in a protected group. Utilizing tools from search-based software engineering and generative AI, we present a randomized algorithm that samples a statistically significant set of points from the tail of ML outcome distributions even if the input dataset lacks a sufficient number of relevant samples. We conducted several experiments on four ML models (deep neural networks, logistic regression, and random forests) over 10 socially relevant tasks from the literature on algorithmic fairness. First, we evaluate the generative AI methods and find that they generate sufficient samples to infer valid EVT distribution in 95% of cases. Remarkably, we found that the prevalent bias mitigators reduce the average-case discrimination but increase the worst-case discrimination significantly in 5% of cases. We also observed that even the tail-aware mitigation algorithm -- MiniMax-Fairness -- increased the worst-case discrimination in 30% of cases. We propose a novel ECD-based mitigator that improves fairness in the tail in 90% of cases with no degradation of the average-case discrimination.

Figures

Figures reproduced from arXiv: 2501.11597 by the authors.

Figure 1
Figure 1. Counterfactual discrimination of DNN: (Left) The observed CD for white with the threshold red line at 0.12; (Mid-Left) The observed CD for black with the threshold line at 0.20. CD of Mitigated DNN: (Mid-Right) The observed CD for white with the threshold line sets at 0.81; and (Right) The observed CD for black with the threshold line sets at 0.72. Dataset. The Adult Census Income dataset [26] is a binary classifica… view at source ↗
Figure 2
Figure 2. The density of GEV for DNN: (Left) The density for white with the location of 0.15 and scale of 0.03; (Mid-Left) The density for black with the location of 0.28 and the scale of 0.08. The density of GEV for Mitigated DNN: (Mid-Right) The density of GEV distributions for white with the location of 0.84 and scale of 0.02; (Right) The density of GEV distributions for black with the location of 0.77 and the scale of 0.0… view at source ↗
Figure 3
Figure 3. MDS plot Our results indicate that both CTGAN and TVAE are effective in learning and replicating the actual data distribution. However, their ability to capture com￾plex feature relationships varies across datasets. For instance, with the Com￾pas dataset, CTGAN’s per￾formance stands out: it achieves a KL-divergence of 0.98. Conversely, TVAE shows its strength with the Adult dataset, as supported by all four eval￾uat… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 54 canonical work pages

  1. [1]

    Goodfellow, Y

    I. Goodfellow, Y . Bengio, and A. Courville, Deep learning. MIT press, 2016

    2016

  2. [2]

    R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction . MIT press, 2018

    2018

  3. [3]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  4. [4]

    Chatgpt: Optimizing language models for dialogue,

    O. ChatGPT, “Chatgpt: Optimizing language models for dialogue,” https: //openai.com/blog/chatgpt/, 2022, online

    work page 2022

  5. [5]

    Machine bias,

    S. M. Julia Angwin, Jeff Larson and L. Kirchne, “Machine bias,” https://www.propublica.org/article/ machine-bias-risk-assessments-in-criminal-sentencing, 2021, online

    work page 2021

  6. [6]

    ” computer says no!

    D. A. Elyounes, “” computer says no!”: The impact of automation on the discretionary power of public officers,” Vand. J. Ent. & Tech. L. , vol. 23, p. 451, 2020

    work page 2020

  7. [7]

    Automated government for vulnerable citizens: Intermediating rights,

    S. Ranchord ´as and L. Scarcella, “Automated government for vulnerable citizens: Intermediating rights,” SSRN Electronic Journal , 2021

    work page 2021

  8. [8]

    The IRS is targeting the poorest americans,

    D. A. Brown, “The IRS is targeting the poorest americans,” August 2021, [Online; posted 27-July-2021]. [Online]. Available: https://www.theatlantic.com/ideas/archive/2021/07/ how-race-plays-tax-policing/619570/

    work page 2021

Show all 74 references
  1. [9]

    Metamorphic testing and debugging of tax preparation soft- ware,

    S. Tizpaz-Niari, V . Monjezi, M. Wagner, S. Darian, K. Reed, and A. Trivedi, “Metamorphic testing and debugging of tax preparation soft- ware,” in 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) . IEEE, 2023, pp. 138–149

    2023

  2. [10]

    The gender gap in employment and wages,

    B. Petrongolo, “The gender gap in employment and wages,” Nature Human Behaviour, vol. 3, no. 4, pp. 316–318, 2019

    2019

  3. [11]

    Racial differences in access to high- paying jobs and the wage gap between black and white women,

    D. Anderson and D. Shapiro, “Racial differences in access to high- paying jobs and the wage gap between black and white women,” ILR Review, vol. 49, no. 2, pp. 273–286, 1996

    1996

  4. [12]

    Fairness through awareness,

    C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel, “Fairness through awareness,” in Proceedings of the 3rd innovations in theoretical computer science conference , 2012, pp. 214–226

    2012

  5. [13]

    Coles, J

    S. Coles, J. Bawa, L. Trenner, and P. Dorazio, An introduction to statistical modeling of extreme values . Springer, 2001, vol. 208

    2001

  6. [14]

    Fairness testing: testing software for discrimination,

    S. Galhotra, Y . Brun, and A. Meliou, “Fairness testing: testing software for discrimination,” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering , ser. ESEC/FSE 2017. New York, NY , USA: Association for Computing Machinery, 2017, p. 498–510. ...

    2017

  7. [15]

    Fairness risk measures,

    R. Williamson and A. Menon, “Fairness risk measures,” in International Conference on Machine Learning . PMLR, 2019, pp. 6786–6797

    2019

  8. [16]

    A goodness-of-fit test for the distribution tail,

    J. Diebolt, M. Garrido, and S. Girard, “A goodness-of-fit test for the distribution tail,” 2007

    2007

  9. [17]

    Measurement- based worst-case execution time estimation using the coefficient of variation,

    J. Abella, M. Padilla, J. D. Castillo, and F. J. Cazorla, “Measurement- based worst-case execution time estimation using the coefficient of variation,” ACM Transactions on Design Automation of Electronic Systems (TODAES), vol. 22, no. 4, pp. 1–29, 2017

    2017

  10. [18]

    L. Xu, M. Skoularidou, A. Cuesta-Infante, and K. Veeramachaneni, Modeling tabular data using conditional GAN . Red Hook, NY , USA: Curran Associates Inc., 2019

    2019

  11. [19]

    Variational autoencoder based synthetic data generation for imbalanced learning,

    Z. Wan, Y . Zhang, and H. He, “Variational autoencoder based synthetic data generation for imbalanced learning,” in 2017 IEEE Symposium Series on Computational Intelligence (SSCI) , 2017, pp. 1–7

    2017

  12. [20]

    A reductions approach to fair classification,

    A. Agarwal, A. Beygelzimer, M. Dud ´ık, J. Langford, and H. Wallach, “A reductions approach to fair classification,” in International Conference on Machine Learning . PMLR, 2018, pp. 60–69

    2018

  13. [21]

    Bias in machine learning software: Why? how? what to do?

    J. Chakraborty, S. Majumder, and T. Menzies, “Bias in machine learning software: Why? how? what to do?” inProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE 2021. New Yo...

    2021

  14. [22]

    Maat: a novel ensemble approach to addressing fairness and performance bugs for machine learning software,

    Z. Chen, J. M. Zhang, F. Sarro, and M. Harman, “Maat: a novel ensemble approach to addressing fairness and performance bugs for machine learning software,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Softwar...

    2022

  15. [23]

    Don’t lie to me: Avoiding malicious explanations with stealth,

    L. Alvarez and T. Menzies, “Don’t lie to me: Avoiding malicious explanations with stealth,” IEEE Software , vol. 40, no. 3, pp. 43–53, 2023

    2023

  16. [24]

    Minimax group fairness: Algorithms and experiments,

    E. Diana, W. Gill, M. Kearns, K. Kenthapadi, and A. Roth, “Minimax group fairness: Algorithms and experiments,” in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , ser. AIES ’21. New York, NY , USA: Association for Computing Machinery, 2021, p. 66–76. [...

    2021

  17. [25]

    M. R. Leadbetter, G. Lindgren, and H. Rootz ´en, Extremes and related properties of random sequences and processes . Springer Science & Business Media, 2012

    2012

  18. [26]

    UCI machine learning repository,

    D. Dua and C. Graff, “UCI machine learning repository,” 2017. [Online]. Available: https://archive.ics.uci.edu/ml/datasets/census+income

    2017

  19. [27]

    White-box fairness testing through adversarial sampling,

    P. Zhang, J. Wang, J. Sun, G. Dong, X. Wang, X. Wang, J. S. Dong, and T. Dai, “White-box fairness testing through adversarial sampling,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, 2020, pp. 949–960

    2020

  20. [28]

    Neuronfair: Interpretable white-box fairness testing through biased neuron identification,

    H. Zheng, Z. Chen, T. Du, X. Zhang, Y . Cheng, S. Ti, J. Wang, Y . Yu, and J. Chen, “Neuronfair: Interpretable white-box fairness testing through biased neuron identification,” in 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE) , 2022, pp. 1519–1531

    2022

  21. [29]

    Efficient white-box fairness testing through gradient search,

    L. Zhang, Y . Zhang, and M. Zhang, “Efficient white-box fairness testing through gradient search,” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2021, 2021, p. 103–114. [Online]. Available: https://doi.org/10.1145/3...

    2021

  22. [31]

    UCI:heart disease data set,

    “UCI:heart disease data set,” 2001. [Online]. Available: https: //archive.ics.uci.edu/ml/datasets/Heart+Disease

    2001

  23. [32]

    Automated directed fairness testing,

    S. Udeshi, P. Arora, and S. Chattopadhyay, “Automated directed fairness testing,” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering , ser. ASE ’18. New York, NY , USA: Association for Computing Machinery, 2018, p. 98–108. [Online]. ...

    2018

  24. [33]

    Neuronfair: interpretable white-box fairness testing through biased neuron identification,

    H. Zheng, Z. Chen, T. Du, X. Zhang, Y . Cheng, S. Ji, J. Wang, Y . Yu, and J. Chen, “Neuronfair: interpretable white-box fairness testing through biased neuron identification,” in Proceedings of the 44th International Conference on Software Engineering , ser. ICSE ’22. New Yor...

    2022

  25. [34]

    Ctab-gan: Effective table data synthesizing,

    Z. Zhao, A. Kunar, R. Birke, and L. Y . Chen, “Ctab-gan: Effective table data synthesizing,” in Proceedings of The 13th Asian Conference on Machine Learning , ser. Proceedings of Machine Learning Research, V . N. Balasubramanian and I. Tsang, Eds., vol

  26. [35]

    Autogan: An automated human-out-of-the-loop approach for training generative adversarial networks,

    E. Nazari, P. Branco, and G.-V . Jourdan, “Autogan: An automated human-out-of-the-loop approach for training generative adversarial networks,” Mathematics, vol. 11, no. 4, 2023. [Online]. Available: https://www.mdpi.com/2227-7390/11/4/977

    2023

  27. [36]

    Distance correlation gan: Fair tabular data generation with generative adversarial networks,

    A. Rajabi and O. O. Garibay, “Distance correlation gan: Fair tabular data generation with generative adversarial networks,” in Artificial Intelligence in HCI: 4th International Conference, AI-HCI 2023, Held as Part of the 25th HCI International Conference, HCII 2023, Copenhage...

    2023 doi

  28. [37]

    Learning classifiers from synthetic data using a multichannel autoencoder,

    X. Zhang, Y . Fu, A. Zang, L. Sigal, and G. Agam, “Learning classifiers from synthetic data using a multichannel autoencoder,” ArXiv, vol. abs/1503.03163, 2015. [Online]. Available: https://api.semanticscholar. org/CorpusID:8164829

    2015 arXiv

  29. [38]

    Crash data augmentation using variational autoencoder,

    Z. Islam, M. Abdel-Aty, Q. Cai, and J. Yuan, “Crash data augmentation using variational autoencoder,” Accident Analysis and Prevention, vol. 151, p. 105950, 2021. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S000145752031770X

    2021

  30. [39]

    Latent imitator: Generating natural individual discriminatory instances for black-box fairness testing,

    Y . Xiao, A. Liu, T. Li, and X. Liu, “Latent imitator: Generating natural individual discriminatory instances for black-box fairness testing,” in Proceedings of the 32nd ACM SIGSOFT international symposium on software testing and analysis , 2023, pp. 829–841

    2023

  31. [40]

    Methods to distinguish between polynomial and exponential tails,

    J. D. Castillo, J. Daoudi, and R. Lockhart, “Methods to distinguish between polynomial and exponential tails,” Scandinavian Journal of Statistics, vol. 41, no. 2, pp. 382–393, 2014

    2014

  32. [41]

    Biometry: The principles and practice of statistics in biological research 3rd edition wh freeman and co,

    R. Sokal and F. Rohlf, “Biometry: The principles and practice of statistics in biological research 3rd edition wh freeman and co,” New York, 1995

    1995

  33. [42]

    Fairness-aware configuration of machine learning libraries,

    S. Tizpaz-Niari, A. Kumar, G. Tan, and A. Trivedi, “Fairness-aware configuration of machine learning libraries,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 909–920

    2022

  34. [43]

    Fairlearn: A toolkit for assessing and improving fairness in AI,

    S. Bird, M. Dud ´ık, R. Edgar, B. Horn, R. Lutz, V . Milan, M. Sameki, H. Wallach, and K. Walker, “Fairlearn: A toolkit for assessing and improving fairness in AI,” Microsoft, Tech. Rep. MSR-TR-2020-32, May 2020. [On- line]. Available: https://www.microsoft.com/en-us/research/...

    2020

  35. [44]

    UCI machine learning repository (german credit),

    “UCI machine learning repository (german credit),” 2017. [On- line]. Available: https://archive.ics.uci.edu/ml/datasets/statlog+(german+ credit+data)

    2017

  36. [45]

    UCI machine learning repository (bank marketing),

    “UCI machine learning repository (bank marketing),” 2017. [Online]. Available: https://archive.ics.uci.edu/ml/datasets/bank+marketing

    2017

  37. [46]

    Compas software ananlysis,

    ProPublica, “Compas software ananlysis,” https://github.com/propublica/ compas-analysis, 2021, online

    2021

  38. [47]

    UCI:default of credit card clients data set,

    “UCI:default of credit card clients data set,” 2009. [Online]. Available: https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients

    2009

  39. [48]

    Medical expenditure panel survey,

    “Medical expenditure panel survey,” 2014. [Online]. Available: https://meps.ahrq.gov/mepsweb/

    2014

  40. [49]

    Student performance data set,

    “Student performance data set,” 2014. [Online]. Available: https: //archive.ics.uci.edu/ml/datasets/Student+Performance

    2014

  41. [50]

    TensorFlow: Large-scale machine learning on heterogeneous systems,

    M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y . Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Man ´e, R. Monga, S. Moore, D. Murray, C. Olah, M...

    2015

  42. [51]

    Scikit-learn: Machine learning in Python,

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vander- plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay, “Scikit-learn: Machine learning in Python,” Journal of Machine ...

    2011

  43. [52]

    Automated directed fairness testing,

    S. Udeshi, P. Arora, and S. Chattopadhyay, “Automated directed fairness testing,” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering , 2018, pp. 98–108

    2018

  44. [53]

    Fairway: a way to build fair ml software,

    J. Chakraborty, S. Majumder, Z. Yu, and T. Menzies, “Fairway: a way to build fair ml software,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2020, pp. 654–665

    2020

  45. [54]

    Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias,

    R. K. Bellamy, K. Dey, M. Hind, S. C. Hoffman, S. Houde, K. Kannan, P. Lohia, J. Martino, S. Mehta, A. Mojsilovi ´c et al. , “Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias,” IBM Journal of Research and Development , vol. 63, no. 4/5, pp. ...

    2019

  46. [55]

    A software review for extreme value analysis,

    E. Gilleland, M. Ribatet, and A. G. Stephenson, “A software review for extreme value analysis,” Extremes, vol. 16, no. 1, pp. 103–119, 2013

    2013

  47. [56]

    Robust confidence intervals for effect sizes: A comparative study of cohen’s d and cliff’s delta under non-normality and heterogeneous variances,

    M. Hess and J. Kromrey, “Robust confidence intervals for effect sizes: A comparative study of cohen’s d and cliff’s delta under non-normality and heterogeneous variances,” Paper Presented at the Annual Meeting of the American Educational Research Association , 01 2004

    2004

  48. [57]

    Ranking and clustering software cost estimation models through a multiple comparisons algorithm,

    N. Mittas and L. Angelis, “Ranking and clustering software cost estimation models through a multiple comparisons algorithm,” IEEE Transactions on Software Engineering, vol. 39, no. 4, pp. 537–551, 2013

    2013

  49. [58]

    Grouped correlational generative adversarial networks for discrete electronic health records,

    F. Yang, Z. Yu, Y . Liang, X. Gan, K. Lin, Q. Zou, and Y . Zeng, “Grouped correlational generative adversarial networks for discrete electronic health records,” in 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) , 2019, pp. 906–913

    2019

  50. [59]

    A note on the evaluation of generative models,

    L. Theis, A. van den Oord, and M. Bethge, “A note on the evaluation of generative models,” CoRR, vol. abs/1511.01844, 2015. [Online]. Available: https://api.semanticscholar.org/CorpusID:2187805

    2015 arXiv

  51. [60]

    Tabsyndex: a universal metric for robust evaluation of synthetic tabular data,

    V . S. Chundawat, A. K. Tarun, M. Mandal, M. Lahoti, and P. Narang, “Tabsyndex: a universal metric for robust evaluation of synthetic tabular data,” arXiv preprint arXiv:2207.05295 , 2022

    2022 arXiv

  52. [61]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in Proceedings of the 31st International Conference on Neural Information Processing Systems , ser. NIPS’17. Red Hook, NY...

    2017

  53. [62]

    The synthetic data vault,

    N. Patki, R. Wedge, and K. Veeramachaneni, “The synthetic data vault,” in IEEE International Conference on Data Science and Advanced Analytics (DSAA), Oct 2016, pp. 399–410

    2016

  54. [63]

    Borg and P

    I. Borg and P. J. Groenen, Modern multidimensional scaling: Theory and applications. Springer Science & Business Media, 2005

    2005

  55. [64]

    Automated test generation to detect individual discrimination in ai models,

    A. Agarwal, P. Lohia, S. Nagar, K. Dey, and D. Saha, “Automated test generation to detect individual discrimination in ai models,” arXiv preprint arXiv:1809.03260, 2018

    2018 arXiv

  56. [65]

    Neufair: Neural network fairness repair with dropout,

    V . A. Dasu, A. Kumar, S. Tizpaz-Niari, and G. Tan, “Neufair: Neural network fairness repair with dropout,” ser. ISSTA 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 1541–1553. [Online]. Available: https://doi.org/10.1145/3650212.3680380

    2024

  57. [66]

    Adaptive sam- pling for minimax fair classification,

    S. Shekhar, G. Fields, M. Ghavamzadeh, and T. Javidi, “Adaptive sam- pling for minimax fair classification,” Advances in Neural Information Processing Systems, vol. 34, pp. 24 535–24 544, 2021

    2021

  58. [67]

    Characterizing intersectional group fairness with worst-case comparisons,

    A. Ghosh, L. Genuit, and M. Reagan, “Characterizing intersectional group fairness with worst-case comparisons,” in Artificial Intelligence Diversity, Belonging, Equity, and Inclusion . PMLR, 2021, pp. 22–34

    2021

  59. [68]

    Adaptive fairness improvement based on causality analysis,

    M. Zhang and J. Sun, “Adaptive fairness improvement based on causality analysis,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 6–17

    2022

  60. [69]

    Fairness improvement with multiple protected attributes: How far are we?

    Z. Chen, J. M. Zhang, F. Sarro, and M. Harman, “Fairness improvement with multiple protected attributes: How far are we?” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

    2024

  61. [70]

    Worst-case convergence time of ml algorithms via extreme value theory,

    S. Tizpaz-Niari and S. Sankaranarayanan, “Worst-case convergence time of ml algorithms via extreme value theory,” in Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI , ser. CAIN ’24. New York, NY , USA: Association for Co...

    2024

  62. [71]

    Income inequality in the united states, 1913– 1998,

    T. Piketty and E. Saez, “Income inequality in the united states, 1913– 1998,” The Quarterly journal of economics , vol. 118, no. 1, pp. 1–41, 2003

    1913

  63. [72]

    Income and wealth concentration in a historical and interna- tional perspective, uc berkeley and nber, forthcoming in john quigley,

    E. Saez, “Income and wealth concentration in a historical and interna- tional perspective, uc berkeley and nber, forthcoming in john quigley,” in Poverty, the Distribution of Income, and Public Policy, A conference in honor of Eugene Smolensky , 2004

    2004

  64. [73]

    Income Inequality in OECD Countries: Data and Explanations,

    A. B. Atkinson, “Income Inequality in OECD Countries: Data and Explanations,” CESifo Economic Studies , vol. 49, no. 4, pp. 479–513, 12 2003. [Online]. Available: https://doi.org/10.1093/cesifo/49.4.479

    2003 doi

  65. [74]

    Fairness metrics for recommender systems,

    H. Wang, “Fairness metrics for recommender systems,” in 2022 9th international Conference on Wireless Communication and Sensor Net- works (ICWCSN), 2022, pp. 89–92

    2022

  66. [157]

    PMLR, 17–19 Nov 2021, pp. 97–112. [Online]. Available: https://proceedings.mlr.press/v157/zhao21a.html

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.