{"id":"ebf98558-6323-4119-9f80-6966d21685e2","arxiv_id":"2412.15920","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"FATE is a proposed genetic algorithm that searches over fairness-aware data preparation pipelines, but the paper reports no experiments and the empirical claim remains untested.","lead":"This registered report proposes a genetic algorithm, FATE, that selects combinations of standard data preparation steps to make machine learning models fairer while preserving performance. The planned study would compare FATE-selected pipelines against established bias-mitigation preprocessors, offering practitioners a more familiar route to fair models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FATE's fairness score treats disparate impact as if 0 were ideal, so the optimizer is driven to select maximally unfair pipelines.","rationale":"The reader identified the fairness score construction as the weakest assumption, but focused on the lack of normalization and weighting. The more severe issue is that DI's ideal value is 1, not 0, so the score is not merely arbitrarily weighted; it actively rewards unfairness. This is the single most load-bearing concern because FATE's entire search is guided by this fitness function: if the score is wrong, the returned pipelines are optimal only with respect to a self-defeating objective, and both research questions collapse. The concern is internally inconsistent with the paper's own statement that '0 represents the ideal unbiased value' for all three metrics, so it does not depend on external consensus. No implementation or empirical results are provided, so the central claim remains unverified, but the design flaw means that even the planned experiments would test the wrong thing. The protocol is salvageable by redefining the DI component (e.g., |DI - 1| or a variant of 1 - min(DI, 1/DI)), which is why I recommend CONDITIONAL rather than REJECT or UNVERDICTED: the paper should be accepted only after the fitness function is corrected and the experimental design updated accordingly.","tokens_in":11988,"tokens_out":5937,"duration_ms":52215,"concrete_test":"Run the fitness function on two synthetic pipelines with identical PR-AUC. Pipeline A: SPD = 0, EOD = 0, DI = 1. Pipeline B: SPD = -0.1, EOD = -0.1, DI = 0. With alpha = beta = 0.5, fitness_A = 0.5 * PS - 0.5 * 1 and fitness_B = 0.5 * PS - 0.5 * 0.2, so B ranks higher. If B ranks higher, the optimizer is selecting the maximally unfair pipeline. This can be checked analytically in a few lines of Python and requires no real dataset.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central design flaw is in the fitness function (Table I, Section III Step 2). FS is defined as |SPD| + |EOD| + |DI|, and the text states that fairness metrics 'typically range from -1 to 1, with 0 representing the ideal unbiased value.' That statement is true for SPD and EOD but false for DI: the disparate impact ratio is bounded below by 0, and its ideal fair value is 1, not 0. Because the genetic algorithm minimizes FS, a pipeline that drives DI toward 0 receives a lower, better score. Concretely, a perfectly fair classifier with SPD = 0, EOD = 0, DI = 1 yields FS = 1, while a classifier that gives the favorable outcome to no members of the unprivileged group (DI = 0, SPD = -0.1, EOD = -0.1) yields FS = 0.2, which the optimizer prefers. Thus FATE is not merely using an unnormalized composite; it is optimizing an objective that rewards maximal discrimination on the DI component. This is an internal inconsistency, not a dispute over metric choice. Any 'near-optimal' pipeline returned by FATE would be optimal only with respect to this mis-specified score, invalidating RQ1 and the FATE side of the RQ2 comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This registered report proposes FATE, a genetic algorithm that selects fairness-aware 'Data Preparation' pipelines by optimizing a linear combination of predictive performance (PR-AUC) and a composite fairness score built from SPD, EOD, and DI. The planned study will evaluate FATE's efficacy under different genetic-algorithm parameters (RQ1) and compare FATE-selected pipelines against three pre-processing bias mitigation techniques, FairSMOTE, Reweighing, and Disparate Impact Remover (RQ2), using three datasets, four classifiers, and non-parametric tests with effect sizes. The paper contains no experimental results; it contributes the design of FATE, the experimental protocol, and a threats-to-validity analysis. The central methodological problem is the definition of the fairness score, which treats DI as though its ideal value were 0, so the optimizer is driven toward maximally unfair outcomes.","tokens_in":12264,"tokens_out":4495,"duration_ms":40064,"significance":"If the protocol were corrected, the study would address a practitioner-relevant question: whether common data preparation practices, selected near-optimally, can compete with dedicated bias mitigation pre-processing techniques. The paper's strengths are its registered-report format, the use of widely accepted datasets and classifiers, the planned statistical tests with effect sizes, and the commitment to releasing data and scripts. However, the soundness of the planned experiments depends entirely on the correctness of FATE's fitness function, and the current definition contains a load-bearing error that would invalidate the interpretation of both research questions. The absence of experimental results is expected for a registered report and is not itself a defect; the defect is in the proposed methodology.","major_comments":[{"comment":"The fairness score FS = |SPD| + |EOD| + |DI| is internally inconsistent because the ideal value of DI is 1, not 0. The text states that 'fairness metrics typically range from -1 to 1, with 0 representing the ideal unbiased value'; this is true for SPD and EOD but false for DI, which is a ratio bounded below by 0 and equals 1 for a perfectly fair model. Since the genetic algorithm maximizes α·PS − β·FS, minimizing FS drives DI toward 0, i.e., toward the maximally discriminatory outcome in which the unprivileged group receives no favorable classifications. For example, a classifier with SPD=0, EOD=0, and DI=1 has FS=1, while a classifier with SPD=-0.1, EOD=-0.1, and DI=0 has FS=0.2 and would be preferred by the optimizer. Thus FATE's search is guided by an objective that rewards unfairness on the DI component, which would invalidate the interpretation of RQ1 and the FATE arm of RQ2.","section":"Section III, Step #2 and Table I"},{"comment":"The fitness function combines PR-AUC, which is bounded in [0,1], with FS, which is unbounded above because DI can exceed 1 and the absolute values of the three metrics are simply summed without normalization. With α=β=0.5, the fairness term can dominate or become negligible depending on the dataset's base rates, so the trade-off parameter does not have the meaning attributed to it. The authors should normalize each component to a common scale, use a weighted sum with justified weights, or empirically validate the chosen formulation; otherwise the fairness-performance trade-off controlled by α and β is not well-defined.","section":"Section III, Step #2 and Table I"}],"minor_comments":[{"comment":"The DI formula contains an unmatched parenthesis: it is written as P(Y=1|A=0))/(P(Y=1|A=1).","section":"Table I"},{"comment":"The text contains typographical errors: 'sensitive attributes such as age' and sex'' uses stray apostrophes, and 'we plan to ML models to experiment with' is missing an infinitive verb.","section":"Section IV.B.2 and IV.B.3"},{"comment":"The null hypotheses H3a-H3c concern execution time, but the alternative hypotheses are described only generically; the authors should specify whether the tests are two-sided or whether a directional alternative is intended.","section":"Section IV.C.2"},{"comment":"The section says the authors will assess the distribution to select the most appropriate test, but the Wilcoxon and Vargha-Delaney tests are already specified; clarify whether the normality check is meant to choose between parametric and non-parametric alternatives.","section":"Section V, Conclusion Validity"},{"comment":"The running examples report FS values but never show the individual SPD, EOD, and DI contributions; a concrete numerical example contrasting a fair model (DI near 1) with an unfair model (DI near 0) would help clarify the intended behavior of the fitness function.","section":"Section III, Running Examples"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the DI issue in the fitness function is a correctness error, not a matter of metric preference. The authors can likely fix it within the scope of the manuscript by redefining FS as a deviation from the ideal values, e.g., |SPD| + |EOD| + |1 - DI|, and by normalizing the components before combining them with PR-AUC. I recommend major revision rather than rejection because the research question and empirical plan are defensible once the objective function is corrected. The revision should also demonstrate, perhaps with a small synthetic example, that the corrected fitness function ranks fair models above unfair ones."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know before reading this paper. First, the central mechanism of FATE is a genetic algorithm that optimizes a fitness function combining PR-AUC with a fairness score FS = |SPD| + |EOD| + |DI|. Second, that fairness score is not just unnormalized; it inverts the intended direction for disparate impact. The stress-test note is correct and it lands exactly on Table I and Section III Step 2.\n\nThe paper claims all fairness metrics “range from -1 to 1, with 0 representing the ideal unbiased value.” That is true for SPD and EOD, but false for DI. DI is a ratio of favorable outcome probabilities; the ideal value is 1, and 0 means the unprivileged group never receives the favorable outcome. Because the GA maximizes α·PS − β·FS, it will drive DI toward 0, rewarding pipelines that maximize discrimination. Concretely, a perfectly fair classifier (SPD=0, EOD=0, DI=1) gets FS=1, while one that denies all favorable outcomes to the unprivileged group (DI=0, SPD=-0.1, EOD=-0.1) gets FS=0.2 and is preferred. This is an internal contradiction, not a debatable metric choice. It invalidates RQ1 and the FATE side of RQ2.\n\nWhat the paper does well: it is an honest, well-structured registered report. It clearly states that no experiments exist yet, builds on the authors’ prior catalog and survey, recognizes FAIRPIPES as the closest related work and articulates three differences, and plans a reasonable empirical protocol with standard datasets, models, and statistical tests. The threats-to-validity section is thoughtful. The literature coverage is appropriate, and the authors are transparent about using ChatGPT for language polishing.\n\nThe secondary soft spots are minor by comparison: FS sums a ratio with differences without normalization or weighting (α=β=0.5 assumes commensurate scales), the GA parameter grid includes 0% and 100% crossover/mutation rates which are degenerate, and some “practices” like clustering and matching are not obviously data preparation steps in the usual sense. But none of these matter if the fitness function is fixed, which it would need to be before any experiment is run.\n\nWho this is for: researchers in SE4AI interested in pipeline search for fairness, and anyone teaching registered reports or GA design. It is a useful discussion piece but not a citable result. As a referee, I would give major revision, not desk rejection, because the topic is timely and the protocol is otherwise solid. But the DI issue is load-bearing and cannot be waved away.","headline":"FATE's fairness score has a sign error on disparate impact, so the GA would optimize for maximal discrimination; the rest of the protocol is sound but this flaw changes everything.","tokens_in":12783,"tokens_out":3870,"would_cite":false,"duration_ms":33365,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This registered report proposes that a near-optimal choice of everyday fairness-aware data preparation practices can improve both fairness and performance, and introduces FATE, a genetic algorithm for finding such pipelines.","keywords":["machine learning fairness","data preparation","genetic algorithm","pipeline selection","fairness-performance trade-off","bias mitigation","pre-processing","registered report"],"falsifier":"Run FATE on the Adult dataset with Logistic Regression and compare its selected pipeline against a large random sample of data-prep pipelines on held-out SPD, EOD, and DI; if the FATE-selected pipeline is not consistently better than random pipelines on these per-metric fairness values, the claim that the search finds a near-optimal fairness-enhancing pipeline fails.","tokens_in":11830,"feed_emoji":"⚖️","tokens_out":10798,"duration_ms":83927,"temperature":0.7,"pith_summary":"This registered report argues that a carefully chosen combination of ordinary data preparation practices—scaling, resampling, clustering, weighting, matching—can reduce machine-learning bias as effectively as specialized preprocessing bias-mitigation algorithms, while being easier for practitioners to adopt. To test that hypothesis, the authors introduce FATE, a genetic algorithm that searches over pipelines of such fairness-aware practices and scores each candidate with a weighted combination of PR-AUC and three fairness metrics. The study is designed as a confirmatory comparison: FATE-selected pipelines are pitted against established preprocessing techniques on three datasets and four classifiers. If the hypothesis holds, teams could mitigate bias by reusing steps they already know rather than learning dedicated debiasing tools.","feed_headline":"Can ordinary data prep match specialized fairness tools?","feed_subtitle":"FATE would find data-prep pipelines that balance fairness and accuracy without specialized debiasing tools.","key_machinery":"FATE is a genetic-algorithm-based pipeline selector. Each individual in the population is a combination of fairness-aware Data Preparation practices chosen from a catalog that includes standard scaling, min-max scaling, resampling, clustering, inverse probability weighting, and matching. Evolution is steered by a single fitness score that merges predictive performance (PR-AUC) with a composite fairness deviation $|\\text{SPD}| + |\\text{EOD}| + |\\text{DI}|$, using crossover, mutation, and K-fold cross-validation to evaluate each pipeline. The genetic parameters—population size, number of generations, crossover rate, and mutation rate—are swept systematically to find configurations that balance the two objectives.","core_discovery":"The central claim is that a near-optimal selection of fairness-aware Data Preparation practices can jointly improve fairness and predictive performance, making it a practical alternative to specialized pre-processing bias mitigation. FATE operationalizes the claim by encoding candidate pipelines as individuals in a genetic algorithm and evolving them under the fitness function $\\alpha \\cdot \\text{PR-AUC} - \\beta \\cdot (|\\text{SPD}| + |\\text{EOD}| + |\\text{DI}|)$, with $\\alpha$ and $\\beta$ set to $0.5$. The paper positions this as a hypothesis to be confirmed by a registered study: RQ1 measures how well FATE selects near-optimal pipelines under varying genetic-algorithm parameters, and RQ2 compares the best pipelines against FairSMOTE, Reweighing, and Disparate Impact Remover on fairness, performance, and execution time.","pith_inferences":["An implied extension not tested in the report: the same search procedure could be applied to other lifecycle stages, such as in-processing or post-processing, although the report only considers Data Preparation.","A testable refinement would normalize each fairness metric before summing, since SPD and EOD are probability differences while DI is a ratio on a different scale; comparing selection stability under normalized versus raw scores would show whether the fitness function is robust.","FATE uses fixed equal weights ($\\alpha = \\beta = 0.5$); exposing the trade-off as a Pareto front of non-dominated pipelines would let practitioners choose their own fairness-performance preference without rerunning the search.","Because FATE is agnostic to the classifier and dataset, a natural next test is whether pipelines found on binary classification transfer to regression or multi-class tasks when PR-AUC is replaced with an appropriate performance metric."],"forward_implications":["If FATE works as hypothesized, teams can reduce bias by reusing data preparation steps they already know, without adopting specialized pre-processing algorithms.","FATE returns the fairness and performance metrics behind its choice, so practitioners can see why a pipeline was selected.","The parameter sweep will show how genetic-algorithm settings shift the fairness-performance trade-off and which settings are safest.","The comparison against FairSMOTE, Reweighing, and Disparate Impact Remover will indicate whether data preparation pipelines can match or beat dedicated bias-mitigation techniques.","A positive result would make fairness improvement available earlier in the ML lifecycle, before model training, which is where data-driven bias enters."],"supporting_citations":[{"why":"Supplies the catalog of fairness-aware Data Preparation practices that FATE selects among.","marker":"[16]"},{"why":"Provides the practitioner survey showing these practices are already used and perceived as fairness-relevant.","marker":"[17]"},{"why":"Defines the prior automated pipeline-selection approach that FATE extends toward fairness-aware practices.","marker":"[28]"},{"why":"Motivates the choice of SPD, EOD, and DI as the fairness metrics used in the fitness function.","marker":"[29]"},{"why":"Provides FairSMOTE, one of the three pre-processing baselines used in the comparison.","marker":"[6]"},{"why":"Provides Reweighing, the second pre-processing baseline used in the comparison.","marker":"[12]"},{"why":"Provides Disparate Impact Remover, the third pre-processing baseline used in the comparison.","marker":"[13]"},{"why":"Documents the fairness datasets from which the three study datasets are selected.","marker":"[45]"},{"why":"Identifies the four classifiers used as optimization tasks and evaluation models.","marker":"[50]"}],"fun_headline_variants":["Can data prep rival specialized fairness tools? FATE investigates","FATE: Optimizing data prep to balance fairness and accuracy","Fairness without debiasing tools? FATE tests data prep pipelines","Data prep as a fairness lever: FATE's pragmatic approach","Is data prep enough for fairness? FATE explores the trade-off"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire search assumes that the composite fairness score $|\\text{SPD}| + |\\text{EOD}| + |\\text{DI}|$ is a meaningful objective, even though the three metrics are measured on different scales, so every pipeline called 'near-optimal' is optimal only with respect to that sum.","fun_headline_variants_meta":{"raw":{"variants":["Can data prep rival specialized fairness tools? FATE investigates","FATE: Optimizing data prep to balance fairness and accuracy","Fairness without debiasing tools? FATE tests data prep pipelines","Data prep as a fairness lever: FATE's pragmatic approach","Is data prep enough for fairness? FATE explores the trade-off"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1461,"prompt_tokens":878,"completion_tokens":583,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":495}},"tokens_in":494,"tokens_out":583,"duration_ms":5615,"temperature":1.0,"reasoning_tokens":495,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:57:08.357082+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FATE on the Adult dataset with Logistic Regression and compare its selected pipeline against a large random sample of data-prep pipelines on held-out SPD, EOD, and DI; if the FATE-selected pipeline is not consistently better than random pipelines on these per-metric fairness values, the claim that the search finds a near-optimal fairness-enhancing pipeline fails.","supporting_citations":[{"cited_title":"Fairness-aware practices from developers’ perspective: A survey,","cited_arxiv_id":null,"evidence_quote":"Provides the practitioner survey showing these practices are already used and perceived as fairness-relevant."},{"cited_title":"Preprocessing matters: Automated pipeline selection for fair classification,","cited_arxiv_id":null,"evidence_quote":"Defines the prior automated pipeline-selection approach that FATE extends toward fairness-aware practices."},{"cited_title":"Fair enough: Searching for sufficient measures of fairness,","cited_arxiv_id":null,"evidence_quote":"Motivates the choice of SPD, EOD, and DI as the fairness metrics used in the fitness function."},{"cited_title":"Bias in machine learning software: why? how? what to do?","cited_arxiv_id":null,"evidence_quote":"Provides FairSMOTE, one of the three pre-processing baselines used in the comparison."},{"cited_title":"Data preprocessing techniques for classi- fication without discrimination,","cited_arxiv_id":null,"evidence_quote":"Provides Reweighing, the second pre-processing baseline used in the comparison."},{"cited_title":"Certifying and removing disparate impact,","cited_arxiv_id":null,"evidence_quote":"Provides Disparate Impact Remover, the third pre-processing baseline used in the comparison."},{"cited_title":"Algorithmic fairness datasets: the story so far,","cited_arxiv_id":null,"evidence_quote":"Documents the fairness datasets from which the three study datasets are selected."}],"review_version":1}