{"id":"a3da1df2-1e95-4c11-90c4-930c20a806c9","arxiv_id":"2411.16965","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Applying CMA-MAP-Elites to neural network weights produces a two-dimensional map of accuracy versus positive-prediction ratios, enabling selection of high-accuracy models that stay within an 80% rule fairness zone.","lead":"This paper tests whether a quality-diversity search can map how a classifier's accuracy and group fairness trade off, then lets users pick the most accurate model that still meets a fairness threshold. It demonstrates the approach on promotion and income prediction datasets, showing that such maps can reveal bias and quantify the cost of fairness.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The fairness descriptors used to build the map may be ratios of mean network outputs rather than ratios of positive predicted rates, so the fair zone may not actually implement the 80% rule.","rationale":"The paper's contribution is a visualization whose axes are the bias descriptors; those axes define what 'fair zone' means. If the descriptor is the wrong quantity, every downstream claim about trade-offs and fair selection changes. The concern is not merely a missing citation for the 80% rule; it is an internal mismatch between the stated metric (positive prediction rate) and the algorithm (mean prediction). This is checkable from the code or a small recomputation. The paper is otherwise clear and the experimental patterns are plausible, so the verdict stays conditional pending this verification.","tokens_in":9927,"tokens_out":10104,"duration_ms":101601,"concrete_test":"Examine the released code (github.com/catajara/Classif-Bias-Tradeoffs-QD) to see exactly how descriptors are computed. Then re-run the Male biased sample (Table 5) while logging, for each archive solution, both ratio_x (as in the paper) and the ratio of counts of predictions above 0.5 for the two groups. Compare the two values and the resulting fair-zone selection. If the descriptor values or the best-fair model differ, the map axes are mislabeled and Tables 5-9 need recomputation with the corrected ratio.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 states that the descriptors are ratios of positive prediction rates between protected groups (e.g., female vs. male), and Section 4 defines the fair zone as [0.8, 1.25] based on the 80% rule quoted in Section 2.1. However, Algorithm 2 (Step 2) computes each descriptor as mean_xa / (mean_xb + ε), i.e., a ratio of mean predictions for the two groups. A mean network output (sigmoid probability or logit) is not the same as the positive predicted rate used by the 80% rule, which is a count of positive decisions divided by group size. The paper never states that predictions are thresholded to labels before averaging, nor does it report the threshold. If descriptors are mean scores, then a model lying inside the fair zone could have a predicted positive-rate ratio outside [0.8, 1.25], and the 'best fair' models in Tables 4-9 might not comply with the 4/5 rule. The central claim of identifying models within a legal fairness threshold therefore depends on an unstated and potentially incorrect implementation detail.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a quality-diversity approach, specifically CMA-ME, to evolve a population of neural network classifiers and archive them on a two-dimensional map whose axes are ratios of predicted positive rates between protected groups. From the resulting accuracy-colored map, the authors identify the most accurate model overall and the most accurate model inside a \"fair zone\" defined by the 80% rule ([0.8, 1.25]), and compare their accuracy and Euclidean distance from (1,1). Experiments on a Promotion dataset with several synthetic bias conditions and on the Adult dataset produce heat maps and summary tables. The paper claims that this method visualizes the bias-accuracy trade-off, reveals where biased data concentrate high-accuracy models, and lets users select the best model that satisfies a minimum fairness threshold.","tokens_in":29,"tokens_out":4207,"duration_ms":105274,"significance":"If the operationalization is correct, the paper offers a useful and fairly novel way to exploit model multiplicity: instead of optimizing a single accuracy-fairness objective, it illuminates a landscape of trade-offs and lets a human user pick a model. The use of CMA-ME for bias characterization rather than bias mitigation is a reasonable contribution, and the qualitative patterns in Figures 2-7 are plausible: biased datasets concentrate high-accuracy models in map regions matching the data bias. The paper also provides code and does not fit free parameters to produce its central qualitative claim, so the core visualization idea is not circular. However, the central claim that the fair zone implements the legal 80% rule is threatened by an implementation ambiguity in Algorithm 2, and the paper's quantitative claims rest on single stochastic runs with no uncertainty quantification.","major_comments":[{"comment":"The fairness descriptors are computed as mean_xa / (mean_xb + epsilon) and mean_ya / (mean_yb + epsilon), i.e., ratios of mean network outputs, not ratios of positive predicted rates. The 80% rule quoted in Section 2.1 is defined for selection rates: the number of positive decisions divided by group size. The paper never states that the network outputs are thresholded to labels before averaging, nor does it report any decision threshold. Consequently, a model inside the fair zone [0.8, 1.25] could have a predicted-positive-rate ratio outside that interval, and the \"best fair\" models in Tables 4-9 may not actually comply with the 4/5 rule. Please either threshold the outputs before computing the ratios and report the threshold, or explicitly redefine the descriptors as score ratios and argue why the 80% rule applies to those ratios.","section":"Section 3, Algorithm 2 Step 2"},{"comment":"All reported results appear to come from single stochastic CMA-ME runs. Tables 4-9 give point estimates with no error bars, repeated-seed variation, or statistical significance tests. In particular, the headline correlation of 0.9150 in Section 6 is computed from only six experiments, with no confidence interval or significance assessment. Since CMA-ME is a stochastic search algorithm, specific numbers such as the accuracy drops of 0.0106 to 0.0431 may not be reproducible. Please add multiple runs with different seeds and report variance, or substantially temper the quantitative claims and present the paper as a qualitative proof of concept.","section":"Section 4 and Section 6"},{"comment":"The statement that \"The maximum accuracy obtained by the models in our experiment is slightly lower than what could be attained by a sophisticated conventional learning algorithm\" is not supported by any comparison experiment in the paper. No baseline classifier results are reported for the same datasets and train/test splits. Please either add such a comparison or explicitly label this statement as a conjecture rather than a finding.","section":"Section 5"}],"minor_comments":[{"comment":"The opening sentence has a missing space: \"Fairness,theimpartial treatment\" should read \"Fairness, the impartial treatment\".","section":"Abstract"},{"comment":"There is a duplicated article in \"using a a dataset of promotion decisions\"; it should be \"using a dataset\".","section":"Section 2.1"},{"comment":"The text refers to \"(Row 2 Tables 2 and 3)\" for the male-oversampled case, but the oversampled dataset appears in Row 3 of both tables.","section":"Section 4.3"},{"comment":"There are several typos: \"Hiden layers\" should be \"Hidden layers\" in Table 1; \"Euclidian\" should be \"Euclidean\" in the bullet list of Section 4; and \"withing\" should be \"within\" in the same list.","section":"Table 1 and Section 4"},{"comment":"The cross-biased sample is described as having higher promotion rates for male+old and female+young subgroups, but Table 3 only reports marginal rates, all equal to 0.50. Please report the four intersectional subgroup rates so the reader can verify the intended construction.","section":"Section 4.4"},{"comment":"The fair-zone figure lacks axis labels and a detailed caption. Adding axes, the (1,1) reference point, and a clear description of the shaded region would make the definition of the fair zone easier to check.","section":"Figure 1"},{"comment":"The GitHub repository is referenced without a version identifier or DOI; please archive the code and data-generation scripts at a permanent repository for reproducibility.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the QD/evolutionary computation community and the central visualization idea is promising. The main risk is the mismatch between Algorithm 2's mean-output ratios and the 80% rule's selection-rate definition; this directly affects whether the fair-zone models satisfy the legal criterion. The missing error bars and the six-point correlation claim should also be fixed. I do not see a circularity problem, and the literature positioning relative to prior fairness-aware evolutionary work is adequate. If the authors resolve the thresholding issue and re-run or appropriately qualify the quantitative claims, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid application paper, not a breakthrough. The genuinely new bit is using CMA-ME to illuminate the accuracy-versus-bias-descriptor landscape and to pick the most accurate model inside a fairness zone defined by the 80% rule. That is a useful, transparent alternative to single-objective fairness mitigation, and the paper demonstrates it on six datasets/scenarios with plausible qualitative results. The method section is clear enough that a competent practitioner could reimplement it, and the code is linked.\n\nWhat it does well: the framing around model multiplicity and the idea of letting a user see the whole accuracy-bias frontier before choosing a deployment model is honest and practically motivated. The synthetic-bias scenarios are a reasonable way to show how data bias shifts the high-accuracy region, and the qualitative claim that the maps reflect the data's bias structure is backed by the figures. Credit where due: no circularity, no fitted parameters driving the central claim, and the authors cite the relevant prior work on QD and evolutionary fairness, including the Zhang et al. multi-objective line they are explicitly contrasting with.\n\nSoft spots, in proportion. First, the stress-test concern is real and lands on reading Algorithm 2: the descriptors are computed as mean_xa / (mean_xb + epsilon), i.e., ratios of mean network outputs, not ratios of positive predicted rates after thresholding. The paper's text and the 80% rule are about predicted positive rates, and the fair zone is justified by the 4/5 rule. If the networks output scores and those scores are averaged without thresholding, then a model inside the [0.8, 1.25] zone on mean scores may not satisfy the 80% rule on actual classifications. The paper never states that predictions are thresholded before the ratios are computed, nor reports the threshold. That is a load-bearing ambiguity: the phrase 'assuring in all cases a model that complies with the 4/5 rule' (Section 4.6) is not supported by what Algorithm 2 shows. This needs fixing or explicit defense.\n\nSecond, the quantitative claims are thin: single stochastic runs per scenario, no error bars, and the headline correlation of 0.9150 is computed over six experiments without confidence intervals. That is enough to support a 'suggests' but not a 'shows.' Third, the \"best fair\" accuracy numbers in Tables 4-9 appear to be reported for a single selected individual from the map; there is no indication of variance across runs, which matters for a stochastic search method.\n\nMinor: the paper's own limitation discussion is honest about the small data scope and the accuracy gap versus conventional training; I don't think those are flaws so much as context. The transfer of the 80% rule from selection rates to predicted-outcome ratios is asserted, not argued, but that is a standard move in the fairness literature; the real issue is the mean-vs-threshold ambiguity already noted.\n\nWho this is for: practitioners in HR/credit who want a visualization and a model-selection knob, and QD researchers interested in a new application domain. The paper deserves a serious referee: the idea is novel enough, the method is reproducible in principle, and the fair-zone ambiguity is fixable with a modest revision that clarifies the descriptor computation and adds error bars or repeated runs. I would engage with it after those changes.","headline":"A practical QD-based tool for visualizing accuracy-fairness trade-offs, but the 80% rule implementation is sloppier than the paper admits and the statistics are thin.","tokens_in":10670,"tokens_out":803,"would_cite":true,"duration_ms":10295,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Quality-diversity search maps the accuracy-fairness trade-off in classifiers and finds the best model that satisfies the 80% rule.","keywords":["fairness","bias","quality-diversity","CMA-ME","MAP-Elites","80% rule","talent management","model multiplicity"],"falsifier":"Re-running the pipeline on a dataset with known demographic bias and finding that the most accurate models already lie inside the fair zone would falsify the claim that biased data push the high-accuracy region to the biased side of the map; alternatively, evaluating the fair-zone selections under a different fairness measure such as equalized odds and finding them still unfair would show that the 80%-rule transfer is insufficient.","tokens_in":9720,"feed_emoji":"⚖️","tokens_out":7748,"duration_ms":58998,"temperature":0.7,"pith_summary":"This paper argues that quality-diversity optimization, specifically CMA-ME, can map a whole landscape of classifiers varying in both accuracy and bias instead of returning a single model. Using the ratio of predicted positive rates between protected groups as bias descriptors and accuracy as fitness, the method produces a two-dimensional map in which each cell holds the most accurate model for that bias combination. A fair zone based on the 80% rule from US employment law lets a user read off the most accurate model that still counts as fair. Across six dataset scenarios, the paper reports that biased training data concentrate high-accuracy models in the biased region of the map, and that enforcing the 80% rule costs on average about two percentage points of accuracy while reducing a Euclidean bias distance by about 30%. If this holds, practitioners can quantify the price of fairness and choose a model accordingly instead of accepting whatever bias a single training run produces.","feed_headline":"Fairness costs about two accuracy points, new bias map shows","feed_subtitle":"A quality-diversity map shows classifier accuracy vs bias, so users can enforce the 80% rule with minimal loss.","key_machinery":"The central machinery is CMA-ME (Covariance Matrix Adaptation MAP-Elites), a quality-diversity algorithm that fills a discretized map of behavior descriptors with the highest-fitness solution found for each cell. Here the descriptors are the two group-ratio bias measures and the fitness is classification accuracy; the fair zone is the rectangle where both ratios fall between 0.8 and 1.25, and bias magnitude is measured as Euclidean distance from the perfectly balanced point (1,1). This machinery turns the bias-accuracy trade-off into a spatial object that can be inspected, and it converts model multiplicity into an explicit menu of alternatives.","core_discovery":"The central claim is that CMA-ME can illuminate the space of classifiers by evolving weights of a fixed neural-network architecture while scoring each model on accuracy and characterizing it by two bias descriptors, each the ratio of predicted positive rates between groups in a protected attribute. In datasets with deliberately inserted demographic bias, the most accurate model on the map lands outside the fair zone (for example, a female/male predicted ratio of 0.24 in the male-biased promotion sample), while the most accurate model inside the fair zone has a far higher ratio (0.83) at an accuracy cost of about 4 points in that scenario. In the unbiased samples the best model already lies inside the fair zone, and the accuracy difference is essentially zero. The maps themselves show that the region of high accuracy shifts toward the descriptor values that match the bias in the training data, which the authors read as a visual diagnostic of the dataset's bias.","pith_inferences":["The reported 0.915 correlation between accuracy sacrificed and bias-distance reduced suggests a nearly deterministic trade-off frontier; one could test whether this relationship persists across other datasets, protected attributes, and classifier families.","Because the fair zone is defined by the 80% rule applied to predicted positive rates, the map inherits that rule's limitations: it measures only one form of disparate impact, not equalized odds or calibration, so a decision-maker choosing a model from the map should still check it against other fairness definitions.","The same map machinery could be extended to decision trees (for interpretability) and to large vision models via differentiable quality-diversity methods, generalizing the visualization to high-dimensional, non-tabular inputs.","The approach could also serve as an auditing tool: given a deployed classifier, one could locate its bias descriptors on the map and see whether a better, fairer alternative exists in its neighborhood."],"forward_implications":["Users can see, before deployment, exactly how much accuracy they give up to meet a fairness constraint defined by the 80% rule.","The map acts as a diagnostic of the training data: biased data push the high-accuracy region to the biased side of the map, making the data's bias visible without a separate fairness audit.","In every biased scenario tested, a model inside the fair zone was found, showing that satisfying the 80% rule is compatible with near-best accuracy on these tasks.","The method exploits model multiplicity: many classifiers with similar accuracy but different bias profiles exist, and quality-diversity search makes them explicit deployment options.","The approach scales to networks with tens of thousands of parameters and applies naturally to tabular data such as credit scoring and hiring support."],"supporting_citations":[{"why":"introduces CMA-ME, the specific quality-diversity algorithm the method builds on","marker":"[10]"},{"why":"introduces MAP-Elites, the map-illumination framework CMA-ME adapts","marker":"[22]"},{"why":"provides the pyribs library used to implement the CMA-ME search","marker":"[31]"},{"why":"defines the 80% (4/5) rule that determines the fair zone used for model selection","marker":"[5]"},{"why":"supplies the employee promotion dataset used for the talent-management experiments","marker":"[23]"},{"why":"supplies the Adult dataset, the standard fairness benchmark used as the second test case","marker":"[2]"},{"why":"motivates the set-of-models view by showing that similarly accurate classifiers can differ in fairness","marker":"[3]"}],"fun_headline_variants":["Quality-diversity map reveals classifier bias vs accuracy trade-off","Bias map shows fair classifiers with minimal accuracy loss","Visualizing data bias to balance fairness and accuracy","CMA-ME maps the fairness-accuracy trade-off in classifiers","Find fair classifiers with minimal accuracy cost via bias maps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's fairness judgement rests entirely on the ratio of predicted positive rates between protected groups, with the legal 80% rule applied to that ratio, and the paper never argues that this ratio is a valid proxy for the adverse impact the rule was designed to detect.","fun_headline_variants_meta":{"raw":{"variants":["Quality-diversity map reveals classifier bias vs accuracy trade-off","Bias map shows fair classifiers with minimal accuracy loss","Visualizing data bias to balance fairness and accuracy","CMA-ME maps the fairness-accuracy trade-off in classifiers","Find fair classifiers with minimal accuracy cost via bias maps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2748,"prompt_tokens":889,"completion_tokens":1859,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":1781}},"tokens_in":505,"tokens_out":1859,"duration_ms":12399,"temperature":1.0,"reasoning_tokens":1781,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:40:48.964198+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the pipeline on a dataset with known demographic bias and finding that the most accurate models already lie inside the fair zone would falsify the claim that biased data push the high-accuracy region to the biased side of the map; alternatively, evaluating the fair-zone selections under a different fairness measure such as equalized odds and finding them still unfair would show that the 80%-rule transfer is insufficient.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the pyribs library used to implement the CMA-ME search"},{"cited_title":"In: Proceedings of the 2020 genetic and evolutionary computation conference","cited_arxiv_id":null,"evidence_quote":"introduces CMA-ME, the specific quality-diversity algorithm the method builds on"},{"cited_title":"https://www.ecfr.gov/current/title-29/subtitle-B/ chapter-XIV/part-1607?toc=1 (1978), title 29, Subtitle B, Chapter XIV, Part 1607","cited_arxiv_id":null,"evidence_quote":"defines the 80% (4/5) rule that determines the fair zone used for model selection"},{"cited_title":"https://www.kaggle.com/ datasets/arashnic/hr-ana (2019)","cited_arxiv_id":null,"evidence_quote":"supplies the employee promotion dataset used for the talent-management experiments"},{"cited_title":"In: Proceedings of the 2022 ACM Conference on Fairness, Account- ability, and Transparency","cited_arxiv_id":null,"evidence_quote":"motivates the set-of-models view by showing that similarly accurate classifiers can differ in fairness"}],"review_version":1}