{"id":"403efebf-df00-40d8-8362-d9cdf0e43dce","arxiv_id":"2505.00802","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Aggregated local explanations from LIME, SHAP, and DiCE can flag group-level unfairness, but the results shift with aggregation method and protected attribute removal, so explanations must be used cautiously as bias detectors.","lead":"This paper tests whether local post-hoc explainability tools, LIME, SHAP, and DiCE, can act as bias detectors for machine learning models. It finds they can surface group-level disparities in feature contributions, but aggregation choices and removal of the protected attribute strongly affect the fairness conclusions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No negative control: every evaluated model violates distributive fairness, so the 'when violated, procedural unfairness appears' pattern cannot be distinguished from a detector that always alarms.","rationale":"I read the paper as an exploratory empirical study, not a formal proof. The strongest claim is about a conditional association: when distributive fairness is violated, aggregated explanations show procedural-fairness signals. The authors are careful about aggregation choices and protected-attribute removal, and they include multiple datasets and models, which is good supporting evidence. However, the absence of any fair-model condition is a gap in the design: the association is consistent with a trivial 'always positive' pattern. This is more fundamental than the 100-instance sampling issue, because the sampling issue affects the precision of the reported gaps, while the missing negative control affects the validity of the central inference. A single fair-model experiment would settle whether the concern lands. The reader's weakest_assumption (sampling representativeness) is real and should be addressed with confidence intervals or a documented sampling procedure, but I would not let it carry the verdict alone. The verdict remains CONDITIONAL because the concern is addressable and does not invalidate the useful empirical observations, but it does mean the central detector claim is not yet fully supported.","tokens_in":22662,"tokens_out":4935,"duration_ms":58534,"concrete_test":"Train a model on the same Adult/AdultCA/AdultLA splits that satisfies a distributive fairness criterion (e.g., reweighted training or post-hoc thresholding to achieve Demographic Parity or Equalized Odds), verify via the z-tests in Table 1 that the PR/TPR/FPR differences are not significant, and run the Section 4 pipeline unchanged (100 instances per group/outcome; LIME, SHAP, DiCE; signed-mean and absolute-mean aggregation; burden). If the protected-attribute contribution gaps in Tables 2 and 6 and the DiCE burden gaps in Table 4 remain as large, the claimed association is not specific to violated distributive fairness and the central claim fails. If those gaps collapse or reverse, the detector claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 6) is that violations of distributive fairness coincide with procedural-fairness signals in aggregated explanations: unequal protected-attribute contributions or disproportionate use of irrelevant features. In every experimental condition—Adult, AdultCA, AdultLA, COMPAS, Random Forest and XGBoost (Tables 1, 8, 9)—the distributive fairness metrics are violated. There is no condition in which Demographic Parity, Equal Opportunity, or Equalized Odds is satisfied. Therefore the reported association is established only on positive examples. Any detector that always returns 'unfair' would reproduce the result, and the observed signal could reflect the presence of the protected attribute in the model or a property of LIME/SHAP/DiCE aggregation rather than distributive unfairness. RQ2 (Section 4.3) partially addresses this by removing the protected attribute, but the resulting models still violate fairness, so it does not provide the missing negative control. The claim as stated is an implication, and the paper provides no evidence about the case where the antecedent is false; without that contrast, the central 'bias detector' conclusion is underdetermined. The reader's sampling concern is valid, but even a perfectly representative sample would not settle this issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a general pipeline for using local post-hoc explanation methods (LIME, SHAP, DiCE) as bias detectors: it generates individual explanations for instances in demographic groups, aggregates them per group, and compares aggregated contributions across groups to derive fairness-related signals. It addresses four research questions: the relationship between distributive fairness metrics and explanation-based 'procedural' signals (RQ1), the effect of removing the protected attribute (RQ2), the impact of alternative aggregation strategies (RQ3), and the trustworthiness of the resulting explanations via AOPC curves (RQ4). Experiments are run on Adult, AdultCA, AdultLA, and COMPAS with Random Forest and XGBoost models. The paper concludes that when distributive fairness is violated, aggregated explanations show similar signs of procedural unfairness, such as unequal contributions of protected attributes or disproportionate use of irrelevant features, and that explanation-based signals are broadly consistent across methods, while cautioning that pipeline design choices matter.","tokens_in":22824,"tokens_out":3686,"duration_ms":37924,"significance":"If the claimed relationship between distributive unfairness and explanation-based procedural signals were established, the pipeline would offer a practical auditing tool for deployed models where outcome labels or fairness metrics are unavailable. The study is empirically broad and thoughtfully designed in several respects: it aligns explanation targets with the definitions of Demographic Parity, Equal Opportunity, and Equalized Odds; it selects one method per explanation family; it includes two model families and multiple datasets; and it explicitly confronts the aggregation-choice problem, showing with Figure 8 that sign-preserving versus absolute-value aggregation yields different fairness conclusions. The critical risk is that the central implication (Section 6) is tested only on models that all violate distributive fairness, so the evidence cannot distinguish the claimed detector from one that always raises an alarm. A second, compounding risk is that group-level aggregated contributions are computed on small samples of unexplained size and without variance estimates.","major_comments":[{"comment":"The central claim, stated in the conclusion as 'when distributive fairness is violated we get similar signs of procedural unfairness, such as unequal contributions of protected attributes across groups or disproportionate use of irrelevant features,' is an implication whose antecedent is true in every experimental condition. In Tables 1, 8, and 9, all reported differences in PR, TPR, and FPR are statistically significant for every dataset, model, and protected attribute, so there is no condition in which Demographic Parity, Equal Opportunity, or Equalized Odds is satisfied. Consequently, the observed association is established only on positive examples, and a detector that always returns 'unfair' would reproduce the pattern. To support the implication, the paper needs at least one negative control: for example, a model trained with a preprocessing or postprocessing debiasing intervention that achieves (approximately) satisfied fairness metrics, or a synthetic dataset constructed so that the fairness metrics hold, with the prediction that the procedural signals disappear or weaken under such a condition. If such an experiment is not feasible, the conclusion should be explicitly weakened to an observation about the tested unfair models rather than an implication linking distributive and procedural fairness.","section":"Section 6 / Tables 1, 8, 9"},{"comment":"The paper generates explanations for 'a representative number of instances,' namely 100 instances per demographic group and outcome category (50 for AdultLA), but does not specify the sampling procedure, the population from which the sample is drawn, or any variance estimate. All group-level comparisons of feature contributions are point estimates without confidence intervals or standard errors. If the sampled instances are not representative of the full groups, the observed group differences in feature contributions, and hence the fairness signals derived from them, could be sampling artifacts. The authors should state how the instances were sampled (e.g., uniform random, stratified by outcome), justify the sample size, and report bootstrap confidence intervals or at least standard errors for the aggregated contributions in Figures 2-5 and Tables 2 and 6.","section":"Section 4.1 / Figures 2-5, Tables 2 and 6"},{"comment":"The AOPC trustworthiness evaluation is performed only on the Adult dataset with the Random Forest model (Figure 9), yet the conclusion in Section 6 states, 'using the AOPC curve, we conclude that the different explanation methods show consistency' and suggests the explanations can be trusted 'to some extent.' A single dataset-model combination is too narrow a basis for a trustworthiness conclusion in a paper that spans four datasets and two model families. The AOPC analysis should be extended to at least one additional dataset or model family, or the trustworthiness conclusion should be explicitly restricted to the Adult/Random Forest setting.","section":"Section 4.4 / Figure 9"},{"comment":"The AOPC evaluation is mildly self-referential: the perturbation order is derived from the same explanation ranking that is being validated. The paper cites criticism of AOPC (references [24,25]) but does not discuss how the circularity between the perturbation ordering and the evaluated explanation ranking affects the interpretation of Figure 9. A short discussion of this limitation, and of why the comparison against a random ranking partially addresses it, would make the trustworthiness argument more careful.","section":"Section 4.4, paragraph 1"}],"minor_comments":[{"comment":"The AdultLA FN Black entry (22.98) and AdultLA TN Female entry (20.64) are an order of magnitude larger than all other values in the same column and are likely typographical errors or anomalies that need explanation; if they are genuine, the paper should comment on why these subgroups exhibit such extreme distances.","section":"Table 4"},{"comment":"Figure 13 is captioned 'when the protected attribute race is removed' but the surrounding text in Appendix A.3 describes Figure 13 as showing removal of the sex attribute; the captions for Figures 13 and 14 appear to be swapped.","section":"Figures 13 and 14 captions"},{"comment":"The two aggregation formulas near the description of RQ3 are both labeled I_abs, although the second formula sums signed contributions without absolute values; using distinct names (e.g., I_abs and I_mean) or explicit descriptions would prevent confusion.","section":"Section 3, aggregation formulas"},{"comment":"For the DiCE columns, the reported values are percentages of feature changes, but the captions do not state the unit; adding 'percent of feature changes' for DiCE columns would increase clarity.","section":"Tables 2 and 6 captions"},{"comment":"The Burden metric definition uses a distance function c(x_i, x'_i) described only as 'some distance metric such as the Euclidean distance'; since the metric is later used in Table 4, the specific distance used for the experiments should be stated explicitly in Section 4.1 or in the definition.","section":"Section 3, Burden metric"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study with a valuable pipeline and a thoughtful alignment of explanation targets to fairness definitions. However, the central 'bias detector' claim is currently underdetermined because every evaluated model violates distributive fairness; a negative control is necessary to support the implication stated in the conclusion. The sampling description also needs to be strengthened before the group-level comparisons can be taken at face value. These are substantive but fixable issues, which is why I recommend major revision rather than rejection. I would also encourage the authors to provide a reproducibility artifact (code and configuration) given that the paper's contribution is a proposed pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the paper is a solid empirical study, but its central claim is only half-supported. Every model they test violates distributive fairness, so they never show what the explanation signals look like when fairness holds. That means the headline conclusion—when distributive fairness is violated, procedural unfairness signals appear—is consistent with an alarm that always goes off. The stress-test note is right: this needs a negative control.\n\nWhat's new and good: they systematically compare LIME, SHAP, and DiCE across four datasets (Adult, two ACS-derived sets, COMPAS) and two model families, and they think hard about aggregation. The protected-attribute-removal results are the most convincing part: when sex or race is dropped, correlated features like Marital Status and Relationship absorb the contribution, which matches the indirect discrimination story. The DiCE finding that protected groups must change more features, including protected ones, is a nice procedural signal. The AOPC check, though only on Adult, at least shows the rankings are not random.\n\nSoft spots, in proportion: the missing negative control is load-bearing. To support the bias-detector claim, they'd need a fair model or a synthetic condition where distributive fairness holds and show the procedural signals disappear. Without that, the implication is underdetermined. Next, the explanation samples are small—100 instances per group/outcome, 50 for AdultLA—with no sampling procedure, variance, or confidence intervals. That weakens the precision of the aggregated attributions, though it likely doesn't reverse the qualitative patterns. Last, AOPC on a single dataset and no code/data release are minor but real for an empirical paper.\n\nI'd send this to referees. The question matters, the design is broadly sound, and the missing negative control is fixable—add a fair baseline and report sampling procedures. The authors know the literature and write clearly. For your own work, cite it for the aggregation and proxy-feature findings, not for the unqualified 'explanations detect bias' claim.","headline":"A useful empirical study of XAI-as-bias-detector whose central claim lacks a negative control: every model tested is unfair, so the detector signal is never checked when fairness holds.","tokens_in":23376,"tokens_out":2757,"would_cite":true,"duration_ms":27995,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Aggregated local explanations can serve as bias detectors, flagging unequal treatment across demographic groups when standard fairness metrics are violated.","keywords":["explainable AI","algorithmic fairness","local post-hoc explanations","feature attribution","counterfactual explanations","bias detection","group fairness","procedural fairness"],"falsifier":"Construct a classifier on a dataset in which the protected attribute is correlated with the outcome but the classifier is trained under a fairness constraint so that demographic parity, equal opportunity, and equalized odds are satisfied; if the pipeline still shows large unequal protected-attribute contributions across groups, the claimed correspondence would be refuted. Conversely, on a synthetic biased model with no outcome disparity, the explanation signal should be absent if it is a reliable detector.","tokens_in":22459,"feed_emoji":"⚖️","tokens_out":5798,"duration_ms":52762,"temperature":0.7,"pith_summary":"Local post-hoc explanations for individual predictions, when averaged within demographic groups, can serve as bias detectors: in the paper's experiments, every dataset where standard distributive fairness metrics were violated also showed procedural unfairness signals in aggregated attributions, such as a protected attribute pushing one group toward unfavorable outcomes or counterfactuals demanding disproportionate changes from protected groups. The authors build and test a pipeline that generates LIME, SHAP, and DiCE explanations for selected instances, aggregates them per group, and compares the group profiles. They report that removing the protected attribute does not erase these signals; the bias moves to correlated features like marital status and relationship. They also show that the choice of aggregation matters and that the explanation methods' feature rankings beat random ordering under AOPC, giving some grounds for trusting them.","feed_headline":"Explanations reveal bias when fairness metrics fail","feed_subtitle":"When standard outcome-fairness metrics fail, group-level explanations expose where the model treats protected groups unequally.","key_machinery":"The mechanism is the aggregated group-level explanation profile. For LIME and SHAP, each instance in a group and outcome subset contributes a signed feature-attribution vector; the pipeline averages these vectors per group, and the difference between protected and non-protected group averages is the fairness signal. For DiCE, the pipeline aggregates counterfactuals by the percentage of feature changes and by the mean Euclidean distance between factual and counterfactual instances (the Burden metric). The AOPC curve is used as a check on whether the underlying per-instance feature rankings are meaningful.","core_discovery":"The central claim is a correspondence between distributive fairness and explanation-based procedural signals: whenever demographic parity, equal opportunity, or equalized odds are violated, group-level feature attribution also shows unequal contributions, typically with the protected attribute positively favoring the non-protected group and negatively affecting the protected group. In the income datasets used (Adult, AdultCA, AdultLA), LIME and SHAP average contributions for sex are negative for women and positive for men across positive, true-positive, and false-positive subsets, with the largest gaps in Louisiana; DiCE shows women and Black individuals needing to change more features, often including sex or race themselves. When the protected attribute is removed from the model, accuracy drops only slightly and fairness violations persist, with contributions shifting to correlated proxies. The paper reads this as evidence that aggregated local explanations can reveal both direct and indirect discrimination, provided the pipeline steps are chosen deliberately.","pith_inferences":["The correspondence could be made quantitative: one could train a meta-classifier that predicts distributive fairness violations from explanation profiles, yielding an automatic fairness audit tool for settings where outcome labels are unavailable.","The reliance on small explanation samples points to adding variance estimates and statistical tests for group attribution differences, which would turn the qualitative pattern into a rigorous audit procedure.","The findings suggest a caution for mitigation: removing protected attributes from a model is insufficient, and explanation-based auditing should be paired with interventions on correlated features.","The approach could extend to multiple protected attributes and intersectional groups, though pairwise comparisons would then need correction for multiple testing."],"forward_implications":["Auditors can run this pipeline on an already-deployed black-box model to get a procedural-fairness read without retraining or accessing the training data.","Removing a protected attribute from the feature set will not by itself erase the fairness signal; explanations can trace where the bias moves, pointing auditors to proxy features.","Choosing the right aggregation is essential: absolute-value aggregations can hide oppositional bias, while signed aggregation or counterfactual burden shows which group is pushed toward unfavorable outcomes.","Agreement among LIME, SHAP, and DiCE on the same group-level pattern strengthens the case that the observed signal is genuine rather than an artifact of one explainer.","The AOPC result suggests the feature rankings used for these comparisons are informative, supporting the use of the pipeline in practice."],"supporting_citations":[{"why":"Supplies LIME, the local surrogate explainer whose per-instance feature contributions are aggregated across demographic groups.","marker":"[46]"},{"why":"Supplies SHAP, the Shapley-value attribution method used for the mean contribution comparisons across groups.","marker":"[36]"},{"why":"Supplies DiCE, the counterfactual generator underlying the feature-change-percentage and Burden aggregations.","marker":"[40]"},{"why":"Provides the ACS PUMS datasets (California, Louisiana) used as modern fairness benchmarks alongside Adult.","marker":"[14]"},{"why":"Defines the AOPC metric used to evaluate the quality of the explanation-based feature rankings.","marker":"[49]"},{"why":"Defines procedural fairness and feature-selection criteria that motivate using attributions as process-fairness signals.","marker":"[21]"},{"why":"Supplies the preprocessing applied to the Adult dataset before training.","marker":"[28]"},{"why":"Adapts AOPC to tabular data, supporting the evaluation choice for the tabular benchmarks.","marker":"[24]"}],"fun_headline_variants":["Explanations catch bias that fairness metrics miss","When fairness metrics fail, explanations reveal bias","Local explanations expose hidden model bias","Explanations as bias detectors: pitfalls and promise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline explains only 100 instances per demographic group and outcome category (50 for AdultLA) and assumes these are representative of the full groups, but no sampling procedure, variance estimate, or confidence intervals are reported, so the group-level attribution differences could be artifacts of the small samples.","fun_headline_variants_meta":{"raw":{"variants":["Explanations catch bias that fairness metrics miss","When fairness metrics fail, explanations reveal bias","Local explanations expose hidden model bias","Explanations as bias detectors: pitfalls and promise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000147,"raw_usage":{"total_tokens":1155,"prompt_tokens":881,"completion_tokens":274,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":218}},"tokens_in":497,"tokens_out":274,"duration_ms":3184,"temperature":1.0,"reasoning_tokens":218,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:33:55.224547+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a classifier on a dataset in which the protected attribute is correlated with the outcome but the classifier is trained under a fairness constraint so that demographic parity, equal opportunity, and equalized odds are satisfied; if the pipeline still shows large unequal protected-attribute contributions across groups, the claimed correspondence would be refuted. Conversely, on a synthetic biased model with no outcome disparity, the explanation signal should be absent if it is a reliable detector.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies LIME, the local surrogate explainer whose per-instance feature contributions are aggregated across demographic groups."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies DiCE, the counterfactual generator underlying the feature-change-percentage and Burden aggregations."},{"cited_title":"Gummadi, and Adrian Weller","cited_arxiv_id":null,"evidence_quote":"Defines procedural fairness and feature-selection criteria that motivate using attributions as process-fairness signals."},{"cited_title":"BASED-XAI: Breaking Ablation Studies Down for Explainable Artificial Intelligence","cited_arxiv_id":"2207.05566","evidence_quote":"Adapts AOPC to tabular data, supporting the evaluation choice for the tabular benchmarks."}],"review_version":1}