{"id":"fa946a77-5d63-41c7-a3ec-7a1486aa4413","arxiv_id":"2506.11627","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A new annotation-free pipeline converts skin pixels into ITA distributions, measures skin-tone differences with a signed distance, and reweights the loss to reduce the correlation between skin tone and model performance.","lead":"This paper proposes a method to measure individual skin-tone fairness in image classifiers without manual skin-color labels, using per-pixel color distributions and statistical distances. The authors also introduce a loss-weighting scheme that uses Bayesian regression to reduce the measured correlation between skin tone and model performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The signed 'Wasserstein' distance in Eq. (3) is not a well-defined distance: it depends on an unseeded, arbitrarily chosen baseline image, so Table 3's correlations and the reported mitigation could be artifacts of that choice.","rationale":"The reader's weakest assumption is the right target and I agree with it. The paper's own text in Section 3.3.1 states that x0 is selected randomly from the validation set, and no seed or baseline is reported; because every downstream quantity is conditional on x0, the central claim is not reproducible. I do not count the miswritten cross-entropy in Eq. (10) as load-bearing, since Algorithm 2 is internally consistent and uses a standard binary cross-entropy; nor do I rest the rejection on the absence of baselines or error bars alone. The distance metric is the hinge: if the baseline-dependence test shows instability, the Table 3 correlation reductions—including the HAM/VGG F1 case that worsened from −0.497 to −0.600—cannot be interpreted as evidence of bias mitigation. If the test shows the correlations are stable across baselines, the concern would be defused; the proposed check is therefore decisive. The verdict remains REJECT because the central supporting quantity is unvalidated and unreported, and the reader's assessment already captures this correctly.","tokens_in":16779,"tokens_out":4566,"duration_ms":49144,"concrete_test":"Re-run the UTKFace EffNet/ResNet/VGG pipeline exactly as described, but repeat the whole experiment for 50 randomly selected baseline images x0 (or, if code is unavailable, recompute D from the extracted ITA histograms with 50 baselines). Record the prior and posterior Pearson correlations of Table 3. If the signed-W1 correlations vary by more than about ±0.1 across baselines, or if any sign flips, the reported bias detection and mitigation are artifacts of the unseeded baseline choice. As a second arm, replace D with the unsigned W1 and with the median-ITA difference; if the qualitative pattern of Table 3 changes, Eq. (3) is not measuring the claimed skin-tone effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (3) is the foundation of the entire pipeline: D(x0, xi) feeds the Bayesian performance estimator (§3.4), the softmax penalty (§3.5), and every correlation in Table 3. As printed it is not a distance: F(A0) and F(Ai) are written as CDF values evaluated at scalars, yet the integrand |F(A0)−F(Ai)| is integrated over dA and then multiplied by sign(Ai−A0); if F(A0) and F(Ai) are instead meant to be the full CDFs of the pixel-ITA distributions, the integral without the sign is W1, but the extra median sign and the random choice of x0 make the measured signed distance a nonlinear function of an unreported random draw. A different choice of x0 changes not just a global shift but the relative ordering of distances, because W1 to a fixed reference is not monotone in the median across arbitrary histograms. Since no seed or baseline is reported, the prior-training correlations, the Bayesian regressions in Fig. 4, the penalty weights, and the posterior changes in Table 3 are not reproducible. The conclusion that latent within-category bias was detected and mitigated therefore rests on an unvalidated, baseline-dependent quantity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes an annotation-free pipeline for measuring and mitigating skin-color bias in image classifiers. Skin pixels are converted into per-image ITA distributions; a signed \"Wasserstein\" distance from a randomly chosen baseline image is computed; a Bayesian polynomial regression predicts validation performance as a function of this distance; and a softmax-weighted cross-entropy loss penalizes samples whose predicted performance is low. Experiments on UTKFace, CelebA, and HAM with EfficientNet, ResNet50, and VGG16 report correlations between the distance and F1/accuracy before and after the proposed reweighting. The paper claims to detect latent within-category bias and to significantly reduce the correlation with the target evaluation metric.","tokens_in":17076,"tokens_out":6922,"duration_ms":66903,"significance":"If the method worked as claimed, it would address a real gap: most fairness work on skin tone uses coarse categories or requires annotations, while this paper models skin tone as a per-image distribution and targets individual-level fairness. The motivation is clear, the distribution-based treatment of skin color is a sensible direction, and the authors explicitly list limitations in Section 6.2. However, the central distance measure is not correctly defined, the loss equation contains an error, and the experimental evidence is not sufficient to support the headline claims. The contribution is therefore not yet established.","major_comments":[{"comment":"This is not a Wasserstein distance and is not well defined as written. If F denotes the CDF function of a distribution, then F(A0) and F(Ai) are numbers, the integrand |F(A0) − F(Ai)| is constant in A, and the integral over R diverges; if F(A0) is intended to denote an entire CDF, the notation is undefined. Even interpreting Eq. (3) as W1(A0,Ai)·sign(median(Ai)−median(A0)), the result depends on the randomly selected baseline image x0, and the relative ordering of distances to different references is not preserved under a change of baseline. Since D feeds the Bayesian estimator (§3.4), the loss penalty (§3.5), and all correlations in Table 3, and since no seed or baseline is reported, the experimental results are not reproducible. The authors should define a canonical baseline (or average over baselines) and report sensitivity to that choice.","section":"§3.3.1, Eq. (3)"},{"comment":"The cross-entropy term is written with the same log xn for both yn and 1−yn, so the displayed formula is not a valid binary cross-entropy loss. Algorithm 2 implements a correct per-sample BCE, but Eq. (10) cannot be used to reproduce the training. Please correct Eq. (10) and state precisely how the softmax penalty σ and scalar α enter the per-sample and per-batch losses.","section":"§3.5, Eq. (10)"},{"comment":"The evidence for mitigation is statistically weak. Correlations are reported without confidence intervals, significance tests, or repeated runs; the batch size for performance estimation is 1% of validation with no justification. One combination, HAM/VGG F1, moves from −0.497 to −0.600, i.e., it worsens, and this is not commented on. In addition, the Bayesian performance estimator is fit to the base model's validation scores on the same distance D that the loss then penalizes, so the reported reduction in the D–performance correlation is partly an artifact of optimizing the evaluation metric. An independent fairness evaluation (e.g., a held-out test set with a separate estimator, or comparison with standard reweighting and adversarial baselines) is required.","section":"§5, Table 3"},{"comment":"Several free parameters (polynomial degree g, penalty start epoch pe, penalty weight α, batch size, preprocessing thresholds) are tuned per dataset and model without a selection rule or sensitivity analysis. Because the method's generality claim depends on these choices, the authors should report how the results vary with these parameters or provide a principled selection procedure.","section":"§3.4 and Tables 2/4"}],"minor_comments":[{"comment":"The text refers to the \"Washer Stain Distance\" in two places; this should be \"Wasserstein distance\".","section":"§3.2 and §3.3.1"},{"comment":"The caption contains the typo \"Performancce\" and should be corrected.","section":"Table 4 caption"},{"comment":"Table 1 includes an ISIC2024 column, but the experimental section lists only three datasets and does not describe ISIC2024.","section":"Table 1"},{"comment":"The x-axis convention (0 for baseline, positive for lighter, negative for darker) should be stated in the caption or on the axis label, since the current description is not self-contained.","section":"Figure 4"},{"comment":"In every row of Tables 2 and 4, the validation F1 score equals the validation accuracy; this is unusual and should be explained or checked.","section":"Tables 2 and 4"},{"comment":"The source-code link is given as \"this GitHub repository\" without an actual URL, so the code is not locatable from the manuscript.","section":"Appendix C"},{"comment":"The header contains the placeholder \"Editor: My editor\"; this should be removed before submission.","section":"Header"}],"recommendation":"reject","confidential_remarks":"The technical issues are decisive: Eq. (3) is the foundation of the pipeline and is not a well-defined distance, and Eq. (10) is not a valid cross-entropy loss. These are not presentation problems; repairing them requires redefining the method and rerunning the experiments, which is beyond a minor revision. The editorial issues (placeholder editor, missing code URL) further suggest the manuscript is not ready for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a genuinely interesting idea—using per-pixel ITA values as a distribution to capture skin-tone nuance without annotation—but the central distance metric is not actually a Wasserstein distance, the cross-entropy formula in Eq. (10) is wrong, and the experiments lack the baselines and error bars needed to support the claims. I would not accept it as is, but I would not dismiss the general direction.\n\nThe good part is the conceptual shift: instead of collapsing skin tone to a single average ITA or a category, the authors build a distribution of ITA values over skin pixels and then measure a signed distance between distributions. That is a reasonable way to think about within-group variation, and it seems new to combine this with a Bayesian regression–weighted loss for mitigation. The idea of detecting bias that category-based fairness misses is worth taking seriously.\n\nThe problems are load-bearing. Eq. (3) defines the distance as an integral of |F(A0) − F(Ai)| dA times sign(Ai − A0). As written, this is not the Wasserstein distance—the Wasserstein-1 distance between two distributions with CDFs F0 and Fi is ∫|F0(t) − Fi(t)| dt, without a sign factor, and it does not require a reference point. The sign and the arbitrary reference baseline x0 (chosen randomly, no seed reported) mean the correlations in Table 3 and the penalty weights in the mitigation are artifacts of an unreported random draw. A different baseline changes relative orderings, not just a global shift. Eq. (10) is also incorrect: both terms in the cross-entropy use log xn, which is not a valid cross-entropy. The experiments don’t compare to any baseline method, report no error bars or significance tests, and one result (HAM VGG F1) worsened from −0.497 to −0.600 without comment. Appendix C says the code is “available here” but no link appears.\n\nOn the citation pattern: the literature review is adequate and the claim of being the first annotation-free method is overstated, since Chiu et al. and others have worked in that direction, but this is a minor sin.\n\nWhere does that leave us? The paper is not ready for publication, but it has a seed of a plausible method. A major revision that replaces Eq. (3) with a properly defined distance (e.g., signed W1 with a deterministic baseline or a rank-based measure), fixes the loss, adds baselines and error bars, and reports seeds could make a contribution. The paper is for readers who study algorithmic fairness in computer vision, especially those interested in continuous sensitive attributes and annotation-free evaluation. I’d send it to a serious referee rather than desk-reject, because the flaws are concrete and fixable, and the underlying question matters.\n\nRecommendation: reject the current version, but invite a revise-and-resubmit with a corrected measure and proper experiments.","headline":"Per-pixel ITA distributions are a promising idea, but Eq. (3) is not a Wasserstein distance and the experimental evidence is too weak; this needs a major revision, not acceptance.","tokens_in":17592,"tokens_out":3511,"would_cite":false,"duration_ms":31453,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that individual-level skin-tone bias in image classifiers can be detected and mitigated without annotations, using per-pixel skin-tone distributions and a Bayesian-regression-weighted loss.","keywords":["fairness","skin color","sensitive attributes","Individual Typology Angle","Wasserstein distance","Bayesian regression","loss reweighting","image classification"],"falsifier":"Recompute the signed distance $D(A_0,A_i)$ for the same validation images using several different randomly chosen baseline images $x_0$; if the rank ordering of images by distance changes materially across baselines, or if the correlation values in Table 3 no longer reproduce, the reported bias signal is an artifact of baseline choice rather than a stable property of skin tone.","tokens_in":16578,"feed_emoji":"⚖️","tokens_out":11658,"duration_ms":95991,"temperature":0.7,"pith_summary":"This paper argues that fairness for image classifiers should treat skin color as a continuous, per-pixel tensor rather than a predefined category such as 'light' or 'dark'. It proposes a pipeline that converts the skin pixels in each image into a probability distribution of Individual Typology Angle (ITA) values and measures the signed statistical distance from a baseline image. A Bayesian regression model then predicts how model performance (F1 score or accuracy) varies across that distance, and the training loss is reweighted to penalize skin tones where performance is predicted to be worse. On three image datasets, the method exposes performance differences hidden within a single skin-tone category and, in most tested settings, reduces the correlation between skin-tone nuance and performance. The paper asserts this is the first annotation-free technique to evaluate and mitigate individual-level skin-tone bias in image classification.","feed_headline":"Skin-tone bias in image classifiers tamed without annotations","feed_subtitle":"Pixel-level skin-tone distance plus Bayesian loss reweighting cuts the correlation between skin color and accuracy.","key_machinery":"The central mechanism is the signed distance measure of Eq. (3), applied to per-image ITA (Individual Typology Angle) distributions computed from skin pixels: $D(A_0,A_i)=\\int_{-\\infty}^{\\infty}|F(A_0)-F(A_i)|\\,dA\\cdot\\operatorname{sign}(A_i-A_0)$, where ITA is a CIELab-based skin-tone angle. This distance turns a tensor of skin pixels into a one-dimensional directional quantity that says how far and in which direction an individual's skin-tone distribution sits relative to a baseline image. A Bayesian polynomial regression then maps that distance to expected per-batch performance (F1 score or accuracy), and the training loss is reweighted by the softmax of the inverted performance estimate, with the penalty switched on after a chosen epoch. The distance measure, the regression, and the reweighted cross-entropy loss together carry the argument.","core_discovery":"The paper's central claim is that latent, individual-level skin-tone bias in image classifiers can be detected and mitigated without annotations, by treating skin color as a tensor of pixel values, converting it into a probability distribution, and measuring a directional statistical distance from a baseline. On this view, conventional group-fairness checks that compare coarse categories like 'light' and 'dark' can report parity while hiding within-category performance differences. The results section reports that biases latent within the same category could be detected, and that weighting the loss function according to nuanced skin-color differences significantly reduced the correlation with the target evaluation metric. The paper also asserts that no prior research has achieved a fair model without annotations using only detected skin color nuances.","pith_inferences":["Editorial inference: because Eq. (3) depends on a randomly chosen baseline image, the paper leaves open a natural robustness check: average distances over many baselines and re-run Table 3; material changes in rank order would make the fairness signal baseline-dependent.","Editorial inference: the same distance-plus-Bayesian-reweight pipeline could be extended to other continuous sensitive attributes expressed as images, such as apparent age or body shape, not just skin color.","Editorial inference: the method's applicability is bounded by the quality of the skin-pixel detector, so combining it with learned skin segmentation could extend it to datasets the paper lists as out of scope, such as images with tiny or multiple skin regions."],"forward_implications":["Group-fairness metrics that coarse-categorize skin tone can certify parity while hidden individual-level bias remains; this method claims to reveal and reduce that within-category bias.","The same signed-distance construction extends Equal Opportunity, Demographic Parity, and Equalized Odds to continuous sensitive attributes.","Fairness evaluation no longer requires human skin-tone annotations, only a skin-pixel detector and a validation set for the Bayesian prior.","In most tested model-dataset combinations, posterior training weakens the correlation between skin-tone distance and F1/accuracy, with the largest reported reductions on the HAM skin-lesion dataset."],"supporting_citations":[{"why":"Defines Equal Opportunity, the group-fairness criterion the paper generalizes to continuous sensitive attributes.","marker":"Hardt et al. (2016)"},{"why":"Introduces individual fairness as 'similar individuals should be treated similarly', which the paper operationalizes for skin-tone distributions.","marker":"Dwork et al. (2012)"},{"why":"Documents accuracy disparities across skin tones in commercial classifiers, motivating the need for skin-tone fairness evaluation.","marker":"Buolamwini and Gebru (2018)"},{"why":"Shows experts disagree on skin-tone grouping, supporting the paper's argument against rigid categorization.","marker":"Groh et al. (2022)"},{"why":"Supplies ITA as the skin-tone measure whose pixel-level distribution the method constructs.","marker":"Kinyanjui et al. (2019)"},{"why":"Supplies the geometric motivation for using Wasserstein distance to capture distribution shifts.","marker":"Cai and Lim (2022)"},{"why":"Prior optimal-transport fairness work that the paper distinguishes by focusing on predicted-score distributions rather than input distributions.","marker":"Chiappa and Pacchiano (2021)"},{"why":"Evidence that human skin-tone annotations are subjective, supporting the annotation-free design.","marker":"Barrett et al. (2023)"},{"why":"Provides the HAM skin-lesion dataset used for the experiments and the validation prior.","marker":"Tschandl et al. (2018, 2020)"}],"fun_headline_variants":["No-annotation fairness fix for skin-tone bias","Bayesian method tames skin-tone bias in image AI","Hidden skin-tone bias in classifiers found and fixed","Unannotated skin-tone fairness via Bayesian weighting","Tensor data reveals and reduces skin-tone bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the signed distance in Eq. (3) is a meaningful measure of skin-tone difference, even though it is not the true Wasserstein distance and depends on which image is randomly chosen as the baseline, because if this quantity is meaningless the detected correlations and the reweighted-loss mitigation collapse.","fun_headline_variants_meta":{"raw":{"variants":["No-annotation fairness fix for skin-tone bias","Bayesian method tames skin-tone bias in image AI","Hidden skin-tone bias in classifiers found and fixed","Unannotated skin-tone fairness via Bayesian weighting","Tensor data reveals and reduces skin-tone bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1341,"prompt_tokens":886,"completion_tokens":455,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":378}},"tokens_in":502,"tokens_out":455,"duration_ms":4492,"temperature":1.0,"reasoning_tokens":378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:03:38.382974+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the signed distance $D(A_0,A_i)$ for the same validation images using several different randomly chosen baseline images $x_0$; if the rank ordering of images by distance changes materially across baselines, or if the correlation values in Table 3 no longer reproduce, the reported bias signal is an artifact of baseline choice rather than a stable property of skin tone.","supporting_citations":[{"cited_title":"Skin deep: Investigating subjectivity in skin tone annotations for computer vision benchmark datasets","cited_arxiv_id":null,"evidence_quote":"Evidence that human skin-tone annotations are subjective, supporting the annotation-free design."},{"cited_title":"The ham10000 dataset, a large col- lection of multi-source dermatoscopic images of common pigmented skin lesions","cited_arxiv_id":null,"evidence_quote":"Provides the HAM skin-lesion dataset used for the experiments and the validation prior."}],"review_version":1}