{"id":"28e32c2f-97f0-42e6-b2f4-14211fa01f72","arxiv_id":"2501.14414","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A systematization of the literature showing that dataset size and group distance to the decision boundary, not the choice of DP algorithm, are likely the decisive factors in privacy-induced unfairness.","lead":"This survey organizes research on why differential privacy makes machine learning models more unfair to some demographic groups. It builds a four-layer taxonomy of contributing factors and argues that the size of the training set and how far each group sits from the decision boundary are the most decisive causes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sufficiency claim in Section V-B does not follow from the individual necessity claims; joint sufficiency is asserted without direct evidence.","rationale":"The reader's weakest_assumption correctly identifies the invalid necessity-to-sufficiency inference as the load-bearing weakness. The entire contribution of the causal analysis, and the paper's headline finding, rests on this inference, since the abstract and conclusion emphasize that dataset size and boundary distance are decisive. Without direct evidence of the conjunction, the practical recommendation to focus on lower-layer factors is premature; the paper's own Section VII-A admits the lack of ablation and scaled analyses. I agree with the reader's conditional stance: the taxonomy and systematic review are valuable, but the sufficiency claim must either be softened to a conjecture or supported by a dedicated experiment. Therefore no change to the CONDITIONAL verdict is needed.","tokens_in":22324,"tokens_out":3778,"duration_ms":33911,"concrete_test":"Run a 2x2 factorial experiment on a fixed dataset (e.g., Adult or CelebA) with a fixed DP-SGD configuration: (1) small vs large training subset, and (2) groups with equal vs disparate distances to the decision boundary, the latter manipulated by feature rescaling or a synthetic distribution with controllable class-conditional Gaussians. For each cell, measure group fairness gaps (e.g., accuracy parity, equalized odds) before and after DP. If the small plus disparate cell does not exhibit a significantly larger DP-induced disparity increase than the large plus disparate and small plus equal cells, the joint sufficiency claim is falsified. Include the small plus equal and large plus disparate cells to also test the necessity claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central causal conclusion (Section V-B) is that small dataset size and disparate group distance to the decision boundary are jointly likely sufficient for DP to exacerbate disparities. This is derived solely from the preceding claims that each factor is individually likely necessary. That inference is invalid: necessity of A and necessity of B does not imply sufficiency of A and B, and no reviewed study varies both factors together. Section VII-A concedes that the evidence base is small, non-comparable, and lacks ablation or scaled analyses. The qualitative necessity argument for dataset size is asymptotic (noise decreases with n) and never defines a 'small' threshold, so the sufficiency claim is not crisply falsifiable. The theoretical bounds cited (Mangold et al. O(sqrt(p/n)) disparity bound; Tran et al.'s dependence on the Hessian) are upper bounds on disparity, not lower bounds, so they cannot establish that the conjunction guarantees manifestation. The paper's own hedge with 'likely' does not repair the asserted implication, which remains unsupported by the surveyed evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This SoK paper surveys the literature on how differential privacy (DP) exacerbates performance disparities across demographic groups in supervised machine learning. The authors propose a four-layer taxonomy of contributing factors (DP technique, ML algorithm and hyperparameters, training dataset, underlying distribution), review 20 in-scope studies, and conduct a causal analysis that assigns each factor a judgment of necessity or non-necessity for the exacerbation effect. The paper concludes in Section V-B that, besides DP noise addition, small dataset size and disparate group distance to the decision boundary are the only likely necessary factors, and that their joint presence is likely sufficient for the issue to manifest. The survey also reviews mitigation strategies and identifies open research directions. The central contribution is the taxonomy and the systematization of a heterogeneous body of literature, but the main causal conclusion—the sufficiency claim—is not supported by the evidence presented.","tokens_in":22496,"tokens_out":1876,"duration_ms":19202,"significance":"If the causal conclusions were justified, the paper would redirect research and mitigation efforts away from DP-specific mechanisms toward dataset size and distributional geometry, which would be a valuable contribution to a maturing field. The survey is well organized: it is the first to my knowledge to systematically categorize contributing factors across the ML pipeline, it documents the experimental details of the 20 surveyed studies in Table IV, and it is appropriately cautious in several places, explicitly acknowledging in Section VII-A the lack of ablation studies and the non-comparability of existing experimental settings. The paper also correctly emphasizes under-explored lower-layer factors and the need for multi-notion fairness evaluation. However, the central sufficiency claim exceeds what the surveyed evidence can support, and the causal inference methodology has structural weaknesses that the paper itself partially concedes.","major_comments":[{"comment":"The central conclusion that the simultaneous presence of small dataset size and group distance disparity is 'likely to be sufficient' does not follow from the preceding necessity claims. Necessity of A and necessity of B do not imply sufficiency of their conjunction, and no reviewed study varies both factors together. Section VII-A also concedes that the literature lacks scaled analyses and interaction studies. This inference is load-bearing because it is the paper's main novel conclusion; it should be removed or substantially weakened to a hypothesis, not presented as a causal finding.","section":"Section V-B"},{"comment":"The necessity judgment for small dataset size is not crisply falsifiable because no threshold for 'small' is ever defined, and the supporting theory is asymptotic ('noise decreases with n'). The cited theoretical results in [17] and [22] provide upper bounds on disparity or sufficient conditions for approximate fairness, not lower bounds showing that small n necessarily produces exacerbation. Thus the evidence supports an attenuation effect of large datasets, but not a necessity claim for small ones.","section":"Section V-A, dataset size paragraph"},{"comment":"The claim that noise addition is necessary 'by definition' is presented as a logical argument, but the premise that 'if the decision boundary was not perturbed, there would be no exacerbation effect' conflates the mechanism with the phenomenon. DP-induced disparity could in principle arise through other channels (e.g., clipping-induced gradient misalignment, as reviewed for [11]), and the paper itself treats clipping as non-necessary but contributing. The necessity of noise addition should be reframed as a definitional assumption, not a result of the causal analysis.","section":"Section V-A, noise addition paragraph"},{"comment":"The cross-study causal inference assumes that counterexamples from heterogeneous settings can rule out necessity in general, and that convergence of findings across different fairness notions supports necessity claims. The paper acknowledges this diversity but does not address the risk that differences in datasets, models, and fairness metrics make the findings non-comparable. For example, a counterexample to group-imbalance necessity on one dataset is treated as proof that group imbalance is not necessary for the exacerbation issue at all, without a formal or even informal argument that the counterexample setting is representative. This weakens all the '⊗' and 'H#' judgments in Table I.","section":"Appendix B and Section V"}],"minor_comments":[{"comment":"The description of the Google Scholar search and the selection of 20 papers is not accompanied by a PRISMA-style flow diagram or a list of excluded papers; adding a table of exclusion reasons would improve reproducibility.","section":"Section III"},{"comment":"Equation (3) defines unfairness as M(hθ;A) − M(hθ;B), but it is not specified which direction corresponds to 'unfairness' in the surveyed studies; the text later refers to 'against B' but the sign convention is ambiguous across metrics.","section":"Section II-C"},{"comment":"The table marks cells with checkmarks but does not indicate the magnitude or direction of effects; a reader cannot tell which studies found exacerbation versus mitigation, which would help assess the strength of the causal evidence.","section":"Table IV"},{"comment":"The justification for placing 'group distance to the decision boundary' in the distribution layer rather than the model layer is reasonable, but the paper should acknowledge that this factor is operationally defined only after a model is trained, since the decision boundary is a function of the model; this affects whether it can be considered a pre-existing distributional property.","section":"Section IV-D"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful systematization, but the sufficiency claim is the main advertised contribution and it is unsupported. I believe a revision that removes or reframes the sufficiency conclusion, and honestly presents the causal analysis as hypothesis-generation rather than inference, would be publishable. The taxonomy itself is a genuine contribution. No concerns about novelty or self-citation beyond the normal expectations for a SoK paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a genuinely useful systematization of the PPML-fairness literature, and the four-layer taxonomy is worth borrowing. But the headline causal conclusion — that small dataset size plus disparate boundary distance is jointly sufficient — does not follow from the evidence the authors themselves marshal. The paper is worth a serious referee, but that sufficiency claim needs to be walked back or directly argued.\n\nWhat's new and good: this is the first SoK that focuses specifically on private learning and fairness, not privacy-fairness broadly. The taxonomy (DP technique, model, dataset, distribution) is clean and does real organizational work; Table I's necessity/sufficiency symbols and Table IV's per-study experimental details are exactly what a newcomer needs to navigate a contradictory literature. The mitigation review with its limitations (group label disclosure, repeated querying, computational cost) is practical and well done. I also appreciate that the authors hedge most factor-level claims as 'likely' and explicitly flag in Section VII-A that the evidence base is small and non-comparable.\n\nThe soft spot is Section V-B. The paper concludes that because dataset size and group-boundary distance are each likely necessary, their conjunction is likely sufficient. That inference is invalid: necessity of A and necessity of B tells you nothing about sufficiency of A∧B, and no reviewed study varies both factors together. The qualitative argument for small dataset size is asymptotic and never defines a 'small' threshold, so the sufficiency claim is not crisply falsifiable. The cited theoretical bounds (e.g., Mangold et al.'s O(sqrt(p/n)) disparity bound) are upper bounds on disparity, not lower bounds, so they cannot establish that the conjunction guarantees the effect will manifest. The 'likely' hedge is honest but does not repair the logical gap.\n\nA minor related point: the cross-study necessity inferences are inherently shaky because the 20 studies use different fairness metrics, DP mechanisms, and datasets. The authors know this; it's why they hedge. But the sufficiency step is the one place where the hedge stops matching the evidence.\n\nBottom line: this paper should be read by anyone working on DP and fairness, mainly for the taxonomy and the organized evidence table. The causal conclusions should be treated as hypotheses, not results. I'd suggest a revision that either removes the joint-sufficiency sentence or replaces it with a clearly labeled conjecture plus a sketch of what direct evidence would look like.","headline":"A useful SoK with a clean taxonomy, but the headline joint-sufficiency claim in Section V-B does not follow from the surveyed evidence and should be softened or reworked.","tokens_in":22978,"tokens_out":2115,"would_cite":true,"duration_ms":19575,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Private learning's unfairness traces to two data conditions","keywords":["differential privacy","machine learning fairness","disparate impact","DP-SGD","causal necessity","taxonomy","group fairness"],"falsifier":"Run DP-SGD on synthetic data that independently controls total dataset size and group distance to the decision boundary: if disparate impact appears when the dataset is small but group distances are equal, then group distance is not necessary; if disparate impact does not appear when both a small dataset and unequal distances are present, the joint-sufficiency claim fails.","tokens_in":22120,"feed_emoji":"⚖️","tokens_out":4209,"duration_ms":45722,"temperature":0.7,"pith_summary":"This survey asks why differentially private machine learning hurts some demographic groups more than others. Reviewing 20 studies, the authors build a four-layer taxonomy of contributing factors and analyze which are causally necessary. They conclude that, besides the noise addition intrinsic to differential privacy, only two factors are likely necessary: a small training dataset and differences between groups in distance to the model's decision boundary. The paper further suggests that these two factors together are likely sufficient to trigger disparate impact. If correct, efforts to make private learning fair should focus on data and distribution, not only on the privacy algorithm.","feed_headline":"Private learning's unfairness traces to two data conditions","feed_subtitle":"A survey of 20 studies finds dataset size and decision-boundary gaps, not the DP algorithm, drive disparate harm.","key_machinery":"The central organizing object is a four-layer taxonomy: DP technique, ML algorithm and hyperparameters, training dataset, and underlying distribution. The analysis applies a necessity-and-sufficiency test to each factor in the taxonomy, pooling evidence across the 20 surveyed studies. The key mechanisms are the inverse relationship between dataset size and DP sensitivity, which controls how much noise is needed, and the role of the Hessian and confidence margin in linking group distance to the decision boundary to noise-induced error.","core_discovery":"The paper's central claim is that the exacerbation of performance disparities by differential privacy is not caused by the specific DP mechanism, such as DP-SGD's gradient clipping or the choice of algorithm. The authors argue that DP noise addition is necessary because perturbing the decision boundary is the essence of differential privacy, while clipping and algorithm choice are not necessary. Based on cross-study evidence, they identify dataset size and group distance to the decision boundary as the only other factors likely necessary, and they infer that the joint presence of these two factors is likely sufficient for disparate impact to manifest. This redirects attention from the internals of DP algorithms to the dataset and distribution layers of the machine learning pipeline.","pith_inferences":["Because the model itself helps determine the decision boundary, a testable extension is to vary model capacity on the same data and observe whether boundary-distance differences mediate the disparate effect.","No surveyed study varies dataset size and group boundary distance together, so the sufficiency claim is an extrapolation that a synthetic-data experiment manipulating both factors independently could settle.","If the claim generalizes, fairness audits of private models should measure dataset size and group boundary distances before DP is applied, not only after.","The taxonomy suggests that interventions at the data and distribution layers, such as collecting more samples from underrepresented groups, may be more cost-effective than DP-layer fixes, but this remains untested."],"forward_implications":["Mitigation strategies that only modify the DP algorithm cannot eliminate disparate impact if the dataset and distribution factors remain unfavorable.","For large datasets, DP's disparity amplification is predicted to fade toward the non-private baseline, so small-data domains such as medicine and niche applications need special care.","Measuring or reducing differences in group distance to the decision boundary becomes a practical fairness lever, for example through decision-boundary regularization.","Evaluations of new DP algorithms should report dataset size and group boundary distances, or cross-study comparisons will remain contradictory.","If the joint-sufficiency claim holds, a small dataset with unequal group boundary distances is a red-flag condition for disparate impact regardless of which DP algorithm is used."],"supporting_citations":[{"why":"supplies the original observation that DP-SGD has disparate impact and isolates clipping and noise as contributing components.","marker":"[3]"},{"why":"proves a bound showing DP-induced fairness disparity decreases with dataset size and depends on confidence margin, supporting both necessity claims.","marker":"[17]"},{"why":"decomposes group excessive risk into clipping and noise contributions tied to the Hessian, establishing the mechanism behind decision-boundary distance.","marker":"[10]"},{"why":"shows that enforcing similar decision-boundary distances across groups in private logistic regression mitigates risk disparity.","marker":"[15]"},{"why":"proves that above a dataset-size threshold privacy and approximate equal opportunity can be satisfied simultaneously.","marker":"[22]"},{"why":"provides empirical evidence that a large, high-quality private medical model does not show significant disparate impact.","marker":"[23]"},{"why":"shows that a perfectly balanced dataset still exhibits DP-induced disparity, serving as a counterexample against group imbalance necessity.","marker":"[11]"},{"why":"demonstrates that extreme group imbalance is needed to worsen disparities, weakening the case for group imbalance as a necessary factor.","marker":"[25]"}],"fun_headline_variants":["DP unfairness traced to data size and decision boundaries","Privacy models get unfair due to data traits, not algorithm","Study: dataset conditions, not DP method, drive disparate harm","What makes private learning unfair? The data, not the algorithm"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The causal conclusions pool evidence from 20 studies that use different fairness notions, datasets, and DP algorithms, treating a counterexample in one setting as proof that a factor is not necessary in general and treating the observed necessity of two factors as evidence of their joint sufficiency even though no study varies them together.","fun_headline_variants_meta":{"raw":{"variants":["DP unfairness traced to data size and decision boundaries","Privacy models get unfair due to data traits, not algorithm","Study: dataset conditions, not DP method, drive disparate harm","What makes private learning unfair? The data, not the algorithm"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1323,"prompt_tokens":873,"completion_tokens":450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":382}},"tokens_in":489,"tokens_out":450,"duration_ms":5343,"temperature":1.0,"reasoning_tokens":382,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:09:42.949560+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DP-SGD on synthetic data that independently controls total dataset size and group distance to the decision boundary: if disparate impact appears when the dataset is small but group distances are equal, then group distance is not necessary; if disparate impact does not appear when both a small dataset and unequal distances are present, the joint-sufficiency claim fails.","supporting_citations":[{"cited_title":"Differential privacy has bounded impact on fairness in classification,","cited_arxiv_id":null,"evidence_quote":"proves a bound showing DP-induced fairness disparity decreases with dataset size and depends on confidence margin, supporting both necessity claims."},{"cited_title":"Differentially private empirical risk minimization under the fairness lens,","cited_arxiv_id":null,"evidence_quote":"decomposes group excessive risk into clipping and noise contributions tied to the Hessian, establishing the mechanism behind decision-boundary distance."},{"cited_title":"Achieving differential privacy and fairness in logistic regression,","cited_arxiv_id":null,"evidence_quote":"shows that enforcing similar decision-boundary distances across groups in private logistic regression mitigates risk disparity."},{"cited_title":"On the compatibility of privacy and fairness,","cited_arxiv_id":null,"evidence_quote":"proves that above a dataset-size threshold privacy and approximate equal opportunity can be satisfied simultaneously."},{"cited_title":"Preserving fairness and diagnostic accuracy in private large-scale ai models for medical imaging,","cited_arxiv_id":null,"evidence_quote":"provides empirical evidence that a large, high-quality private medical model does not show significant disparate impact."},{"cited_title":"Disparate impact in differential privacy from gradient misalignment,","cited_arxiv_id":null,"evidence_quote":"shows that a perfectly balanced dataset still exhibits DP-induced disparity, serving as a counterexample against group imbalance necessity."}],"review_version":1}