{"id":"95f87a02-a00e-4057-860a-527a06991549","arxiv_id":"2412.18904","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"FedCFA replaces local latent factors with global average features to generate counterfactual samples, improving federated global model accuracy under heterogeneous data.","lead":"FedCFA trains federated models by generating counterfactual samples that replace local data features with features from a global average dataset, aiming to weaken spurious correlations that skew the aggregated model. The paper reports accuracy gains and faster convergence over six FL baselines across six datasets, with the largest gains under strong non-IID partitions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Global average dataset (Eqs. 3–4) constructed by pixel-space averaging likely yields off-manifold images and soft labels; counterfactual samples (Eqs. 6–8) inherit this, and the pointwise-to-Wasserstein step (Eqs. 16–17) is invalid, leaving the central alignment claim unproven.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing gap: the global average dataset, constructed by averaging raw inputs, may not approximate the global data distribution, and the counterfactual samples built from it may be OOD. This is the same concern I find most critical. The proof section compounds the issue by jumping from a pointwise replacement inequality (Eq. 16) to a Wasserstein distance inequality (Eq. 17) without any coupling argument; even a true pointwise reduction for each sample paired with the same global-average feature does not imply a reduction in optimal-transport cost between the full distributions. Because Lpos and Lneg are the only novel components beyond a standard classification loss, the entire mechanistic justification of FedCFA rests on the validity of these counterfactual samples and the distribution-alignment claim. If those samples are OOD blurry mixtures, the paper does not demonstrate why the method works, even if the empirical tables are accurate. The concern is severe enough to keep the verdict CONDITIONAL: the empirical results, if reproducible, may still reflect a useful method, but the manuscript as written does not establish the proposed mechanism. I therefore do not move the reader's verdict. The concrete test of measuring feature-space Wasserstein distance before and after transformation directly checks whether the claimed alignment actually occurs, which would settle whether the concern lands.","tokens_in":16066,"tokens_out":8627,"duration_ms":80228,"concrete_test":"Using a trained FedCFA encoder, compute the empirical Wasserstein distance (or MMD) in feature space between the global factor distribution P_Fg and (a) original local factors P_Fk and (b) counterfactually transformed local factors P_F'k on a held-out set of CIFAR10 Dir60(0.6) clients. If W(P_Fg, P_F'k) does not decrease relative to W(P_Fg, P_Fk), then Eq. 14 is contradicted and the central alignment claim fails; if it decreases consistently, the off-manifold concern is empirically resolved. Visual inspection of the generated global average images and counterfactual samples should accompany this check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The method's novel training signal depends on a 'global average dataset' built by averaging raw pixels within random subsets (Section 'Global Average Dataset Construction', Eqs. 3–4). For natural images, the mean of a set of images is a blur that lies off the data manifold, and averaging one-hot labels yields soft distributions that do not correspond to any real class. The counterfactual transformations (Eqs. 6–8) replace local factors with factors extracted from these averaged images, so the resulting positive/negative samples are likely out-of-distribution rather than global-distribution-aligned. The central claim that local distributions are 'aligned with the global' therefore rests on an unverified premise: that pixel-space averages faithfully approximate the global data distribution. The proof in the 'Proof' section does not rescue this. Eq. 16 is asserted without derivation, and even if each replaced coordinate moves closer to a fixed global-average point, the Wasserstein distance between the full factor distributions need not decrease (Eq. 17) because optimal transport couplings are not the fixed pointwise pairing. This is load-bearing: Lpos and Lneg are the only novel training signals beyond the classification loss, and if the counterfactual samples are OOD, the reported improvements in Tables 1–3 may stem from implicit regularization or other effects rather than the claimed Simpson's Paradox mitigation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FedCFA, a federated learning framework that aims to mitigate Simpson's Paradox in model aggregation. The method constructs a global average dataset by averaging raw inputs (and labels) across clients, extracts factor representations with an encoder, and generates counterfactual positive/negative samples by replacing selected local factors with features of the global average data. A factor decorrelation (FDC) loss based on Pearson correlation is added to encourage factor independence. The authors claim that these counterfactual samples align local distributions with the global distribution and show improved accuracy and convergence over six baselines on six datasets under IID and Non-IID partitions. The paper includes a proof attempt via Wasserstein distance, extensive tables of classification accuracy, ablations of the counterfactual modules, and a synthetic Simpson's Paradox dataset experiment.","tokens_in":16371,"tokens_out":4910,"duration_ms":49643,"significance":"If correct, FedCFA would be a useful new approach to a real and under-studied failure mode in federated learning: Simpson's Paradox where a global model aggregated from client models contradicts the true global distribution. The paper gives a concrete algorithmic recipe, compares against six baselines on six datasets, provides ablations of each loss component, and constructs a dedicated Simpson's Paradox benchmark. These are valuable contributions. However, the central distribution-alignment claim is supported by a flawed proof and by experiments whose reproducibility is limited by unreleased code, single-run rounds-to-target results, and hyperparameter selection on the same datasets. The empirical improvements are plausible but not yet established at the standard needed for publication.","major_comments":[{"comment":"The Wasserstein proof is invalid. Eq. (16) does not follow from the construction in Eqs. (6)-(7): only the selected factors are replaced, so for any j not in the key-factor set, ||fg - f'_k,j|| = ||fg - f_k,j|| and the strict inequality is false. Even for replaced factors, the equality f'_k,j = fbar_g,j holds only for the specific global-average factor used, not for every element of Fbar_g. Moreover, the step from a pointwise norm inequality to an inequality between infima over different couplings in Eq. (17) is not justified: the optimal transport couplings for (P_Fbar_g, P_F'_k) and (P_Fbar_g, P_F_k) are generally different, so a pointwise bound cannot be transferred to the infimum. Since this proof is the only theoretical support for the paper's central claim that counterfactual transformation brings local data closer to the global distribution, it must be repaired or replaced by direct empirical distribution-distance measurements (e.g., MMD or Wasserstein estimates before and after counterfactual transformation).","section":"Proof (Eqs. 13-17)"},{"comment":"The method assumes that a pixel-space average of raw inputs and a corresponding average of labels faithfully approximate the global data distribution. For natural images, the mean of a set of images is typically off the data manifold and can be uninterpretable, and averaging one-hot labels produces soft label vectors that do not correspond to any actual class. The counterfactual samples in Eqs. (6)-(8) inherit these properties, so the claim that Lpos and Lneg align local distributions with the global distribution rests on an unverified premise. The paper should provide evidence that the constructed global average dataset is a faithful proxy, for example by visualizing generated counterfactual samples, measuring distribution distance before/after transformation, and ablating the average-dataset size B and subset size n.","section":"Global Average Dataset Construction (Eqs. 3-4)"},{"comment":"Several hyperparameters (lambda_pos, lambda_neg, lambda_corr, topk, hook, and the global average dataset size B) are tuned on the same datasets for which the final results are reported, with best results selected. Figure 6 and Table 5 show that accuracy varies by several points with these settings; for instance, Table 5 reports an 80.01 accuracy at topk=24 but 78.53 at topk=8, and hook=2 gives 80.01 while hook=4 gives 75.31. Without a fixed train/validation/test split for hyperparameter selection or a nested protocol, the margins in Tables 1-3 may reflect selection bias. The authors should specify the exact protocol, report the chosen configuration, and give results on held-out test partitions across multiple seeds.","section":"Experiments: hyperparameter selection and validation protocol"},{"comment":"No code or data are released, and Table 3 reports rounds-to-target without error bars or repetition counts. The empirical claim is the main support for the method, so the paper should include code or detailed pseudocode, dataset partition metadata, the number of random seeds, and confidence intervals for all headline numbers, including Table 3 and Figure 4. Table 3 should also clarify how the target accuracies were set before running baselines, since targets appear close to FedCFA's achieved values.","section":"Experiments: reproducibility and statistical reporting"}],"minor_comments":[{"comment":"The construction of Yneg as a weighted average of the local label and the global average label is not explained; if Ybar_g is a soft label derived from averaged one-hot vectors, the resulting Yneg is a convex combination that may not correspond to any valid class. Please clarify how the classification loss Lneg handles these soft labels.","section":"Eq. (8)"},{"comment":"The baseline is called FedPVR in Section 'Baseline Methods' and FedPRV in Table 1; please make the naming consistent.","section":"Baseline list and Table 1"},{"comment":"The Simpson's Paradox experiment reports only a learning curve without final accuracy values, error bars, or a comparison with FedProx/SCAFFOLD; quantitative results and repetitions should be provided.","section":"Figure 4"},{"comment":"The server receives local average datasets (Xbar_k, Ybar_k) from clients and broadcasts the global average dataset. The privacy implications of sharing data-derived averages are not discussed; in a federated setting this requires at least a brief treatment.","section":"Algorithm 1, lines 13-16"},{"comment":"The summation index |F0| appears in the denominator and range of Eq. (10), while the factor-set size is elsewhere denoted |F| or |F0|; please standardize the notation.","section":"Eq. (10)"},{"comment":"The reduction from W(P_Fg, .) to W(P_Fbar_g, .) via 'Since we construct a global average dataset to approximate the global data distribution' is asserted without a quantitative approximation bound; if this step is retained, it needs a precise statement.","section":"Proof, sentence before Eq. (15)"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an interesting and timely problem, and the proposed framework is concrete enough to be worth pursuing. However, the current manuscript is not publishable in its present form: the Wasserstein proof is unsound, the global-average-data assumption is not validated, and the empirical evaluation lacks code, data, and a rigorous hyperparameter protocol. These issues are load-bearing for the central claim but are fixable within the manuscript's scope if the authors can either provide a correct theoretical argument or reframe the claim as empirical and support it with distribution-distance measurements and a fully reproducible evaluation. I would recommend requesting a major revision rather than rejecting, because the underlying idea has merit and the reported gains, if confirmed, would be a useful contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FedCFA is a genuinely different twist on heterogeneity: instead of aligning client and server models, it replaces selected latent factors with features from a global average dataset, then trains on the resulting counterfactual samples. That is not FedMix with a new name. FedMix mixes at the input level; FedCFA operates on factors, and the FDC decorrelation loss is a sensible addition. The experimental picture is consistent: on CIFAR10/100, Tiny-ImageNet, and the Simpson's-paradox MNIST setup, it beats six baselines in accuracy and often in rounds-to-target. The ablations are real, and the t-SNE visualizations support the claim that the counterfactual samples pull local features toward global ones. If the numbers reproduce, this is a useful contribution to federated learning.\n\nThe soft spots are the usual ones, but two are load-bearing. The 'Proof' section does not prove anything: Eq. 16 asserts the pointwise inequality and Eq. 17 jumps to a Wasserstein ordering. The optimal transport coupling is not the pointwise pairing, so the conclusion does not follow. I would be surprised if this is fixable as stated. The second is the global average dataset. Averaging raw pixels and one-hot labels gives blurry images and soft labels that are probably off the data manifold. If the counterfactual samples are out-of-distribution, then the gains may come from regularization, not from Simpson's paradox mitigation. The paper's own Figure 4 (the MNIST paradox experiment) is the most direct evidence for the mechanism, and it only compares three methods with no error bars. Table 3 also lacks error bars. No code or data is released, and hyperparameters like lambda_pos, lambda_neg, lambda_corr, topk, and hook are tuned on the same datasets where best results are reported. That is a reproducibility problem, not a fatal one.\n\nOn balance: the idea is worth refereeing. I would not desk-reject this. But I would not accept it in current form either. Send it out with strong reviewers; ask for code, for error bars on the rounds table, and for either a valid proof or an explicit statement that the Wasserstein argument is heuristic. Also ask for an analysis of whether the global average images are on-manifold, because that determines whether the stated mechanism is what is actually working.","headline":"A plausible empirical method for heterogeneous FL with a broken theoretical justification; worth refereeing, but only with code and a rewritten proof.","tokens_in":16926,"tokens_out":2213,"would_cite":false,"duration_ms":20917,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedCFA uses counterfactual generation to align local data with the global distribution, mitigating Simpson's Paradox in federated learning.","keywords":["Federated learning","Simpson's paradox","Counterfactual learning","Data heterogeneity","Non-IID data","Factor decorrelation","Contrastive learning","Model aggregation"],"falsifier":"Compute the Wasserstein distance W(P_{F̄g}, P_{F'_k}) and W(P_{F̄g}, P_{F_k}) with the trained encoder on a natural-image dataset; if the post-counterfactual distance is not smaller, or if human raters cannot recognize the class of the generated counterfactual images, the claimed distribution-alignment mechanism does not hold.","tokens_in":15829,"feed_emoji":"🤖","tokens_out":5060,"duration_ms":44782,"temperature":0.7,"pith_summary":"The paper argues that federated learning's standard averaging can fall victim to Simpson's Paradox: a trend that holds in each client's local data disappears or reverses in the aggregated global model, so the global model encodes wrong feature-label associations. The authors propose FedCFA, which builds a small global-average dataset on the server, sends it to clients, and uses it to generate counterfactual positive and negative samples for each local batch. Replacing a sample's low- or high-gradient latent factors with the corresponding factors from global-average data pulls each client's training distribution toward the global one and teaches the model to ignore spurious associations. If the claim is right, the same framework that fixes Simpson's Paradox also improves accuracy and convergence speed under both Non-IID and IID partitions, on image and text datasets.","feed_headline":"Counterfactual samples cure Simpson's paradox in federated learning","feed_subtitle":"Replacing local features with global-average ones lifts accuracy and cuts communication rounds.","key_machinery":"The central mechanism is the global average dataset D̄g = {X̄g, Ȳg}, built by each client averaging B random subsets of its local data and the server aggregating those averages; this dataset serves as the reference distribution for global data. A counterfactual transformation module extracts latent factors with an encoder, selects the top-k low- or high-gradient factors using decoder gradients, and replaces them with the corresponding factors from the global average data, producing positive samples that keep the original label and negative samples that receive a weighted counterfactual label. A factor decorrelation (FDC) loss penalizes the mean absolute Pearson correlation coefficient between factor variables, forcing the encoder to produce more independent factors so that the counterfactual samples cleanly isolate the features being manipulated.","core_discovery":"The paper's central claim is that Simpson's Paradox in federated aggregation can be mitigated by counterfactual data generation guided by global average data, rather than by aligning client models to a global model that may itself be untrustworthy. FedCFA replaces critical local latent factors with factors extracted from a global average dataset, producing positive and negative counterfactual samples that make each client's local distribution closer to the global distribution. The authors further claim that adding a factor decorrelation loss makes the extracted factors more independent and improves counterfactual sample quality. Experiments on six datasets report that FedCFA achieves higher global model accuracy than six baselines after 500 communication rounds and reaches target accuracy in fewer rounds.","pith_inferences":["The Wasserstein-distance proof assumes that every replaced factor moves closer to the corresponding global-average factor; in practice, on natural images, some replacements may land off the data manifold, so the observed gains could partly come from a regularizing effect rather than from pure distribution alignment.","Because the global average dataset is shared with all clients, it communicates aggregate statistics about the training data; a privacy-focused extension would need to perturb or distill this shared reference before sending it to clients.","A testable extension is to construct the global reference in latent space or from per-class prototypes instead of pixel-space averages; that would likely improve counterfactual quality on datasets with large intra-class variation, where pixel averages quickly become unrecognizable."],"forward_implications":["On CIFAR10 and CIFAR100 under six partitions, FedCFA reports the highest top-1 accuracy after 500 rounds, with the largest reported margin being 7.75 percentage points over the best baseline on CIFAR100 with Dir100(0.6).","In CIFAR10 target-accuracy tests, FedCFA reaches the targets in 375 to 453 rounds while most baselines fail to reach them within 1000 rounds, implying lower communication cost.","Ablation studies show that both counterfactual modules contribute to the accuracy gain and that their combined effect exceeds the sum of the individual effects, and adding the FDC loss further improves accuracy.","FedCFA maintains its advantage when the number of clients grows from 60 to 100 and also under IID partitions, suggesting the method is not limited to strongly heterogeneous label distributions."],"supporting_citations":[{"why":"Defines FedAvg, the basic aggregation baseline that this paper's Simpson's Paradox critique targets and that FedCFA must outperform.","marker":"(McMahan et al. 2017)"},{"why":"FedMix, the closest prior method that also augments local data with global average data; FedCFA is compared against it and claims to go further by disrupting spurious feature-label links.","marker":"(Yoon et al. 2021)"},{"why":"SCAFFOLD, a control-variate baseline that corrects client drift; its failure under Simpson's Paradox is used to motivate FedCFA.","marker":"(Karimireddy et al. 2020)"},{"why":"FedProx, a proximal-regularization baseline used as a comparison and as a representative of global-model alignment approaches.","marker":"(Li et al. 2020b)"},{"why":"q-FedAvg, a fairness-oriented aggregation baseline used in the main comparison.","marker":"(Li et al. 2019)"},{"why":"FedPVR, a partial variance reduction baseline used in the main comparison.","marker":"(Li et al. 2023b)"}],"fun_headline_variants":["Counterfactual data fix Simpson's paradox in FL","Flip local data to beat Simpson's paradox","Global-average swaps cure Simpson's paradox","FedCFA: counterfactuals defeat aggregation paradox","Make local data mimic global to dodge paradox"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a pixel-wise average of each client's data subsets, pooled across clients, faithfully represents the global data distribution, and that replacing factors with features from that average yields valid, class-preserving counterfactual samples.","fun_headline_variants_meta":{"raw":{"variants":["Counterfactual data fix Simpson's paradox in FL","Flip local data to beat Simpson's paradox","Global-average swaps cure Simpson's paradox","FedCFA: counterfactuals defeat aggregation paradox","Make local data mimic global to dodge paradox"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00076,"raw_usage":{"total_tokens":3347,"prompt_tokens":889,"completion_tokens":2458,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":2388}},"tokens_in":505,"tokens_out":2458,"duration_ms":15112,"temperature":1.0,"reasoning_tokens":2388,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:20:45.067951+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the Wasserstein distance W(P_{F̄g}, P_{F'_k}) and W(P_{F̄g}, P_{F_k}) with the trained encoder on a natural-image dataset; if the post-counterfactual distance is not smaller, or if human raters cannot recognize the class of the generated counterfactual images, the claimed distribution-alignment mechanism does not hold.","supporting_citations":[],"review_version":1}