Initialization Matters: Unraveling the Impact of Pre-Training on Federated Learning

Divyansh Jhunjhunwala; Gauri Joshi; Pranay Sharma; Zheng Xu

REVIEW 3 major objections 4 minor 1 cited by

Initialization Matters: Unraveling the Impact of Pre-Training on Federated Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that pre-trained initialization removes the non-IID penalty in FedAvg because data heterogeneity and local steps only degrade filters that are misaligned at initialization, and pretraining leaves fewer such filters.

desk verdict First initialization-dependent test error bound for FedAvg, with a clean explanation of pre-training benefits, but the bridge to real pre-training relies on an unquantified margin that the paper never states. read the letter →

arxiv 2502.08024 v1 pith:HOBKKVZZ submitted 2025-02-11 cs.LG cs.DC

Divyansh Jhunjhunwala , Pranay Sharma , Zheng Xu , Gauri Joshi This is my paper · ORCID

classification cs.LGcs.DC

keywords federatedlearningFedAvgpre-trainedinitializationdataheterogeneityfilteralignmenttwo-layerCNNbenignoverfittingsignal-to-noiseratio

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish why starting federated learning from a pre-trained model sharply reduces the test-accuracy loss caused by non-identical client data, without changing the FedAvg algorithm. It studies a two-layer ReLU convolutional network on a binary classification task where each input is a signal patch plus a noise patch, and it labels each first-layer filter as aligned or misaligned at initialization according to the sign of its correlation with the label signal. The central theoretical claim is that data heterogeneity and the number of local steps only slow down learning of the signal on misaligned filters, while noise memorization is unaffected, so the test-error bound depends directly on the fraction of misaligned filters. A pre-trained initialization matters because it typically starts with fewer misaligned filters, and in the limiting case of all filters aligned the paper recovers the centralized training bound, meaning the non-IID gap disappears.

What carries the argument

The argument is carried by a signal-noise decomposition of the CNN filter weights (Proposition 1): after any number of FedAvg rounds, each filter weight is the initial weight plus a signal coefficient $\Gamma^{(t)}_{j,r}$ times the signal vector plus noise coefficients $P^{(t)}_{j,r,k,i}$ times the per-client noise vectors. The paper defines a filter as aligned at initialization when $\langle w^{(0)}_{j,r}, j\mu\rangle \geq 0$ (Definition 1), and then proves two growth lemmas: aligned filters learn the signal at a rate independent of heterogeneity and local steps, while misaligned filters learn it at a rate scaled by $h + (1-h)/\tau$; noise memorization grows with $\tau$ but is unaffected by $h$ or alignment. The ratio of signal learning to noise memorization, which Theorem 2 exponentiates, is exactly where this asymmetry enters, and it reduces to the centralized bound when $\tau=1$ or when all filters are aligned.

What would settle it

Construct a synthetic experiment with known signal $\mu$ and a pretraining signal $\mu^{(pre)}$ orthogonal to $\mu$, count the misaligned filters at initialization, and run FedAvg with varying $\tau$ and heterogeneity $h$. If test error fails to increase with $\tau$ or $h$ when a nonzero misaligned fraction is present, or if it increases with $\tau$ even when all filters are aligned, the theorem's mechanism is contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is a test-error bound for FedAvg on a two-layer ReLU CNN. In the benign overfitting regime, where $\text{SNR}^2 \gtrsim 1/\sqrt{nd}$, the test error satisfies approximately $$L_{0-1}^D($W^{{(T)}}$) \leq \frac{1}{2}\sum_{j\in\{\pm1\}}\exp\!\left(-\frac{n}{d}\left[\frac{|A_j|}{m}\,\text{SNR}^2 + \left(1-\frac{|A_j|}{m}\right)\text{SNR}^2\left(h+\frac{1-h}{\tau}\right)\right]^2\right),$$ where $A_j$ is the set of filters aligned with the label signal at initialization, $h$ measures label heterogeneity across clients, and $\tau$ is the number of local steps. The load-bearing assertion is that heterogeneity $h$ and local steps $\tau$ enter the bound only through the misaligned fraction $1-|A_j|/m$; aligned filters contribute the full signal-to-noise ratio regardless of $h$ and $\tau$, while noise memorization is independent of both. Consequently, the non-IID generalization gap in FedAvg is attributed to filter alignment at initialization, not to optimization noise or aggregation variance alone, and pre-training helps because it reduces the misaligned fraction.

Load-bearing premise

The load-bearing premise is that after enough centralized pre-training every filter points at the pretraining signal, and that the downstream task's signal is close enough to the pretraining signal for the same pointing to hold at the start of federated training; the paper does not independently verify this for real image transfers such as ImageNet to CIFAR-10.

Editorial extensions

If this is right

If all filters are aligned at initialization, FedAvg's test-error bound matches centralized training, so non-IID data should stop hurting generalization even with many local steps.
The benefit of pre-trained initialization should grow as heterogeneity increases and as clients take more local steps, because both amplify the penalty on the misaligned fraction.
Local steps can be used freely in the first stage of training to cut communication when initialization is aligned, but with random initialization large $\tau$ is predicted to worsen non-IID generalization.
Tasks whose signal is farther from the pre-training signal should show more misaligned filters and therefore larger gains from pre-training, matching the paper's ImageNet-to-CIFAR-10 versus ImageNet-to-TinyImageNet comparison.
The theorem predicts that training error convergence can be decoupled from the non-IID generalization penalty: convergence is governed by the two-stage analysis, while generalization is governed by alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: if filter alignment at initialization is the causal mechanism, then interventions that increase alignment on a given downstream task without changing FedAvg, such as projecting initial filters toward class-wise signal directions or selecting a pretrained checkpoint with higher alignment, should produce the same non-IID gains as pretraining itself.
Editorial extension: the fraction of filters whose output signs flip during federated training could serve as a practical diagnostic for whether a pretrained checkpoint will resist heterogeneity; ranking checkpoints by this fraction should predict their non-IID test performance.
Editorial extension: the same alignment lens may transfer to differentially private federated training, where public pretraining is known to help; if private noise mainly perturbs misaligned filters, alignment at initialization could explain part of that benefit, though the paper does not make this claim.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper delivers what it promises: the first test-error bound for FedAvg that explicitly depends on initialization, and an aligned/misaligned filter decomposition that gives a clean explanation for why pre-training shrinks the non-IID generalization gap. The bound is internally coherent, recovers the centralized result at τ=1, and the synthetic experiments track the predicted trends. The two-stage convergence proof follows the Cao/Kou template and is reasonably careful, though not machine-checked. This is a solid contribution to the feature-learning analysis of FL.

The soft spot is Section 3.5. Lemma 3 guarantees sign alignment with the pre-training signal µ^(pre), but converting that into alignment with the downstream signal µ requires a positive margin: the filter's overlap with jµ^(pre) must exceed the worst-case perturbation from the weight component orthogonal to µ^(pre). The paper handwaves this with "if ∥µ−µ^(pre)∥ is small" and never states the allowable drift as a function of SNR, n, d. The stress-test note's specific claim that noise always dominates is not conclusive—Lemma 3's proof actually yields a signal component of order n∥µ∥²/(σ_p²d), which can be large—but the missing margin is a genuine gap. The real-data misalignment measure in Eq. (8) is defined against the final model's output signs, so it measures sign stability during fine-tuning, not true signal alignment; it cannot independently confirm that pre-trained filters are aligned with the downstream signal. That weakness is in proportion: the conditional theorem stands, but the advertised practical implication is not fully established.

The paper is honest about the assumption and does not fake evidence, but it should either quantify the drift condition or soften the practical conclusion. A serious referee should engage with this. I would send it to review, and I would ask the authors for an explicit margin condition and a more direct check of filter alignment on real data. The central theorem is worth citing; the transfer claim needs revision.

Referee Report

3 major / 4 minor

Summary. The paper studies a two-layer ReLU CNN with a signal-noise data model and gives a theoretical analysis of FedAvg under label heterogeneity. It defines a filter as aligned at initialization when ⟨w^(0)_{j,r}, jµ⟩ ≥ 0, proves a training-convergence bound, and derives a test-error bound (Theorem 2) whose exponent depends on the heterogeneity parameter h and the number of local steps τ only through the fraction of misaligned filters. It then argues via Lemma 3 that centralized pre-training aligns all filters with the pre-training signal and that, when the pre-training and downstream signals are close, FedAvg recovers the centralized bound. The paper reports synthetic simulations and real-data FL experiments (ResNet18, SqueezeNet) as verification.

Significance. If the conditional Theorem 2 is taken alone, the paper makes a solid technical contribution: the test-error bound is non-vacuous and interpretable, the appendix contains a detailed proof, the bound reduces to the centralized Kou et al. bound at τ=1, and the synthetic experiments in Figures 2–3 track the predicted qualitative trends, including the predicted τ- and h-dependence for misaligned filters. The paper would be substantially strengthened, and the advertised practical conclusion would be justified, if the transfer step in §3.5 were turned into a quantitative margin condition and tested with a non-circular alignment measure. As written, the conditional theory is sound, but the central practical claim about pre-training reducing the non-IID FedAvg gap is not yet independently established.

major comments (3)

[§3.5 and Theorem 2] The advertised application of Theorem 2 to a pre-trained initialization is not covered by the theorem's hypotheses. Theorem 2 and its proof require Condition 1, in particular C4, and the proof repeatedly uses Gaussian-init facts: Lemma 6 controls |⟨w^(0)_{j,r},µ⟩| and |⟨w^(0)_{j,r},ξ⟩| by σ0, Lemma 7 gives at least 0.4m activated filters, and Lemma 32 uses ||w^(0)_{j,r}|| = Θ(σ0√d). When W^(0) = W^(pre,*), the initialization is neither Gaussian nor small in the sense of C4, and the local-GD analysis in Appendix C has not been re-run for such initializations. An explicit extension of Theorem 2 (or a separate theorem) to deterministic, bounded-norm initializations is needed before the conclusion A_j=[m] can be combined with the test-error bound.
[§3.5, Lemma 3 and following paragraph] Even if one accepts an extension of Theorem 2 to arbitrary initializations, the transfer argument is incomplete. Lemma 3 only establishes ⟨w^(pre,*), jµ^(pre)⟩ ≥ 0, and the following paragraph asserts that 'if ∥µ−µ^(pre)∥ is small' then every filter is aligned with µ. The proof of Lemma 3 protects the overlap with µ^(pre) because pre-training noise is orthogonal to µ^(pre), but the drift term ⟨w^(pre,*), j(µ−µ^(pre))⟩ is not controlled: after T1 pre-training steps the positive noise coefficients sum to Θ(n) (Theorem 3 and Lemma 2 with τ=1), and their contribution to the drift inner product can scale as n∥µ−µ^(pre)∥/(σ_p d). To conclude alignment one needs a per-filter margin lower bound on ⟨w^(pre,*), jµ^(pre)⟩ and a matching upper bound on the drift overlap. The paper states no such quantitative condition and does not verify it for the ImageNet-to-CIFAR/TinyImageNet transfers in Section 4.
[§4, Eq. (8)] The real-data alignment measure is defined relative to the final trained model: A(w^(t)) compares sign patterns with W^(T). A filter is called misaligned only if its initialization sign pattern disagrees with the final sign pattern. With this definition, a filter can be counted as aligned even if its initialization overlap with the true downstream signal is negative, provided the final model's sign pattern agrees with the initialization pattern; conversely, the measure will automatically improve when pre-training produces a better final model. This makes the empirical claim 'pre-trained initialization leads to fewer misaligned filters' dependent on the very stability property (Lemma 27) that the transfer argument needs to establish for real CNNs. The experiments are therefore suggestive but cannot independently confirm the premise in Definition 1 and Lemma 3. The authors should either use a notion of alignment based on measured signal or class statistics, or explicitly label Eq. (8) as a heuristic and remove it from the verification chain.

minor comments (4)

[Equation (3)] The parameter space in the FL objective should be R^{2md}, not R^{2d}, since the CNN has 2m filters each of dimension d.
[Section 2, data model] The sentence 'We assume orthogonality just for simplicity of analysis and can be easily relaxed as done in .' has a dangling reference; a citation is missing.
[Section 4, Eq. (8)] The paper should specify which convolutional filters are included in Eq. (8), how many filters are counted, the batch size used, and how the threshold A(w^(t)) < 0 is applied; these details are needed to reproduce the reported percentages.
[Theorem 2] The lower-bound statement in Theorem 2 ('test error ≥ 0.1') would benefit from an explicit 'with probability ≥ 1−δ' and a note that 0.1 is a constant depending on the constants in Condition 1.

Circularity Check

1 steps flagged · score 4.0 of 10

The conditional test-error theorem is self-contained; partial circularity arises in the real-data validation, where 'misalignment at initialization' is measured relative to the final trained model, so the empirical confirmation is partly constructed from the outcome it purports to explain.

self definitional [Section 4, Eq. (8) and surrounding text]
"Therefore, we propose to use the sign of the output produced by a filter at the end of training as a reference for alignment at any given round. ... A(w(t)) := Σ_{x∈B,l∈[p]} sign(F_l(w(t), x)) sign(F_l(w(T), x)). We say that the weight w(t) at round t is misaligned if A(w(t)) < 0, because this implies that the sign of the output ... eventually changed for a majority of the inputs, hence indicating that the filter was misaligned at round t."

This empirical measure defines 'misaligned at round t' as disagreement with the final model w(T), and the paper then uses this measure to conclude that pre-trained initialization yields fewer misaligned filters and therefore better final accuracy (Figures 5-6). The step is circular because the inference from sign change to initial misalignment presupposes Lemma 27, the very alignment-stability property the real-data experiments are claimed to corroborate. Consequently the measured 'misalignment' is an outcome-relative quantity (output-sign drift), not the Definition 1 signal-overlap quantity, and its correlation with test accuracy is partly built in by construction.

full rationale

The central derivation, Theorem 2, is not circular: it is a conditional bound parameterized by the independently defined set A_j of filters aligned with the downstream signal, it is derived from Lemmas 1-2 and the signal-noise decomposition (Proposition 1), and it reduces to the external centralized bound of Kou et al. (2023) at tau=1. Lemma 3 is a real theorem with a proof, not a self-citation, and the bound depends on conditions that do not include the target result. Section 3.5's transfer step ('if ||mu - mu^(pre)|| is small, all filters are aligned') is, however, under-specified: Lemma 3 guarantees only non-negative overlap with mu^(pre) and supplies no explicit positive margin, so the allowable drift is unquantified; this is a correctness/assumption gap rather than a circular reduction. The concrete circularity is in the empirical bridge of Section 4: Eq. (8) defines real-data 'misalignment at initialization' relative to the final model w(T), so the observed correlation between this misalignment count and final accuracy cannot independently verify the theory. Because the mathematical main result stands on its own and the circularity is confined to the real-data validation proxy, the score is 4 rather than higher.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The central theorem is conditional on a clean signal-noise data model and strong overparameterization conditions. No constants are fitted in the theory. The main non-standard assumption is the alignment-transfer premise for pre-training, which essentially assumes the mechanism the paper claims to explain. The conceptual entity 'aligned/misaligned filters' is not independently measurable on real data.

assumptions (4)

domain assumption Each input consists of two patches, one is yµ and the other is Gaussian noise orthogonal to µ.
Section 2 data-generation model; the signal-noise decomposition relies on this structural assumption.
standard math Dimension d, width m, sample size n, initialization variance σ0, and learning rate η satisfy Assumptions 1-6 (Condition 1).
Appendix C uses these to ensure enough activations, bounded coefficients, and a nearly convex second stage of training.
ad hoc to paper After T1 pre-training iterations all filters align with µ^{(pre)}, and the downstream signal µ is close to µ^{(pre)} so alignment transfers.
Section 3.5; this is the load-bearing premise connecting pre-training to fewer misaligned filters.
domain assumption Client data differ only in label proportions, summarized by scalar h, with equal client dataset sizes.
Equation (1) defines h; feature distribution shift and unequal client sizes are not modeled.

invented entities (1)

Aligned/misaligned filter classification at initialization
purpose: Separates heterogeneity-sensitive signal learning from heterogeneity-independent noise memorization; it is the central quantity in the test error bound.
The classification is defined relative to the unknown signal µ. On real data the paper's proxy (Eq. 8) uses the final model as reference, so the notion cannot be independently falsified outside the paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Initialization Matters: Unraveling the Impact of Pre-Training on Federated Learning." pith.science (2026). https://pith.science/paper/HOBKKVZZ

@misc{pith2026250208024,
  author       = {Pith},
  title        = {Pith review of: Initialization Matters: Unraveling the Impact of Pre-Training on Federated Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HOBKKVZZ}},
  note         = {Machine review of arXiv:2502.08024}
}

read the original abstract

Initializing with pre-trained models when learning on downstream tasks is becoming standard practice in machine learning. Several recent works explore the benefits of pre-trained initialization in a federated learning (FL) setting, where the downstream training is performed at the edge clients with heterogeneous data distribution. These works show that starting from a pre-trained model can substantially reduce the adverse impact of data heterogeneity on the test performance of a model trained in a federated setting, with no changes to the standard FedAvg training algorithm. In this work, we provide a deeper theoretical understanding of this phenomenon. To do so, we study the class of two-layer convolutional neural networks (CNNs) and provide bounds on the training error convergence and test error of such a network trained with FedAvg. We introduce the notion of aligned and misaligned filters at initialization and show that the data heterogeneity only affects learning on misaligned filters. Starting with a pre-trained model typically results in fewer misaligned filters at initialization, thus producing a lower test error even when the model is trained in a federated setting with data heterogeneity. Experiments in synthetic settings and practical FL training on CNNs verify our theoretical findings.

Figures

Figures reproduced from arXiv: 2502.08024 by the authors.

**Figure 1.** Test accuracy (%) on CIFAR10 with SqueezeNet model (Iandola et al., 2016) under random and pretrained initializations for FL and centralized training. Pre-training benefits FL more than centralized setting and significantly reduces the gap between IID and non-IID FL model performance. performance than a model trained on Independent and Identically Distributed (IID) data (Kairouz et al., 2021; Li et al., 2020; Yang … view at source ↗

**Figure 2.** Empirical results on synthetic dataset to verify the upper bound on test error in Theorem 2. We fix the training error [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Signal learning and noise memorization for our CNN model in the IID [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 2.** Figure 2: Figure 2a shows the test error as a function of [PITH_FULL_IMAGE:figures/full_fig_p006_2.png]

**Figure 2.** Figure 2: Figure Figure 3a shows that in the IID set [PITH_FULL_IMAGE:figures/full_fig_p007_2.png]

**Figure 4.** Figure 4: Initial alignment of the filters in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: The percentage of misaligned filters (see Equation (8) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA
cs.LG 2025-06 conditional novelty 6.0 of 10

FedRPCA decomposes federated LoRA client updates with Robust PCA into common and client-specific components, averaging the common part and scaled-averaging the sparse part, which improves accuracy and convergence over...

Reference graph

Works this paper leans on

18 extracted references · 15 canonical work pages · cited by 1 Pith paper

[1]

For j ̸= yk,i, Fj( fW(v′) j,k , xk,i) ≤ 0.5

work page
[2]

For j = yk,i, Fj( fW(v′) j,k , xk,i) ≥ 1 m Pm r=1 P (v′) j,r,k,i − 0.25

work page
[3]

Proof of 1

yk,if ( fW(v′) k , xk,i) ≥ 1 m Pm r=1 P (v′) yk,i,r,k,i − 0.75. Proof of 1. First note that for j ̸= yk,i from Lemma 11 we have, ⟨ ew(v′) j,r,k, µ⟩ ≤ ⟨w(0) j,r , µ⟩. (32) since G(v′) j,r,k ≥ 0 by the induction hypothesis. Also from Lemma 11 for j ̸= yk,i we have, ⟨ ew(v′) j,r,k, ξk,i⟩ ≤ ⟨w(0) j,r , ξk,i⟩ + P(v′) j,r,k,i + 4 r log(6n2/δ) d nα (a) ≤ ⟨w(0) j...

work page
[5]

Here we take κ = 5 and C1 = 6.75

˜S(0) j,r ⊆ ˜S(v) j,r where ˜S(0) j,r := n k ∈ [K], i∈ [N ] : yk,i = j, ⟨ ew(v) j,r,k, ξk,i⟩ ≥0 o , and hence ˜S(v) j,r ≥ n 8 . Here we take κ = 5 and C1 = 6.75. 27 Initialization Matters: Unraveling the Impact of Pre-Training on Federated Learning Proof of 1. We will use a proof by induction. For v = 0 , it is simple to verify that 1 holds since P (0) j,...

work page
[13]

1 m Pm r=1 h P (v) yk,i,r,k,i − P (v) yk′ ,i′ ,r,k′,i′ i ≤ κ for all k, k′ ∈ [K], i, i′ ∈ [N ]

work page
[14]

yk,if ( fW(v) k , xk,i) − yk′,i′f ( fW(v) k′ , xk′,i′) ≤ C1 for all k, k′ ∈ [K] and i, i′ ∈ [N ]. 3. ℓ′ (v) k′ ,i′ ℓ′ (v) k,i ≤ C2 = exp(C1) for all k, k′ ∈ [K] and i, i′ ∈ [N ]

work page
[15]

S(0) k,i ⊆ S(v) k,i where S(v) k,i := n r ∈ [m] : ⟨ ew(v) yk,i,r,k, ξk,i⟩ ≥0 o , and hence S(v) k,i ≥ 0.4m for all k ∈ [K], i∈ [N ]

work page
[16]

Here we take κ = 5 and C1 = 6.75

˜S(0) j,r ⊆ ˜S(v) j,r where ˜S(v) j,r := n k ∈ [K], i∈ [N ] : yk,i = j, ⟨ ew(v) j,r,k, ξk,i⟩ ≥0 o , and hence ˜S(v) j,r ≥ n 8 . Here we take κ = 5 and C1 = 6.75. C.3. First Stage of Training. Define, T1 = C3nm ησ 2pdτ (54) where C3 = Θ(1) is some large constant. In this stage, our goal is to show that P (T1) yk,i,r∗,k,i ≥ 2 for all r∗ such that r∗ ∈ S(0) ...

work page

Show all 18 references

[17]

For any j ∈ {±1}, r∈ [m] : Γ(t) j,r ≤ η∥µ∥2 2 m Pt−1 t′=0 Pτ −1 s=0 maxk,i ℓ′(t′,s) k,i
[18]

44 Initialization Matters: Unraveling the Impact of Pre-Training on Federated Learning

For any r ∈ Aj : Γ(t) j,r ≥ η∥µ∥2 2 4m Pt−1 t′=0 Pτ −1 s=0 min(k,i) ℓ′(t′,s) k,i . 44 Initialization Matters: Unraveling the Impact of Pre-Training on Federated Learning
[19]

For any r /∈ Aj : Γ(t) j,r ≥ η∥µ∥2 2 4m Pt−1 t′=0 mink,i ℓ′(t′,0) k,i + h Pτ −1 s=1 mink,i ℓ′(t′,s) k,i . Proof. Unrolling the iterative update in equation 16 we have, Γ(t) j,r = η nm t−1X t′=0 τ −1X s=0 X k,i (−ℓ′(t′,s) k,i ) · σ ′ ⟨w(t′,s) j,r,k , yk,iµ⟩ · ∥µ∥2 2 . (68) Proo...
[20]

P k,i P (t) j,r,k,i ≤ 3ησ 2 pd 2m Pt−1 t′=0 Pτ −1 s=0 maxk,i ℓ′(t′,s) k,i
[21]

P k,i P (t) j,r,k,i ≥ ησ 2 pd 16m Pt−1 t′=0 Pτ −1 s=0 min(k,i)∈ ˜S(t′ ,s) j,r ℓ′(t′,s) k,i where ˜S(t′,s) j,r := n k ∈ [K], i∈ [N ] : ⟨w(t′,s) j,r,k , ξk,i⟩ ≥0 o . Proof. From equation 18 we have the following update equation for P (t) j,r,k,i. X k,i P (t) j,r,k,i = X k,i P (t...

2022
[60]

acl-industry.60/

URL https://aclanthology.org/2023. acl-industry.60/. Yang, C., Wang, Q., Xu, M., Chen, Z., Bian, K., Liu, Y ., and Liu, X. Characterizing impacts of heterogeneity in federated learning upon large-scale smartphone data. In Proceedings of the Web Conference 2021, pp. 935–946, 20...

2023 arXiv
[2006]

Proceedings 3, pp. 265–284. Springer, 2006. Fallah, A., Mokhtari, A., and Ozdaglar, A. Generalization of model-agnostic meta-learning algorithms: Recurring and unseen tasks. Advances in Neural Information Processing Systems, 34:5469–5480, 2021. Fanì, E., Camoriano, R., Caputo,...

2006 arXiv
[2009]

Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P

URL https://www.cs.toronto.edu/ ~kriz/learning-features-2009-TR.pdf . Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and under- perform out-of-distribution. International Conference on Learning Representations, 2022. Le...

2009 arXiv
[2010]

Gupta, S., Huang, Y ., Zhong, Z., Gao, T., Li, K., and Chen, D

URL http://proceedings.mlr.press/ v9/glorot10a.html. Gupta, S., Huang, Y ., Zhong, Z., Gao, T., Li, K., and Chen, D. Recovering private text in federated learning of lan- guage models. Advances in Neural Information Process- ing Systems, 35:8130–8143, 2022. He, K., Zhang, X., ...

2022 arXiv
[2022]

(Huang et al., 2021) study the generalization of FedAvg on wide two-layer ReLU networks with homogeneous data

derive information-theoretic bounds, but these bounds require specific forms of loss functions and cannot capture effects of heterogeneity. (Huang et al., 2021) study the generalization of FedAvg on wide two-layer ReLU networks with homogeneous data. (Collins et al., 2022) stu...

2021

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

For j ̸= yk,i, Fj( fW(v′) j,k , xk,i) ≤ 0.5

work page

[2] [2]

For j = yk,i, Fj( fW(v′) j,k , xk,i) ≥ 1 m Pm r=1 P (v′) j,r,k,i − 0.25

work page

[3] [3]

Proof of 1

yk,if ( fW(v′) k , xk,i) ≥ 1 m Pm r=1 P (v′) yk,i,r,k,i − 0.75. Proof of 1. First note that for j ̸= yk,i from Lemma 11 we have, ⟨ ew(v′) j,r,k, µ⟩ ≤ ⟨w(0) j,r , µ⟩. (32) since G(v′) j,r,k ≥ 0 by the induction hypothesis. Also from Lemma 11 for j ̸= yk,i we have, ⟨ ew(v′) j,r,k, ξk,i⟩ ≤ ⟨w(0) j,r , ξk,i⟩ + P(v′) j,r,k,i + 4 r log(6n2/δ) d nα (a) ≤ ⟨w(0) j...

work page

[4] [5]

Here we take κ = 5 and C1 = 6.75

˜S(0) j,r ⊆ ˜S(v) j,r where ˜S(0) j,r := n k ∈ [K], i∈ [N ] : yk,i = j, ⟨ ew(v) j,r,k, ξk,i⟩ ≥0 o , and hence ˜S(v) j,r ≥ n 8 . Here we take κ = 5 and C1 = 6.75. 27 Initialization Matters: Unraveling the Impact of Pre-Training on Federated Learning Proof of 1. We will use a proof by induction. For v = 0 , it is simple to verify that 1 holds since P (0) j,...

work page

[5] [13]

1 m Pm r=1 h P (v) yk,i,r,k,i − P (v) yk′ ,i′ ,r,k′,i′ i ≤ κ for all k, k′ ∈ [K], i, i′ ∈ [N ]

work page

[6] [14]

yk,if ( fW(v) k , xk,i) − yk′,i′f ( fW(v) k′ , xk′,i′) ≤ C1 for all k, k′ ∈ [K] and i, i′ ∈ [N ]. 3. ℓ′ (v) k′ ,i′ ℓ′ (v) k,i ≤ C2 = exp(C1) for all k, k′ ∈ [K] and i, i′ ∈ [N ]

work page

[7] [15]

S(0) k,i ⊆ S(v) k,i where S(v) k,i := n r ∈ [m] : ⟨ ew(v) yk,i,r,k, ξk,i⟩ ≥0 o , and hence S(v) k,i ≥ 0.4m for all k ∈ [K], i∈ [N ]

work page

[8] [16]

Here we take κ = 5 and C1 = 6.75

˜S(0) j,r ⊆ ˜S(v) j,r where ˜S(v) j,r := n k ∈ [K], i∈ [N ] : yk,i = j, ⟨ ew(v) j,r,k, ξk,i⟩ ≥0 o , and hence ˜S(v) j,r ≥ n 8 . Here we take κ = 5 and C1 = 6.75. C.3. First Stage of Training. Define, T1 = C3nm ησ 2pdτ (54) where C3 = Θ(1) is some large constant. In this stage, our goal is to show that P (T1) yk,i,r∗,k,i ≥ 2 for all r∗ such that r∗ ∈ S(0) ...

work page