Pith. sign in

REVIEW 4 major objections 4 minor 53 references

DICE: Data Influence Cascade in Decentralized Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DICE is the first tractable estimator of how a training batch's influence spreads hop by hop through a decentralized network, showing a node's contribution is shaped by its data, its network position, and loss curvature along the path.

desk verdict One-hop DICE is a clean, novel estimator with honest experiments, but the multi-hop Theorem 2 carries an extra learning-rate factor and a Hessian index error, so the central cascade claim as printed does not follow from the proof. read the letter →

arxiv 2507.06931 v1 pith:NLRGU5W4 submitted 2025-07-09 cs.LG cs.DCcs.MAcs.SIstat.ML

Tongtian Zhu , Wenhao Li , Can Wang , Fengxiang He This is my paper
classification cs.LGcs.DCcs.MAcs.SIstat.ML MSC 68T0568W15
keywords datainfluenceestimationdecentralizedlearningcascadegossipaveragingvaluationmulti-hopincentivemechanismsanomalydetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles a question that arises once machine learning leaves the central server: how much did each participant's data actually contribute to the models of the other participants? In decentralized learning, a training batch changes its own node's parameters, those parameters are mixed with neighbors' by gossip averaging, and the perturbation keeps traveling — influence cascades through the network topology. The paper defines a ground-truth cascade (DICE-GT) for this process and derives a tractable linearized estimator (DICE-E) valid for any number of hops. The central claim is that multi-hop influence factorizes into the originating update, the communication weights along each path, and Hessian terms capturing the loss curvature at every intermediate node. If the estimate holds, it gives decentralized networks the missing accounting tool for fair incentives, collaborator selection, and detection of malicious participants.

What carries the argument

The load-bearing object is the multi-hop DICE-E identity in Theorem 2, which decomposes the r-hop influence of a batch into a product along each propagation path: the scaled communication weights $\prod_{s=1}^{\rho} W^{t+s-1}_{k_s,k_{s-1}}$ (topology), the Hessian terms $\prod_{s=2}^{\rho}(I - \eta^{t+s-1}H(\theta^{t+s-1}_{k_s}; z^{t+s-1}_{k_s}))$ (curvature of the loss at intermediate nodes), the originating update $\Delta_j(\theta^t_j, z^t_j)$ (the batch's own optimization step), and the endpoint test gradient $\nabla L(\theta^{t+\rho}_{k_\rho}; z')$. The argument that carries the proof is a recursive first-order Taylor expansion: removing the batch shrinks each gossip step by the mixing weight and each intermediate optimization step by $(I - \eta H)$, so the perturbation can be unrolled hop by hop. Because these factors multiply, the identity simultaneously yields the additivity of influence over mini-batches, the exponential-decay claim, and the topological-dependency claim that the experiments then probe with the one-hop version.

What would settle it

On a small chain or tree network, train twice with Decentralized SGD, once with a fixed batch at the root node and once without it, and compare the measured loss changes at two- and three-hop nodes against the DICE-E predictions with r = 2 and r = 3; if the match degrades as the hop count grows, or breaks under a large learning rate or sharp minima, the linearized cascade formula is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that data influence in decentralized learning is a cascade rather than a static attribute: the one-hop ground truth (Definition 2) counts both the loss reduction a batch causes at its own node and the reduction it causes at immediate neighbors through gossip averaging, and the multi-hop ground truth (Definition 3) extends this to all nodes within r hops of the source. Theorem 2 then identifies the estimator DICE-E with a sum over every path of length up to r, in which each path contributes the product of communication weights along the path times products of curvature terms $(I - \eta^{t+s-1}H)$, applied to the originating parameter update $\Delta_j(\theta^t_j, z^t_j)$ and read off by a test gradient at the endpoint. Because each factor is contractive, the formula implies that influence decays with hop count and that a node's reach is governed by its topological importance, so the same data batch exerts different influence depending on where it lives and what the intermediate loss landscapes look like.

Load-bearing premise

The cascade formula assumes each training batch barely moves the parameters and the loss landscape is smooth all along the path, so a simple linear approximation can track the ripple hop by hop; the experiments test only the one-hop version.

Editorial extensions

If this is right

  • A participant can estimate how much a neighbor's or a far-away node's data helped its own model: with access to gradients, mixing weights, and local Hessian-vector products, the DICE-E sum over paths is computable for any hop count r.
  • Influence decays with distance: each additional hop multiplies the contribution by communication weights and $(I - \eta H)$ factors, so distant nodes matter less — the paper's 'ripples in water' picture formalized.
  • Topologically central nodes carry more influence: a node with larger outgoing mixing weights propagates its data's effect more widely, which the paper ties to power asymmetries in the network.
  • The one-hop proximal influence gives practical tools: selecting beneficial collaborators, computing reciprocity factors that flag imbalanced exchanges, and detecting anomalous neighbors whose influence score deviates sharply.
  • Additivity over mini-batches means influence scores for large batches can be computed as sums of per-sample scores, keeping the estimator efficient.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The factorization suggests a practical shortcut the paper leaves implicit: with Hessian terms hard to obtain in deep networks, one could proxy the curvature factors by gradient inner products; the paper does not test whether such a proxy preserves the multi-hop ordering of influence.
  • The exponential-decay result has a design corollary not drawn in the paper: an operator could deliberately shape the mixing matrix to concentrate or equalize influence, using the formula as a tool for tuning contribution fairness rather than merely measuring it.
  • A direct test of the multi-hop claim is missing from the paper: leave-a-batch-out retraining on a small chain or tree network would compare true two- and three-hop loss changes against DICE-E with r = 2 and r = 3, which the current experiments do not do.
  • The reciprocity factor could seed a dynamic reputation system for incentives, but whether nodes following the reciprocity signal reach a stable, fair equilibrium is a game-theoretic question the paper leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper addresses the problem of quantifying the influence of a data instance on the models of other participants in fully decentralized learning. It defines ground-truth one-hop and multi-hop influence cascades (Definitions 2 and 3), derives first-order approximations called DICE-E (Proposition 1 and Theorem 2), and presents experiments measuring one-hop alignment, anomaly detection, and qualitative topological effects. The paper's central claim is that the r-hop influence cascade is determined by an interplay of the data, the communication topology, and the curvature of the loss landscape, and that DICE is the first tractable multi-hop influence estimator for decentralized learning.

Significance. If the multi-hop formula can be corrected and validated, DICE would be a valuable first step: the one-hop estimator is parameter-free, derived directly from the ground-truth definition, and the alignment experiments cover several datasets, architectures, topologies, and hyperparameters. The paper also demonstrates a concrete application to anomaly detection. The limitations are substantial, however: Theorem 2 as stated contains algebraic inconsistencies, the multi-hop approximation is not tested for r≥2, and the proof assumes SGD-type intermediate updates despite the generality claimed in Algorithm 1. These issues currently leave the central multi-hop claim unsupported.

major comments (4)
  1. [4.2 / Theorem 2, Appendix C.3] The r-hop formula in Theorem 2 does not reduce to Proposition 1. For ρ=0, Theorem 2 gives -η^t q_j ∇L(θ^t_j; z')^T Δ_j, and for ρ=1 it gives -η^t q_{k1} W^t_{k1,j} ∇L(θ^{t+1}_{k1}; z')^T Δ_j, whereas Proposition 1, derived from the same Definition 2, gives the same expressions without η^t. Because Δ_j(θ^t_j,z^t_j)=O_j(θ^t_j,z^t_j)-θ^t_j is the parameter update (for SGD, -η^t∇L), the extra η^t counts the learning rate twice. The same double-counting appears in the base step of the proof in C.3, where θ^{t+1}_{k1}-θ^{t+1}_{k1\z^t_j} is written as -η^t W^t_{k1,j}Δ_j, while Algorithm 1 directly gives W^t_{k1,j}Δ_j. Removing the leading η^t from Theorem 2 makes the ρ=1 case consistent with Proposition 1, but as stated the theorem is internally inconsistent.
  2. [4.2 / Theorem 2, C.2, C.3] The Hessian index in Theorem 2 is inconsistent with the proof. The statement's curvature product involves H evaluated at the receiver node k_s, while the surrounding text even writes H(θ^{t+s}_{k_s};...); however, the recursion in C.3 and the two-hop formula in C.2 require H evaluated at the predecessor k_{s-1} at time t+s-1, so for ρ=2 this is H(θ^{t+1}_{k_1}; z^{t+1}_{k_1}). This is not a notational nuisance: the curvature term is supposed to describe the linearization of the local update at the intermediate node that actually propagates the influence, and the receiver's Hessian at that time is not the object appearing in the derivation.
  3. [5, Appendix D] The multi-hop theorem is never empirically validated. All alignment plots (Figure 3 and Figures D.1-D.12) compare one-hop DICE-E with one-hop DICE-GT; there is no experiment comparing the r-hop approximation of Theorem 2 with the r-hop ground truth of Definition 3 for any r≥2. Since the proof relies on nested first-order Taylor expansions whose error compounds across hops, the claim that DICE-E gives accurate multi-hop cascades, and the associated 'exponential decay' statement, are unsupported. This missing validation is load-bearing given the algebraic inconsistencies in the theorem.
  4. [3 / Algorithm 1, C.3] Theorem 2 is claimed for arbitrary optimizers in Algorithm 1, but its proof linearizes the intermediate local updates as SGD steps. In C.3, the difference at k_{ρ-1} is written as (θ^{t+ρ-1} - θ^{t+ρ-1}_{\z}) - η^{t+ρ-1}(∇L(...)-∇L(...)), which is only valid when O_{k_{ρ-1}} is gradient descent with learning rate η. For a general optimizer, the correct linearization would involve the Jacobian of O with respect to its parameter input, not simply I - ηH. The theorem should either be restricted to the SGD case or generalized accordingly.
minor comments (4)
  1. [4.3, Eq. (5)] Equation (5) defines proximal influence with a minus sign, making it the negative of the one-hop neighbor contribution from Proposition 1 when Δ_j is the SGD update. The text interprets 'high' proximal influence as indicating a loss-increasing neighbor, so the sign convention should be stated explicitly and reconciled with Proposition 1.
  2. [5, Anomaly Detection] The anomaly detection paragraph refers to 'Figure 3', but the anomaly results are shown in Figure 4; this cross-reference should be corrected.
  3. [Definition 5] In the proximal reciprocity factor R^t_{k,j}, the data arguments in the gradients appear swapped: the numerator's ∇L(θ^t_j; z^t_k) and denominator's ∇L(θ^t_k; z^t_j) should presumably be ∇L(θ^t_j; z^t_j) and ∇L(θ^t_k; z^t_k), respectively.
  4. [General] There are several typos: 'Regrading' in Section 2, 'condcut' in Appendix D.2.2, 'trainingg' in Appendix D.4.1, 'an completely new perspective' in Section 2, and 'the most anomalies' in Section 5.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular reduction: DICE-E is a Taylor approximation of the paper's own DICE-GT counterfactual, validated against that ground truth without fitted parameters.

full rationale

The central derivation (Proposition 1, Theorem 2, Appendices C.1-C.3) begins from the ground-truth counterfactual loss differences in Definitions 2 and 3 and approximates them by first-order Taylor expansions plus a recursion over Algorithm 1's local-update and gossip-averaging equations. The resulting DICE-E formula contains no parameter fitted to DICE-GT; the experiments compare DICE-E directly to DICE-GT (Section 5 and Appendix D.2), so the observed alignment is an empirical check rather than a quantity forced by construction. The claimed dependence on data, communication topology, and curvature is read out of the derived formula, not imposed as an input. Self-citations (Zhu et al. 2022, 2023b) appear only in background on decentralized generalization and are not used to justify the DICE derivation, so they are not load-bearing. The internal discrepancy between Theorem 2's eta^t factor and Proposition 1, and the Hessian-index mismatch between the theorem statement and the proofs in Appendices C.2/C.3, are algebraic-consistency and correctness concerns, not circularity: neither reduces a prediction to an input by definition. Similarly, the repeated Taylor linearization for r >= 2 is an unverified approximation assumption, but it is not a circular step. The paper is self-contained relative to its own ground-truth measure, and no load-bearing argument collapses into a self-citation or a fitted parameter.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard smoothness assumptions from the influence-function literature and on the choice of a one-step counterfactual as the ground truth. No parameters are fitted to data: DICE-E is a closed-form first-order approximation with no adjustable coefficients. The only invented notion is the 'influence cascade' itself, which is a definitional extension of TracIn to gossip paths rather than a postulated entity with independent evidence requirements. The main fragile input is the linearization assumption in Theorem 2, which is unverified for multi-hop propagation.

assumptions (5)
  • domain assumption First-order Taylor expansions are valid for the loss and gradient differences along the propagation path
    Proposition 1 and Theorem 2 replace L(θ+Δ) - L(θ) by ∇L(θ)^T Δ and replace gradient differences by Hessian-vector products, requiring small parameter perturbations and smooth loss. This is invoked in Appendix C.1 and C.3 and is the load-bearing approximation for the cascade formula.
  • domain assumption DICE-GT, defined by removing a batch from one local update, is the appropriate gold-standard influence measure
    Definitions 2 and 3 quantify influence via a one-step counterfactual loss difference rather than full leave-one-out retraining; the entire approximation targets this quantity, so its validity as a measure of contribution is assumed.
  • domain assumption The gossip matrix W is row-stochastic and the relevant entries are known to participants
    Algorithm 1 and the proximal influence formula in Definition 4 assume participants know W_{k,j} and q_k; the paper invokes convergence guarantees for row-stochastic gossip from prior work.
  • domain assumption Neighbors share gradients necessary to compute proximal influence
    Definition 4 states participant k has access to ∇L(θ^t_j; z^t_j), which requires gradient sharing beyond parameter sharing; the paper argues this does not significantly reduce privacy because gradients can be reconstructed from parameters (citing Mrini et al. 2024), but this is an assumption about the communication protocol.
  • domain assumption The empirical DICE-GT in experiments is computed as a one-step removal counterfactual
    Appendix D.2 says DICE-GT is 'the loss reduction after removing Bt_j from node j at the t-th iteration', which is a simplified version of Definition 2 that ignores longer-term retraining effects; the approximation is validated against this simplified target.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DICE: Data Influence Cascade in Decentralized Learning." pith.science (2026). https://pith.science/paper/NLRGU5W4

@misc{pith2026250706931,
  author       = {Pith},
  title        = {Pith review of: DICE: Data Influence Cascade in Decentralized Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NLRGU5W4}},
  note         = {Machine review of arXiv:2507.06931}
}
read the original abstract

Decentralized learning offers a promising approach to crowdsource data consumptions and computational workloads across geographically distributed compute interconnected through peer-to-peer networks, accommodating the exponentially increasing demands. However, proper incentives are still in absence, considerably discouraging participation. Our vision is that a fair incentive mechanism relies on fair attribution of contributions to participating nodes, which faces non-trivial challenges arising from the localized connections making influence ``cascade'' in a decentralized network. To overcome this, we design the first method to estimate \textbf{D}ata \textbf{I}nfluence \textbf{C}ascad\textbf{E} (DICE) in a decentralized environment. Theoretically, the framework derives tractable approximations of influence cascade over arbitrary neighbor hops, suggesting the influence cascade is determined by an interplay of data, communication topology, and the curvature of loss landscape. DICE also lays the foundations for applications including selecting suitable collaborators and identifying malicious behaviors. Project page is available at https://raiden-zhu.github.io/blog/2025/DICE/.

Figures

Figures reproduced from arXiv: 2507.06931 by the authors.

Figure 1
Figure 1. A semantic visualization of influence cascade in decentralized learning with ResNet-18 on [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A comparative illustration of server-based learning versus decentralized learning. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Alignment between one-hop DICE-GT (vertical axis) and DICE-E (horizontal axis) on a [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Anomaly detection on exponential graph with 32 nodes. Each node uses a 512-sample [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 36 canonical work pages

  1. [1]

    Let G = (V, E) denote a decentralized communication topology, where V is the set of n nodes and E is the set of edges

    To accommodate such cases while still ensuring the convergence of decentralized SGD (Yuan et al., 2019; Xin et al., 2019), we introduce a relaxed condition: Definition A.2 (Row Stochastic Matrix). Let G = (V, E) denote a decentralized communication topology, where V is the set of n nodes and E is the set of edges. For any G = ( V, E), the row stochastic g...

    work page 2019

  2. [2]

    The proof is given below

    can be linearly approximated as follows: I (1) DICE-E(zt j, z′) = − qj ∇L(θt j; z′)⊤∆j(θt j, zt j) − X k∈N (1) out (j) qk W t k,j ∇L(θt+1 k ; z′)⊤∆j(θt j, zt j), (C.1) where ∆j(θt j, zt j) = Oj(θt j, zt j) − θt j. The proof is given below. Proof. Recall from Definition 2 that the one-hop DICE-GT is defined by I (1) DICE-GT(zt j, z′) = qj L(θ t+ 1 2 j ; z′...

    work page 2025

  3. [3]

    andH(θt+1 k ; zt+1 k ) denotes the Hessian matrix of L with respect to θt+1 k evaluated at zt+1 k

    can be approximated as I (2) DICE-E(zt j, z′) = I (1) DICE-E(zt j, z′) − X k∈N (1) out (j) X l∈N (1) out (k) ηtqlW t+1 l,k W t k,j ∇L(θt+2 l ; z′)⊤(I − ηt+1H(θt+1 k ; zt+1 k ))∆j(θt j; zt j), (C.2) where ∆j(θt j, zt j) ≜ Oj(θt j, zt j) − θt j. andH(θt+1 k ; zt+1 k ) denotes the Hessian matrix of L with respect to θt+1 k evaluated at zt+1 k . Proof. We beg...

    work page 2025

  4. [4]

    Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli

    URL https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024/#gemini-2-0-flash . Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. In 28th USENIX Secu...

    work page 2024

  5. [7]

    On three-layer data markets

    Alireza Fallah, Michael I Jordan, Ali Makhdoumi, and Azarakhsh Malekian. On three-layer data markets. arXiv preprint arXiv:2402.09697,

    arXiv

  6. [12]

    Heavy-tail phenomenon in decentralized sgd

    Mert Gurbuzbalaban, Yuanhan Hu, Umut Simsekli, Kun Yuan, and Lingjiong Zhu. Heavy-tail phenomenon in decentralized sgd. arXiv preprint arXiv:2205.06689,

    arXiv

  7. [14]

    Federated learning of a mixture of global and local models

    Filip Hanzely and Peter Richt´arik. Federated learning of a mixture of global and local models. arXiv preprint arXiv:2002.05516,

    arXiv 2002

  8. [16]

    Algorithmic progress in language models

    Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, and Jaime Sevilla. Algorithmic progress in language models. arXiv preprint arXiv:2403.05812,

    arXiv

Show all 53 references
  1. [17]

    An empirical analysis of compute-optimal large language model training

    12 Published as a conference paper at ICLR 2025 Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George v...

    2025

  2. [18]

    ICML 2024 Tutorial

    URL https://ml-data-tutorial.org/. ICML 2024 Tutorial. Sami Jaghouar, Jack Min Ong, Manveer Basra, Fares Obeid, Jannik Straube, Michael Keiblinger, Elie Bakouch, Lucas Atkins, Maziyar Panahi, Charles Goddard, et al. Intellect-1 technical report. arXiv preprint arXiv:2412.01152,

    2024 arXiv

  3. [19]

    Subpopulation data poisoning attacks

    Matthew Jagielski, Giorgio Severi, Niklas Pousette Harger, and Alina Oprea. Subpopulation data poisoning attacks. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security,

    2021

  4. [20]

    Commodification of compute

    Jesper Kristensen, David Wender, and Carl Anthony. Commodification of compute. arXiv preprint arXiv:2406.19261,

    arXiv

  5. [21]

    Gradient-based learning applied to document recognition

    13 Published as a conference paper at ICLR 2025 Yann LeCun, L´eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324,

    2025

  6. [22]

    Textbooks are all you need ii: phi-1.5 technical report

    Yuanzhi Li, S´ebastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463,

    arXiv

  7. [25]

    The AI index 2024 annual report

    Nestor Maslej, Loredana Fattorini, Raymond Perrault, Vanessa Parli, Anka Reuel, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, and Jack Clark. The AI index 2024 annual report. Technical report, AI...

    2024

  8. [27]

    Beyond neural scaling laws: beating power law scaling via data pruning

    15 Published as a conference paper at ICLR 2025 Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Advances in Neural Information Processing Systems,

    2025

  9. [28]

    Decentralized hyper-gradient computation over time-varying directed networks

    Naoyuki Terashita and Satoshi Hara. Decentralized hyper-gradient computation over time-varying directed networks. arXiv preprint arXiv:2210.02129,

    arXiv

  10. [29]

    Measure contribution of participants in federated learning

    Guan Wang, Charlie Xiaoqian Dang, and Ziye Zhou. Measure contribution of participants in federated learning. In 2019 IEEE International Conference on Big Data (Big Data), pp. 2597–2604,

    2019

  11. [30]

    Data shapley in one training run

    Jiachen T Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia. Data shapley in one training run. arXiv preprint arXiv:2406.11011,

    arXiv

  12. [31]

    A framework for incentivized collaborative learning

    Xinran Wang, Qi Le, Ahmad Faraz Khan, Jie Ding, and Ali Anwar. A framework for incentivized collaborative learning. arXiv preprint arXiv:2305.17052, 2023b. Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Selecting influential data for tar...

    arXiv

  13. [32]

    16 Published as a conference paper at ICLR 2025 Ran Xin, Chenguang Xi, and Usman A. Khan. Frost—fast row-stochastic optimization with uncoor- dinated step-sizes. EURASIP Journal on Advances in Signal Processing, 2019(1):1,

    2025

  14. [33]

    idml: Incentivized decentralized machine learning

    Haoxiang Yu, Hsiao-Yuan Chen, Sangsu Lee, Sriram Vishwanath, Xi Zheng, and Christine Julien. idml: Incentivized decentralized machine learning. arXiv preprint arXiv:2304.05354,

    arXiv

  15. [34]

    A comprehensive survey of incentive mechanism for federated learning

    Rongfei Zeng, Chao Zeng, Xingwei Wang, Bo Li, and Xiaowen Chu. A comprehensive survey of incentive mechanism for federated learning. arXiv preprint arXiv:2106.15406,

    arXiv

  16. [35]

    Decentralized SGD and average-direction SAM are asymptotically equivalent

    17 Published as a conference paper at ICLR 2025 Tongtian Zhu, Fengxiang He, Kaixuan Chen, Mingli Song, and Dacheng Tao. Decentralized SGD and average-direction SAM are asymptotically equivalent. In Proceedings of the 40th International Conference on Machine Learning, 2023b. 18...

    2025

  17. [36]

    Data influence estimators are broadly categorized into static and dynamic approaches5

    and defenses (Hammoudeh & Lowd, 2022). Data influence estimators are broadly categorized into static and dynamic approaches5. Specifically, static approaches include both retraining-based and one-point methods. Retraining-based methods, such as leave-one-out (Cook, 1977), Shap...

    2022

  18. [37]

    Fu et al

    and scaled for larger models (Guo et al., 2021; Schioppa et al., 2022), including LLMs (Grosse et al., 2023). Fu et al. (2022) extend influence function to Bayesian inference. While these static influence measures have elegant theoretical foundations, they are limited in chara...

    2022

  19. [38]

    The practicality of dynamic influence estimators is demonstrated by their applications in improving training processes in modern setups (Xia et al., 2024)

    and In-Run Data Shapley (Wang et al., 2024), which track the influence of training data points by averaging gradient similarities over time. The practicality of dynamic influence estimators is demonstrated by their applications in improving training processes in modern setups ...

    2023

  20. [39]

    Additionally, privacy concerns in decentralized learning are also critical, with efforts focusing on differentially privacy (Cyffers et al., 2024; Allouah et al.,

    optimization problems. Additionally, privacy concerns in decentralized learning are also critical, with efforts focusing on differentially privacy (Cyffers et al., 2024; Allouah et al.,

    2024

  21. [40]

    Theoretical Development of Decentralized Learning

    and data reconstruction attacks (Mrini et al., 2024). Theoretical Development of Decentralized Learning. In terms of optimization, earlier works on decentralized optimization (Nedic & Ozdaglar, 2009; Sayed, 2014; Yuan et al., 2016b; Lian et al.,

    2024

  22. [42]

    has employed a combination of data parallel and model parallel to collaboratively train large models with up to billions of param- eters. For a comprehensive overview of large-scale deep learning training, including data, model architecture, optimization strategies, budget con...

    2024

  23. [44]

    is a special case of Algorithm 1 where the averaging step is performed globally. This demonstrates that our framework accommodates decentralized learning with dynamic communication topologies and is applicable to both federated and decentralized learning paradigms, even though...

    2025

  24. [45]

    (2021) employing cosine similarity to recursively partition clients after convergence and Kim et al

    use client gradient similarities to form clusters, with Sattler et al. (2021) employing cosine similarity to recursively partition clients after convergence and Kim et al. (2024) dynamically applying spectral clustering to organize clients based on gradient features during tra...

    2021

  25. [48]

    , kρ) such that ks ∈ N(1) out (ks−1) for s = 1,

    can be approximated as I (r) DICE-E(zt j, z′) = − rX ρ=0 X (k1,...,kρ)∈P (ρ) j ηt qkρ ρY s=1 W t+s−1 ks,ks−1 ! ∇L θt+ρ kρ ; z′ ⊤ × ρY s=2 I − ηt+s−1 H θt+s−1 ks ; zt+s−1 ks ! ∆j(θt j, zt j), (C.3) where ∆j(θt j, zt j) ≜ Oj(θt j, zt j) − θt j, where k0 = j, P (ρ) j denotes the ...

    2025

  26. [49]

    to train 27 Published as a conference paper at ICLR 2025 three-layer MLPs (Rumelhart et al., 1986), three-layer CNNs (LeCun et al., 1998), and ResNet-18 (He et al.,

    2025

  27. [50]

    The number of participants (one GPU as a participant) is set to 16 and 32, with each participant holding 512 samples

    on subsets of MNIST (LeCun et al., 1998), CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and Tiny ImageNet (Le & Yang, 2015). The number of participants (one GPU as a participant) is set to 16 and 32, with each participant holding 512 samples. For sensitivity analysis, we eval...

    1998

  28. [51]

    power asymmetries

    and its first-order approximation, one-hop DICE-E (see Proposition 1). One-hop DICE-E I(1) DICE-E(Bt j ,z′) is computed as the sum of one-sample DICE-E within the mini-batch Bt j thanks to the additivity (see Equation (4)). DICE-GT IDICE-GT(1)(Bt j ,z′) is calculated by measur...

    2025

  29. [52]

    influence cascade

    in decentralized learning. To support the theoretical finding, we examine the one-hop DICE-E values of the same batch on participants with vastly different topological importance. Figure 1 illustrates the one-hop DICE-E influence scores of an identical data batch across partic...

    2025

  30. [53]

    The numerical labels on the nodes indicate the corresponding participants, aligning with the participant indices in Figure D.22

    computed for the same data batch across different participants. The numerical labels on the nodes indicate the corresponding participants, aligning with the participant indices in Figure D.22. D.4.2 M ULTI-HOP INFLUENCE CASCADE To better illustrate the communication structure ...

    2025

  31. [1960]

    Studying large language model generalization with influence functions

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296,

    arXiv

  32. [1982]

    Unraveling the impact of training samples

    Daiwei Chen, Jane Zhang, and Ramya Korlakai Vinayak. Unraveling the impact of training samples. In ICLR Blogposts 2024 , 2024a. URL https://iclr-blogposts.github. io/2024/blog/unraveling-the-impact-of-training-samples/ . https://iclr- blogposts.github.io/2024/blog/unraveling-t...

    2024

  33. [1985]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  34. [2008]

    Three approaches for personalization with applications to federated learning

    Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619,

    2002 arXiv

  35. [2014]

    Distributed subgradient methods for multi-agent optimization

    14 Published as a conference paper at ICLR 2025 Angelia Nedic and Asuman Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control, 54(1):48–61,

    2025

  36. [2016]

    Byzantine-robust decentralized learning via clippedgossip

    Lie He, Sai Praneeth Karimireddy, and Martin Jaggi. Byzantine-robust decentralized learning via clippedgossip. arXiv preprint arXiv:2202.01545,

    arXiv

  37. [2017]

    Lu & De Sa (2021) present a systematic framework for federated and decentralized learning by categorizing decentralization into three distinct layers

    lay the groundwork for understanding convergence. Lu & De Sa (2021) present a systematic framework for federated and decentralized learning by categorizing decentralization into three distinct layers. Koloskova et al. (2020) unify synchronous decentralized gradient descent alg...

    2021

  38. [2018]

    Consent in crisis: The rapid decline of the ai data commons

    Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, et al. Consent in crisis: The rapid decline of the ai data commons. arXiv preprint arXiv:2407.14933,

    arXiv

  39. [2019]

    Diloco: Distributed low- communication training of language models

    Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low- communication training of language models. arXiv preprint arXiv:2311.08105,

    arXiv

  40. [2020]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  41. [2021]

    11 Published as a conference paper at ICLR 2025 Hongchang Gao, Bin Gu, and My T. Thai. On the convergence of distributed stochastic bilevel optimization algorithms over a network. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, vo...

    2025

  42. [2022]

    Identifying a training-set attack’s target using renormalized influence estimation

    Zayd Hammoudeh and Daniel Lowd. Identifying a training-set attack’s target using renormalized influence estimation. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security,

    2022

  43. [2023]

    Petals: Collaborative inference and fine-tuning of large models

    Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and fine-tuning of large models. In Proceedings of the 61st Annual Meeting of the Association for Computatio...

    arXiv

  44. [2024]

    Cambridge bitcoin electricity consumption index (CBECI)

    10 Published as a conference paper at ICLR 2025 CCAF. Cambridge bitcoin electricity consumption index (CBECI). https://ccaf.io/cbnsi/ cbeci,

    2025

  45. [2025]

    FastIF: Scalable influence functions for efficient model interpretation and debugging

    Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. FastIF: Scalable influence functions for efficient model interpretation and debugging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.