REVIEW 3 major objections 5 minor 1 cited by
Learning In Chaos: Efficient Autoscaling and Self-Healing for Multi-Party Distributed Training
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A greedy shard-assignment rule makes cross-region cluster joins sub-second
desk verdict Nice system-level idea; the optimization is a known greedy in disguise, and the <20ms latency headline is not supported by the measurements as reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a multi-neighbor state replication model with equal-sized model shards, carried by two structural results. First, monotonicity on a divisibility chain ($s_1\mid s_2$ implies $\theta^{(s_1)}\le\theta^{(s_2)}$) fixes the shard size and turns the MINLP into a fixed-dimension 0-1 MILP. Second, an equivalence between P3 and a multi-way merge problem over per-neighbor completion-time sequences turns the MILP into a greedy choice: always hand the next shard to the neighbor whose next-finish time is smallest, which is exactly $K$ steps of merging sorted sequences and is optimal by the standard multi-way merge argument. The greedy algorithm is Algorithm 1, running in $O(mK)$ time or $O((K-m)\log m)$ with a min-heap.
What would settle it
Run a controlled WAN test with fixed total bytes but different shard counts and compare the greedy assignment to a MILP optimum that includes a per-stream setup cost: if actual replication time changes with shard count, or if the greedy assignment's measured makespan exceeds the MILP optimum beyond noise, Assumption 1 and the optimality claim fail.
Extended reading notes
Core claim
The paper's central claim is that the optimal multi-neighbor state replication problem, an intractable mixed-integer nonlinear program in the shard size and assignment, can be solved exactly by a simple greedy rule. The key step is a monotonicity result: along a chain of shard sizes where each divides the next, the optimal replication time is non-decreasing, so the smallest atomic shard size is optimal and shard size can be fixed. With shard size fixed, the problem becomes the 0-1 MILP P3, which the paper proves equivalent to a multi-way merge of arithmetic progressions $T_u=\{r_u+p_u, r_u+2p_u, \dots\}$, where $r_u$ is the propagation-plus-synchronization delay for neighbor $u$ and $p_u$ is the per-shard transmission cost. Because the greedy rule of repeatedly assigning the next shard to the neighbor with the least next-finish time picks the $K$ smallest elements of the merged sequences, it minimizes the makespan and is optimal. The measured consequence is that a new node's state replication, previously minutes-long with checkpoint-restart or single-source fetch, drops to about one second.
Load-bearing premise
The load-bearing premise is that the time to copy a training state from a neighbor depends only on how many bytes that neighbor is asked to send, not on how the bytes are split into shards; if per-shard connection and packetization overheads matter, the greedy solution may no longer be optimal.
Editorial extensions
If this is right
- A node can join a running synchronous WAN training cluster with about one second of interruption, and scale-in, connect-link, and disconnect-link finish within 20 ms, so elastic membership no longer requires stopping the job or a central GPU scheduler.
- Scale-out delay grows linearly in the model's shard count and stays flat, sometimes decreasing, as the cluster grows, because additional neighbors add parallel bandwidth instead of longer transfer paths.
- Cluster-wide GPU idle time stays under 10 seconds during scaling because only a few neighbor nodes replicate state while the rest continue computing, making frequent one-at-a-time joins and exits affordable.
- Model accuracy moves smoothly between the fixed-cluster curves after a single join or exit, and a sequence of churn events converges close to the fixed larger-cluster accuracy, so adding or removing data under churn need not disrupt training.
- The shard-assignment plan can be computed and applied while all-reduce and gradient computation are still running, so the planning latency of roughly 160 ms for standard models is hidden rather than added to the training critical path.
Reading between the lines
- The same greedy rule would apply to pulling any shardable state from multiple holders, such as fine-tuning adapters, embeddings, or optimizer moments, as long as transfer cost is linear in bytes per stream.
- If Assumption 1 is relaxed to include a fixed cost per stream, the optimal assignment may stop being pure byte-balancing for very small shards; this is directly testable by adding a setup-cost term to the multi-way merge problem.
- The reported 160 ms planning time for AlexNet does not automatically generalize to billions of parameter shards; the heap-based $O(K\log m)$ variant and the all-reduce overlap would need re-measurement at LLM scale.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Chaos, a multi-party distributed training system for cross-region WAN clusters that must handle node and link churn without a central GPU scheduler. Chaos uses multi-neighbor state replication with model sharding: the replication problem is formulated as a MINLP, reduced via a monotonicity result to a 0-1 MILP, and then solved by a greedy algorithm that is argued to be optimal through an equivalence to multi-way merge. The system design includes a cluster monitor, peer negotiation protocols for scale-out, scale-in, connect-link, and disconnect-link, and an overlap analysis showing how much of the scaling delay is hidden by training. The evaluation compares Chaos with Pollux, Elan, and Autoscaling on scale-out delay and cluster idle time, reports convergence behavior under churn, and claims that scale-in, connect-link, and disconnect-link complete within 20ms.
Significance. If the claims hold, the paper addresses a real and growing problem: elastic, self-governed multi-party training over WANs. The optimization section is a genuine strength: the reduction from the MINLP to a multi-way merge formulation is self-contained, the greedy optimality proof is standard and correctly applied under the stated divisibility and linearity assumptions, and the complexity analysis is clear. The system design, with control-plane scaling protocols that overlap with training, is reasonable and the evaluation covers several models and cluster sizes. However, the two headline empirical claims, sub-second scale-out and sub-20ms self-healing/autoscaling actions, are not fully established as stated; the 20ms claim in particular appears to be either measured over a restricted interval or physically inconsistent with the testbed's link latency. The convergence experiments are suggestive but lack details on data partitioning, so the model-quality conclusions should be viewed as preliminary.
major comments (3)
- [Section VI-D, Fig. 11, Table I] The claim that connect-link completes within 20ms is not established by the reported measurement and appears inconsistent with the testbed. Section VI-A sets link latency to 10-20ms one-way, so a TCP handshake for a new socket costs at least one RTT, i.e., 20-40ms, and the connect-link protocol in Section IV-B also involves a request to the scheduler, socket establishment, and cluster-monitor measurement before the policy update. The paper must report the full end-to-end delay, state exactly which time interval Fig. 11 measures (e.g., only the policy-update phase after the socket is up), or revise the claim. In addition, scale-in and disconnect-link are asserted to behave like connect-link but are not directly measured; the paper should measure them or clearly label the 20ms entries in Table I as projections.
- [Section IV-A and Table I] The 20ms scale-in delay conflates the scaling action with failure detection. The cluster monitor detects node failure only after a heartbeat timeout, and the text says heartbeats are sent every few seconds. Thus the end-to-end time from a node failure to resumed training necessarily includes this timeout, which is far larger than 20ms. Table I's Event Delay for 'Node exits or failure' should be interpreted as the negotiation-only delay, and the abstract and Section VII should not claim that failure handling itself completes within 20ms.
- [Section III-A, Assumption 1] The optimality result is conditional on Assumption 1, which states that each neighbor's completion time depends only on the total bytes assigned and ignores per-shard overhead such as packetization, handshake, and per-stream scheduling. This is a load-bearing assumption for Theorem 1, the MINLP-to-MILP reduction, and the greedy optimality claim. Since the paper's own motivation involves very large K and many small shards, the authors should either provide empirical evidence that per-shard overhead is negligible in their setting or discuss the sensitivity of the greedy solution to this assumption. Without such a discussion, the claim that Algorithm 1 'yields the optimal solution' is only an optimality result for the linear model, not for the real system.
minor comments (5)
- [Section VI-B, Figs. 8, 9, 16] The text says each run was repeated five times, but the figures show no error bars, confidence intervals, or per-run dispersion. For claims about delays that differ by orders of magnitude across systems, reporting mean and standard deviation would substantially strengthen the presentation.
- [Section VI-D] Fig. 11 shows only connect-link delay. If scale-in and disconnect-link are claimed to behave identically, the measurement should be provided; alternatively, the text should state clearly that these values are inferred from the connect-link experiment rather than measured.
- [Section III-A, Eqs. (1)-(6)] The notation for propagation delay is inconsistent: Eq. (1) uses t_prop_{u→vnew}, while later formulations use t_prop^u and related variants. The notation should be unified, and the units of t_trans should be stated explicitly.
- [Section VI-E and Figs. 12-15] The convergence experiments do not describe how the training data is partitioned among nodes or how a joining node's data is selected, which makes it hard to interpret claims about data being 'injected' or 'withdrawn'. A sentence describing the data split per node would be helpful.
- [Section IV-B and Fig. 5] The protocols rely on a scheduler for coordination, while the paper emphasizes a fully self-governed setting. The relationship between 'no central scheduler' and the scheduler role in Fig. 5 should be clarified, for example by describing the scheduler as an elected coordinator rather than an authority.
Circularity Check
No material circularity; optimality reduces to standard multi-way merge and delays are externally benchmarked.
full rationale
Chaos's central optimization claim is not circular. In Section III-B, the paper reduces P3 to P4, a multi-way merge problem with sorted sequences, and invokes the standard greedy optimality result [38] for that problem. Proposition 1 supplies the equivalence, and Lemma 1 is the standard greedy argument; no fitted parameter is later renamed as a prediction. Algorithm 1 takes measured propagation, transmission, and synchronization delays as inputs and optimizes the replication makespan, while scale-out delays are compared against external baselines (Pollux, Elan, Autoscaling) on a separately configured testbed. The paper itself states 'Although no new algorithm is proposed,' which is an admission that the greedy rule is a known result rather than a disguised input. The self-citations ([35]-[37] as complementary techniques, and [36] for the WAN emulation configuration) are not load-bearing for the optimality or delay claims. Assumption 1 (ignoring per-shard overhead) is an explicit modeling idealization: Theorem 1 and the greedy optimality are conditional on it, and the theorem is not used to justify the assumption, so this is a correctness risk rather than circularity. Some empirical statements are unsupported, notably Section VI-D's assertion that the under-20 ms connect-link result 'also applies to scale-in and disconnect-link' without separate measurements, and the reported sub-20 ms figures sit close to the 10-20 ms link latency. That is an evidence/completeness concern, not a circular derivation. Overall, the derivation chain is self-contained.
Assumptions & free parameters
free parameters (2)
- atomic shard size s_min =
H, the size of the smallest tensor/vector (e.g., transformer hidden dimension)
- heartbeat timeout for failure detection
assumptions (6)
- domain assumption Per-shard overhead is zero: completion time depends only on total bytes assigned to each neighbor (Assumption 1).
- domain assumption The model state can be split into equal-sized shards, and an atomic shard size s_min divides the total parameter count.
- domain assumption All neighbors hold identical model weights at replication time, so the new node may fetch any shard from any neighbor.
- domain assumption Link propagation delay, transmission delay, and synchronization delay are accurately measurable in the control plane.
- domain assumption A designated scheduler coordinates scaling events and can fail over via periodic state backup.
- domain assumption Linux tc bandwidth and latency emulation is representative of real cross-region WAN behavior.
Cite this review
Pith. "Pith review of Learning In Chaos: Efficient Autoscaling and Self-Healing for Multi-Party Distributed Training." pith.science (2026). https://pith.science/paper/SVQDZ3RW
@misc{pith2026250512815,
author = {Pith},
title = {Pith review of: Learning In Chaos: Efficient Autoscaling and Self-Healing for Multi-Party Distributed Training},
year = {2026},
howpublished = {\url{https://pith.science/paper/SVQDZ3RW}},
note = {Machine review of arXiv:2505.12815}
}
read the original abstract
Node and link churn in multi-party, cross-region clusters over wide-area networks (WANs) often disrupts distributed training. However, checkpoint-based recovery and cloud-centric autoscaling react slowly and assume centralized control, which is misaligned with the self-governed setup where institutions can freely join and leave. This paper proposes Chaos, a multi-party distributed training system with self-healing and autoscaling, enabling robust and elastic training under churn. It speeds up autoscaling via multi-neighbor state replication and model sharding. We formalize the sharding and assignment as a MINLP that captures WAN heterogeneity, and reduce it to a tractable MILP by analyzing its monotonicity on a divisibility chain. By establishing an equivalence, we derive a greedy algorithm that follows optimality rules and yields the optimal solution in polynomial time. Chaos uses a cluster monitor to track resource and topology changes, and handles scaling events through peer negotiation protocols, enabling fully self-governed autoscaling among institutions. Experiments show that Chaos has substantially lower scale-out delay than Pollux, Elan, and Autoscaling, and handles scale-in, connect-link, and disconnect-link events within 20ms. It also delivers the lowest idle time, showing superior resource use and scalability as the cluster grows.
Figures
Figures from the paper (12 more)
Forward citations
Cited by 1 Pith paper
-
Toward Edge General Intelligence with Multiple-Large Language Model (Multi-LLM): Architecture, Trust, and Orchestration
A survey of multi-LLM systems in edge computing, covering architectures, enabling technologies, trust mechanisms, applications, and open datasets for edge general intelligence.
Reference graph
Works this paper leans on
-
[1]
Federated learning for predicting clinical outcomes in patients with covid-19,
I. Dayan, H. R. Roth, A. Zhonget al., “Federated learning for predicting clinical outcomes in patients with covid-19,”Nature Medicine, vol. 27, no. 10, pp. 1735–1743, 2021
work page 2021
-
[2]
Federated learning for breast density classification: A real-world implementation,
H. R. Roth, K. Chang, P. Singhet al., “Federated learning for breast density classification: A real-world implementation,” inMICCAI Work- shop on Domain Adaptation and Representation Transfer. Springer, 2020, pp. 181–191
work page 2020
-
[3]
Federated learning improves site performance in multicenter deep learning without data sharing,
K. V . Sarma, S. Harmon, T. Sanfordet al., “Federated learning improves site performance in multicenter deep learning without data sharing,” Journal of the American Medical Informatics Association, vol. 28, no. 6, pp. 1259–1264, 2021
work page 2021
-
[4]
L. Peng, G. Luo, A. Walkeret al., “Evaluation of federated learning variations for covid-19 diagnosis using chest radiographs from 42 us and european hospitals,”Journal of the American Medical Informatics Association, vol. 30, no. 1, pp. 54–63, 2022
work page 2022
-
[5]
Federated learning enables big data for rare cancer boundary detection,
S. Pati, U. Baid, B. Edwardset al., “Federated learning enables big data for rare cancer boundary detection,”Nature Communications, vol. 13, no. 1, p. 7346, 2022
work page 2022
-
[6]
W. Heyndrickx, L. Mervin, T. Morawietzet al., “Melloddy: Cross- pharma federated learning at unprecedented scale unlocks benefits in qsar without compromising proprietary information,”Journal of Chem- ical Information and Modeling, vol. 64, no. 7, pp. 2331–2344, 2023
work page 2023
-
[7]
Z. Zhang, Y . Yang, Y . Daiet al., “Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models,” inAnnual Meeting of the Association of Computational Lin- guistics, 2023, pp. 9963–9977
work page 2023
-
[8]
Federated fine-tuning of large language models under heterogeneous tasks and client resources,
J. Bai, D. Chen, B. Qianet al., “Federated fine-tuning of large language models under heterogeneous tasks and client resources,” inAdvances in Neural Information Processing Systems, vol. 37, 2024, pp. 14 457– 14 483
work page 2024
Show all 38 references
-
[9]
Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations,
Z. Wang, Z. Shen, Y . Heet al., “Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations,” inAdvances in Neural Information Processing Systems, vol. 37, 2024, pp. 22 513– 22 533
2024
-
[10]
Federated residual low-rank adaptation of large language models,
Y . Yan, C.-M. Feng, W. Zuoet al., “Federated residual low-rank adaptation of large language models,” inInternational Conference on Learning Representations, 2025
2025
-
[11]
Fedex-lora: Exact aggrega- tion for federated and efficient fine-tuning of large language models,
R. Singhal, K. Ponkshe, and P. Vepakomma, “Fedex-lora: Exact aggrega- tion for federated and efficient fine-tuning of large language models,” in 63rd Annual Meeting of the Association for Computational Linguistics, 2025, pp. 1316–1336
2025
-
[12]
Improving lora in privacy-preserving federated learning,
Y . Sun, Z. Li, Y . Li, and B. Ding, “Improving lora in privacy-preserving federated learning,” inInternational Conference on Learning Represen- tations, 2024, pp. 1–17
2024
-
[13]
Unicron: Economizing self-healing llm training at scale,
T. He, X. Li, Z. Wanget al., “Unicron: Economizing self-healing llm training at scale,”arXiv preprint arXiv:2401.00134, 2023
2023 arXiv
-
[14]
Datastates-llm: Lazy asynchronous checkpointing for large language models,
A. Maurya, R. Underwood, M. M. Rafiqueet al., “Datastates-llm: Lazy asynchronous checkpointing for large language models,” in33rd International Symposium on High-Performance Parallel and Distributed Computing, 2024, pp. 227–239
2024
-
[15]
Checkfreq: Frequent, fine-grained dnn checkpointing,
J. Mohan, A. Phanishayee, and V . Chidambaram, “Checkfreq: Frequent, fine-grained dnn checkpointing,” in19th USENIX Conference on File and Storage Technologies, 2021, pp. 203–216
2021
-
[16]
Just-in-time checkpointing: Low cost error recovery from deep learning training failures,
T. Gupta, S. Krishnan, R. Kumaret al., “Just-in-time checkpointing: Low cost error recovery from deep learning training failures,” in19th European Conference on Computer Systems, 2024, pp. 1110–1125
2024
-
[17]
Check-n-run: A check- pointing system for training deep learning recommendation models,
A. Eisenman, K. K. Matam, S. Ingramet al., “Check-n-run: A check- pointing system for training deep learning recommendation models,” in 19th USENIX Symposium on Networked Systems Design and Implemen- tation, 2022, pp. 929–943
2022
-
[18]
Gemini: Fast failure recovery in distributed training with in-memory checkpoints,
Z. Wang, Z. Jia, S. Zhenget al., “Gemini: Fast failure recovery in distributed training with in-memory checkpoints,” inProceedings of the 29th Symposium on Operating Systems Principles, 2023, pp. 364–381
2023
-
[19]
Pollux: Co-adaptive clus- ter scheduling for goodput-optimized deep learning,
A. Qiao, S. K. Choe, S. J. Subramanyaet al., “Pollux: Co-adaptive clus- ter scheduling for goodput-optimized deep learning,” in15th USENIX Symposium on Operating Systems Design and Implementation, 2021
2021
-
[20]
Rubick: Exploiting job re- configurability for deep learning cluster scheduling,
X. Zhang, H. Zhao, W. Xiaoet al., “Rubick: Exploiting job re- configurability for deep learning cluster scheduling,”arXiv preprint arXiv:2408.08586, 2024
2024 arXiv
-
[21]
Optimus: an efficient dynamic resource scheduler for deep learning clusters,
Y . Peng, Y . Bao, Y . Chenet al., “Optimus: an efficient dynamic resource scheduler for deep learning clusters,” inProceedings of the 13th EuroSys Conference, 2018, pp. 1–14
2018
-
[22]
Deepboot: Dynamic scheduling system for training and inference deep learning tasks in gpu cluster,
Z. Chen, X. Zhao, C. Zhi, and J. Yin, “Deepboot: Dynamic scheduling system for training and inference deep learning tasks in gpu cluster,” IEEE Transactions on Parallel and Distributed Systems, 2023
2023
-
[23]
Universal checkpointing: A flexible and efficient distributed checkpointing system for large-scale dnn training with reconfigurable parallelism,
X. Lian, S. A. Jacobs, L. Kurilenkoet al., “Universal checkpointing: A flexible and efficient distributed checkpointing system for large-scale dnn training with reconfigurable parallelism,” inUSENIX Annual Technical Conference, 2025, pp. 1519–1534
2025
-
[24]
Paddlepaddle: An open-source deep learning platform from industrial practice,
Y . Ma, D. Yu, T. Wu, and H. Wang, “Paddlepaddle: An open-source deep learning platform from industrial practice,”Frontiers of Data and Domputing, vol. 1, no. 1, pp. 105–115, 2019
2019
-
[25]
Elasticdl: A kubernetes-native deep learning framework with fault-tolerance and elastic scheduling,
J. Zhou, K. Zhang, F. Zhuet al., “Elasticdl: A kubernetes-native deep learning framework with fault-tolerance and elastic scheduling,” in16th ACM International Conference on Web Search and Data Mining, 2023, pp. 1148–1151
2023
-
[26]
Dl2: A deep learning-driven scheduler for deep learning clusters,
Y . Peng, Y . Bao, Y . Chenet al., “Dl2: A deep learning-driven scheduler for deep learning clusters,”IEEE Transactions on Parallel and Dis- tributed Systems, vol. 32, no. 8, pp. 1947–1960, 2021
1947
-
[27]
Elastic parameter server: Accelerating ml training with scalable resource scheduling,
S. Wang, A. Pi, and X. Zhou, “Elastic parameter server: Accelerating ml training with scalable resource scheduling,”IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 5, pp. 1128–1143, 2021
2021
-
[28]
Elastic deep learning in multi-tenant gpu clusters,
Y . Wu, K. Ma, X. Yanet al., “Elastic deep learning in multi-tenant gpu clusters,”IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 1, pp. 144–158, 2021
2021
-
[29]
Elastic resource sharing for distributed deep learning,
C. Hwang, T. Kim, S. Kimet al., “Elastic resource sharing for distributed deep learning,” in18th USENIX Symposium on Networked Systems Design and Implementation, 2021, pp. 721–739
2021
-
[30]
Elan: Towards generic and efficient elastic training for deep learning,
L. Xie, J. Zhai, B. Wuet al., “Elan: Towards generic and efficient elastic training for deep learning,” inIEEE 40th International Conference on Distributed Computing Systems, 2020, pp. 78–88
2020
-
[31]
Resource elasticity in distributed deep learning,
A. Or, H. Zhang, and M. Freedman, “Resource elasticity in distributed deep learning,”Proceedings of Machine Learning and Systems, vol. 2, pp. 400–411, 2020
2020
-
[32]
Comparing decentralized learning to federated learning when training deep neural networks under churn,
J. Vikstr ¨om, “Comparing decentralized learning to federated learning when training deep neural networks under churn,” Dissertation, KTH Royal Institute of Technology, Stockholm, Sweden, 2021
2021
-
[33]
Federated learning with dynamic client arrival and departure: Convergence and rapid adaptation via initial model construction,
Z. Chang, D. Han, S. Hosseinalipouret al., “Federated learning with dynamic client arrival and departure: Convergence and rapid adaptation via initial model construction,”arXiv preprint arXiv:2410.05662, 2024
2024 arXiv
-
[34]
Mimic: Combating client dropouts in federated learning by mimicking central updates,
Y . Sun, Y . Mao, and J. Zhang, “Mimic: Combating client dropouts in federated learning by mimicking central updates,”IEEE Transactions on Mobile Computing, vol. 23, no. 7, pp. 7572–7584, 2023
2023
-
[35]
Esync: Accelerating intra-domain federated learning in heterogeneous data centers,
Z. Li, H. Zhou, T. Zhouet al., “Esync: Accelerating intra-domain federated learning in heterogeneous data centers,”IEEE Transactions on Services Computing, vol. 15, no. 4, pp. 2261–2274, 2020
2020
-
[36]
Accelerating geo-distributed ma- chine learning with network-aware adaptive tree and auxiliary route,
Z. Li, W. Feng, W. Caiet al., “Accelerating geo-distributed ma- chine learning with network-aware adaptive tree and auxiliary route,” IEEE/ACM Transactions on Networking, no. 01, pp. 1–16, 2024
2024
-
[37]
Data heterogeneity-robust federated learning via group client selection in industrial iot,
Z. Li, Y . He, H. Yuet al., “Data heterogeneity-robust federated learning via group client selection in industrial iot,”IEEE Internet of Things Journal, vol. 9, no. 18, pp. 17 844–17 857, 2022
2022
-
[38]
The art of computer programming,
E. K. Donaldet al., “The art of computer programming,”Sorting and searching, vol. 3, no. 426-458, p. 4, 1999
1999
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.