Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism

Alexander Long; Gil Avraham; Sameera Ramasinghe; Thalaiyasingam Ajanthan; Yan Zuo

REVIEW 4 major objections 5 minor 63 references

Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By confining transformer projection weights to a shared low-dimensional subspace, this paper claims up to 100x communication compression in model-parallel training, with lossless forward and backward transfers and convergence matching…

desk verdict Forward-pass subspace compression is clever and the experiments are strong, but the lossless backprop claim fails because residual skip connections carry raw gradients outside the shared subspace. read the letter →

arxiv 2506.01260 v3 pith:KPZN3ERK submitted 2025-06-02 cs.LG

Sameera Ramasinghe , Thalaiyasingam Ajanthan , Gil Avraham , Yan Zuo , Alexander Long This is my paper

classification cs.LG

keywords Low-rankcompressionPipelineparallelismDecentralizedtrainingModelTransformernetworksLosslesscommunicationRankcollapseSubspace-constrainedoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the main obstacle to decentralized model-parallel training—the need to ship activations and activation gradients between GPUs—can be removed by a structural constraint rather than lossy compression. The proposal confines the rows of a transformer's projection matrices to one shared low-dimensional subspace, and because transformer layer outputs are recursive sums of projected terms, the residual activations then lie in that subspace and can be transmitted as short coordinates with exact reconstruction at the next stage. The same subspace is used in the backward pass, where the paper claims gradients can likewise be reconstructed without loss. The practical payoff is up to 100x communication compression: billion-parameter models trained over 80Mbps links are reported to converge at the same wall-clock speed as centralized 100Gbps clusters, including an 8B model spread across four geographic regions.

What carries the argument

The central object is the shared k-dimensional subspace S with orthonormal basis U_k ∈ $R^{{d×k}}$, which gives one coordinate system in which both forward activations and backward gradient tensors are exactly representable with k coordinates instead of d. It works through two properties: (1) the recursion/row-space closure, where $X^{{l+1}}$ = Σ_i (X_hidden^i W_p2^i + X_concat^i W_p1^i) + PE + TE means that if Row(W_p2^i) and Row(W_p1^i) lie in S then residual activations lie in S, so U_kU_k^T acts as identity; and (2) the optimizer-in-the-subspace property, where AdamW with decoupled weight decay and a row-wise constant adaptive rate keeps W_p2 rows in S without iterative projection. The design also uses a Grassmann-manifold step to slowly rotate U_k every few hundred iterations, plus a fixed-plus-trainable decomposition of the token embedding so the high-rank part does not need to be retransmitted.

What would settle it

Train the subspace-compressed transformer and periodically measure the off-subspace component (I − U_kU_k^T)∇L($X^{{l+1}}$) for a layer with a residual skip connection; if at any checkpoint that component's norm is a non-negligible fraction of the total gradient norm, the backward pass is lossy and the reported losslessness is refuted.

Watch

Extended reading notes

Core claim

The central claim is that transformer layer outputs have a recursive structure — $X^{{l+1}}$ = ∑_{i≤l}(X_hidden^i W_p2^i + X_concat^i W_p1^i) + PE + TE — and because Row(AB) ⊆ Row(B), confining the rows of all projection matrices to a fixed k-dimensional subspace S (orthonormal basis U_k) forces the residual activations \hat{X}^{l+1} = $X^{{l+1}}$ − PE − TE into S. Then \hat{X}^{l+1} = \hat{X}^{l+1}U_kU_k^T, so sending only the short vector \hat{X}^{l+1}U_k and multiplying by the shared U_k^T recovers $X^{{l+1}}$ exactly after adding back the fixed positional and token-embedding terms. The same subspace is claimed to make the backward pass lossless: because Row(W_p2) ⊆ S, projecting ∇L($X^{{l+1}}$) onto S does not change the gradient propagated through W_p2, and the recursive chain extends this to earlier layers without approximation. A modified AdamW with row-wise constant scaling keeps W_p2 inside S, an infrequent Grassmann-manifold step lets S drift with the optimization, and the token embedding is split into a fixed high-rank part sent once plus a trainable low-rank part. On 2B- and 8B-parameter LLaMA-style models, the paper reports up to 100x communication compression with convergence matching centralized 100Gbps training on links as slow as 60–80Mbps.

Load-bearing premise

Backward-pass losslessness holds only if the gradient of the loss with respect to a layer's output, ∇L($X^{{l+1}}$), lies in the shared subspace S; the paper's backward-pass derivation (Appendix A) follows the path through W_p2 and does not cover the residual skip connection, which passes ∇L($X^{{l+1}}$) unchanged to earlier layers.

Editorial extensions

If this is right

Model-parallel decentralized training of billion-parameter transformers becomes practical over consumer internet links: the 8B LLaMA run matches centralized wall-clock convergence across four geographic regions at 60–350Mbps.
Communication drops by up to 100x (k=40 vs d=4096) with no convergence loss, and at 80Mbps the compressed model's perplexity beats the uncompressed 100Gbps centralized baseline on the same wall-clock budget.
Lossy compression schemes (top-k, SVD, quantization) fail at 100x in model-parallel settings, so the lossless subspace construction is what makes the reported throughput gains real.
Inference also speeds up roughly 100x at low bandwidth and about 3x even at 100Gbps, meaning centralized clusters can benefit from the same compression.
Memory overhead stays near-constant at about 400MB as sequence length and worker count grow, because the fixed embeddings are ephemeral and are discarded before attention computation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If backward-pass losslessness is genuine, then activation gradients in trained transformers must themselves be nearly confined to the subspace S; a direct empirical check would be to report the per-layer ratio ||(I − U_kU_k^T)∇L(X^{l+1})|| / ||∇L(X^{l+1})|| during training.
The residual skip connection is the natural stress point, since it carries ∇L(X^{l+1}) unchanged to earlier layers; the paper's Appendix A derivation follows the path through W_p2 and would need an explicit argument for that identity path to close the losslessness claim.
The subspace constraint appears to act as an implicit regularizer, with compressed models slightly outperforming centralized baselines in perplexity; if confirmed, the constraint could be used deliberately even outside distributed training.
The method fixes k=40 a priori; a natural extension is to grow k over training or make it layer-dependent, trading communication cost for model capacity as optimization converges.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The new idea here is real: force projection matrices into a shared low-rank subspace so transformer activations become exactly reconstructable after compression. The forward-pass construction is mathematically sound, and the 100x communication figure is plausible from the architecture. The experiments are also serious — an 8B model trained over 60–80Mbps links matching centralized convergence is a headline result if it holds up.

But the backward-pass losslessness claim does not survive contact with the architecture. The Appendix A proof (Eqs. 19–24) shows only that the path through W_p2 is invariant under projection because Row(W_p2) is contained in S. It ignores the residual skip connection. Eq. 16 correctly identifies that the residual gradient is ∇L(X^{l+1}) itself, and Eqs. 26–27 include it as an additive term, but by that point in the compressed scheme the received gradient has already been replaced by P_S g. For a minimal residual layer y = xW + x with Row(W) in S, the true gradient is gW^T + g, while the compressed scheme gives gW^T + P_S g, which differ whenever g has an outside-S component. Nothing in the architecture confines layer-output gradients to S: attention and MLP weights are not constrained, and the final loss gradient is arbitrary. So the central claim — lossless gradient compression with no convergence degradation — is not supported by the paper's theory.

The convergence lemmas also rest on assumptions (low-rank gradients, diminishing variance) that are validated only on the same training setup, not independently. The empirical results lack code, seeds, and error bars, so the strong performance claims are hard to verify. The ablations are thoughtful and the pretrained-checkpoint rank analysis is a nice touch, but they do not repair the theoretical gap.

This paper is for researchers in decentralized training and compression. The forward-pass trick alone might be worth keeping if the backward pass is reframed as approximate with explicit error bounds on the residual component. As it stands, the paper overclaims. I would send it to review rather than desk reject — the idea is novel, the flaw is specific and fixable, and the empirical evidence, if released, could be valuable. A serious referee should focus on the backward-pass gradient path and on bounds for the unprojected residual component.

Referee Report

4 major / 5 minor

Summary. The paper proposes a communication-compression scheme for pipeline-model-parallel training of transformer networks. The method restricts the row spaces of the two projection matrices in each transformer block to a shared, slowly drifting low-dimensional subspace S, and decomposes token embeddings into a fixed high-rank part and a trainable low-rank part. Under this construction, the forward-pass activations (after subtracting the fixed embedding contribution) are claimed to lie exactly in S, so communicating only the S-coordinates of each layer output achieves lossless forward reconstruction. The paper further claims that the same constraint makes backward-pass gradient communication lossless: projecting the incoming activation gradient onto S is said not to change any downstream gradient flow. The empirical section reports up to 100x communication compression, convergence matching centralized 100Gbps training over 80Mbps links, and an 8B-parameter LLaMA-style model trained across geographically distributed GPUs.

Significance. If the lossless-reconstruction claim were correct, this would be a substantial contribution: it would be the first communication-compression method for model-parallel training that avoids the error-accumulation problem identified in prior work, and the reported wall-clock results at 80Mbps would be practically important for decentralized training. The paper also contains a useful formal observation (Theorem B.1) that lossy layerwise compression errors can compound exponentially with depth, which sharpens the motivation for exact reconstruction. However, the backward-pass losslessness proof has a genuine gap, and the empirical claims are presented without error bars, seeds, or code. As a result, the central theoretical justification for the method's headline 'no convergence degradation' claim is not established.

major comments (4)

[Section 4.4 and Appendix A, Eqs. (16), (26)-(27), (32)-(34)] The claim that backward-pass gradient compression is lossless is false as stated. The proof shows invariance only for the path through W_p2: Eq. (19)-(24) demonstrate that P_S g (W_p2)^T = g (W_p2)^T when Row(W_p2) is contained in S. However, the residual skip connection in Eq. (2) means the gradient w.r.t. X_attn includes the identity term ∇L(X^{l+1}) itself, as stated in Eq. (16) and used in Eq. (26). In the actual compressed scheme, the received gradient is P_S g, not g, because Eq. (9)-(10) reconstruct only P_S g = (g U_k) U_k^T. For a minimal residual layer y = xW + x with Row(W) ⊆ S, the true gradient is g W^T + g, while the compressed scheme provides g W^T + P_S g; these differ by (I - P_S)g. Nothing in the architecture confines ∇L(X^{l+1}) to S: the final-loss gradient is arbitrary, and attention and MLP weights are unconstrained. Thus the equality claimed in Eq. (34) does not hold, and the central 'lossless' statement in the abstract and Section 4.4 is not supported.
[Appendix B, Theorem B.1, and Section 8.6 (Fig. 3)] Theorem B.1 shows that any layerwise gradient-compression error of size e can accumulate exponentially with depth when the Jacobian norm exceeds 1. Since the backward pass is actually lossy (as shown above), the depth-invariance experiments in Fig. 3 and the 32-layer experiment in Fig. 5 cannot be explained by the paper's theory. The paper provides no error bound for the residual-path error (I-P_S)g, nor any analysis of how this error propagates through the unconstrained attention/MLP weights. Without such an analysis, the empirical depth invariance is an unexplained observation rather than a validation of the losslessness claim.
[Section 7 (Statements 7.2, 7.3) and Appendix C.1, Fig. 7] The low-rankness assumption is validated circularly. Fig. 7 measures stable ranks of gradients of the projection matrices in the same 8-layer, C4-trained architecture used for the main experiments, and then this observation is used to justify the subspace confinement that the method itself enforces. Moreover, the assumption needed for the backward pass is not that weight gradients are low-rank, but that activation gradients ∇L(X^{l+1}) lie (approximately) in S; Fig. 7 does not measure activation gradients, and the paper offers no independent evidence for this stronger assumption. The convergence guarantee in Statement 7.4 (Proposition E.1) applies to constrained optimization of the weights but does not account for the compression-induced perturbation of the gradients, so it does not cover the actual algorithm.
[Section 8 (Tables 1-2, Figs. 2, 5)] The empirical results are reported without error bars, multiple seeds, or code, and several claims ('even surpasses centralized', 'slightly improving' over centralized at 8B scale) are based on single runs. Given that the central theoretical guarantee is invalid, the convergence-matching claim needs substantially stronger empirical support, including iteration-level loss curves (not only wall-clock curves) and a clear statement of the variance across runs. As submitted, the evidence is not sufficient to establish the headline 'no convergence degradation' conclusion.

minor comments (5)

[Title and abstract] The phrase 'Protocol Models' in the title and keywords does not match the content and seems to be a leftover; the paper is about subspace networks.
[Section 8.2, Table 1] The perplexity values for the decentralized uncompressed baseline (e.g., 925 on OpenWebText) suggest the model diverged or was severely undertrained; it would be helpful to state whether these are 12-hour wall-clock values or converged values.
[Appendix references] Some cross-references are inconsistent: Section 4.1 refers to 'Appendix A' for empirical validation, but the relevant validation appears in Appendix C.1; Section 4.3.1 refers to 'Appendix 8.8' instead of the appendix letter.
[Throughout] There are several typos, e.g., 'neglegible' (Section 8.8), 'LlaMa' (Sections 8.1, G), and 'Puralis' in the affiliation line; a careful proofread is needed.
[Section 4.5 and Eq. (11)] The Grassmann update is described as minimizing the norm of the residual gradient at the last compressed layer, but it is not discussed how this update interacts with the losslessness claim; even a small change in U_k makes the forward-pass reconstruction only approximately lossless, and this effect is not analyzed.

Circularity Check

1 steps flagged · score 6.0 of 10

Backward-pass losslessness is assumed, not derived: the recovery equation gUU^T = g holds only if gradients already lie in S, and the proof skips the residual identity path.

self definitional [Section 4.4, Eqs. (9)-(10); Appendix A, Eqs. (16), (27), (34)]
"(∇𝐿(X𝑙+1))compressed = ∇𝐿(X𝑙+1)U𝑘 ... and subsequently fully recovered in the previous layer 𝑙 as: (∇𝐿(X𝑙+1))recovered = (∇𝐿(X𝑙+1))compressedU⊤𝑘 = ∇𝐿(X𝑙+1) ... this formulation ensures that the gradient flow to the computational graph prior to X𝑙+1 remains lossless ... The residual gradient from the skip connection is given by (∇𝑙(X𝑙attn))residual = ∇𝑙(X𝑙+1)."

The recovery equation asserts gUU^T = g, an identity that holds if and only if g is in Col(U_k) = S — exactly the losslessness the paper claims to prove. The Appendix A derivation (Eqs. 19-24) checks only the path through W_p2, relying on Row(W_p2) ⊆ S, but the paper's own Eq. 16 and Eq. 27 propagate the raw gradient ∇L(X^{l+1}) directly into ∇L(X^l_attn) through the residual identity path, with no matrix whose rows lie in S. In the compressed scheme, g has already been replaced by P_S g = gUU^T, so this residual term changes by (I-P_S)g whenever g has an outside-S component.

full rationale

The forward-pass compression is genuinely self-contained: because the paper explicitly constrains Row(W_p2), Row(W_p1) ⊆ S, its Eqs. (4)-(7) correctly show that residual activations lie in S and are exactly recovered from XU_k. The backward-pass claim, however, is circular in the specific sense that Eq. (10)/(34) writes the desired conclusion gUU^T = g as a derived equality, while the only proof offered handles the W_p2 term and silently carries the unprojected residual gradient g through the skip connection. The extensive empirical results (2B and 8B runs, 80Mbps comparisons) are real and not themselves fabricated, but the paper's advertised theoretical guarantee of lossless backward propagation and hence 'no convergence degradation' rests on the unproven identification of the true gradient with its projection onto S. This is a partial circularity: the method's headline losslessness is assumed by construction rather than derived, so the score is 6 rather than 8-10.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The ledger is dominated by the subspace dimension k=40 and the unstated gradient-in-S assumption. The latter is the load-bearing premise that makes backward compression lossless, and it is not proven.

free parameters (4)

Subspace dimension k = 40
Chosen to achieve 100x compression (Section 8.1); not derived from theory.
U_k initialization = isotropic Gaussian noise
Set in Section 8.1; seed not reported.
Grassmann update interval = 500 iterations
Set in Section 4.5; not justified by a convergence criterion.
Grassmann step size eta = not specified
Section 4.5 introduces eta but no value is given in the experimental setup.

assumptions (4)

domain assumption Projection matrices in trained transformers are effectively low-rank due to rank collapse.
Empirically validated in Section 4.1 and Appendix C.1, and supported by cited prior work; used to justify the subspace constraint.
ad hoc to paper The gradient of the loss with respect to a layer output, ∇L(X^{l+1}), lies in the shared subspace S, so projecting it is lossless.
Assumed implicitly in Section 4.4 and Appendix A; false for residual networks because the skip connection passes ∇L(X^{l+1}) directly to earlier layers.
domain assumption Gradient variance diminishes as training converges, so AdamW updates become asymptotically confined to the gradient subspace.
Invoked in Theorem C.2 and Section C.1.2; relies on prior literature and the paper's own empirical observations.
standard math Constrained projected gradient descent on non-convex functions achieves O(1/T) stationarity convergence.
Used in Proposition E.1 and Theorem E.2; a standard result in non-convex optimization.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism." pith.science (2026). https://pith.science/paper/KPZN3ERK

@misc{pith2026250601260,
  author       = {Pith},
  title        = {Pith review of: Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KPZN3ERK}},
  note         = {Machine review of arXiv:2506.01260}
}

read the original abstract

Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While existing compression techniques are effective in data-parallel, they do not extend to model parallelism. Unlike data-parallel training, where weight gradients are exchanged, model-parallel requires compressing activations and activation gradients as they propagate through layers, accumulating compression errors. We propose a novel compression algorithm that compresses both forward and backward passes, enabling up to 99% compression with no convergence degradation with negligible memory/compute overhead. By leveraging a recursive structure in transformer networks, we predefine a low-dimensional subspace to confine the activations and gradients, allowing full reconstruction in subsequent layers. Our method achieves up to 100x improvement in communication efficiency and enables training billion-parameter-scale models over low-end GPUs connected via consumer-grade internet speeds as low as 80Mbps, matching the convergence of centralized datacenter systems with 100Gbps connections with model parallel.

Figures

Figures reproduced from arXiv: 2506.01260 by the authors.

**Figure 1.** Rank collapse in projection matrices. Consistent with Statements 7.2 and 7.3 (and Theorems C.2, D.1 - Appendix), we empirically observe a natural rank collapse in the projection matrices (of non-compressed models). Shown is an 8-layer, 2B-parameter model, with the stable (effective) ranks of the projection matrices for the 4 th (middle) and 7 th (penultimate) layers plotted over training steps. et al., 2018; Li et a… view at source ↗

**Figure 2.** Convergence in low-bandwidth settings. From left to right: OpenWebText, WikiText, and BookCorpus. In each plot, the training curves are presented against wall-clock time for an 8-layer (2B) model. Decentralized models utilize 80Mbps connections while the centralized model has datacenter-grade 100Gbps links. Our compressed model achieves on-par convergence to the centralized model, even under a 80Mbps bandwidth budge… view at source ↗

**Figure 3.** Performance against depth. Two key observations: (1) Our compression matches (even slightly exceeds) with centralized baseline as the # layers increases [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Figures from the paper (12 more)

**Figure 4.** Figure 4: Throughput Gain. As bandwidth becomes increasingly limited, the compressed models achieve a significantly higher throughput gain in both inference (left) and training (right). Results are shown for 8-layer (2B) models. This partially offsets communication bottlenecks w…

**Figure 5.** Figure 5: Training convergence and throughput on an 8B LLaMA model. All runs use 64 L4 GPUs distributed over 8 instances. Centralized instances reside within one region (min bandwidth 16 Gbps), while decentralized instances span 4 regions (min bandwidth 60 Mbps), highlighting p…

**Figure 7.** Figure 7: Low dimensionality of the gradients of projection matrices. We measure the stable ranks of gradients of the projection matrices over the course of training. In each plot, the green dashed line indicates the maximum rank. As shown, all the weight matrices consistently m…

**Figure 8.** Figure 8: Convergence with different batch sizes. Each block shows the training curves (with respect to training steps) and throughput for an 8-layer, 2B-parameter model on C4. Decentralized configurations use 80 Mbps connections, while the centralized configuration uses datacen…

**Figure 9.** Figure 9: Convergence with different batch sizes. Each block shows the training curves (with respect to wall-clock time) and throughput for an 8-layer, 2B-parameter model on C4. Decentralized configurations use 80 Mbps connections, while the centralized configuration uses datace…

**Figure 10.** Figure 10: Convergence with different context lengths. Each block shows the training curves (with respect to training steps) and throughput for an 8-layer, 2B-parameter model on C4. Decentralized configurations use 80 Mbps connections, while the centralized configuration uses da…

**Figure 11.** Figure 11: Convergence with different context lengths. Each block shows the training curves (with respect to wall-clock time) and throughput for an 8-layer, 2B-parameter model on C4. Decentralized configurations use 80 Mbps connections, while the centralized configuration uses d…

**Figure 12.** Figure 12: Convergence with increasing number of layers. Note that as the number of layer increase, our model consistently matches (even slightly exceeds) the centralized model. This is in stark contrast to lossy compression schemes, where the model convergence severely degrades…

**Figure 13.** Figure 13: Throughput across bandwidth constraints. We limit network bandwidth between GPUs and measure throughput during inference and training. The compressed model consistently achieves significantly higher throughput than the non-compressed model. Notably, even at 100Gbps, c…

**Figure 14.** Figure 14: Effect of the (Grassman) subspace updates. An 8-layer model is trained on C4 for this experiment. Since this performance gap seems to keep increasing towards the end of training, we emphasize the importance of infrequent subspace updates [PITH_FULL_IMAGE:figures/full…

**Figure 15.** Figure 15: Effect of fixed token embedding. We decompose the token embeddings to a fixed high rank embedding and a dynamic low rank embedding. As shown in the figure, we observe inferior convergence when there is no such decomposition. An 8-layer model is trained on C4 for this …

**Figure 16.** Figure 16: Stable ranks of output projection matrices (normalized by the maximum possible rank) across different frontier models. Statistics are computed on official fully pretrained checkpoints. All the models demonstrate extremely low ranks, solidifying our theoretical argume…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 33 canonical work pages

[1]

Transformers learn through gradual rank increase

Emmanuel Abbe, Samy Bengio, Enric Boix-Adsera, Etai Littwin, and Joshua Susskind. Transformers learn through gradual rank increase. Advances in Neural Information Processing Systems, 36, 2024

work page 2024
[2]

Qsgd: Communication-efficient sgd via gradient quantization and encoding

Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficient sgd via gradient quantization and encoding. Advances in neural information processing systems, 30, 2017

work page 2017
[3]

Dissecting adam: The sign, magnitude and variance of stochastic gradients

Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning, pages 404--413. PMLR, 2018

work page 2018
[4]

signsgd: Compressed optimisation for non-convex problems

Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pages 560--569. PMLR, 2018

work page 2018
[5]

Forest-of-thought: Scaling test-time compute for enhancing llm reasoning

Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. Forest-of-thought: Scaling test-time compute for enhancing llm reasoning. arXiv preprint arXiv:2412.09078, 2024

arXiv 2024
[6]

Does compressing activations help model parallel training? Proceedings of Machine Learning and Systems, 6: 0 239--252, 2024

Song Bian, Dacheng Li, Hongyi Wang, Eric Xing, and Shivaram Venkataraman. Does compressing activations help model parallel training? Proceedings of Machine Learning and Systems, 6: 0 239--252, 2024

work page 2024
[7]

Low-rank gradient descent

Romain Cosson, Ali Jadbabaie, Anuran Makur, Amirhossein Reisizadeh, and Devavrat Shah. Low-rank gradient descent. IEEE Open Journal of Control Systems, 2023

work page 2023
[8]

DeepSeek LLMs , 2023

DeepSeek AI Team . DeepSeek LLMs , 2023. URL https://deepseek.com/

work page 2023

Show all 63 references

[9]

A simple convergence proof of adam and adagrad

Alexandre D \'e fossez, L \'e on Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of adam and adagrad. arXiv preprint arXiv:2003.02395, 2020

2003 arXiv
[10]

8-bit optimizers via block-wise quantization

Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861, 2021

2021 arXiv
[11]

Distributed deep learning in open collaborations

Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, Anton Sinitsin, Dmitry Popov, Dmitry V Pyrkin, Maxim Kashirin, Alexander Borzunov, Albert Villanova del Moral, et al. Distributed deep learning in open collaborations. Advances in Neural Information Processing ...

2021
[12]

Attention is not all you need: Pure attention loses rank doubly exponentially with depth

Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793--2803. PMLR, 2021

2021
[13]

Diloco: Distributed low-communication training of language models

Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc'Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low-communication training of language models. arXiv preprint arXiv:2311.08105, 2023

2023 arXiv
[14]

The llama 3 herd of models

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

2024 arXiv
[15]

Openwebtext corpus

Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

2019
[16]

Gradient descent happens in a tiny subspace

Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754, 2018

2018 arXiv
[17]

Training compute-optimal large language models

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models
[18]

Gpipe: Efficient training of giant neural networks using pipeline parallelism

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019

2019
[19]

Error feedback fixes signsgd and other gradient compression schemes

Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International Conference on Machine Learning, pages 3252--3261. PMLR, 2019

2019
[20]

Adam: A method for stochastic optimization

Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

2014 arXiv
[21]

Big transfer (bit): General visual representation learning

Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, ...

2020
[22]

Decentralized stochastic optimization and gossip algorithms with compressed communication

Anastasia Koloskova, Sebastian Stich, and Martin Jaggi. Decentralized stochastic optimization and gossip algorithms with compressed communication. In International Conference on Machine Learning, pages 3478--3487. PMLR, 2019

2019
[23]

A unified theory of decentralized sgd with changing topology and local updates

Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning, pages 5381--5393. PMLR, 2020

2020
[24]

Imagenet classification with deep convolutional neural networks

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012

2012
[25]

Convergence of adam under relaxed assumptions

Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. Convergence of adam under relaxed assumptions. Advances in Neural Information Processing Systems, 36, 2024 a

2024
[26]

Learning on transformers is provable low-rank and sparse: A one-layer analysis

Hongkang Li, Meng Wang, Shuai Zhang, Sijia Liu, and Pin-Yu Chen. Learning on transformers is provable low-rank and sparse: A one-layer analysis. In 2024 IEEE 13rd Sensor Array and Multichannel Signal Processing Workshop (SAM), pages 1--5. IEEE, 2024 b

2024
[27]

Pytorch distributed: Experiences on accelerating data parallel training

Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704, 2020

2006 arXiv
[28]

Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent

Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30, 2017

2017
[29]

Torchtitan: One-stop pytorch native solution for production ready llm pre-training

Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, et al. Torchtitan: One-stop pytorch native solution for production ready llm pre-training. arXiv preprint arXiv:2410.06511, 2024

arXiv
[30]

Deep gradient compression: Reducing the communication bandwidth for distributed training

Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv preprint arXiv:1712.01887, 2017

2017 arXiv
[31]

Adam + : A stochastic method with adaptive variance reduction

Mingrui Liu, Wei Zhang, Francesco Orabona, and Tianbao Yang. Adam + : A stochastic method with adaptive variance reduction. arXiv preprint arXiv:2011.11985, 2020

2011 arXiv
[32]

Decoupled weight decay regularization

I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

2017 arXiv
[33]

Pointer sentinel mixture models, 2016

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

2016
[34]

Efficient large-scale language model training on gpu clusters using megatron-lm

Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedin...

2021
[35]

Decoupled momentum optimization

Bowen Peng, Jeffrey Quesnelle, and Diederik P Kingma. Decoupled momentum optimization. arXiv preprint arXiv:2411.19870, 2024

2024
[36]

Ai and compute

How Much Longer Can Computing Power and Drive Artificial Intelligence Progress. Ai and compute
[37]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019

2019
[38]

Pangu- \ Sigma \ : Towards trillion parameter language model with sparse heterogeneous computing

Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing Huang, Yadao Wang, Weichao Wang, Pengfei Li, Xiaoda Zhang, Alexander Podolskiy, Grigory Arshinov, et al. Pangu- \ Sigma \ : Towards trillion parameter language model with sparse heterogeneous computing. arXiv preprint arXiv:2303.1...

2023 arXiv
[39]

Activations and gradients compression for model-parallel training

Mikhail I Rudakov, Aleksandr Nikolaevich Beznosikov, Ya A Kholodov, and Alexander Vladimirovich Gasnikov. Activations and gradients compression for model-parallel training. In Doklady Mathematics, volume 108, pages S272--S281. Springer, 2023

2023
[40]

Towards crowdsourced training of large neural networks using decentralized mixture-of-experts

Max Ryabinin and Anton Gusev. Towards crowdsourced training of large neural networks using decentralized mixture-of-experts. Advances in Neural Information Processing Systems, 33: 0 3659--3672, 2020

2020
[41]

Moshpit sgd: Communication-efficient decentralized training on heterogeneous unreliable devices

Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk, and Gennady Pekhimenko. Moshpit sgd: Communication-efficient decentralized training on heterogeneous unreliable devices. Advances in Neural Information Processing Systems, 34: 0 18195--18211, 2021

2021
[42]

Swarm parallelism: Training large models can be surprisingly communication-efficient

Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov. Swarm parallelism: Training large models can be surprisingly communication-efficient. In International Conference on Machine Learning, pages 29416--29440. PMLR, 2023

2023
[43]

Inheritune: Training smaller yet more attentive language models, 2024

Sunny Sanyal, Ravid Shwartz-Ziv, Alexandros G Dimakis, and Sujay Sanghavi. Inheritune: Training smaller yet more attentive language models, 2024. URL https://arxiv. org/abs/2404.08634

2024
[44]

Scaling llm test-time compute optimally can be more effective than scaling model parameters

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

2024 arXiv
[45]

1-bit adam: Communication efficient large-scale training with adam’s convergence speed

Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He. 1-bit adam: Communication efficient large-scale training with adam’s convergence speed. In International Conference on Machine Learning, pages 10118--10...

2021
[46]

Scaling the summit: deploying the world’s fastest supercomputer

Ver \'o nica G Vergara Larrea, Wayne Joubert, Michael J Brim, Reuben D Budiardja, Don Maxwell, Matt Ezell, Christopher Zimmer, Swen Boehm, Wael Elwasif, Sarp Oral, et al. Scaling the summit: deploying the world’s fastest supercomputer. In High Performance Computing: ISC High P...

2019
[47]

Powersgd: Practical low-rank gradient compression for distributed optimization

Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 32, 2019

2019
[48]

Kalman gradient descent: Adaptive variance reduction in stochastic optimization

James Vuckovic. Kalman gradient descent: Adaptive variance reduction in stochastic optimization. arXiv preprint arXiv:1810.12273, 2018

2018 arXiv
[49]

Variance reduction for stochastic gradient optimization

Chong Wang, Xi Chen, Alexander J Smola, and Eric P Xing. Variance reduction for stochastic gradient optimization. Advances in neural information processing systems, 26, 2013

2013
[50]

Atomo: Communication-efficient learning via atomic sparsification

Hongyi Wang, Scott Sievert, Shengchao Liu, Zachary Charles, Dimitris Papailiopoulos, and Stephen Wright. Atomo: Communication-efficient learning via atomic sparsification. Advances in neural information processing systems, 31, 2018

2018
[51]

Pufferfish: Communication-efficient models at no extra cost

Hongyi Wang, Saurabh Agarwal, and Dimitris Papailiopoulos. Pufferfish: Communication-efficient models at no extra cost. Proceedings of Machine Learning and Systems, 3: 0 365--386, 2021

2021
[52]

Efficient distributed learning with sparsity

Jialei Wang, Mladen Kolar, Nathan Srebro, and Tong Zhang. Efficient distributed learning with sparsity. In International conference on machine learning, pages 3636--3645. PMLR, 2017

2017
[53]

Cocktailsgd: Fine-tuning foundation models over 500mbps networks

Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, and Ce Zhang. Cocktailsgd: Fine-tuning foundation models over 500mbps networks. In International Conference on Machine Learning, pages 36058--36076. PMLR, 2023

2023
[54]

Gradient sparsification for communication-efficient distributed optimization

Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gradient sparsification for communication-efficient distributed optimization. Advances in Neural Information Processing Systems, 31, 2018

2018
[55]

Error compensated quantized sgd and its applications to large-scale distributed optimization

Jiaxiang Wu, Weidong Huang, Junzhou Huang, and Tong Zhang. Error compensated quantized sgd and its applications to large-scale distributed optimization. In International conference on machine learning, pages 5325--5333. PMLR, 2018

2018
[56]

A spectral condition for feature learning

Greg Yang, James B Simon, and Jeremy Bernstein. A spectral condition for feature learning. arXiv preprint arXiv:2310.17813, 2023

2023 arXiv
[57]

Stochastic gradient variance reduction by solving a filtering problem

Xingyi Yang. Stochastic gradient variance reduction by solving a filtering problem. arXiv preprint arXiv:2012.12418, 2020

2012 arXiv
[58]

Decentralized training of foundation models in heterogeneous environments

Binhang Yuan, Yongjun He, Jared Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy S Liang, Christopher Re, and Ce Zhang. Decentralized training of foundation models in heterogeneous environments. Advances in Neural Information Processing Systems, 35: 0 25464--25477, 2022

2022
[59]

Convergence guarantees for rmsprop and adam in generalized-smooth non-convex optimization with affine noise variance

Qi Zhang, Yi Zhou, and Shaofeng Zou. Convergence guarantees for rmsprop and adam in generalized-smooth non-convex optimization with affine noise variance. arXiv preprint arXiv:2404.01436, 2024

2024 arXiv
[60]

Zero initialization: Initializing neural networks with only zeros and ones

Jiawei Zhao, Florian Sch \"a fer, and Anima Anandkumar. Zero initialization: Initializing neural networks with only zeros and ones. arXiv preprint arXiv:2110.12661, 2021

2021 arXiv
[61]

Galore: Memory-efficient llm training by gradient low-rank projection

Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024

2024 arXiv
[62]

Pytorch fsdp: experiences on scaling fully sharded data parallel

Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023

2023 arXiv
[63]

Aligning books and movies: Towards story-like visual explanations by watching movies and reading books

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), ...

2015

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Transformers learn through gradual rank increase

Emmanuel Abbe, Samy Bengio, Enric Boix-Adsera, Etai Littwin, and Joshua Susskind. Transformers learn through gradual rank increase. Advances in Neural Information Processing Systems, 36, 2024

work page 2024

[2] [2]

Qsgd: Communication-efficient sgd via gradient quantization and encoding

Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficient sgd via gradient quantization and encoding. Advances in neural information processing systems, 30, 2017

work page 2017

[3] [3]

Dissecting adam: The sign, magnitude and variance of stochastic gradients

Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning, pages 404--413. PMLR, 2018

work page 2018

[4] [4]

signsgd: Compressed optimisation for non-convex problems

Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pages 560--569. PMLR, 2018

work page 2018

[5] [5]

Forest-of-thought: Scaling test-time compute for enhancing llm reasoning

Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. Forest-of-thought: Scaling test-time compute for enhancing llm reasoning. arXiv preprint arXiv:2412.09078, 2024

arXiv 2024

[6] [6]

Does compressing activations help model parallel training? Proceedings of Machine Learning and Systems, 6: 0 239--252, 2024

Song Bian, Dacheng Li, Hongyi Wang, Eric Xing, and Shivaram Venkataraman. Does compressing activations help model parallel training? Proceedings of Machine Learning and Systems, 6: 0 239--252, 2024

work page 2024

[7] [7]

Low-rank gradient descent

Romain Cosson, Ali Jadbabaie, Anuran Makur, Amirhossein Reisizadeh, and Devavrat Shah. Low-rank gradient descent. IEEE Open Journal of Control Systems, 2023

work page 2023

[8] [8]

DeepSeek LLMs , 2023

DeepSeek AI Team . DeepSeek LLMs , 2023. URL https://deepseek.com/

work page 2023