Pith. sign in

REVIEW 3 major objections 5 minor 98 references

One Rank at a Time: Cascading Error Dynamics in Sequential Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that errors made while learning one rank-1 component at a time compound multiplicatively through later steps, with amplification set by the gaps between singular values.

desk verdict Sensible qualitative story about error compounding in sequential rank-1 learning, but the main theorems as stated are not proven due to a gap-indexing mismatch. read the letter →

arxiv 2505.22602 v1 pith:OFTAD5AG submitted 2025-05-28 cs.LG cs.AImath.OC

Mahtab Alizadeh Vandchali , Fangshuo (Jasper) Liao , Anastasios Kyrillidis This is my paper
classification cs.LGcs.AImath.OC MSC 15A1862J0568T05
keywords sequentiallearninglow-ranklinearregressionerrorpropagationdeflationsingularvaluegaprank-1updatesLoRAgeneralizationbounds
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that sequential learning by rank-1 deflation — repeatedly solving one rank-1 regression and subtracting it from the residual — does not accumulate errors chaotically. Its central theorem gives an explicit worst-case bound: if the per-step numerical errors, measured as $\|\delta_k\|_F$, stay below half the relevant singular-value gap, then the final training error is at most the tail of the spectrum plus a sum of weighted products of those per-step errors, with amplification factors $(2+6\sigma^*_j/T^*_k)$ controlled by singular-value gaps. The practical content is that early rank-1 mistakes are multiplied through every later deflation step, so a fixed computation budget should be front-loaded onto the first components. If true, this turns “sequence of rank-1 updates” from a heuristic into a predictable procedure with tuneable accuracy, relevant to low-rank regression and parameter-efficient fine-tuning.

What carries the argument

The carrying object is the deflation residual recursion. Lemma 2 and Lemma 3 show that the distance between the ideal deflated matrix $Y^*_k$ and the actually maintained matrix $Y_k$ satisfies $\|Y_{k+1}-Y^*_{k+1}\|_F \le (2+3\sigma^*_k/T_k)\|Y^*_k-Y_k\|_F + \|\delta_kX\|_F$, where $T_k$ is the local singular-value gap and $\delta_k$ is the rank-1 solver error. Wedin’s sin-$\theta$ theorem and Weyl’s inequality convert that recursion into the explicit product factors $2+6\sigma^*_j/T^*_k$ that appear in all of the paper’s bounds.

What would settle it

Construct a data matrix whose top two singular values are within 1% of each other, run the inexact sequential algorithm with a deliberately inaccurate first rank-1 solve, and measure the final error across increasing rank $r$; the theorem predicts the error remains inside its product bound, while a divergent or much larger error in this small-gap regime would falsify the claim.

Watch

Extended reading notes

Core claim

Given a low-rank linear regression task $Y \approx WX$ with rank-$r$ $W$, the paper analyzes the inexact sequential algorithm that estimates one pair $(a_k,b_k)$ at a time from the current residual $Y_k$ and then forms $Y_{k+1}=Y_k-b_ka_k^\top X$. Theorem 1 states that, whenever the accumulated error $E(k)=\sigma_{\max}(X)\sum_{k'=0}^{k-1}\|\delta_{k'}\|_F\prod_{j=k'+1}^{k-1}(2+6\sigma^*_j/T^*_k)$ is below half the gap $\min_{j>k}|\sigma^*_k-\sigma^*_j|$, the reconstruction error satisfies $\|Y-\sum_{k=1}^r b_ka_k^\top X\|_F \le \sum_{k=r+1}^p \sigma^*_k + \sigma_{\max}(X)\sum_{k=1}^r\sum_{k'=0}^{k}\|\delta_{k'}\|_F\prod_{j=k'+1}^{k}(2+6\sigma^*_j/T^*_k)$. The message is that errors compound multiplicatively with factors determined by the singular-value spectrum, and that this compounding is controlled as long as the solver errors are small relative to the spectral gaps. The same mechanism yields component-wise and whole-model generalization bounds in the noiseless and noisy-label cases.

Load-bearing premise

The whole bound rests on each accumulated error staying below half the gap between the current singular value and all later ones; when singular values are nearly identical the paper supplies no guarantee, and this is exactly where cascades are worst.

Editorial extensions

If this is right

  • Under a fixed iteration budget, putting more optimization steps into early rank-1 components yields smaller final reconstruction error than equal or back-loaded schedules.
  • For matrices with slowly decaying singular values, the bound requires each subproblem to be solved more accurately; otherwise error amplification grows with each step.
  • The noisy-label bound shows a bias-variance trade-off: too-small $r$ leaves signal unexplained, while too-large $r$ overfits noise.
  • Because rank-1 components are learned sequentially, one can stop once accuracy is sufficient, avoiding the need to preselect the LoRA rank in advance.
  • Sequential rank-1 methods require more total training iterations than simultaneous rank-$r$ optimization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The analysis suggests a testable scheduling principle beyond the paper’s experiments: allocate per-component compute roughly inversely to the singular-value gap $T^*_k$, since that factor multiplies all downstream errors.
  • The same deflation-recursion view may transfer to residual-stream architectures, where each low-rank adapter update plays the role of a deflation step; the paper only demonstrates this on feedforward fine-tuning, so this is an extension.
  • The bounds could be converted into an online stopping rule: estimate $\|\delta_k\|_F$ during training and stop when the predicted compounded error crosses a tolerance, a direction the paper lists as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies sequential low-rank linear regression, in which a rank-r model is built by solving rank-1 problems one at a time and deflating the label matrix after each step. The main theoretical claim is Theorem 1, which bounds the training error of this inexact sequential algorithm by the tail of the singular value spectrum plus a sum of per-step numerical errors δ_k amplified by products of factors of the form (2 + O(σ*_j / T*_k)), where T*_k is a spectral gap. Theorems 2 and 3 give analogous component-wise and generalization bounds under noiseless and noisy labels, respectively. The paper also reports synthetic experiments on linear regression and LoRA-style fine-tuning, showing qualitatively that front-loading computational resources to early components improves final accuracy.

Significance. If the bounds are correct, the paper offers a quantitative characterization of error accumulation in sequential/hierarchical rank-1 learning, with explicit dependence on singular-value gaps and no fitted constants. The analysis is parameter-free with respect to the data: δ_k is an externally defined per-step error, T*_k is read from the spectrum, and the experimental validation does not feed back into the derivation. The qualitative prediction that early errors compound more severely is falsifiable and is supported by the experiments. However, the significance is currently tempered by a load-bearing indexing inconsistency in the main theorems, which means the central quantitative claim is not established as written.

major comments (3)
  1. [§3, Eq. (8); Appendix B.4, Eq. (32)] Theorem 1 as stated is not established by its proof. The theorem's error condition and final bound use a single spectral gap T*_k inside the product over j, whereas the recurrence unrolled in Eq. (32) produces per-step factors (2 + 3σ*_j / T_j). The inequality in Eq. (32) that passes from the single-gap T*_k product to the per-step T*_j product does not hold in general. For example, with σ* = [10, 1, 0.9, 0.1] and k = 2, the single-gap product for k' = 0 is 2 + 3·10 / 0.1 = 302, while the per-step product is 2 + 3·10 / 9 ≈ 5.33, so the claimed ≤ direction fails. Consequently, the error quantity E(k) defined in Theorem 1 does not dominate the quantity the proof actually bounds, and the condition of Lemma 3 can fail exactly in the regimes the theorem is meant to cover. The same T*_k versus T*_j inconsistency appears in Theorem 2 (Eqs. (10)–(11)) and Theorem 3 (Eq. (12)). The theorems and proofs must be consistently re-derived with per-step gaps T*_j in both the precondition and the bound, or a correct relationship between the two gap objects must be proved under the stated assumptions.
  2. [§4, Theorem 3, Eq. (12)] The first sum in Eq. (12) is ∑_{k=r+1}^{r⋆} σ_r(W*), which uses the fixed index r in every term. This is not the truncated-SVD residual that appears in Theorem 2 and in the proof sketch; it should be ∑_{k=r+1}^{r⋆} σ_k(W*). The same index error is repeated in Appendix C.3, so the stated bound on the generalization error is incorrect as written.
  3. [Appendix C.1] The proof of Theorem 2 begins with the assertion 'Utilizing Lemma 3, we could get that ∥b⋆_k a⋆⊤_k X − b_k a⊤_k X∥_F ≤ (2 + 6σ⋆_k/T⋆_k) + σmax(X)∥δ_k∥_F'. This inequality is dimensionally inconsistent: Lemma 3 bounds the left-hand side by a multiple of ∥Y*_k − Y_k∥_F, not by a spectral quotient plus a separate noise term. The subsequent derivation of Eq. (10) therefore does not follow as written. This appears to be a manifestation of the same indexing/bookkeeping problem noted in Theorem 1, but it leaves the component-wise generalization bound unsupported.
minor comments (5)
  1. [Definition 1, Eq. (7)] The definition δk := b_k a_k^⊤ − b_k a_k^⊤ contains a typo: both terms are identical. It should be b_k a_k^⊤ − \bar{b}_k \bar{a}_k^⊤ (or equivalent) with the exact rank-1 solution denoted by a different symbol.
  2. [Front matter] The 'Paper Meta-Analysis Card' inserted after the abstract appears to be an internal editorial document (it contains a date and a correspondence line) that was accidentally included in the manuscript. It should be removed, especially since Section 6 already states the limitations.
  3. [Theorem 2 statement] The condition σmin(X) ≥ 0 is trivially true for any matrix; presumably σmin(X) > 0 is intended, since the proof divides by σmin(X).
  4. [Algorithm 2 and Definition 1] The subroutine iteration count is denoted t in Algorithm 2 and in the surrounding text, but Definition 1 uses T. Please use a consistent notation throughout.
  5. [Appendix C.1, Eq. (11) derivation] The final sum in Eq. (11) starts at k' = 1, while Theorem 1 and Eq. (10) use k' = 0. The off-by-one should be reconciled.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the error-propagation bound is a parameter-free perturbation analysis whose inputs are external to the theorem.

full rationale

The derivation chain is self-contained. Theorem 1 and its generalization in Theorem 4 are perturbation arguments: the quantities δ_k are externally defined subroutine errors, the gaps T⋆_k and T⋆_j are spectral quantities read off the data matrix, and the multiplicative factors arise from Wedin's theorem applied to the pair Y_k versus Y⋆_k. No parameter is fitted to data, no experimental observation is fed back into the bound, and no load-bearing step is justified by a self-citation; the authors' own prior works appear only in peripheral related-work enumerations and do not carry the argument. The skeptical concern that the proof of Theorem 4 passes from a T⋆_k-indexed product to a T⋆_j-indexed product with an inequality that may not hold is a correctness or indexing issue, not circularity: even if the proof does not establish the theorem as written, it does not reduce the theorem to its own assumptions. The experimental validation of front-loaded iteration schedules is qualitative and illustrative rather than a fitted input to the theory. Therefore no circular step is present.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No free parameters are fitted to data: delta_k is a defined error quantity and T*_k is a spectral gap read off from the data matrix. The analysis relies on classical theorems (Eckart-Young-Mirsky, Wedin, Weyl) and on modeling assumptions about Gaussian X, full column rank, and additive Gaussian label noise. The spectral-gap condition is an operating-regime premise: outside it, the stated bounds are vacuous.

assumptions (6)
  • standard math Eckart-Young-Mirsky theorem for the optimal rank-1 approximation of a matrix.
    Used to identify b*_k a*_k^T X with the top singular triple of Y*_k in Lemma 1 (Section 2, Appendix B.1).
  • standard math Wedin's sin-theta perturbation theorem for singular subspaces.
    Used in Lemma 3 (Appendix B.3) to bound the angle between top singular vectors of Y*_k and Y_k in terms of ||Y*_k - Y_k|| / T_k.
  • standard math Weyl's singular value inequality for perturbations.
    Used in Lemma 3 and Lemma 4 to bound singular value shifts under additive noise or deflation error.
  • domain assumption X is sampled entrywise from a standard normal distribution followed by row normalization, with full column rank and n >= d.
    Assumed in Section 2 for identifiability of (a_k, b_k) in Lemma 1 and for sigma_min(X) > 0 used in Theorem 2.
  • domain assumption Label noise E_ij ~ N(0, epsilon^2) in Theorem 3.
    Needed for Lemma 4 and for the noise-scale condition epsilon <= O(T*_min / (sqrt(n) + sqrt(log(1/gamma)))) that preserves the ordering of rank-1 components.
  • domain assumption Spectral-gap condition E(k) < 1/2 min_{j>k} |sigma*_k - sigma*_j|.
    Stated before Eq. (8) in Theorem 1 and analogously in Theorems 2 and 3. The bounds are vacuous outside this regime, so this is a load-bearing operating premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of One Rank at a Time: Cascading Error Dynamics in Sequential Learning." pith.science (2026). https://pith.science/paper/OFTAD5AG

@misc{pith2026250522602,
  author       = {Pith},
  title        = {Pith review of: One Rank at a Time: Cascading Error Dynamics in Sequential Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OFTAD5AG}},
  note         = {Machine review of arXiv:2505.22602}
}
read the original abstract

Sequential learning -- where complex tasks are broken down into simpler, hierarchical components -- has emerged as a paradigm in AI. This paper views sequential learning through the lens of low-rank linear regression, focusing specifically on how errors propagate when learning rank-1 subspaces sequentially. We present an analysis framework that decomposes the learning process into a series of rank-1 estimation problems, where each subsequent estimation depends on the accuracy of previous steps. Our contribution is a characterization of the error propagation in this sequential process, establishing bounds on how errors -- e.g., due to limited computational budgets and finite precision -- affect the overall model accuracy. We prove that these errors compound in predictable ways, with implications for both algorithmic design and stability guarantees.

Figures

Figures reproduced from arXiv: 2505.22602 by the authors.

Figure 1
Figure 1. Impact of iteration allocation strategy under a fixed iteration budget. [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Test accuracy of sequential rank-1 LoRA when adapting to new classes across the [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Marker sizes: relative efficiency of each config. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: α  β  γ denotes sequential training with α, β and γ epochs for each component. Not all combinations are shown. Sequential training paths [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Comparison of singular value decay under different profiles. [PITH_FULL_IMAGE:figures/full_fig_p029_5.png]
Figure 6
Figure 6. Figure 6: Effect of singular value profile on sequential learning performance. [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]
Figure 7
Figure 7. Figure 7: Impact of noise level. Left: Gaussian noise. Right: Sparse noise. Effect of iteration allocation strategies in noisy settings. To investigate mitigation strate￾gies, we first evaluate how different iteration allocation strategies perform under noisy conditions [PITH_F…
Figure 8
Figure 8. Figure 8: Comparison of iteration allocation strategies under different noise levels. The [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]
Figure 9
Figure 9. Figure 9: Effect of singular value profiles under noise. Power-law decay consistently [PITH_FULL_IMAGE:figures/full_fig_p031_9.png]
Figure 10
Figure 10. Figure 10: Number of iterations required to reach reconstruction error thresholds for dif [PITH_FULL_IMAGE:figures/full_fig_p032_10.png]
Figure 11
Figure 11. Figure 11: Test accuracy of sequential rank-1 LoRA components when adapting to new [PITH_FULL_IMAGE:figures/full_fig_p034_11.png]
Figure 12
Figure 12. Figure 12: The bubble sizes represent the relative efficiency of each configuration. [PITH_FULL_IMAGE:figures/full_fig_p035_12.png]
Figure 13
Figure 13. Figure 13: For every rank-r architecture, we depict how well the model performs (the variance bars indicate how good or bad the model ends up, depending on the different number of epochs we spend on each component of the low-rank sequential architecture). 36 [PITH_FULL_IMAGE:fi…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

98 extracted references · 60 canonical work pages

  1. [1]

    Recent advances in hierarchical reinforce- ment learning

    Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforce- ment learning. Discrete event dynamic systems, 13:341–379, 2003

    2003

  2. [2]

    Hierarchical reinforcement learning: A comprehensive survey

    Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical reinforcement learning: A comprehensive survey. ACM Computing Surveys (CSUR), 54 (5):1–35, 2021

    2021

  3. [3]

    Learning to compose skills

    Himanshu Sahni, Saurabh Kumar, Farhan Tejani, and Charles Isbell. Learning to compose skills. arXiv preprint arXiv:1711.11289, 2017

    arXiv 2017

  4. [4]

    Towards modular LLMs by building and reusing a library of LoRAs

    Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti, Laurent Charlin, Nicolas Le Roux, Matheus Pereira, Lucas Caccia, and Alessandro Sordoni. Towards modular LLMs by building and reusing a library of LoRAs. arXiv preprint arXiv:2405.11157, 2024

    arXiv 2024

  5. [5]

    Joint prompt optimization of stacked llms using variational inference

    Alessandro Sordoni, Eric Yuan, Marc-Alexandre Côté, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. Joint prompt optimization of stacked llms using variational inference. Advances in Neural Information Processing Systems, 36, 2024

    2024

  6. [6]

    Multi-head adapter routing for cross-task generalization

    Lucas Page-Caccia, Edoardo Maria Ponti, Zhan Su, Matheus Pereira, Nicolas Le Roux, and Alessandro Sordoni. Multi-head adapter routing for cross-task generalization. Advances in Neural Information Processing Systems, 36, 2024

    2024

  7. [7]

    The structure of scientific revolutions

    Dudley Shapere. The structure of scientific revolutions. The Philosophical Review, 73(3): 383–394, 1964

    1964

  8. [8]

    Models of cognitive development

    Ken Richardson. Models of cognitive development. Psychology Press, 2019

    2019

Show all 98 references
  1. [9]

    Learning and memory.Proceedings of the National Academy of Sciences, 97(23):12403–12404, 2000

    Hideyuki Okano, Tomoo Hirano, and Evan Balaban. Learning and memory.Proceedings of the National Academy of Sciences, 97(23):12403–12404, 2000

    2000

  2. [10]

    Cognitive development

    Susan Carey and Ellen M Markman. Cognitive development. In Cognitive science, pages 201–254. Elsevier, 1999

    1999

  3. [11]

    The role of deliberate practice in the acquisition of expert performance

    K Anders Ericsson, Ralf T Krampe, and Clemens Tesch-Römer. The role of deliberate practice in the acquisition of expert performance. Psychological review, 100(3):363, 1993

    1993

  4. [12]

    How people learn: Brain, mind, experience, and school: Expanded edition , volume 1

    National Research Council, Division of Behavioral, Board on Behavioral, Sensory Sciences, Committee on Developments in the Science of Learning with additional material from the Committee on Learning Research, and Educational Practice. How people learn: Brain, mind, experience,...

    2000

  5. [13]

    Orthogonal gradient descent for continual learning

    Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pages 3762–3773. PMLR, 2020

    2020

  6. [14]

    Continual learning in low-rank orthogonal subspaces

    Arslan Chaudhry, Naeemullah Khan, Puneet Dokania, and Philip Torr. Continual learning in low-rank orthogonal subspaces. Advances in Neural Information Processing Systems, 33:9900–9911, 2020. 12

    2020

  7. [15]

    Multitask learning

    Rich Caruana. Multitask learning. Machine learning, 28:41–75, 1997

    1997

  8. [16]

    An overview of multi-task learning in deep neural networks

    Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017

    2017 arXiv

  9. [17]

    Learning to learn by gradient descent by gradient descent

    Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016

    2016

  10. [18]

    Multi-task learning with deep neural networks: A survey

    Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796, 2020

    2009 arXiv

  11. [19]

    Understanding deep learning requires rethinking generalization

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2017

    2017 arXiv

  12. [20]

    Understanding batch normalization

    Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normalization. Advances in neural information processing systems, 31, 2018

    2018

  13. [21]

    Understanding dropout

    Pierre Baldi and Peter J Sadowski. Understanding dropout. Advances in neural informa- tion processing systems, 26, 2013

    2013

  14. [22]

    Optimization for deep learning: An overview

    Ruo-Yu Sun. Optimization for deep learning: An overview. Journal of the Operations Research Society of China, 8(2):249–294, 2020

    2020

  15. [23]

    Understanding the difficulty of training transformers

    Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. arXiv preprint arXiv:2004.08249, 2020

    2004 arXiv

  16. [24]

    Understanding the robustness in vision transformers

    Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference on Machine Learning, pages 27378–27394. PMLR, 2022

    2022

  17. [25]

    Rethinking calibration of deep neural networks: Do not be afraid of overconfidence

    Deng-Bao Wang, Lei Feng, and Min-Ling Zhang. Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. Advances in Neural Information Processing Systems, 34:11809–11820, 2021

    2021

  18. [26]

    Rethinking bias- variance trade-off for generalization of neural networks

    Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias- variance trade-off for generalization of neural networks. In International Conference on Machine Learning, pages 10767–10777. PMLR, 2020

    2020

  19. [27]

    Learning in order! a sequential strategy to learn invariant features for multimodal sentiment analysis

    Xianbing Zhao, Lizhen Qu, Tao Feng, Jianfei Cai, and Buzhou Tang. Learning in order! a sequential strategy to learn invariant features for multimodal sentiment analysis. In Proceedings of the 32nd ACM International Conference on Multimedia , pages 9729–9738, 2024

    2024

  20. [28]

    Sequential learning in the dense associative memory

    Hayden McAlister, Anthony Robins, and Lech Szymanski. Sequential learning in the dense associative memory. arXiv preprint arXiv:2409.15729, 2024

    2024 arXiv

  21. [29]

    Multi-modal mixture of experts represetation learning for sequential recommendation

    Shuqing Bian, Xingyu Pan, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, and Ji-Rong Wen. Multi-modal mixture of experts represetation learning for sequential recommendation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 11...

    2023

  22. [30]

    Robustness via deep low-rank representations

    Amartya Sanyal, Varun Kanade, Philip HS Torr, and Puneet K Dokania. Robustness via deep low-rank representations. arXiv preprint arXiv:1804.07090, 2018

    2018 arXiv

  23. [31]

    Jolliffe

    I.T. Jolliffe. Rotation of principal components: choice of normalization constraints. Journal of Applied Statistics, 22(1):29–35, 1995

    1995

  24. [32]

    Matrix factorization techniques for recommender systems

    Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30–37, 2009

    2009

  25. [33]

    Low rank subspace clustering

    René Vidal and Paolo Favaro. Low rank subspace clustering. Pattern Recognition Letters, 43:47–61, 2014. 13

    2014

  26. [34]

    Exact matrix completion via convex optimiza- tion

    Emmanuel Candes and Benjamin Recht. Exact matrix completion via convex optimiza- tion. Communications of the ACM, 55(6):111–119, 2012

    2012

  27. [35]

    Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization

    Benjamin Recht, Maryam Fazel, and Pablo A Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review, 52 (3):471–501, 2010

    2010

  28. [36]

    Estimation of high-dimensional low-rank matrices1

    Angelika Rohde and Alexandre B Tsybakov. Estimation of high-dimensional low-rank matrices1. The Annals of Statistics, 39(2):887–930, 2011

    2011

  29. [37]

    Robust PCA with compressed data

    Wooseok Ha and Rina Foygel Barber. Robust PCA with compressed data. Advances in Neural Information Processing Systems, 28, 2015

    2015

  30. [38]

    Analysis of a complex of statistical variables into principal compo- nents

    Harold Hotelling. Analysis of a complex of statistical variables into principal compo- nents. Journal of educational psychology, 24(6):417, 1933

    1933

  31. [39]

    Deflation methods for sparse pca

    Lester Mackey. Deflation methods for sparse pca. Advances in neural information processing systems, 21, 2008

    2008

  32. [40]

    The Schur complement and its applications, volume 4

    Fuzhen Zhang. The Schur complement and its applications, volume 4. Springer Science & Business Media, 2006

    2006

  33. [41]

    Sparse eigen methods by DC programming

    Bharath K Sriperumbudur, David A Torres, and Gert RG Lanckriet. Sparse eigen methods by DC programming. In Proceedings of the 24th international conference on Machine learning, pages 831–838, 2007

    2007

  34. [42]

    Projection and deflation method for partial pole assignment in linear state feedback

    Youcef Saad. Projection and deflation method for partial pole assignment in linear state feedback. IEEE Transactions on Automatic Control, 33(3):290–297, 1988

    1988

  35. [43]

    A comparison of eigenvalue methods for principal component analysis

    Y Danisman, MF Yilmaz, A Ozkaya, and I Comlekciler. A comparison of eigenvalue methods for principal component analysis. Appl. and Comput. Math, 13:316–331, 2014

    2014

  36. [44]

    Projection approximation subspace tracking

    Bin Yang. Projection approximation subspace tracking. IEEE Transactions on Signal processing, 43(1):95–107, 1995

    1995

  37. [45]

    Robust subspace learning: Robust PCA, robust subspace tracking, and robust subspace recovery

    Namrata Vaswani, Thierry Bouwmans, Sajid Javed, and Praneeth Narayanamurthy. Robust subspace learning: Robust PCA, robust subspace tracking, and robust subspace recovery. IEEE signal processing magazine, 35(4):32–55, 2018

    2018

  38. [46]

    Streaming PCA and subspace tracking: The missing data case

    Laura Balzano, Yuejie Chi, and Yue M Lu. Streaming PCA and subspace tracking: The missing data case. Proceedings of the IEEE, 106(8):1293–1310, 2018

    2018

  39. [47]

    The ideal continual learner: An agent that never forgets

    Liangzu Peng, Paris Giampouras, and René Vidal. The ideal continual learner: An agent that never forgets. In International Conference on Machine Learning, pages 27585–27610. PMLR, 2023

    2023

  40. [48]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In Interna- tional Conference on Learning Representations, 2021

    2021

  41. [49]

    Guaranteed minimum rank approximation from linear observations by nuclear norm minimization with an ellipsoidal constraint

    Kiryung Lee and Yoram Bresler. Guaranteed minimum rank approximation from linear observations by nuclear norm minimization with an ellipsoidal constraint. arXiv preprint arXiv:0903.4742, 2009

    2009 arXiv

  42. [50]

    Interior-point method for nuclear norm approxi- mation with application to system identification

    Zhang Liu and Lieven Vandenberghe. Interior-point method for nuclear norm approxi- mation with application to system identification. SIAM Journal on Matrix Analysis and Applications, 31(3):1235–1256, 2009

    2009

  43. [51]

    Guaranteed rank minimization via singular value projection

    Prateek Jain, Raghu Meka, and Inderjit S Dhillon. Guaranteed rank minimization via singular value projection. In Advances in Neural Information Processing Systems, pages 937–945, 2010

    2010

  44. [52]

    Admira: Atomic decomposition for minimum rank approximation

    Kiryung Lee and Yoram Bresler. Admira: Atomic decomposition for minimum rank approximation. IEEE Transactions on Information Theory, 56(9):4402–4416, 2010. 14

    2010

  45. [53]

    Kyrillidis and V

    A. Kyrillidis and V . Cevher. Matrix recipes for hard thresholding methods.Journal of mathematical imaging and vision, 48(2):235–265, 2014

    2014

  46. [54]

    Kyrillidis and V

    A. Kyrillidis and V . Cevher. Recipes on hard thresholding methods. InComputational Advances in Multi-Sensor Adaptive Processing (CAMSAP), 2011 4th IEEE International Workshop on, pages 353–356. IEEE, 2011

    2011

  47. [55]

    Khanna and A

    R. Khanna and A. Kyrillidis. IHT dies hard: Provable accelerated iterative hard thresholding. arXiv preprint arXiv:1712.09379, 2017

    2017 arXiv

  48. [56]

    Accelerated stochastic power iteration

    Peng Xu, Bryan He, Christopher De Sa, Ioannis Mitliagkas, and Chris Re. Accelerated stochastic power iteration. In Amos Storkey and Fernando Perez-Cruz, editors, Pro- ceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of ...

  49. [57]

    A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization

    Samuel Burer and Renato DC Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95(2):329–357, 2003

    2003

  50. [58]

    Provable inductive matrix completion

    Prateek Jain and Inderjit S Dhillon. Provable inductive matrix completion. arXiv preprint arXiv:1306.0626, 2013

    2013 arXiv

  51. [59]

    Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees

    Yudong Chen and Martin J Wainwright. Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees. arXiv preprint arXiv:1509.03025, 2015

    2015 arXiv

  52. [60]

    A nonconvex optimization framework for low rank matrix estimation

    Tuo Zhao, Zhaoran Wang, and Han Liu. A nonconvex optimization framework for low rank matrix estimation. In Advances in Neural Information Processing Systems, pages 559–567, 2015

    2015

  53. [61]

    A convergent gradient descent algorithm for rank minimization and semidefinite programming from random linear measurements

    Qinqing Zheng and John Lafferty. A convergent gradient descent algorithm for rank minimization and semidefinite programming from random linear measurements. In Advances in Neural Information Processing Systems, pages 109–117, 2015

    2015

  54. [62]

    S. Tu, R. Boczar, M. Simchowitz, M. Soltanolkotabi, and B. Recht. Low-rank solutions of linear matrix equations via Procrustes flow. In Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48 , pages 964–973. JMLR. org, 2016

    2016

  55. [63]

    Provable compressed sensing quantum state tomography via non-convex methods

    Anastasios Kyrillidis, Amir Kalev, Dohyung Park, Srinadh Bhojanapalli, Constan- tine Caramanis, and Sujay Sanghavi. Provable compressed sensing quantum state tomography via non-convex methods. npj Quantum Information, 4(1):36, 2018

    2018

  56. [64]

    Non-square matrix sensing without spurious local minima via the burer-monteiro approach

    Dohyung Park, Anastasios Kyrillidis, Constantine Caramanis, and Sujay Sanghavi. Non-square matrix sensing without spurious local minima via the burer-monteiro approach. arXiv preprint arXiv:1609.03240, 2016

    2016 arXiv

  57. [65]

    Guaranteed matrix completion via non-convex factor- ization

    Ruoyu Sun and Zhi-Quan Luo. Guaranteed matrix completion via non-convex factor- ization. IEEE Transactions on Information Theory, 62(11):6535–6579, 2016

    2016

  58. [66]

    Dropping convexity for faster semi-definite optimization

    Srinadh Bhojanapalli, Anastasios Kyrillidis, and Sujay Sanghavi. Dropping convexity for faster semi-definite optimization. In Conference on Learning Theory, pages 530–582, 2016

    2016

  59. [67]

    Global optimality of local search for low rank matrix recovery

    Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Global optimality of local search for low rank matrix recovery. InAdvances in Neural Information Processing Systems, pages 3873–3881, 2016

    2016

  60. [68]

    Finding low-rank solutions to matrix problems, efficiently and provably

    Dohyung Park, Anastasios Kyrillidis, Constantine Caramanis, and Sujay Sanghavi. Finding low-rank solutions to matrix problems, efficiently and provably. arXiv preprint arXiv:1606.03168, 2016. 15

    2016 arXiv

  61. [69]

    No spurious local minima in nonconvex low rank problems: A unified geometric analysis

    Rong Ge, Chi Jin, and Yi Zheng. No spurious local minima in nonconvex low rank problems: A unified geometric analysis. arXiv preprint arXiv:1704.00708, 2017

    2017 arXiv

  62. [70]

    A non-euclidean gradient descent framework for non- convex matrix factorization

    Ya-Ping Hsieh, Yu-Chun Kao, Rabeeh Karimi Mahabadi, Yurtsever Alp, Anastasios Kyrillidis, and Volkan Cevher. A non-euclidean gradient descent framework for non- convex matrix factorization. Technical report, Institute of Electrical and Electronics Engineers, 2017

    2017

  63. [71]

    Kyrillidis, A

    A. Kyrillidis, A. Kalev, D. Park, S. Bhojanapalli, C. Caramanis, and S. Sanghavi. Prov- able quantum state tomography via non-convex methods. npj Quantum Information, 4 (36), 2018

    2018

  64. [72]

    Fast quantum state reconstruction via accelerated non-convex programming

    Junhyung Lyle Kim, George Kollias, Amir Kalev, Ken X Wei, and Anastasios Kyril- lidis. Fast quantum state reconstruction via accelerated non-convex programming. In Photonics, volume 10, page 116. MDPI, 2023

    2023

  65. [73]

    Matrix computations

    Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013

    2013

  66. [74]

    Eigengame: {PCA} as a nash equilibrium

    Ian Gemp, Brian McWilliams, Claire Vernade, and Thore Graepel. Eigengame: {PCA} as a nash equilibrium. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=NzTU59SYbNq

    2021

  67. [75]

    Eigengame unloaded: When playing games is better than optimizing

    Ian Gemp, Brian McWilliams, Claire Vernade, and Thore Graepel. Eigengame unloaded: When playing games is better than optimizing. arXiv preprint arXiv:2102.04152, 2021

    2021 arXiv

  68. [76]

    Implicit regularization in deep matrix factorization, 2019

    Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization, 2019. URL https://arxiv.org/abs/1905.13655

    2019 arXiv

  69. [77]

    Du, and Jason D

    Jikai Jin, Zhiyuan Li, Kaifeng Lyu, Simon S. Du, and Jason D. Lee. Understanding incremental learning of gradient descent: A fine-grained analysis of matrix sensing,

  70. [78]

    Implicit balancing and regularization: Generalization and convergence guarantees for overparameterized asymmetric matrix sensing

    Mahdi Soltanolkotabi, Dominik Stöger, and Changzhi Xie. Implicit balancing and regularization: Generalization and convergence guarantees for overparameterized asymmetric matrix sensing. In Gergely Neu and Lorenzo Rosasco, editors,Proceedings of Thirty Sixth Conference on Learn...

    2023

  71. [79]

    Efficient compression of overparameterized deep models through low-dimensional learning dynamics, 2024

    Soo Min Kwon, Zekai Zhang, Dogyoon Song, Laura Balzano, and Qing Qu. Efficient compression of overparameterized deep models through low-dimensional learning dynamics, 2024. URL https://arxiv.org/abs/2311.05061

    2024 arXiv

  72. [80]

    Zhi-Yong Wang, Xiao Peng Li, Hing Cheung So, and Abdelhak M. Zoubir. Adaptive rank-one matrix completion using sum of outer products. IEEE Transactions on Circuits and Systems for Video Technology , 33(9):4868–4880, 2023. doi: 10.1109/TCSVT.2023. 3250651

    2023 doi

  73. [81]

    Contin- ual learning with low rank adaptation, 2023

    Martin Wistuba, Prabhu Teja Sivaprasad, Lukas Balles, and Giovanni Zappella. Contin- ual learning with low rank adaptation, 2023. URL https://arxiv.org/abs/2311. 17601

    2023

  74. [82]

    Orthogonal subspace learning for language model continual learning, 2023

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning, 2023. URL https://arxiv.org/abs/2310.14152

    2023 arXiv

  75. [83]

    AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023

    2023 arXiv

  76. [84]

    Merging LoRAs like playing LEGO: Pushing the modularity of LoRA to extremes through rank-wise clustering

    Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, and Fei Wu. Merging LoRAs like playing LEGO: Pushing the modularity of LoRA to extremes through rank-wise clustering. arXiv preprint arXiv:2409.16167, 2024. 16

    2024 arXiv

  77. [85]

    Pareto low-rank adapters: Efficient multi-task learning with preferences

    Nikolaos Dimitriadis, Pascal Frossard, and Francois Fleuret. Pareto low-rank adapters: Efficient multi-task learning with preferences. arXiv preprint arXiv:2407.08056, 2024

    2024 arXiv

  78. [86]

    Mixture-of-subspaces in low-rank adaptation

    Taiqiang Wu, Jiahao Wang, Zhe Zhao, and Ngai Wong. Mixture-of-subspaces in low-rank adaptation. arXiv preprint arXiv:2406.11909, 2024

    2024 arXiv

  79. [87]

    Towards modular LLMs by building and reusing a library of LoRAs

    Oleksiy Ostapenko, Zhan Su, Edoardo Ponti, Laurent Charlin, Nicolas Le Roux, Lucas Caccia, and Alessandro Sordoni. Towards modular LLMs by building and reusing a library of LoRAs. In Forty-first International Conference on Machine Learning

  80. [88]

    Training neural networks from scratch with parallel low-rank adapters

    Minyoung Huh, Brian Cheung, Jeremy Bernstein, Phillip Isola, and Pulkit Agrawal. Training neural networks from scratch with parallel low-rank adapters. arXiv preprint arXiv:2402.16828, 2024

    2024 arXiv

  81. [89]

    Chain of LoRA: Efficient fine-tuning of language models via residual learning

    Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of LoRA: Efficient fine-tuning of language models via residual learning. arXiv preprint arXiv:2401.04151, 2024

    2024 arXiv

  82. [90]

    Perturbation bounds in connection with singular value decomposition

    Per-Åke Wedin. Perturbation bounds in connection with singular value decomposition. BIT, 12(1):99–111, March 1972. ISSN 0006-3835. doi: 10.1007/BF01932678. URL https://doi.org/10.1007/BF01932678

    1972 doi

  83. [91]

    ground-truth

    H. Weyl. Das asymptotische verteilungsgesetz der eigenwerte linearer partieller dif- ferentialgleichungen (mit einer anwendung auf die theorie der hohlraumstrahlung). Mathematische Annalen, 71:441–479, 1912. URL http://eudml.org/doc/158545. 17 A Proof of Theorem 1 The proof fo...

    1912

  84. [94]

    Given that ba⊤ is a rank-1 matrix, the product ba⊤X is also rank-1

    = arg min a∈Rd,b∈Rm 1 2 Y⋆ 1 − ba⊤X 2 F . Given that ba⊤ is a rank-1 matrix, the product ba⊤X is also rank-1. Therefore, ac- cording to the Eckart-Young-Mirsky theorem, the best rank-1 approximation of Y⋆ 1 = Y = Pp i=1 σ⋆ i u⋆ i v⋆⊤ i , based on its Singular Value Decompositi...

  85. [95]

    Singular Value Profiles: We vary the singular value distribution of W⋆ to analyze how the spectrum of ground truth influences error propagation

  86. [96]

    Noise Variations: We introduce different types and levels of noise to assess the robustness of sequential rank-1 learning to perturbations

  87. [97]

    more-first

    Iteration allocation strategies: We evaluate three different iteration allocation strategies: (a) Equal: Same number of optimization iterations to each rank-1 component. (b) More First: More iterations allocated to the earlier components and fewer to later ones. (c) Less First...

  88. [98]

    1→10→10” and “10→1→1

    consistently achieve higher parameter efficiency compared to standard LoRA. However, sequential rank-1 models require more total training to achieve comparable accuracy, thus creating a tradeoff to be taken in consideration in practice, but still maintain favorable parameter-t...

  89. [2018]

    URL https://proceedings.mlr.press/v84/xu18a.html

  90. [2023]

    URL https://arxiv.org/abs/2301.11500

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.