Pith. sign in

REVIEW 4 major objections 4 minor 56 references

SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SMMF claims that square-matricizing any rank-d momentum tensor and factorizing it into two vectors cuts optimizer memory by up to 96% while retaining AdamNC-comparable regret.

desk verdict The empirical optimizer is practical and the experimental campaign is serious, but the regret analysis in Theorem 4.1 does not go through as stated; the paper deserves review, not rejection out of hand. read the letter →

arxiv 2412.08894 v2 pith:YCEYYEPP submitted 2024-12-12 cs.LG cs.AI

Kwangryeol Park , Seulki Lee This is my paper · ORCID
classification cs.LGcs.AI
keywords square-matricizationmemory-efficientoptimizernon-negativematrixfactorizationmomentumAdamregretboundadaptivelearningrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes SMMF, an optimizer that stores the first and second momentum tensors of Adam-style training in factorized form, reducing optimizer memory by up to 96% relative to existing memory-efficient optimizers such as Adafactor, CAME, and SM3. The central idea is square-matricization: any rank-d momentum tensor is reshaped into the matrix closest to a square, then factorized into two vectors by one non-negative matrix factorization, with a binary sign matrix preserving the signs of the first momentum. The paper proves a regret bound for SMMF matching the AdamNC bound, $O(\sqrt{T})$ under two scheduling conditions, arguing that factorization need not sacrifice convergence. Experiments on CNNs and Transformers report competitive accuracy and perplexity with drastically lower memory.

What carries the argument

Square-matricization is the central mechanism: given a rank-$d$ tensor with $N$ elements, SMMF chooses $\hat{n}, \hat{m}$ with $\hat{n}\hat{m} = N$ that minimize $|\hat{n} - \hat{m}|$, reshapes the tensor into $\mathbb{R}^{\hat{n} \times \hat{m}}$, and applies one-time rank-1 NNMF to obtain row and column vectors whose outer product approximates the matrix. A decompression-to-compression loop reconstructs the momentum, updates it with the current gradient, and factorizes again; a binary sign matrix restores sign information for the first momentum. Theorems 3.1 and 3.2 establish that minimizing $|n-m|$ also minimizes $n+m$, which is what makes the square shape memory-optimal.

What would settle it

Run SMMF on a convex problem and record the compression errors $e_{m,t}$ and $e_{v,t}$ from Lemma E.3. If the substitution $\tilde{g}_{m,t}^2 \le \zeta_1^2 \tilde{g}_{v,t}$ used in Lemma E.5 fails at any step, or the error sum for the sign-restored first momentum is not zero, Theorem 4.1's derivation breaks; one can also check directly whether the empirical regret stays below the bound in Equation (2).

Watch

Extended reading notes

Core claim

On its own terms, SMMF claims that the memory cost of adaptive optimization can be reduced from $O(n_1 \cdots n_d)$ to $O(\hat{n} + \hat{m})$ per momentum for tensors of any rank, by reshaping each momentum into the closest-to-square matrix $\mathbb{R}^{\hat{n} \times \hat{m}}$ and factorizing it into two vectors via non-negative matrix factorization. The first momentum's signs are kept in a 1-bit binary matrix, and the second momentum is non-negative and factorized directly. Theorem 4.1 states that, under the two scheduling conditions (a) and (b) imported from AdamNC, the regret satisfies the same upper bound as AdamNC, namely $O(\sqrt{T})$ for convex objectives, with the explicit bound given in Equation (2). The paper takes this bound, together with experiments on CNNs and Transformers, as evidence that the factorization does not sacrifice convergence while saving up to 96% of optimizer memory.

Load-bearing premise

The regret proof assumes that the errors introduced by compressing and decompressing the momentums are small enough that the AdamNC convergence argument still applies, but the proof never bounds those errors.

Editorial extensions

If this is right

  • Optimizer memory becomes $O(\hat{n} + \hat{m})$ per momentum, so high-rank tensors such as CNN kernels no longer force memory to grow with the product of all tensor dimensions.
  • Under the two AdamNC scheduling conditions, SMMF inherits an $O(\sqrt{T})$ regret bound in the convex setting, matching Adam-family optimizers.
  • Transformer-base, which needs about 1.4 GiB with Adam, trains with roughly 0.8 GiB end-to-end under SMMF, bringing it closer to memory-constrained devices.
  • SMMF applies to vectors, matrices, and rank-$d$ tensors alike, unlike Adafactor and CAME which target matrix-shaped momentums.
  • Training time rises modestly, about 1.2 to 1.6 times Adam, while optimizer memory drops by up to 96%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the compression-error gap in the proof is closed, the same square-matricization trick could likely combine with 8-bit quantization of the factor vectors to push memory even lower.
  • Because square reshaping deliberately discards the original coordinate layout, a stress test comparing SMMF against reshaping into a non-square shape would reveal whether the memory benefit is purely from factorization or from the square geometry itself.
  • SMMF's decompression-first scheme keeps the full current gradient before compression, so it may be particularly suited to fine-tuning tasks where small precise updates matter, such as instruction tuning of large language models.
  • The 1-bit sign matrix dominates memory for small tensors like biases; the paper itself notes that binary matrix factorization could reduce that overhead further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SMMF, an optimizer that square-matricizes arbitrary-rank momentum tensors and compresses the first and second moments into row and column vectors by one NNMF step, storing a binary sign matrix for the first moment. It claims up to 96% memory reduction relative to Adafactor/CAME/SM3, comparable empirical performance across CNN and Transformer tasks, and a regret bound over convex objectives, Theorem 4.1, matching AdamNC's O(sqrt(T)) rate. The theoretical analysis is centered on bounding regret terms after absorbing NNMF decompression-compression errors e_m,t and e_v,t. The empirical section reports extensive experiments with optimizer memory, end-to-end memory, and task metrics.

Significance. If the claims were established, SMMF would fill a useful niche: a factorization-based optimizer that works for any tensor rank and comes with a regret analysis, something Adafactor and CAME lack. The square-matricization idea in Section 3.1 is simple and Theorems 3.1 and 3.2 appear correct. The empirical study is broad, includes many models and tasks, and code is provided. However, the central theoretical guarantee is not supported by the proof as written, and the formal memory-complexity statement ignores the sign matrix. The paper's significance is therefore currently limited to a practical memory-saving proposal with strong empirical results but unsupported asymptotic claims.

major comments (4)
  1. [Appendix E, Lemma E.5, Eq. (70)] Equation (70) replaces sum_j beta^{T-j} tilde{g}_{m,j,i}^2 with zeta_1^2 sum_j beta^{T-j} tilde{g}_{v,j,i} on the justification 'since tilde{g}_{m,j,i}^2 and tilde{g}_{v,j,i} are positive.' Positivity alone does not imply an elementwise inequality of the form tilde{g}_{m,j,i}^2 <= zeta_1^2 tilde{g}_{v,j,i} for a fixed zeta_1. No bound on the compression-error terms e_{m,t} and e_{v,t} is given, and Lemma E.7's zero-sum property cannot be used for the first momentum because SMMF applies NNMF to |M| and restores signs, so the reconstruction error of M is not the error of NNMF on a non-negative matrix. Consequently the chain (69)-(75) does not prove Lemma E.5, and Theorem 4.1's O(sqrt(T)) regret bound is unsupported.
  2. [Appendix E, Lemma E.5, Eqs. (70)-(71)] Independently of the zeta_1 issue, the proof drops the accumulated sum: Eq. (70) has sum_{t=1}^{T-1} eta_t m_{t,i}^2/sqrt(v_{t,i}) on the right-hand side, but Eq. (71) only keeps the newly bounded term. Unless an induction over T with a dominating inequality is supplied, the displayed inequalities do not imply the claimed bound for S_T = sum_{t=1}^T eta_t m_{t,i}^2/sqrt(v_{t,i}).
  3. [Section 3 and Algorithm 4] The formal memory complexity O_M(hat n + hat m) and O_V(hat n + hat m) in Section 3 ignores the sign matrix S in {0,1}^{hat n x hat m} stored by Algorithm 4. For a square-matricized tensor with hat n approximately hat m approximately sqrt(N), the sign matrix alone is N bits, so the asymptotic optimizer-state size is Theta(N) bits, not O(sqrt(N)) as implied. Including the sign matrix may still give a constant-factor saving over the original 32-bit momentum, and Table 1 does count S in the measured memory, but the formal claim as stated is false and should be revised.
  4. [Section 4, Theorem 4.1] The theorem assumes the Reddi et al. conditions (a)-(b) 'can be satisfied by properly scheduling beta_{2,t}', but the paper never proves that the default schedule used in Algorithm 8, beta_{2,t}=1-t^gamma, satisfies them. In addition, condition (b) as printed has the factor (1 - beta_{2,j} tilde{g}_{v,j,i}), which is structurally different from the Reddi et al. condition (1 - beta_{2,j}) tilde{g}_{v,j,i}, and zeta_1 is not connected to any inequality in the conditions. Thus even if Lemma E.5 were repaired, the theorem would not be linked to the algorithm as implemented.
minor comments (4)
  1. [Appendix E, Lemma E.3, Eq. (40)] The recurrence for hat v_t writes hat g_{m,j}; it should presumably be hat g_{v,j}.
  2. [Table 3] The table reports NaN for GPT-2 with Adafactor; this is an unusual result and the caption or text should explain the configuration or the reason this cell is excluded.
  3. [Algorithm 4] The notation 1_d is defined but never used; the algorithm actually uses 1_{hat n} and 1_{hat m}, which should be defined.
  4. [General presentation] There are typos such as 'full-trining' in Section 5 and 'SCMF' in Appendix L tables, and a duplicated citation in Appendix J ('Il-Post (Landro et al. 2022a) (Landro et al. 2022a)').

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 4.1 is an attempted external regret proof with a gap, not a construction that reduces to its own inputs.

full rationale

SMMF's regret analysis follows the AdamNC proof template and imports conditions (a) and (b) from Reddi, Kale, and Kumar (2019); this is external support rather than a self-citation chain. The compression error terms e_m,t and e_v,t are introduced as defined quantities, not fitted parameters, and the final regret bound is not obtained by renaming a fitted quantity as a prediction. Lemma E.7's zero-sum property is a real algebraic fact for the nonnegative second momentum and is applied there; it is not used to force the conclusion for the sign-restored first momentum. The genuine weaknesses identified by the reader are proof gaps, not circularity: Lemma E.5 replaces \tilde g_m^2 with \zeta_1^2 \tilde g_v solely because both quantities are positive, and no proof is given that such a finite \zeta_1 exists uniformly; likewise the satisfiability of conditions (a) and (b) for SMMF's actual \beta_2 schedule is asserted, not verified. These are correctness concerns about an incomplete derivation, but they do not exhibit a step where the claimed result is equivalent to its own input by definition, by fitted construction, or by a load-bearing self-citation. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The key unproven input is the bound on compression errors in the regret analysis, which is an ad hoc assumption rather than a derived quantity.

free parameters (3)
  • decay_rate gamma = -0.5 (CNN), -0.8 (Transformer)
    Hand-tuned per model family in Algorithm 8 and Appendix L; the paper states the recommended range -0.5 to -0.8 and that performance depends on architecture.
  • weight_decay multiplier = 1/1000 to 1/2000 of AdamW value
    Appendix F reports that weight decay must be 1000-2000 times smaller than AdamW to avoid loss spikes; tuned empirically.
  • growth_rate lambda = 0.999
    Used in the beta1 schedule in Algorithm 8; chosen as a standard value, not derived.
assumptions (4)
  • standard math Convexity and standard inequalities used in the regret proof (Lemmas E.1, E.2, E.8, Cauchy-Schwarz, Young's inequality).
    Invoked in Appendix E as standard analysis tools.
  • domain assumption The decompression error matrix from NNMF preserves the total sum of the 2nd momentum (Lemma E.7).
    Holds for non-negative V under the row/column-sum factorization, but the proof is not extended to the sign-restored first momentum M.
  • ad hoc to paper Compression errors can be absorbed into constants zeta1 and zeta2 so that the AdamNC regret chain remains valid (Theorem 4.1, Lemma E.5).
    No bound on e_m,t or e_v,t is stated or proved; the substitution of tilde g_m^2 by tilde g_v in Lemma E.5 is unverified.
  • ad hoc to paper The beta2 schedule (beta2_t = 1 - t^gamma) satisfies Reddi et al. conditions (a) and (b).
    Stated in Theorem 4.1 with the comment that conditions can be satisfied by scheduling beta2_t, but no verification is given for the chosen gamma.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization." pith.science (2026). https://pith.science/paper/YCEYYEPP

@misc{pith2026241208894,
  author       = {Pith},
  title        = {Pith review of: SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YCEYYEPP}},
  note         = {Machine review of arXiv:2412.08894}
}
read the original abstract

We propose SMMF (Square-Matricized Momentum Factorization), a memory-efficient optimizer that reduces the memory requirement of the widely used adaptive learning rate optimizers, such as Adam, by up to 96%. SMMF enables flexible and efficient factorization of an arbitrary rank (shape) of the first and second momentum tensors during optimization, based on the proposed square-matricization and one-time single matrix factorization. From this, it becomes effectively applicable to any rank (shape) of momentum tensors, i.e., bias, matrix, and any rank-d tensors, prevalent in various deep model architectures, such as CNNs (high rank) and Transformers (low rank), in contrast to existing memory-efficient optimizers that applies only to a particular (rank-2) momentum tensor, e.g., linear layers. We conduct a regret bound analysis of SMMF, which shows that it converges similarly to non-memory-efficient adaptive learning rate optimizers, such as AdamNC, providing a theoretical basis for its competitive optimization capability. In our experiment, SMMF takes up to 96% less memory compared to state-of-the-art memory efficient optimizers, e.g., Adafactor, CAME, and SM3, while achieving comparable model performance on various CNN and Transformer tasks.

Figures

Figures reproduced from arXiv: 2412.08894 by the authors.

Figure 1
Figure 1. (Left) The validation top-1 accuracy of Mo￾bileNetV2 on ImageNet. (Right) The validation mAP50 of YOLOv5s on COCO of the five optimizers. Transformer Models and Tasks (Full-Trining) (Optimizer and End-to-End Memory [GiB]), Model Performance Dataset WMT32k Model Adam Adafactor SM3 CAME SMMF Transformer (0.7, 1.4) (0.4, 1.1) ( 0.4, 1.1) (0.4, 1.1) ( .01, 0.8) (base) 6.6 6.6 7.8 6.6 6.7 Transformer (2.1, 4.2) (1.1, 3.2… view at source ↗
Figure 2
Figure 2. The test perplexity of the Transformer-base model [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. (Left) The validation top-1 accuracy of MobileNetV2 on ImageNet. (Right) The test perplexity of the Transformer￾base model on WMT32k during full-training steps. The graphs, based on measurements from the Wandb 1 platform, show the training time for two tasks, i.e., image classification and full-training . The left graph corresponds to training MobileNetV2 on ImageNet, and the right graph corresponds to training the … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The loss graph of LLaMA-2-7B with LoRA fine-tuned on Alpaca dataset during the 1000 steps. The blue line indicates [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 20 canonical work pages

  1. [1]

    Abdulkadirov, R.; Lyakhov, P.; and Nagornov, N. 2023. Survey of Optimization Algorithms in Modern Neural Networks. Mathematics, 11(11): 2466

    work page 2023

  2. [2]

    Aghajanyan, A.; Zettlemoyer, L.; and Gupta, S. 2020. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. arXiv:2012.13255

    arXiv 2020

  3. [3]

    Amari, S.-i. 2010. Information geometry in optimization, machine learning and statistical inference. Frontiers of Electrical and Electronic Engineering in China, 5: 241--260

    work page 2010

  4. [4]

    Anil, R.; Gupta, V.; Koren, T.; and Singer, Y. 2019. Memory Efficient Adaptive Optimization. In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d Alch\' e -Buc, F.; Fox, E.; and Garnett, R., eds., Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc

    work page 2019

  5. [5]

    Bojar, O.; Buck, C.; Federmann, C.; Haddow, B.; Koehn, P.; Leveling, J.; Monz, C.; Pecina, P.; Post, M.; Saint-Amand, H.; et al. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation, 12--58

    work page 2014

  6. [6]

    Bojar, O. r.; Chatterjee, R.; Federmann, C.; Graham, Y.; Haddow, B.; Huck, M.; Jimeno Yepes, A.; Koehn, P.; Logacheva, V.; Monz, C.; Negri, M.; Neveol, A.; Neves, M.; Popel, M.; Post, M.; Rubino, R.; Scarton, C.; Specia, L.; Turchi, M.; Verspoor, K.; and Zampieri, M. 2016. Findings of the 2016 Conference on Machine Translation. In Proceedings of the First...

    work page 2016

  7. [7]

    Chen, T.; Xu, B.; Zhang, C.; and Guestrin, C. 2016. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174

    arXiv 2016

  8. [8]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

Show all 56 references
  1. [9]

    Dong, K.; Zhou, C.; Ruan, Y.; and Li, Y. 2020. MobileNetV2 Model for Image Classification. In 2020 2nd International Conference on Information Technology and Computer Application (ITCA), 476--480

    2020

  2. [10]

    Finesso, L.; and Spreij, P. 2006. Nonnegative matrix factorization and I-divergence alternating minimization. Linear Algebra and its Applications, 416(2): 270--287

    2006

  3. [11]

    Google. 2022. Coral Dev Board Micro. Https://coral.ai/products/dev-board-micro/

    2022

  4. [12]

    R.; Millman, K

    Harris, C. R.; Millman, K. J.; van der Walt, S. J.; Gommers, R.; Virtanen, P.; Cournapeau, D.; Wieser, E.; Taylor, J.; Berg, S.; Smith, N. J.; Kern, R.; Picus, M.; Hoyer, S.; van Kerkwijk, M. H.; Brett, M.; Haldane, A.; del R \' i o, J. F.; Wiebe, M.; Peterson, P.; G \' e rard...

    2020

  5. [13]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770--778

    2016

  6. [14]

    Hinton, G.; Srivastava, N.; and Swersky, K. 2012. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8): 2

    2012

  7. [15]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685

    2021 arXiv

  8. [16]

    P.; and Ba, J

    Kingma, D. P.; and Ba, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  9. [17]

    Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images

    2009

  10. [18]

    Kumar, R.; Panigrahy, R.; Rahimi, A.; and Woodruff, D. 2019. Faster Algorithms for Binary Matrix Factorization. In Chaudhuri, K.; and Salakhutdinov, R., eds., Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Res...

    2019

  11. [19]

    Lan, Z.; Chen, M.; Goodman, S.; Gimpel, K.; Sharma, P.; and Soricut, R. 2020. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv:1909.11942

    2020 arXiv

  12. [20]

    Landro, N.; Gallo, I.; La Grassa, R.; and Federici, E. 2022 a . Two New Datasets for Italian-Language Abstractive Text Summarization. Information, 13(5)

    2022

  13. [21]

    Landro, N.; Gallo, I.; La Grassa, R.; and Federici, E. 2022 b . Two New Datasets for Italian-Language Abstractive Text Summarization. Information, 13(5)

    2022

  14. [22]

    D.; and Seung, H

    Lee, D. D.; and Seung, H. S. 1999. Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755): 788--791

    1999

  15. [23]

    Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2019. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. arXiv:1910.13461

    2019 arXiv

  16. [24]

    Li, C.; Farkhoor, H.; Liu, R.; and Yosinski, J. 2018. Measuring the Intrinsic Dimension of Objective Landscapes. arXiv:1804.08838

    2018 arXiv

  17. [25]

    L.; and Dollár, P

    Lin, T.-Y.; Maire, M.; Belongie, S.; Bourdev, L.; Girshick, R.; Hays, J.; Perona, P.; Ramanan, D.; Zitnick, C. L.; and Dollár, P. 2015. Microsoft COCO: Common Objects in Context. arXiv:1405.0312

    2015 arXiv

  18. [26]

    C.; and Nocedal, J

    Liu, D. C.; and Nocedal, J. 1989. On the limited memory BFGS method for large scale optimization. Mathematical programming, 45(1-3): 503--528

    1989

  19. [27]

    Liu, Y.; Gu, J.; Goyal, N.; Li, X.; Edunov, S.; Ghazvininejad, M.; Lewis, M.; and Zettlemoyer, L. 2020. Multilingual Denoising Pre-training for Neural Machine Translation. arXiv:2001.08210

    2020 arXiv

  20. [28]

    Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692

    2019 arXiv

  21. [29]

    Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. arXiv:1711.05101

    2019 arXiv

  22. [30]

    Luo, Y.; Ren, X.; Zheng, Z.; Jiang, Z.; Jiang, X.; and You, Y. 2023. CAME : Confidence-guided Adaptive Memory Efficient Optimization. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ...

    2023

  23. [31]

    Martens, J. 2016. Second-order optimization for neural networks. University of Toronto (Canada)

    2016

  24. [32]

    N.; Gulcehre, C.; and Xiang, B

    Nallapati, R.; Zhou, B.; dos santos, C. N.; Gulcehre, C.; and Xiang, B. 2016. Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond. arXiv:1602.06023

    2016 arXiv

  25. [33]

    B.; and Lapata, M

    Narayan, S.; Cohen, S. B.; and Lapata, M. 2018. Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. ArXiv, abs/1808.08745

    2018 arXiv

  26. [34]

    OpenAI; Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; Avila, R.; Babuschkin, I.; Balaji, S.; Balcom, V.; Baltescu, P.; Bao, H.; Bavarian, M.; Belgum, J.; Bello, I.; Berdine, J.; Bernadett-Shapi...

    2024 arXiv

  27. [35]

    Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017. Automatic Differentiation in PyTorch. In NIPS 2017 Workshop on Autodiff

    2017

  28. [36]

    Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9

    2019

  29. [37]

    Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1): 5485--5551

    2020

  30. [38]

    Rajpurkar, P.; Jia, R.; and Liang, P. 2018. Know What You Don't Know: Unanswerable Questions for SQuAD. arXiv:1806.03822

    2018 arXiv

  31. [39]

    Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv:1606.05250

    2016 arXiv

  32. [40]

    J.; Kale, S.; and Kumar, S

    Reddi, S. J.; Kale, S.; and Kumar, S. 2019. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237

    2019 arXiv

  33. [41]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. arXiv:2112.10752

    2022 arXiv

  34. [42]

    Ruder, S. 2016. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747

    2016 arXiv

  35. [43]

    C.; and Fei-Fei, L

    Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; Berg, A. C.; and Fei-Fei, L. 2015. ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Vision (IJCV), 115(3): 211--252

    2015

  36. [44]

    Shazeer, N.; and Stern, M. 2018. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. In Dy, J.; and Krause, A., eds., Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, 4596--4604. PMLR

    2018

  37. [45]

    Shi, Z.; Wang, L.; and Shi, L. 2014. Approximation method to rank-one binary matrix factorization. In 2014 IEEE International Conference on Automation Science and Engineering (CASE), 800--805

    2014

  38. [46]

    Takase, S.; Kiyono, S.; Kobayashi, S.; and Suzuki, J. 2024. Spike No More: Stabilizing the Pre-training of Large Language Models. arXiv:2312.16903

    2024 arXiv

  39. [47]

    Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  40. [48]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lample, G. 2023 a . LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971

    2023 arXiv

  41. [49]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; Bikel, D.; Blecher, L.; Ferrer, C. C.; Chen, M.; Cucurull, G.; Esiobu, D.; Fernandes, J.; Fu, J.; Fu, W.; Fuller, B.; Gao, C.; Goswami, V.; Goyal, N....

    2023 arXiv

  42. [50]

    Ultralytics. 2021. YOLOv5 : A state-of-the-art real-time object detection system. https://docs.ultralytics.com

    2021

  43. [51]

    N.; Kaiser, .; and Polosukhin, I

    Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  44. [52]

    Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461

    2018 arXiv

  45. [53]

    Zhang, Z.; and Xu, Z.-Q. J. 2023. Loss Spike in Training Neural Networks. arXiv:2305.12133

    2023 arXiv

  46. [54]

    Zhu, Y.; Kiros, R.; Zemel, R.; Salakhutdinov, R.; Urtasun, R.; Torralba, A.; and Fidler, S. 2015. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In The IEEE International Conference on Computer Vision (ICCV)

    2015

  47. [55]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  48. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.