Pith. sign in

REVIEW 3 major objections 4 minor 62 references

Split Forward Gradients (Split-FG) trains a 16M-parameter GPT-2-style transformer from scratch to validation perplexity 387 with no backward pass through the trunk.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 20:25 UTC pith:UQTGCJSA

load-bearing objection A genuinely new split estimator with a solid variance identity, but the paper overstates its convergence guarantee and leans on single-seed results. the 3 major comments →

arxiv 2607.16612 v1 pith:UQTGCJSA submitted 2026-07-18 stat.ML cs.LG

Backpropagation-Free Trunk Training via the Split Forward Gradients

Tian Qin , Wei-Min Huang This is my paper
classification stat.ML cs.LG MSC 68T07
keywords forward gradientsbackpropagation-free trainingvariance reductiontransformer pretrainingAdamJacobian-vector productmemory-efficient trainingperplexity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that forward-mode gradient estimation can train deep networks from scratch if the network is split at an intermediate representation: the output head is differentiated exactly, while only the trunk is estimated with Jacobian-vector products. The split cuts the dominant variance term from the full gradient norm to the trunk gradient norm, so models whose head holds most parameters — like transformers with large vocabulary readouts — see a large variance reduction. On WikiText-103, a 16M-parameter GPT-2-style model trained this way reaches validation perplexity 387, versus 668 for a frozen random trunk and 2,885 for pure forward gradients (backpropagation reaches 150). The paper also identifies a failure mode: naive Adam updates of noisy trunk coordinates are worse than not training the trunk at all, and scaling the trunk step by rho = 0.03 reverses this. If right, this gives a practical backprop-free pretraining recipe for small transformers and a partial answer to why prior forward-gradient methods stalled at scale.

Core claim

The central claim is that decomposing the gradient at a hidden state h makes forward-mode training viable: Split-FG computes the head gradient and the loss derivative u = dL/dh exactly in closed form, then estimates only the trunk gradient by averaging K random directional derivatives <u, J_trunk v_k> v_k. This estimator is unbiased and replaces the dominant per-coordinate variance term ||grad L||^2 with ||grad_trunk L||^2, reducing variance by roughly P_trunk/P_total. Empirically, the paper shows a 16M GPT-style model reaches validation perplexity 387 compared with 668 for a frozen random trunk and 2,885 for a matched pure forward-gradient baseline, while backpropagation reaches 150; peak m

What carries the argument

The central object is the split at an intermediate hidden state h. Algorithm 1 first computes the trunk representation, the exact head gradient, and u = dL/dh; then, for each of K Gaussian tangents v_k in trunk-parameter space, it computes the Jacobian-vector product Delta_h_k = J_trunk v_k, scores it by the inner product d_k = <u, Delta_h_k>, and forms the trunk estimate as the average of d_k v_k. This push-score-estimate chain rule requires no backward pass through the trunk and keeps the estimator unbiased, reducing per-coordinate variance from ||grad L||^2 to ||grad_trunk L||^2. The practical companion is the trunk step scale rho, a per-group learning-rate multiplier that slows noisy tru

Load-bearing premise

The load-bearing premise is that the variance-reduction analysis and the trunk-step-scale fix transfer to the actual Adam implementation with global gradient clipping, since the convergence proof covers only an idealized predictable preconditioner and the GPT result further depends on a tied-embedding surrogate and validation-selected rho.

What would settle it

Train the same small8 GPT with rho = 0.03 but without the tied-embedding readout surrogate (propagating the trunk JVP through the vocabulary lookup instead); if perplexity rises toward the frozen-trunk control or worse, the readout surrogate is doing the work, whereas staying near 387 would confirm the split itself carries the result.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Backprop-free pretraining of small transformers becomes feasible: a 16M GPT reaches 387 perplexity versus 2,885 for pure forward gradients at matched tangent count and tuning.
  • Peak GPU memory drops by up to 35% relative to matched backpropagation, with zero reverse-mode passes through the trunk, a regime suited to forward-only hardware.
  • Variance reduction is governed by the head/trunk parameter split, so architectures with large vocabulary readouts benefit for free; the method degrades when the trunk dominates unless the network is restructured with a light trunk and heavy exact head.
  • The frozen-trunk control exposes a noise-blind step-size failure mode, and a simple per-group trunk learning-rate scale rho reverses it across a 100x grid of values.
  • The O(1/T) nonconvex convergence guarantee holds for an idealized predictable preconditioner, with the forward-gradient variance penalty scaling as P_trunk/K rather than P_total/K.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the noise-blind step-size explanation is correct, an adaptive SNR-aware scheduling rule could replace the hand-set rho and may transfer to larger trunks; one testable check is whether naive Adam causes large trunk displacement from initialization.
  • The GPT result relies on a tied-embedding readout surrogate and a validation-selected rho, so removing the surrogate (or extending the convergence proof to cover it) would clarify whether the split itself, rather than those heuristics, carries the result.
  • The heavy-head/light-trunk restructuring suggests a broader design principle for forward-only training: push parameters into exactly-differentiated layers and keep the forward-mode trunk small, which improves both variance and activation memory.
  • Conditioned tangents that scale each tangent coordinate by a running gradient-magnitude estimate are a natural next step; the paper reports preliminary evidence of better sample efficiency but no final gain under full annealing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. Split-FG partitions a network at an intermediate representation into a trunk and a head. The head gradient is computed exactly in closed form, while the trunk gradient is estimated by K forward-mode Jacobian-vector products with Gaussian tangents. The paper derives the unbiasedness of this estimator, proves a per-coordinate variance reduction from ||∇θL||² to ||∇θ_trunkL||², and gives a convergence theorem for an idealized predictable-preconditioner update. Experiments include a toy variance diagnostic (within 6.8% of the energy-ratio prediction), tabular benchmarks, CIFAR-10/100 with a heavy-head/flat-trunk system, and a 16M-parameter GPT-2-style model on WikiText-103. The GPT runs show a failure mode: naive trunk training is worse than freezing the trunk, and rescaling the trunk step by ρ=0.03 gives validation perplexity 386.7 versus 667.6 for the frozen trunk and 2,885 for a matched pure forward-gradient baseline, with a 34.5% reduction in peak memory and a 3.2× increase in step time. The paper is transparent about several limitations, including single-seed runs, validation-selected hyperparameters, gradient clipping, and a tied-embedding surrogate.

Significance. If the empirical results hold, Split-FG provides a credible backprop-free training recipe for small transformers, with a measured variance-reduction mechanism and a practical fix (per-group step scaling) for a failure mode that is invisible without a frozen-trunk control. The variance analysis is sound and verified on a controlled toy model; the frozen-trunk control and the explicit disclosure of hyperparameter selection are good scientific practice. The paper does not claim state-of-the-art accuracy and honestly acknowledges that the theoretical guarantee covers an idealized algorithm rather than the exact implementation. The main weakness is that the abstract and contribution list advertise an 'Adam-style convergence guarantee' that the proofs do not deliver for the algorithm actually run, and the headline GPT result rests on a validation-selected ρ and a tied-embedding heuristic not covered by the theory. These issues are fixable by scoping the claims precisely.

major comments (3)
  1. [Abstract, §3, App. C–D] The advertised 'Adam-style convergence guarantee' is not established for the algorithm run in the experiments. Theorem 1 (Assumption 2) requires a predictable diagonal preconditioner fixed before the step-t tangents are drawn; the actual Adam update in all large experiments computes its second-moment denominator from the current stochastic estimate, so A_t is not predictable. Appendix C states the theorem 'does not capture the exact Adam implementation,' and Appendix D states that global-norm clipping introduces bias not covered by Section 3. Since the abstract and contribution list present the guarantee as a property of Split-FG, the paper should either prove a comparable result for the actual update (including clipping) or explicitly scope the guarantee to an idealized lagged/preconditioned update and remove the word 'Adam-style' from the abstract.
  2. [§3, App. C, App. G.7] The GPT experiments use a tied-embedding readout surrogate: the vocabulary table's output projection is trained exactly, but its token-lookup role is held fixed outside the trunk JVP, and Appendix C says 'no fixed L-smooth, lower-bounded objective telescopes' for that surrogate and 'the tied table's update is a heuristic we disclose.' Consequently, the unbiasedness and convergence properties of the strict Split-FG estimator do not apply to the full parameter update that produced the headline perplexity 386.7. The paper should state this limitation in the main text near Table 3, not only in the appendix, and should not describe the GPT run as covered by Theorem 1.
  3. [§4.4, App. G.2, G.4] The central empirical claim rests on a single seed and on ρ=0.03, selected on the validation split from a small grid. Appendix G.4 shows ρ=0.1 gives better perplexity at both budgets (365.0 vs 386.7 full budget), so the headline configuration is not the grid optimum. The paper's mechanistic explanation ('noise-blind Adam') is explicitly 'consistent with this mechanism rather than a proof.' This is acceptable as a proof-of-concept, but the abstract's wording ('reverses this result', 'removes the failure') should be tempered, and the robustness of the ρ-fix across seeds and larger trunks should be acknowledged as unverified.
minor comments (4)
  1. [App. G.3] The heading 'GPT COREABLATION' appears with no content; the core ablation table (Table 14) appears under the G.4 heading. Please add the missing text or renumber.
  2. [Related Work] The reference for 'discriminative fine-tuning' is a bare '(?)'. A proper citation should be supplied.
  3. [§4.5] The phrase 'reported 100× grid' is ambiguous: Table 4 has five grid points spanning a factor of 100 in ρ. Clarify that it is a 5-point log-spaced sweep, not a 100-point grid.
  4. [Abstract and Table 2] The CIFAR-10 and CIFAR-100 numbers are called 'development evaluation' in the text (Appendices F–F.1) because the official test split was inspected during development, but the abstract reports them without this caveat. Please carry the 'development' qualifier into the abstract.

Circularity Check

0 steps flagged

No significant circularity; the variance identities and convergence theorem are self-contained, with disclosed idealization gaps that are limitations rather than circular steps.

full rationale

The paper's central derivation is self-contained. The variance reduction claim (Eq. 6/10) is an algebraic consequence of the Gaussian-tangent identity E[vv^T]=I and Isserlis' theorem, not a quantity fitted from the data; the toy experiment measures empirical estimator variances and compares them to this closed-form prediction. The convergence theorem (Theorem 1) is a standard descent lemma under explicitly stated assumptions (L-smoothness, predictable bounded diagonal preconditioner, unbiasedness, bounded adaptive-noise ratio), and its R_max bound is derived from the same second-moment identity; it does not assume its conclusion. The paper explicitly discloses the gap between the theorem and the running algorithm: Appendix C says the predictable-preconditioner analysis 'does not capture the exact Adam implementation,' Appendix D says global-norm clipping 'introduces a bias not covered by the analysis of Section 3,' and Appendix C states the tied-embedding readout surrogate has no fixed L-smooth telescoping objective. These are precision/limitation issues about the advertised 'Adam-style guarantee,' not circular reasoning, because the proof is independent of the empirical GPT results. The GPT perplexity numbers are measured outcomes, not quantities derived by construction from the estimator's definition; the trunk-step scale rho is a validation-selected hyperparameter whose effect is ablated, not a fitted parameter renamed as a prediction. There are no load-bearing self-citations: the method, proofs, and controls are presented in the paper, and cited prior work is used for standard results and baselines. Accordingly, no circular step is exhibited, and the correct circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The central estimator introduces no new physical entities. The main load-bearing choices are the trunk-step scale rho and the idealized preconditioner assumptions for the convergence theorem. The paper is transparent that the theorem does not cover the actual Adam-with-clipping optimizer, and that the GPT success depends on a validation-selected rho and a tied-embedding surrogate. The variance formula itself is a derived identity, not a fitted parameter.

free parameters (4)
  • trunk step scale rho = 0.03 (optimum in later grid: 0.1)
    The central GPT result depends on scaling the trunk Adam step by rho=0.03; the paper states this was fixed by a 10%-budget exploratory run on the validation split, and the later grid finds rho=0.1 better.
  • tangent count K = 4 for main runs
    K is chosen rather than derived; it trades variance against wall-clock time, and the paper measures that perplexity improves monotonically with K.
  • CIFAR head inner width H = 64 (CIFAR-10), 128 (CIFAR-100)
    The heavy-head architecture uses a two-layer linear head with chosen width H; the paper credits this factored head with +2.5 points on CIFAR-10.
  • global learning rate and schedule = 1.5e-3 for GPT, 1e-3 for image
    Standard optimizer hyperparameters chosen by screening; the pure-FG baseline is tuned over a six-point grid while Split-FG uses its own schedule.
axioms (6)
  • standard math Isserlis theorem for Gaussian fourth moments
    Used in Section 2.1 and Appendix B to compute E[v_j v_l v_i^2] for the variance of the forward-gradient estimator.
  • standard math F is L-smooth and bounded below
    Assumption 1 in Section 3; standard nonconvex optimization assumption for the convergence theorem.
  • ad hoc to paper The preconditioner A_t is predictable and uniformly bounded (Assumption 2)
    This idealized assumption makes the theorem tractable but does not hold for the actual Adam implementation used in the experiments, which the paper acknowledges in Section C.
  • ad hoc to paper The adaptive noise ratio R_t is almost surely bounded (Assumption 3)
    Assumption 3 bounds the noise relative to the preconditioned gradient energy; the paper derives a worst-case bound, but the actual clipped Adam estimator is not unbiased, so the assumption is not verified for the reported runs.
  • domain assumption Head gradients are available in closed form for linear CE/MSE heads
    The method's exact-head property relies on the head being linear or a factored linear pair; this restricts the architecture class and is stated in Sections 2.3 and 4.3.
  • domain assumption Global gradient clipping does not invalidate the empirical comparisons
    Appendix D states that clipping introduces bias not covered by the analysis; the paper argues comparisons are fair because the protocol is uniform across methods, but this is an assumption about fairness rather than a proven property.

pith-pipeline@v1.3.0-alltime-deepseek · 25274 in / 11430 out tokens · 118130 ms · 2026-08-01T20:25:18.168430+00:00 · methodology

0 comments
read the original abstract

Backpropagation makes training deep networks memory intensive because it must store intermediate activations. Forward-mode methods avoid this cost, but their gradient estimates become increasingly noisy as the number of trained parameters grows. We introduce Split Forward Gradient (Split-FG), which splits a network at an intermediate representation: it computes the output head gradient exactly and estimates only the trunk gradient with a Jacobian--vector product. This reduces estimator variance and requires no backward pass through the trunk, while retaining an Adam-style convergence guarantee. Our experiments reveal an important practical failure mode. On WikiText-103, naive forward-gradient training of the trunk performs worse than leaving a randomly initialized trunk frozen, likely because Adam updates every noisy, under-determined trunk coordinate too aggressively. Simply using a much smaller learning rate for the trunk reverses this result: a $16$M-parameter GPT-2-style model reaches validation perplexity $387$, compared with $668$ for the frozen-trunk control and $2{,}885$ for a matched pure forward-gradient baseline (backpropagation reaches $150$). Split-FG also produces the strongest backprop-free results on our tabular benchmarks and reaches $60.5\%$ on CIFAR-10 and $35.2\%$ on CIFAR-100 with a heavy-head design. It reduces peak memory by up to $35\%$ relative to matched backpropagation, although the performance gap widens as the forward-mode trunk grows.

Figures

Figures reproduced from arXiv: 2607.16612 by Tian Qin, Wei-Min Huang.

Figure 1
Figure 1. Figure 1: Frozen-trunk control, naive Split-FG (K=4), and ρ-scaled Split-FG on WikiText-103, same one-epoch cosine schedule (single seed). Naive trunk training tracks above the frozen control throughout; the ρ-scaled trunk drops below it from the first checkpoint and passes the frozen run’s final perplexity in about a quarter of the budget [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Controlled toy variance experiment (Section 4.1). Left: total empirical variance over trunk [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Split-FG tangent-count sweep on WikiText-103 with the reduced trunk step ( [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 8 linked inside Pith

  1. [1]

    Pearlmutter, Don Syme, Frank Wood, and Philip Torr

    Atılım Güneş Baydin, Barak A. Pearlmutter, Don Syme, Frank Wood, and Philip Torr. Gradients without backpropagation, 2022. URL https://arxiv.org/abs/2202.08587

    Pith/arXiv arXiv 2022

  2. [2]

    Deepzero: Scaling up zeroth-order optimization for deep model training

    Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Konstantinos Parasyris, Jiancheng Liu, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order optimization for deep model training. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=qBWhjsNPEY

    2024

  3. [3]

    Training deep nets with sublinear memory cost, 2016

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost, 2016. URL https://arxiv.org/abs/1604.06174

    Pith/arXiv arXiv 2016

  4. [4]

    The reversible residual network: Backpropagation without storing activations

    Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedi...

    2017

  5. [5]

    Tabm: Advancing tabular deep learning with parameter-efficient ensembling

    Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. Tabm: Advancing tabular deep learning with parameter-efficient ensembling. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=Sd4wYYOhmY

    2025

  6. [6]

    Evaluating Derivatives

    Andreas Griewank and Andrea Walther. Evaluating Derivatives. Society for Industrial and Applied Mathematics, second edition, 2008. doi:10.1137/1.9780898717761. URL https://epubs.siam.org/doi/abs/10.1137/1.9780898717761

    doi:10.1137/1.9780898717761 2008

  7. [8]

    The forward-forward algorithm: Some preliminary investigations, 2022

    Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations, 2022. URL https://arxiv.org/abs/2212.13345

    Pith/arXiv arXiv 2022

  8. [11]

    Isserlis

    L. Isserlis. On a formula for the product-moment coefficient of any order of a normal frequency distribution in any number of variables. Biometrika, 12 0 (1/2): 0 134--139, 1918. ISSN 00063444, 14643510. URL http://www.jstor.org/stable/2331932

    arXiv 1918

  9. [12]

    Generalization through memorization: Nearest neighbor language models

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=HklBjCEKvH

    2020

  10. [13]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980

    Pith/arXiv arXiv 2017

  11. [14]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https://api.semanticscholar.org/CorpusID:18268744

    2009

  12. [16]

    Lee, Danqi Chen, and Sanjeev Arora

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=Vota6rFhBQ

    2023

  13. [17]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Byj72udxe

    2017

  14. [18]

    Beren Millidge, Alexander Tschantz, and Christopher L. Buckley. Predictive coding approximates backprop along arbitrary computation graphs. Neural Computation, 34 0 (6): 0 1329--1368, 2022. doi:10.1162/neco_a_01497

    doi:10.1162/neco_a_01497 2022

  15. [19]

    Direct feedback alignment provides learning in deep neural networks

    Arild N kland. Direct feedback alignment provides learning in deep neural networks. In Neural Information Processing Systems, 2016. URL https://api.semanticscholar.org/CorpusID:2843914

    2016

  16. [20]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI, 2019. URL https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf. Accessed: 2024-11-15

    2019

  17. [21]

    a fl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, G \

    Hubert Ramsauer, Bernhard Sch \"a fl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, G \"u nter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need. In International Conference on Learning Representations, 2021. URL https://openreview.net/fo...

    2021

  18. [22]

    Scaling forward gradient with local losses

    Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=JxpBP1JM15-

    2023

  19. [23]

    Rumelhart, Geoffrey E

    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, 323: 0 533--536, 1986. URL https://api.semanticscholar.org/CorpusID:205001834

    1986

  20. [24]

    Evolution strategies as a scalable alternative to reinforcement learning, 2017

    Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning, 2017. URL https://arxiv.org/abs/1703.03864

    Pith/arXiv arXiv 2017

  21. [25]

    Reverse differentiation via predictive coding

    Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, Rafał Bogacz, and Zhenghua Xu. Reverse differentiation via predictive coding. Proceedings of the ... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, 36: 0 8150 -- 8158, 2021. URL https://api.semanticscholar.org/CorpusID:237532780

    2021

  22. [27]

    Harris , Scott Skirlo , Mihika Prabhu , Tom Baehr-Jones , Michael Hochberg , Xin Sun , Shijie Zhao , Hugo Larochelle , Dirk Englund , and Marin Solja c i \'c

    Yichen Shen , Nicholas C. Harris , Scott Skirlo , Mihika Prabhu , Tom Baehr-Jones , Michael Hochberg , Xin Sun , Shijie Zhao , Hugo Larochelle , Dirk Englund , and Marin Solja c i \'c . Deep learning with coherent nanophotonic circuits . Nature Photonics, 11 0 (7): 0 441--446, July 2017. doi:10.1038/nphoton.2017.93

    doi:10.1038/nphoton.2017.93 2017

  23. [28]

    Learning by directional gradient descent

    David Silver, Anirudh Goyal, Ivo Danihelka, Matteo Hessel, and Hado van Hasselt. Learning by directional gradient descent. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=5i7lJLuhTm

    2022

  24. [29]

    James C. R. Whittington and Rafał Bogacz. An approximation of the error backpropagation algorithm in a predictive coding network with local hebbian synaptic plasticity. Neural computation, 29: 0 1229 -- 1262, 2017. URL https://api.semanticscholar.org/CorpusID:13651627

    2017

  25. [30]

    Natural evolution strategies

    Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. Journal of Machine Learning Research, 15 0 (27): 0 949--980, 2014. URL http://jmlr.org/papers/v15/wierstra14a.html

    2014

  26. [31]

    Tuning large neural networks via zero-shot hyperparameter transfer

    Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\...

    2021

  27. [32]

    Large batch training of convolutional networks, 2017

    Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks, 2017. URL https://arxiv.org/abs/1708.03888

    Pith/arXiv arXiv 2017

  28. [33]

    Large batch optimization for deep learning: Training bert in 76 minutes, 2020

    Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes, 2020. URL https://arxiv.org/abs/1904.00962

    Pith/arXiv arXiv 2020

  29. [34]

    Nature , year=

    Learning representations by back-propagating errors , author=. Nature , year=

  30. [35]

    2022 , eprint=

    Gradients without Backpropagation , author=. 2022 , eprint=

    2022

  31. [36]

    2017 , eprint=

    Evolution Strategies as a Scalable Alternative to Reinforcement Learning , author=. 2017 , eprint=

    2017

  32. [37]

    Journal of Machine Learning Research , year =

    Daan Wierstra and Tom Schaul and Tobias Glasmachers and Yi Sun and Jan Peters and Jürgen Schmidhuber , title =. Journal of Machine Learning Research , year =

  33. [38]

    Thirty-seventh Conference on Neural Information Processing Systems , year=

    Fine-Tuning Language Models with Just Forward Passes , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=

  34. [39]

    Frontiers in Computational Neuroscience , VOLUME=

    Scellier, Benjamin and Bengio, Yoshua , TITLE=. Frontiers in Computational Neuroscience , VOLUME=. 2017 , URL=. doi:10.3389/fncom.2017.00024 , ISSN=

    arXiv 2017

  35. [40]

    Frontiers in Neuroscience , VOLUME=

    Laborieux, Axel and Ernoult, Maxence and Scellier, Benjamin and Bengio, Yoshua and Grollier, Julie and Querlioz, Damien , TITLE=. Frontiers in Neuroscience , VOLUME=. 2021 , URL=. doi:10.3389/fnins.2021.633674 , ISSN=

    arXiv 2021

  36. [41]

    Neural computation , year=

    An Approximation of the Error Backpropagation Algorithm in a Predictive Coding Network with Local Hebbian Synaptic Plasticity , author=. Neural computation , year=

  37. [42]

    , journal=

    Millidge, Beren and Tschantz, Alexander and Buckley, Christopher L. , journal=. Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs , year=

  38. [43]

    Proceedings of the

    Reverse Differentiation via Predictive Coding , author=. Proceedings of the ... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence , year=

  39. [44]

    2022 , eprint=

    The Forward-Forward Algorithm: Some Preliminary Investigations , author=. 2022 , eprint=

    2022

  40. [45]

    Neural Information Processing Systems , year=

    Direct Feedback Alignment Provides Learning in Deep Neural Networks , author=. Neural Information Processing Systems , year=

  41. [46]

    The Eleventh International Conference on Learning Representations , year=

    Scaling Forward Gradient With Local Losses , author=. The Eleventh International Conference on Learning Representations , year=

  42. [47]

    2008 , doi =

    Griewank, Andreas and Walther, Andrea , title =. 2008 , doi =

    2008

  43. [48]

    International Conference on Learning Representations , year=

    Hopfield Networks is All You Need , author=. International Conference on Learning Representations , year=

  44. [49]

    International Conference on Learning Representations , year=

    Generalization through Memorization: Nearest Neighbor Language Models , author=. International Conference on Learning Representations , year=

  45. [50]

    Isserlis , journal =

    L. Isserlis , journal =. On a Formula for the Product-Moment Coefficient of any Order of a Normal Frequency Distribution in any Number of Variables , urldate =

  46. [51]

    Deep Learning , author=

  47. [52]

    The Thirteenth International Conference on Learning Representations , year=

    TabM: Advancing tabular deep learning with parameter-efficient ensembling , author=. The Thirteenth International Conference on Learning Representations , year=

  48. [53]

    2009 , url=

    Learning Multiple Layers of Features from Tiny Images , author=. 2009 , url=

    2009

  49. [54]

    International Conference on Learning Representations , year=

    Pointer Sentinel Mixture Models , author=. International Conference on Learning Representations , year=

  50. [55]

    2017 , eprint=

    Adam: A Method for Stochastic Optimization , author=. 2017 , eprint=

    2017

  51. [56]

    Nature Photonics , keywords =

    Deep learning with coherent nanophotonic circuits. Nature Photonics , keywords =. doi:10.1038/nphoton.2017.93 , archivePrefix =. 1610.02365 , primaryClass =

    Pith/arXiv arXiv 2017

  52. [57]

    Hughes and Momchil Minkov and Yu Shi and Shanhui Fan , journal =

    Tyler W. Hughes and Momchil Minkov and Yu Shi and Shanhui Fan , journal =. Training of photonic neural networks through in situ backpropagation and gradient measurement , volume =. 2018 , url =. doi:10.1364/OPTICA.5.000864 , abstract =

    doi:10.1364/optica.5.000864 2018

  53. [58]

    2016 , eprint=

    Training Deep Nets with Sublinear Memory Cost , author=. 2016 , eprint=

    2016

  54. [59]

    The Reversible Residual Network: Backpropagation Without Storing Activations , url =

    Gomez, Aidan N and Ren, Mengye and Urtasun, Raquel and Grosse, Roger B , booktitle =. The Reversible Residual Network: Backpropagation Without Storing Activations , url =

  55. [60]

    The Twelfth International Conference on Learning Representations , year=

    DeepZero: Scaling Up Zeroth-Order Optimization for Deep Model Training , author=. The Twelfth International Conference on Learning Representations , year=

  56. [61]

    2020 , eprint=

    Large Batch Optimization for Deep Learning: Training BERT in 76 minutes , author=. 2020 , eprint=

    2020

  57. [62]

    Universal Language Model Fine-tuning for Text Classification

    Howard, Jeremy and Ruder, Sebastian. Universal Language Model Fine-tuning for Text Classification. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018. doi:10.18653/v1/P18-1031

    doi:10.18653/v1/p18-1031 2018

  58. [63]

    Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer , url =

    Yang, Ge and Hu, Edward and Babuschkin, Igor and Sidor, Szymon and Liu, Xiaodong and Farhi, David and Ryder, Nick and Pachocki, Jakub and Chen, Weizhu and Gao, Jianfeng , booktitle =. Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer , url =

  59. [64]

    2016 , volume =

    He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian , booktitle =. 2016 , volume =. doi:10.1109/CVPR.2016.90 , url =

    doi:10.1109/cvpr.2016.90 2016

  60. [65]

    Language Models are Unsupervised Multitask Learners , url =

    Radford, Alec and Wu, Jeffrey and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya , biburl =. Language Models are Unsupervised Multitask Learners , url =. OpenAI , keywords =

  61. [66]

    International Conference on Learning Representations , year=

    Learning by Directional Gradient Descent , author=. International Conference on Learning Representations , year=

  62. [67]

    2017 , eprint=

    Large Batch Training of Convolutional Networks , author=. 2017 , eprint=

    2017