Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Scalable Thermodynamic Second-order Optimization

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Thermodynamic K-FAC replaces cubic matrix inversions with a per-layer cost of $O(bn^2 + n^2\kappa^2)$ instead of $O(bn^2 + n^3)$, making second-order optimization approach first-order per-step cost while the advantage grows with layer…

desk verdict A clean mapping of K-FAC onto thermodynamic linear algebra with a genuinely useful quantization study, but the headline asymptotic advantage rests on an unexamined condition-number assumption. read the letter →

arxiv 2502.08603 v1 pith:GSZVSS3C submitted 2025-02-12 cs.ET cs.LG

Kaelan Donatella , Samuel Duffield , Denis Melanson , Maxwell Aifer , Phoebe Klett , Rajath Salegame , Zach Belateche , Gavin Crooks
show 2 more authors
Antonio J. Martinez Patrick J. Coles
This is my paper · ORCID
classification cs.ETcs.LG
keywords thermodynamiccomputingK-FACsecond-orderoptimizationnaturalgradientdescentlinearalgebraacceleratorquantizationrobustnessKronecker-factoredapproximatecurvatureanaloghardware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that K-FAC, a popular second-order optimizer, can be made practical for wide neural networks by running its per-layer matrix inversions on thermodynamic hardware. The central claim is that replacing digital inversion with a thermodynamic linear-systems solver changes the per-layer cost from $O(bn^2 + n^3)$ to $O(bn^2 + n^2\kappa^2)$, so the advantage over digital K-FAC grows linearly with layer width $n$. Because K-FAC factorizes the curvature into Kronecker factors of dimension roughly $n$, the hardware need not scale to billions of parameters, making training-time acceleration plausible. Numerical experiments suggest the method tolerates quantization noise, preserving K-FAC's convergence edge over Adam even at moderate output precision. The paper also estimates wall-clock speedups on standard vision and graph workloads, driven by hardware assumptions of roughly one microsecond RC time constants and 50 Gb/s transfer bandwidth.

What carries the argument

The central object is the thermodynamic linear algebra solver: a system of coupled harmonic oscillators (e.g., RC circuits) evolving under an Ornstein–Uhlenbeck process $dx = -(M x - b)dt + \mathcal{N}(0, 2\beta^{-1}dt)$, whose stationary distribution $x \sim \mathcal{N}(M^{-1}b, \beta^{-1}M^{-1})$ has mean solving $Mx = b$ and covariance proportional to $M^{-1}$. Thermodynamic K-FAC applies this solver to the Kronecker factors $G_\ell$ and $A_{\ell-1}$ using the linear-systems method, solving one system per column of $D\Theta_\ell$ and one per row of the intermediate, for $2n$ solves per layer, each taking $O(n\kappa^2)$ time on thermodynamic hardware. The K-FAC block-diagonal approximation is what makes the device size practical, because each factor has dimension roughly equal to the layer width rather than the full parameter count.

What would settle it

Measure how the condition numbers of the Kronecker factors $A_\ell$ and $G_\ell$ scale with layer width $n$ during real training runs (e.g., on a vision transformer or graph neural network); if $\kappa$ grows roughly linearly with $n$, the $O(n^2\kappa^2)$ term becomes cubic and the claimed advantage over digital K-FAC vanishes. Alternatively, a hardware prototype measuring end-to-end per-iteration wall-clock time for a single wide linear layer could test the RC and transfer-bandwidth assumptions against the predicted speedups.

Watch

Extended reading notes

Core claim

The core claim is that the K-FAC per-layer update, $U_\ell = G_\ell^{-1} D\Theta_\ell A_{\ell-1}^{-1}$, can be executed on a thermodynamic computer—a network of coupled stochastic harmonic oscillators whose equilibrium samples solve linear systems and yield inverses—by solving $2n$ linear systems per layer instead of performing a digital matrix inversion. This changes the per-layer iteration cost from $O(bn^2 + n^3)$ to $O(bn^2 + n^2\kappa^2)$ and the per-layer memory from $O(bn + n^2)$ to $O(bn)$ when no exponential moving average is used, so the advantage over digital K-FAC grows with layer width. The paper further claims that the block-diagonal structure of K-FAC keeps the matrices involved at dimension about $n$, so the thermodynamic device only needs to scale with layer width rather than total parameter count. Simulations on standard benchmark workloads show K-FAC retains second-order convergence benefits under quantization noise, and estimated speedups from the inversion fraction translate into end-to-end training speedups of roughly tens of percent.

Load-bearing premise

The central load-bearing premise is that the condition number $\kappa$ of the Kronecker factors stays bounded or grows slowly as layer width $n$ increases, since the paper assumes $\kappa$ is at most a constant in its complexity table but does not analyze how $\kappa$ scales with $n$; a secondary hardware assumption is the roughly one-microsecond RC time constant and 50 Gb/s transfer bandwidth used for the estimated speedups.

Editorial extensions

If this is right

  • Per-iteration cost of K-FAC approaches that of first-order methods, $O(bn^2 + n^2\kappa^2)$ versus SGD's $O(bn^2)$, while retaining second-order convergence benefits.
  • The advantage over digital K-FAC grows with layer width $n$, making wide layers the clearest target for thermodynamic acceleration.
  • Thermodynamic K-FAC works with output precision around 8 bits, remaining competitive with full-precision Adam, which guides hardware precision requirements.
  • Using the linear-systems method avoids constructing matrix inverses explicitly, reducing both memory and the number of operations.
  • On the benchmark workloads considered, the estimated wall-clock speedup to reach a target validation metric is substantial, especially for the graph neural network workload.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If in practice the condition number $\kappa$ of the Kronecker factors grows only slowly with layer width, the $O(n^2\kappa^2)$ term could make K-FAC competitive with first-order optimizers per wall-clock step at large widths; but if $\kappa$ grows linearly with $n$, the claimed advantage disappears and the method is no better than cubic per layer.
  • The same thermodynamic linear-algebra acceleration should extend to other block-diagonal curvature approximations, such as generalized Gauss–Newton or block-diagonal Fisher methods, since only the per-layer solves are offloaded.
  • A direct empirical test on real networks—measuring how $\kappa$ scales with width during training—would convert the asymptotic claim into a concrete engineering target for thermodynamic hardware.
  • The quantization-robustness result suggests that even a mixed-signal implementation with moderate output resolution could be viable, provided the diagonal-dominant quantization scheme keeps the factor matrices positive definite.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This manuscript proposes Thermodynamic K-FAC, an algorithm that offloads the per-layer matrix inversions and linear solves of K-FAC to a thermodynamic computer modeled as an Ornstein-Uhlenbeck process. The central theoretical claim is that Thermodynamic K-FAC has per-layer runtime O(bn^2 + n^2κ^2) and memory O(bn), versus O(bn^2 + n^3) and O(bn + n^2) for digital K-FAC, so that the advantage over digital K-FAC grows with layer width n, provided the Kronecker factors have condition number at most κ. The paper also reports profiling of K-FAC update costs on MLPs and GPT architectures, quantization-robustness experiments on CIFAR-10 with 6–16 bit integer precision, and estimated wall-clock speedups for ImageNet-ViT and OGBG-GNN based on profiled inversion fractions and assumed hardware constants.

Significance. If the complexity claim holds with κ bounded independently of n, the result is significant: it would bring K-FAC's per-iteration cost close to that of first-order methods while retaining second-order convergence benefits, and the block-diagonal dimension of the involved matrices (on the order of a thousand) makes the hardware proposal plausible. The paper has concrete strengths: the complexity derivation is transparent given the imported O(nκ^2) solver bound from Ref. [9]; the quantization experiments are a genuine robustness test across several precision levels; and the wall-clock panels in Fig. 3 are honestly labeled as estimates with assumptions stated in Appendix E. No machine-checked proofs or full hardware measurements are claimed, and the absence of those is not itself a flaw; the load-bearing issue is the unanalyzed scaling of the condition number κ.

major comments (3)
  1. [§4.4, Table 1; also §1 and Conclusion] The central asymptotic claim that the advantage "grows linearly with n" is not established by the analysis as written. The O(n^2κ^2) term in Table 1 only improves on the digital O(n^3) term when κ = o(√n), and the stated linear-in-n advantage requires κ = O(1). The paper simply states "we assume that the Kronecker factors all have condition numbers at most κ" (Table 1) and provides neither an analysis nor measurements of how κ scales with n. Since A_ℓ and G_ℓ are empirical covariance matrices, their conditioning is not a free parameter: in the GPT configuration n=768, b=64 (Appendix C), the unregularized factors are rank-deficient, and the effective κ after damping can be very large and can grow with n. Concretely, κ ~ n^{1/2} yields O(n^3) for the thermodynamic term and no asymptotic advantage, while κ ~ n yields O(n^4). Please report empirical κ(n) for the Kronecker factors in the experiments, or give an analytic argument bounding κ, and restate the complexity with the measured κ(n). A similar caveat applies to b: the O(bn^2) gradient term is common to both methods, so the comparison in Table 1 assumes b stays fixed as n grows.
  2. [Table 1, memory column; §4.4] The O(bn) memory entry for Thermodynamic K-FAC cannot be a total per-layer memory complexity: the weight matrix W_ℓ and the n×n gradient matrix DΘ_ℓ must be stored or streamed to form the right-hand sides of the linear systems in Eq. (15), which alone is Ω(n^2). If the table is intended to count only auxiliary memory on the thermodynamic device, excluding digital weights and gradients, that metric must be defined explicitly and the SGD/Adam row must be interpreted accordingly; as written, the linear memory advantage asserted in the Conclusion is not supported. Please either correct the asymptotic memory counts or clarify the memory accounting convention.
  3. [Fig. 3 and Appendix E] The quantitative wall-clock predictions depend on three assumed hardware constants—RC = 1 μs, 50 Gb/s transfer bandwidth, and 16-bit precision—but no sensitivity analysis is given. Because the abstract advertises "substantial speedups for large-scale vision and graph problems," please add a sensitivity range over these constants (e.g., RC ∈ [0.1, 10] μs and bandwidth ∈ [10, 100] Gb/s), or otherwise state how the predicted curves and the reported OGBG-GNN speedup change under plausible variations. This would separate conclusions that are robust from those that depend on optimistic hardware parameters.
minor comments (5)
  1. [§4.3, footnote 2] The statement that "there is some evidence that the quadratic dependence on κ can be improved in the average-case, which will be published in forthcoming work" is unsupported in this manuscript; please either remove it or provide a citation, since the reported complexity relies on the worst-case bound from Ref. [9].
  2. [§4.1] The stated complexity of the conjugate-gradient approach, "O(n^2κ)", should be clarified; for SPD systems the standard bound is O(n^2 κ^{1/2} log(1/ε)) per solve (or a preconditioned variant), and the text should specify whether κ is the condition number of the unpreconditioned system and whether the stated cost is per solve or total.
  3. [Appendix E, Eq. (24)] Equation (24) uses κ_0 for the noise variance, while κ is used throughout the paper for the condition number; this notation collision should be fixed to avoid confusion.
  4. [§4.5] The phrase "we strategies to reduce these errors" appears to have a missing verb; it should read "we study strategies to reduce these errors" or similar.
  5. [Fig. 3 and §5.1] For the AlgoPerf results, no number of seeds or variance is reported; please add this information or state explicitly that the curves are single runs, consistent with the 5-run reporting used in Fig. 4.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central complexity claim follows from a cited thermodynamic solver result and the K-FAC update equations, and the estimated speedup curves are explicitly labeled estimates rather than fitted predictions.

full rationale

The paper's derivation chain is self-contained in the relevant sense. The K-FAC update U_ell = G_ell^{-1} DTheta_ell A_{ell-1}^{-1} is imported from the independently published K-FAC literature, and the thermodynamic solver's O(n kappa^2) per-system complexity is cited from Ref. [9], a published, parameter-free result with stated assumptions; the same-group authorship of that reference does not make the citation circular because the result is externally checkable and is not defined in terms of the present paper's conclusions. In Section 4.2 the stationary distribution of the Ornstein-Uhlenbeck process is re-derived explicitly, giving the mean as the linear-system solution and the covariance as the inverse, so the mapping from physical dynamics to linear algebra is shown rather than assumed. The Table 1 complexity entries are obtained by substituting this solver cost into the standard K-FAC per-layer computation; no parameter is fitted to force the claimed n-scaling. The Figure 3 'ThermoKFAC (est.)' curves are honest estimates, constructed from measured inversion time fractions and stated hardware constants (Appendix E), not from fitting the predicted curves to the baseline curves. The main weakness is the unexamined assumption that the Kronecker-factor condition number kappa is bounded as n grows; if kappa grows with width the advertised asymptotic advantage weakens or disappears. That is a correctness or assumption-support concern, not circularity, because the assumption is stated as an input to the complexity analysis rather than being derived from the conclusion. Overall no load-bearing step reduces to its own inputs by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central complexity claim leans on the O(n kappa^2) thermodynamic solver from prior work and on bounded condition numbers; the wall-clock predictions lean on chosen hardware constants. No new physical entities are introduced.

free parameters (4)
  • Hardware time constant RC = 1 microsecond (R=10^3 ohm, C=1 nF)
    Chosen in Appendix E as the characteristic relaxation time; directly determines the estimated speedups in Fig. 3.
  • Digital transfer speed = 50 Gb/s
    Assumed in Appendix E for moving Kronecker factors to hardware; a slower transfer rate reduces the estimated speedups.
  • Precision bits for solver = 16 bits
    Assumed in Appendix E when estimating speedups; fewer bits are said to give larger speedups, so this is a chosen working point.
  • Damping lambda = 0.001 (CIFAR), 0.001 (ImageNet), 0.005 (OGBG)
    Hyperparameters from Table 3; they regularize Kronecker factors and hence bound the effective condition number, indirectly controlling the kappa dependence of the runtime.
assumptions (5)
  • domain assumption Equilibrium distribution of the OU process is x ~ N(M^-1 b, beta^-1 M^-1)
    Used in Section 4.2 to justify reading linear-system solutions and matrix inverses from samples of the thermodynamic device.
  • domain assumption K-FAC approximation E[a tensor g] approx E[a] tensor E[g] and block-diagonal Fisher approximation
    Eq. (11) and surrounding text; this approximation is empirically motivated, not theoretically controlled, and is inherited from Ref. [13].
  • domain assumption Thermodynamic solver solves an n-dimensional linear system in O(n kappa^2) time
    Stated in Section 4.3 and attributed to Ref. [9] by the same authors; the paper relies on this complexity result without proving it.
  • ad hoc to paper The Kronecker factors have condition number at most kappa independent of problem size
    Assumed in Tables 1 and 2 and Section 4.4; no scaling analysis with n is provided, and this assumption is necessary for the claimed linear advantage.
  • domain assumption The conservative diagonal-shift quantization keeps the quantized matrix positive semidefinite
    Section 4.5.1, based on Ref. [41]; used to make low-precision simulations stable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scalable Thermodynamic Second-order Optimization." pith.science (2026). https://pith.science/paper/GSZVSS3C

@misc{pith2026250208603,
  author       = {Pith},
  title        = {Pith review of: Scalable Thermodynamic Second-order Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GSZVSS3C}},
  note         = {Machine review of arXiv:2502.08603}
}
abstract

Many hardware proposals have aimed to accelerate inference in AI workloads. Less attention has been paid to hardware acceleration of training, despite the enormous societal impact of rapid training of AI models. Physics-based computers, such as thermodynamic computers, offer an efficient means to solve key primitives in AI training algorithms. Optimizers that normally would be computationally out-of-reach (e.g., due to expensive matrix inversions) on digital hardware could be unlocked with physics-based hardware. In this work, we propose a scalable algorithm for employing thermodynamic computers to accelerate a popular second-order optimizer called Kronecker-factored approximate curvature (K-FAC). Our asymptotic complexity analysis predicts increasing advantage with our algorithm as $n$, the number of neurons per layer, increases. Numerical experiments show that even under significant quantization noise, the benefits of second-order optimization can be preserved. Finally, we predict substantial speedups for large-scale vision and graph problems based on realistic hardware characteristics.

Figures

Figures reproduced from arXiv: 2502.08603 by the authors.

Figure 1
Figure 1. Overview of the thermodynamic algorithm for K-FAC. On the left is shown a two-layer neural network with weight matrices W1 and W2 and activations a1, a2, a3 that are stored on a digital device. From these quantities Kronecker factors Aℓ and Bℓ are computed and sent to the thermodynamic solver, which inverts them or solves a linear system where they enter as the positive semi-definite matrix. Then, the result is sent… view at source ↗
Figure 2
Figure 2. Profiling of the K-FAC update for different architectures. Panel (a): K-FAC update time contributions for an MLP with a fixed depth of 50, with varying number of neurons n on each layer. Panel (b): K-FAC update time contributions for a GPT architecture (based on Ref. [38]). with varying embedding dimension, which is the number of neurons n in the linear layers. Panel (c): GPT architecture with varying vocabulary siz… view at source ↗
Figure 3
Figure 3. Results on ImageNet and OGBG. Panels (a-b): validation loss and validation accuracy for the NAdamW (the baseline given by AlgoPerf), K-FAC and Thermodynamic K-FAC (estimated) optimizers as a function of the wall-clock time for training a ViT on ImageNet. Panels (c-d): validation loss and validation mean-average precision (mAP) for the Nesterov (baseline), K-FAC and Thermodynamic K-FAC (estimated) optimizers as a fun… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Effect of quantization on K-FAC training accuracy. Validation accuracy from training a ResNet on image classification with either the Adam optimizer or with the K-FAC optimizer for various levels of precision (integer 6, 8, 12 and 16 bits, and floating-point 32 bits at…
Figure 5
Figure 5. Figure 5: Circuit diagram of a possible implementation of the thermodynamic solver. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Solving the compute crisis with physics-based ASICs

    cs.ET 2025-07 unverdicted novelty 4.0 of 10

    A coalition of academic and industry researchers argues that chips exploiting natural physical dynamics, rather than enforcing digital abstractions, could dramatically cut AI computing costs.

Reference graph

Works this paper leans on

48 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [9]

    Maxwell Aifer, Kaelan Donatella, Max Hunter Gordon, Samuel Duffield, Thomas Ahle, Daniel Simpson, Gavin Crooks, and Patrick J. Coles. Thermodynamic linear algebra. npj Unconven- tional Computing, 1(1):13, 2024. doi: 10.1038/s44335-024-00014-0

    work page doi:10.1038/s44335-024-00014-0 2024

  2. [1]

    Natural gradient works efficiently in learning.Neural computation, 10(2):251– 276, 1998

    Shun-Ichi Amari. Natural gradient works efficiently in learning.Neural computation, 10(2):251– 276, 1998. URL http://cognet.mit.edu/journal/10.1162/089976698300017746

    doi:10.1162/089976698300017746 1998

  3. [2]

    Fast convergence of natural gradient descent for over-parameterized neural networks

    Guodong Zhang, James Martens, and Roger B Grosse. Fast convergence of natural gradient descent for over-parameterized neural networks. Advances in Neural Information Processing Systems, 32, 2019

    2019

  4. [3]

    New insights and perspectives on the natural gradient method

    James Martens. New insights and perspectives on the natural gradient method. The Journal of Machine Learning Research, 21(1):5776–5851, 2020. URL https://arxiv.org/abs/1412. 1193

    work page 2020

  5. [4]

    A natural policy gradient

    Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001

    2001

  6. [5]

    A survey of actor- critic reinforcement learning: Standard and natural policy gradients

    Ivo Grondman, Lucian Busoniu, Gabriel AD Lopes, and Robert Babuska. A survey of actor- critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, part C (applications and reviews), 42(6):1291–1307, 2012

    work page 2012

  7. [6]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InProceedings of the 3rd International Conference on Learning Representations (ICLR), 2015. URL http: //arxiv.org/abs/1412.6980

    arXiv 2015

  8. [7]

    Crooks, James Crutchfield, et al

    Tom Conte, Erik DeBenedictis, Natesh Ganesh, Todd Hylton, John Paul Strachan, R Stanley Williams, Alexander Alemi, Lee Altenberg, Gavin E. Crooks, James Crutchfield, et al. Ther- modynamic computing. arXiv preprint arXiv:1911.01968, 2019. URL https://arxiv.org/ abs/1911.01968

    arXiv 1911

Show all 48 references
  1. [8]

    Coles, Collin Szczepanski, Denis Melanson, Kaelan Donatella, Antonio J

    Patrick J. Coles, Collin Szczepanski, Denis Melanson, Kaelan Donatella, Antonio J. Mar- tinez, and Faris Sbahi. Thermodynamic AI and the fluctuation frontier. In 2023 IEEE International Conference on Rebooting Computing (ICRC) , pages 1–10. IEEE, 2023. doi: 10.1109/ICRC60800.2...

    2023

  2. [12]

    Optimization methods for large-scale machine learning

    Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60(2):223–311, 2018. URL https://epubs.siam.org/doi/10. 1137/16M1080173. 12

    2018

  3. [13]

    Optimizing neural networks with kronecker-factored approx- imate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approx- imate curvature. In International conference on machine learning, pages 2408–2417. PMLR,

  4. [14]

    Kronecker-factored curvature approximations for recurrent neural networks

    James Martens, Jimmy Ba, and Matt Johnson. Kronecker-factored curvature approximations for recurrent neural networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/pdf?id=HyMTkQZAb

    2018

  5. [15]

    Kronecker-factored approximate curvature for modern neural network ar- chitectures

    Runa Eschenhagen, Alexander Immer, Richard Turner, Frank Schneider, and Philipp Hennig. Kronecker-factored approximate curvature for modern neural network ar- chitectures. In Advances in Neural Information Processing Systems , volume 36,

  6. [16]

    Efficient subsampled gauss-newton and natural gradient methods for training neural networks

    Yi Ren and Donald Goldfarb. Efficient subsampled gauss-newton and natural gradient methods for training neural networks. arXiv preprint arXiv:1906.02353, 2019. URL https://arxiv. org/abs/1906.02353

    1906 arXiv

  7. [18]

    Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model

    Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. Advances in neural information processing systems, 32, 2019

    2019

  8. [19]

    URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 6a6679e3d5b9f7d5f09cdb79a5fc3fd8-Paper-Conference.pdf

    2023

  9. [20]

    Time complexity of in-memory solution of linear systems

    Zhong Sun, Giacomo Pedretti, Piergiulio Mannocci, Elia Ambrosi, Alessandro Bricalli, and Daniele Ielmini. Time complexity of in-memory solution of linear systems. IEEE Transactions on Electron Devices, 67(7):2945–2951, 2020

    2020

  10. [21]

    Thermodynamic neural network

    Todd Hylton. Thermodynamic neural network. Entropy, 22(3):256, 2020. doi: 10.3390/ e22030256. URL https://www.mdpi.com/1099-4300/22/3/256

    2020

  11. [22]

    A thermodynamic treatment of intelligent systems

    Natesh Ganesh. A thermodynamic treatment of intelligent systems. In 2017 IEEE International Conference on Rebooting Computing (ICRC) , pages 1–4, 2017. doi: 10.1109/ICRC.2017. 8123676

    2017 doi

  12. [23]

    Thermodynamic computing via autonomous quantum thermal machines

    Patryk Lipka-Bartosik, Martí Perarnau-Llobet, and Nicolas Brunner. Thermodynamic computing via autonomous quantum thermal machines. Science Advances, 10(36):eadm8792, 2024

    2024

  13. [24]

    Solving matrix equations in one step with cross-point resistive arrays

    Zhong Sun, Giacomo Pedretti, Elia Ambrosi, Alessandro Bricalli, Wei Wang, and Daniele Ielmini. Solving matrix equations in one step with cross-point resistive arrays. Proceedings of the National Academy of Sciences, 116(10):4123–4128, 2019

    2019

  14. [25]

    Maxwell Aifer, Samuel Duffield, Kaelan Donatella, Denis Melanson, Phoebe Klett, Zach Belateche, Gavin Crooks, Antonio J Martinez, and Patrick J. Coles. Thermodynamic bayesian inference. arXiv preprint arXiv:2410.01793, 2024. URL https://arxiv.org/abs/2410. 01793

    2024 arXiv

  15. [26]

    Patryk-Lipka Bartosik, Kaelan Donatella, Maxwell Aifer, Denis Melanson, Marti Perarnau- Llobet, Nicolas Brunner, and Patrick J. Coles. Thermodynamic algorithms for quadratic programming. arXiv preprint arXiv:2411.14224 , 2024. URL https://arxiv.org/abs/ 2411.14224

    2024 arXiv

  16. [27]

    Martinis, Giovanni Finocchio, and Kerem Y

    Navid Anjum Aadit, Andrea Grimaldi, Mario Carpentieri, Luke Theogarajan, John M. Martinis, Giovanni Finocchio, and Kerem Y . Camsari. Massively parallel probabilistic computing with sparse Ising machines. Nat. Electron., 5(7):460–468, 2022. doi: 10.1038/s41928-022-00774-2

    2022 doi

  17. [28]

    Kaiser, S

    J. Kaiser, S. Datta, and B. Behin-Aein. Life is probabilistic—why should all our computers be deterministic? computing with p-bits: Ising solvers and beyond. In 2022 International Electron Devices Meeting (IEDM), pages 21–4. IEEE, 2022. 13

    2022

  18. [29]

    Thermodynamic computing out of equilibrium

    Stephen Whitelam and Corneel Casert. Thermodynamic computing out of equilibrium. arXiv preprint arXiv:2412.17183, 2024. URL https://arxiv.org/abs/2412.17183

    2024

  19. [30]

    Analog cmos-based resistive processing unit for deep neural network training

    Seyoung Kim, Tayfun Gokmen, Hyung-Min Lee, and Wilfried E Haensch. Analog cmos-based resistive processing unit for deep neural network training. In 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), pages 422–425. IEEE, 2017. URL https://ieeexplor...

    2017

  20. [31]

    Equivalent-accuracy accelerated neural-network training using analogue memory

    Stefano Ambrogio, Pritish Narayanan, Hsinyu Tsai, Robert M Shelby, Irem Boybat, Carmelo Di Nolfo, Severin Sidler, Massimo Giordano, Martina Bodini, Nathan CP Farinha, et al. Equivalent-accuracy accelerated neural-network training using analogue memory. Nature, 558 (7708):60–67...

    2018

  21. [32]

    Romero, Christina Cheng, Pritish Narayanan, Hsinyu Tsai, Robert M

    Giorgio Cristiano, Massimo Giordano, Stefano Ambrogio, Louis P. Romero, Christina Cheng, Pritish Narayanan, Hsinyu Tsai, Robert M. Shelby, and Geoffrey W Burr. Perspective on training fully connected networks with resistive memories: Device requirements for multiple conductanc...

    2018 doi

  22. [33]

    Hardware implementation of memristor-based artificial neural networks

    Fernando Aguirre, Abu Sebastian, Manuel Le Gallo, Wenhao Song, Tong Wang, J Joshua Yang, Wei Lu, Meng-Fan Chang, Daniele Ielmini, Yuchao Yang, et al. Hardware implementation of memristor-based artificial neural networks. Nature Communications, 15(1):1974, 2024. URL https://www...

    1974

  23. [34]

    Reversible computing with fast, fully static, fully adiabatic cmos

    Michael P Frank, Robert W Brocato, Brian D Tierney, Nancy A Missert, and Alexander H Hsia. Reversible computing with fast, fully static, fully adiabatic cmos. In 2020 International Conference on Rebooting Computing (ICRC), pages 1–8. IEEE, 2020

    2020

  24. [35]

    Optimization of graph neural networks with natural gradient descent

    Mohammad Rasool Izadi, Yihao Fang, Robert Stevenson, and Lizhen Lin. Optimization of graph neural networks with natural gradient descent. In 2020 IEEE international conference on big data (big data), pages 171–179. IEEE, 2020

    2020

  25. [36]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. URL https://proceedings.neurips.cc/paper/2017/ file/3f5ee243547d...

    2017

  26. [37]

    Denis Melanson, Mohammad Abu Khater, Maxwell Aifer, Kaelan Donatella, Max Hunter Gordon, Thomas Ahle, Gavin Crooks, Antonio J Martinez, Faris Sbahi, and Patrick J. Coles. Thermodynamic computing system for AI applications. arXiv preprint arXiv:2312.04836, 2023. URL https://arx...

    2023 arXiv

  27. [38]

    Andrej Karpathy. NanoGPT. https://github.com/karpathy/nanoGPT, 2022

    2022

  28. [39]

    Deep learning via hessian-free optimization

    James Martens et al. Deep learning via hessian-free optimization. In ICML, volume 27, pages 735–742, 2010. URL https://www.cs.toronto.edu/~asamir/cifar/HFO_James.pdf

    2010

  29. [40]

    Maxwell Aifer, Denis Melanson, Kaelan Donatella, Gavin Crooks, Thomas Ahle, and Patrick J. Coles. Error mitigation for thermodynamic computing, 2024. URL https://arxiv.org/ abs/2401.16231

    2024 arXiv

  30. [41]

    Hanebeck

    Christopher Funk, Benjamin Noack, and Uwe D. Hanebeck. Conservative quantization of covariance matrices with applications to decentralized information fusion. Sensors, 21(9), 2021. ISSN 1424-8220. doi: 10.3390/s21093059. URL https://www.mdpi.com/1424-8220/21/ 9/3059

    2021 doi

  31. [42]

    Higham and Srikara Pranesh

    Nicholas J. Higham and Srikara Pranesh. Exploiting lower precision arithmetic in solving symmetric positive definite linear systems and least squares problems. SIAM Journal on Scientific Computing, 43(1):A258–A277, 2021. doi: 10.1137/19M1298263. URL https: //doi.org/10.1137/19M1298263

    2021 doi

  32. [43]

    Benchmarking neural network training algorithms

    George E Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, et al. Benchmarking neural network training algorithms. arXiv preprint arXiv:2306.07179, 2023. 14

    2023 arXiv

  33. [44]

    Scalable second order optimization for deep learning

    Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020

    2002 arXiv

  34. [45]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139–1147. PMLR, 2013

    2013

  35. [46]

    Asdl: A unified interface for gradient preconditioning in pytorch

    Kazuki Osawa, Satoki Ishikawa, Rio Yokota, Shigang Li, and Torsten Hoefler. Asdl: A unified interface for gradient preconditioning in pytorch. arXiv preprint arXiv:2305.04684, 2023. A Energy-based viewpoint Consider the following energy-based viewpoint of our work. Training a ...

    2023 arXiv

  36. [49]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. URL https://arxiv.org/abs/1711.05101

    2017 arXiv

  37. [2015]

    URL https://proceedings.mlr.press/v37/martens15.html

  38. [2020]

    URL https://arxiv.org/abs/2006.02409

    2006 arXiv

  39. [2023]

    URL https://arxiv.org/abs/2311.12759

    arXiv

  40. [2024]

    URL https://arxiv.org/abs/2405.13817

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.