Can Hessian-Based Insights Support Fault Diagnosis in Attention-based Models?

Mohammad Masudur Rahman; Sigma Jahan

REVIEW 5 major objections 6 minor 55 references

Can Hessian-Based Insights Support Fault Diagnosis in Attention-based Models?

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Hessian-based diagnostics reveal fragile regions and parameter couplings in attention models that gradient norms miss.

desk verdict Plausible exploratory study of Hessian diagnostics for attention models, but the headline claim about gradients is unbacked and the off-diagonal Hessian 'correlations' are raw second derivatives, not correlations. read the letter →

arxiv 2506.07871 v1 pith:LEG627VR submitted 2025-06-09 cs.LG cs.SE

Sigma Jahan , Mohammad Masudur Rahman This is my paper · ORCID

classification cs.LGcs.SE

keywords AttentionMechanismsHessianAnalysisCurvatureParameterInterdependenciesLossSurfaceFaultDiagnosisSecond-orderDerivativesTrainingInstability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether second-derivative (Hessian) information about the loss surface can locate faults in attention-based models better than first-order gradient checks. It claims that two Hessian-derived diagnostics do this: curvature measures (largest eigenvalue and trace) identify concave, sharply curved regions where training is fragile, and off-diagonal Hessian entries reveal parameter couplings through which a fault in one attention component cascades into another. Across three architectures—HAN, a 3D-CNN with cross-attention, and DistilBERT—the paper reports fragile regions such as word-level attention in HAN (trace $-5.72$), cross-attention in the 3D-CNN ($-2.11$), and the query projection in DistilBERT ($-2.24$), while neighboring layers show stable positive traces. In HAN, a strong negative off-diagonal coupling ($-0.68$) between word- and sentence-level parameters predicted that lowering the learning rate on the coupled word parameters would reduce prediction variability from 30% to 10%, which the intervention confirmed. The paper concludes that Hessian-based metrics complement gradient-based debugging by exposing instability and fault propagation that gradients alone miss.

What carries the argument

The central object is the Hessian matrix $H=\nabla^2_\theta L(\theta)$ of the training loss with respect to model parameters. The paper derives three diagnostic quantities from it: the largest eigenvalue $\lambda_{\max}$, which measures the worst-case curvature along any parameter direction; the trace $\mathrm{Tr}(H)$, whose sign and magnitude indicate whether the local loss surface is convex and stable or concave and fragile; and the off-diagonal entries $H_{ij}$, which quantify how a change in parameter $i$ alters the gradient with respect to parameter $j$, interpreted as a coupling through which faults propagate. Because exact Hessians are costly for large models, the paper estimates these quantities through Hessian-vector products with a reported error under 1% on standard benchmarks, then isolates attention-specific parameter groups by module path and aggregates them to layer level so the diagnostics map onto architectural components like word-level attention or query projection.

What would settle it

Train HAN until word-level attention exhibits a real documented failure such as attention entropy collapse, measure $\mathrm{Tr}(H)$ for that layer, and test whether lowering the learning rate on $W_1$ and $W_2$ still reduces prediction variability. If the trace is not sharply negative or the intervention does not restore stability, the paper's diagnostic pattern is an artifact of the Gaussian noise proxy rather than a signature of real faults.

Watch

Extended reading notes

Core claim

The paper's central claim is that Hessian-based analysis of the loss surface can localize instability and pinpoint fault sources in attention-based models more effectively than gradients alone. The evidence is empirical: for HAN, the Hessian trace at word-level attention is $-5.72$ while sentence-level attention is $+2.86$; for the 3D-CNN, cross-attention gives $-2.11$ while spatial and temporal attention give $+1.92$ and $+4.37$; for DistilBERT, the query projection gives $-2.24$ while key and output projections give $+3.12$ and $+1.75$. In each case, the negative-trace layer is the one whose loss curve reacts sharply to parameter perturbations. The parameter-interaction analysis reads off-diagonal Hessian entries as couplings: in HAN, $W_2$ and $S_1$ show $-0.68$, and a targeted intervention that lowers the learning rate on the coupled word-level parameters reduces the off-diagonal magnitude to $-0.41$ and prediction variability from 30% to 10%. The paper takes these results to show that curvature and interaction information from the Hessian identify fragile attention components and cascading fault paths that gradient norms miss.

Load-bearing premise

The load-bearing premise is that injecting Gaussian noise into attention parameters reproduces the loss-surface geometry of real attention-model faults, so diagnostic conclusions derived from these perturbations transfer to genuine training failures.

Editorial extensions

If this is right

Debuggers can monitor Hessian trace during training to flag layers whose curvature turns negative before a failure becomes visible in the loss.
When the Hessian shows strong coupling between attention components, lowering the learning rate on the coupled parameters is a targeted repair: in HAN the intervention reduced prediction variability from 30% to 10%.
Gradient norms can point to the wrong parameter as the fault source: $W_2$ in HAN showed a large gradient (norm about 1.25) but only Hessian coupling explained how instability reached sentence-level attention.
The same diagnostic pattern recurs across multilevel self-attention (HAN), cross-attention (3D-CNN), and multi-head self-attention (DistilBERT), suggesting the method is not tied to one attention design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's fault model is isotropic Gaussian noise; a real attention-model failure such as attention entropy collapse or head misalignment may create a different loss-surface signature, so the diagnostic thresholds (e.g., a negative trace) are not yet calibrated on natural faults.
The authors interpret raw off-diagonal Hessian entries as causal correlations; a safer reading is that they show local second-order coupling only, and a validation experiment would need to ablate individual parameters and measure downstream loss changes to confirm the cascade direction.
A practical extension suggested by the results is to use per-layer Hessian-trace estimates as a trigger for adaptive learning-rate schedules, lowering rates specifically on negative-curvature layers rather than globally.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things worth knowing. First, the paper is genuinely exploratory: the Hessian measurements on attention submodules (per-layer traces, off-diagonal couplings for HAN, 3D-CNN, DistilBERT) do not appear in prior work, and the authors attempt a perturbation-based sanity check and a learning-rate intervention. Second, the central claim—that Hessians localize instability and pinpoint fault sources 'more effectively than gradients alone'—is not established. There is no gradient baseline, no error bars, and the RQ2 result treats a raw second partial derivative as a correlation. The stress-test note is correct: off-diagonal Hessian entries are scale-dependent and non-causal, and the -0.68 value is not a normalized correlation coefficient.

Where credit is due: using PyHessian with stated approximation error is reasonable; reporting layer-level aggregates makes the results interpretable; and the perturbation curves do give some independent evidence that the negative-trace layers are actually sensitive. The RQ1 story, while not novel, is at least internally consistent. The intervention on HAN (lowering LR on W1/W2) is a nice proof-of-concept, even if it cannot validate the causal interpretation.

Soft spots, in order of severity. (1) RQ2's fault-propagation claim is not supported: the Hessian off-diagonal is a local curvature coefficient, not a correlation or a mechanism, and no normalization is given. (2) The abstract's 'more effective than gradients' is asserted, not measured: no quantitative comparison to any first-order diagnostic. (3) The fault injection uses Gaussian noise with an unreported alpha, and all results are single runs with no confidence intervals, so the 5%-30% prediction variability figure is unverifiable. (4) No code or data artifact, which is unfortunate given the paper's empirical thrust. These are fixable, but they affect the headline claim.

Who this is for: researchers in software engineering for deep learning who want a quick look at whether Hessian diagnostics are worth trying on attention models. They will get a useful map of where the obvious pitfalls are, not a validated method. I would not cite it as evidence in the next year, but I would send it to a serious referee: the questions are legitimate, the measurements are transparent enough to audit, and a demanding review could push the authors to add baselines, error bars, an artifact, and a mathematically honest treatment of Hessian off-diagonals. As it stands, a conditional reject with a clear path to revision is the right call.

Referee Report

5 major / 6 minor

Summary. The paper studies whether Hessian-based metrics can support fault diagnosis in attention-based models. It presents an empirical study on three architectures (HAN, 3D-CNN, DistilBERT) in which Hessian trace and top eigenvalues are used to identify 'fragile' layers (RQ1), and off-diagonal Hessian elements are interpreted as parameter interdependencies that explain fault propagation (RQ2). The authors add controlled Gaussian parameter perturbations, report loss-sensitivity plots, and describe a learning-rate intervention on HAN that reduces prediction variability. The central claim is that these Hessian-based metrics localize instability and pinpoint fault sources more effectively than gradients alone.

Significance. If substantiated, the result would be practically useful: a cheap second-order diagnostic that identifies unstable attention components and cascading interdependencies could complement existing gradient-based software-debugging tools for deep learning. The paper has genuine strengths: it covers three architecturally distinct attention models, uses PyHessian with a stated approximation target, couples curvature findings to an independent perturbation-sensitivity experiment (Figure 2), and attempts a targeted intervention that reduces prediction variability. The main limitation is that the central comparative claim ('more effectively than gradients alone') is not supported by any quantitative head-to-head evaluation, and the interaction metric underlying RQ2 is not a validated measure of causal fault propagation.

major comments (5)

[Section 3, Answering RQ2, Table 2] The off-diagonal Hessian entries reported in Table 2 are raw second partial derivatives, but the text repeatedly calls them 'correlations' and reads them causally, e.g., 'small perturbations in W2 can cascade to S1'. A Hessian entry is scale-dependent: if parameter theta_i is rescaled by c_i, the off-diagonal entry scales by c_i*c_j, so the ranking of couplings can change under a reparameterization that leaves the model function unchanged. No normalization is stated, and no argument is given for why a local curvature coefficient at one parameter point measures how a perturbation propagates through training dynamics. Since RQ2 and the 'pinpoint fault sources' portion of the abstract depend on this interpretation, the interaction analysis is not currently established.
[Abstract and Section 3, RQ1/RQ2] The paper claims that Hessian-based metrics are 'more effective than gradients alone', but no quantitative baseline comparison is reported. There is no evaluation protocol that measures, for example, precision/recall of identified fragile layers, agreement with independently labeled faults, or the diagnostic benefit of Hessian features over gradient-norm features. The narrative contrast with gradients (e.g., 'gradients showed large updates in W2 ... but could not explain how this instability spread to S1') is illustrative, not a measured comparison. Without a head-to-head metric, the central comparative claim is unsubstantiated.
[Section 2.5 and Figure 2] Several quantities needed to assess the results are missing: the perturbation magnitude alpha is not reported, the prediction-variability values (5% to 30% and back to 10%) are single numbers with no confidence intervals, number of seeds, or significance tests, and the Hessian traces in Table 1 are single point estimates. The loss-sensitivity curves in Figure 2 have no axis labels, error bars, or per-panel model/layer identification. These omissions make it impossible to judge whether the qualitative trends are robust or within noise, and they undermine the reproducibility of the empirical claims.
[Section 3, RQ1 and RQ2 intervention] RQ1 is partly circular: fragile regions are defined via sharp curvature and then identified via Hessian curvature. The independent perturbation-sensitivity experiment in Figure 2 mitigates this, but it is only shown for selected layers and is not quantified as a correlation or agreement measure across all layers. In addition, the RQ2 intervention lowers the learning rate for W1 and W2, which changes the optimizer trajectory globally; the observed decrease in prediction variability could result from the lower learning rate alone rather than from weakening the specific W2-S1 coupling. The intervention therefore does not validate the causal reading of the Hessian entry.
[Section 2.3, Eq. (1)-(3)] The paper uses a negative Hessian trace as evidence that a region is 'concave and unstable', but a negative trace only means that the sum of eigenvalues is negative; the loss surface may be convex in some directions and concave in others. The connection between the sign of the trace and training instability should be justified for the specific loss surfaces studied, and the reported top eigenvalues in Table 1 should be accompanied by the full spectral context or a stated criterion for what counts as 'fragile'.

minor comments (6)

[Section 1, Introduction] There is a duplicated word: 'However, However, they might fall short for more complex networks...' (around the gradient-limitation paragraph).
[Figure 2] The figure lacks axis labels, a legend identifying which panel corresponds to which model/layer, and any indication of the perturbation magnitude alpha used; please add these details.
[Section 2.5] The choice of Gaussian noise with identity covariance and the range of alpha values are not justified; please report how alpha was selected and whether the qualitative findings are stable across different alpha values.
[Section 2.1] The 3D-CNN and DistilBERT model descriptions are brief; specifying the exact repositories, commits, data splits, and training hyperparameters would improve reproducibility.
[References] The reference list mixes peer-reviewed venues with informal sources (e.g., [34], a Towards Data Science article); consider replacing informal citations with archival or peer-reviewed alternatives where available.
[General] The paper does not include an artifact availability statement or link to code/data; given the empirical nature of the work, an artifact (or at least scripts for the perturbation and Hessian computations) would substantially strengthen the contribution.

Circularity Check

1 steps flagged · score 6.0 of 10

RQ1's 'fragile region' identification is definitionally tied to Hessian curvature; independent perturbation experiments only partially break the tautology.

self definitional [Section 3, 'Answering RQ1'; also abstract]
"We analyze the curvature of the loss surface using the Hessian method to identify fragile regions in three attention-based neural networks: HAN, 3D-CNN, and DistilBERT. Fragile regions are areas with sharp curvatures [27,53], where optimization becomes unstable."

The paper defines the target construct 'fragile regions' as 'areas with sharp curvatures' and then uses Hessian curvature analysis (trace, top eigenvalues) as the method to identify them. Since the Hessian trace/eigenvalues are the operational measure of sharp curvature, 'Hessian metrics localize fragile regions' is true by construction: the identification step re-labels a curvature measurement as a diagnosis of instability rather than testing an independent definition of fragility. The circularity is partial: Fig. 2's perturbation-sensitivity curves and the prediction-variability numbers provide external behavioral evidence that the flagged regions are indeed unstable, and the comparison against gradient norms is not definitional.

full rationale

No self-citation chains are load-bearing, and no fitted parameters are renamed as predictions. The main circular element is definitional: RQ1 equates fragility with sharp curvature and then reports that Hessian curvature analysis finds sharp-curvature regions; this step is tautological. The paper partially breaks the tautology with independent perturbation-sensitivity experiments (Fig. 2) and intervention tests (prediction variability dropping from 30% to 10%), so the central claim still has empirical content beyond the definition. RQ2's reading of raw off-diagonal Hessian entries as 'correlations' and fault-propagation evidence is a mathematical/causal validity concern rather than a circularity, because the prediction-variability experiment is an external check; it is therefore not scored as a separate circular step. Overall: one partial definitional circularity, no self-citation dependence, moderate score.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central findings rest on three main assumptions: the Hessian trace sign is treated as a fragility indicator, the Gaussian perturbation proxy stands in for real faults, and off-diagonal Hessian values are interpreted as causal parameter couplings. The perturbation magnitude alpha is a free parameter but its values are not reported. No invented entities are introduced.

free parameters (1)

Perturbation magnitude alpha = not reported
Controls the injected Gaussian noise (theta_perturbed = theta + alpha*delta, Section 2.5). The reported prediction variability changes (5% to 30%) depend on alpha, but the chosen alpha values are not stated, making the fault-simulation results hard to assess or replicate.

assumptions (4)

domain assumption A negative Hessian trace indicates a fragile, unstable region in the loss surface.
Used to label word attention (Tr(H)=-5.72), cross-attention (-2.11), and query projection (-2.24) as fragile in Table 1 and Section 3, citing curvatures from optimization literature [27,36,40,53].
ad hoc to paper Isotropic Gaussian parameter noise is a valid proxy for real faults in attention models.
Section 2.5 introduces theta_perturbed = theta + alpha*delta with delta ~ N(0,I) and states perturbations 'serve as proxies for real-world faults'.
ad hoc to paper Off-diagonal Hessian entries measure fault propagation and correlation between parameters.
Section 3 RQ2 interprets the raw second derivative -0.68 as 'small perturbations in W2 can cascade to S1', a causal/statistical reading not derived from the math.
domain assumption The three selected models are representative of attention-based models for generalizing conclusions.
Section 2.1 selects HAN, 3D-CNN, and DistilBERT to cover modalities and attention designs, and Section 4 extends conclusions to LLMs and general attention architectures.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Can Hessian-Based Insights Support Fault Diagnosis in Attention-based Models?." pith.science (2026). https://pith.science/paper/LEG627VR

@misc{pith2026250607871,
  author       = {Pith},
  title        = {Pith review of: Can Hessian-Based Insights Support Fault Diagnosis in Attention-based Models?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LEG627VR}},
  note         = {Machine review of arXiv:2506.07871}
}

read the original abstract

As attention-based deep learning models scale in size and complexity, diagnosing their faults becomes increasingly challenging. In this work, we conduct an empirical study to evaluate the potential of Hessian-based analysis for diagnosing faults in attention-based models. Specifically, we use Hessian-derived insights to identify fragile regions (via curvature analysis) and parameter interdependencies (via parameter interaction analysis) within attention mechanisms. Through experiments on three diverse models (HAN, 3D-CNN, DistilBERT), we show that Hessian-based metrics can localize instability and pinpoint fault sources more effectively than gradients alone. Our empirical findings suggest that these metrics could significantly improve fault diagnosis in complex neural architectures, potentially improving software debugging practices.

Figures

Figures reproduced from arXiv: 2506.07871 by the authors.

**Figure 2.** Demonstration of Loss Sensitivity to Parameter Perturbations [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 50 canonical work pages

[1]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies . 4171–4186

work page 2019
[2]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS

2017
[3]

Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based rec- ommender system: A survey and new perspectives. Comput. Surveys 52 (2019), 1–38

work page 2019
[4]

Yanming Yang, Xin Xia, David Lo, and John Grundy. 2022. A survey on deep learning for software engineering. Comput. Surveys 54 (2022), 1–73

work page 2022
[5]

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2024. GPT understands, too. AI Open 5 (2024), 208–215

work page 2024
[6]

Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016. Deep learning. Vol. 1. MIT press Cambridge

2016
[7]

Ian Goodfellow and Oriol Vinyals. 2015. Qualitatively characterizing neural network optimization problems. In ICLR

work page 2015
[8]

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. In NeurIPS

work page 2018

Show all 55 references

[9]

Charles Guille-Escuret, Hiroki Naganuma, Kilian Fatras, and Ioannis Mitliagkas
[10]

Fengxiang He, Bohan Wang, and Dacheng Tao. 2020. Piecewise linear activations substantially shape the loss surfaces of neural networks. In ICLR

2020
[11]

Tala Talaei Khoei, Hadjar Ould Slimane, and Naima Kaabouch. 2023. Deep learning: systematic review, models, challenges, and research directions. Neural Computing and Applications 35 (2023), 23103–23124

2023
[12]

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. 2023. Scaling vision transformers to 22 billion parameters. In ICML. 7480–7512

2023
[13]

Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie E Everett, Alexander A Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. 2024. Small-scale prox...

2024
[14]

Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Rama- puram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. 2023. Stabilizing trans- former training by preventing attention entropy collapse. InICML. 40770–40803

2023
[15]

Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. (2020). doi:10.48550/arXiv.2001.08361 arXiv:arXiv:2001.08361

2020 doi arXiv
[16]

Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. 2021. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In ICML. 2793–2803

2021
[17]

Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . 3543–3556

2019
[18]

Bing Bai, Jian Liang, Guanhua Zhang, Hao Li, Kun Bai, and Fei Wang. 2021. Why Attentions May Not Be Interpretable?. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining . 25–34

2021
[19]

Mohammad Wardat, Wei Le, and Hridesh Rajan. 2021. Deeplocalize: Fault local- ization for deep neural networks. In ICSE. 251–262

2021
[20]

Xiaoyu Zhang, Juan Zhai, Shiqing Ma, and Chao Shen. 2021. Autotrainer: An automatic dnn training problem detection and repair system. In ICSE. 359–371

2021
[21]

Jialun Cao, Meiziniu Li, Xiao Chen, Ming Wen, Yongqiang Tian, Bo Wu, and Shing-Chi Cheung. 2022. Deepfd: Automated fault diagnosis and localization for deep learning programs. In ICSE. 573–585

2022
[22]

Eldon Schoop, Forrest Huang, and Bjoern Hartmann. 2021. Umlaut: Debugging deep learning programs using program structure and model behavior. In CHI conference on human factors in computing systems . 1–16

2021
[23]

Goodfellow, Jonathon Shlens, and Christian Szegedy

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In ICLR

2015
[24]

Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. 2018. Hessian-based analysis of large batch training and robustness to adversaries. In NeurIPS

2018
[25]

Sofia Serrano and Noah A Smith. 2019. Is attention interpretable?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . 2931– 2951

2019
[26]

Sidak Pal Singh, Gregor Bachmann, and Thomas Hofmann. 2021. Analytic insights into structure and rank of neural network hessian maps. In NeurIPS

2021
[27]

Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. 2019. An investigation into neural net optimization via hessian eigenvalue density. InICML. 2232–2241

2019
[28]

Yiding Jiang*, Behnam Neyshabur*, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. 2020. Fantastic Generalization Measures and Where to Find Them. In ICLR

2020
[29]

James Martens et al. 2010. Deep learning via hessian-free optimization.. In ICML, Vol. 27. 735–742

2010
[30]

Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language...

2016
[31]

S. G. R. Vinod. 2023. A PyTorch Tutorial to Text Classification. https://github. com/sgrvinod/a-PyTorch-Tutorial-to-Text-Classification. Accessed: 2025-04-21

2023
[32]

Hugging Face. 2024. Transformers Library. https://github.com/huggingface/ transformers. Accessed: 2025-04-21

2024
[33]

Kiet Nguyen. 2023. HMDB51 Recognition. https://github.com/kietngt00/hmdb51- recognition. Accessed: 2025-04-21

2023
[34]

Nicolo Cosimo Albanese. 2023. Implementing the Steepest Descent Algorithm in Python from Scratch. https://towardsdatascience.com/implementing-the- steepest-descent-algorithm-in-python-from-scratch-d32da2906fe2. Towards Data Science, Accessed: 2025-04-21

2023
[35]

PyTorch Team. 2025. Autograd: Automatic Differentiation. https://pytorch.org/ tutorials/beginner/blitz/autograd_tutorial. Accessed: 2025-04-21

2025
[36]

James Martens and Ilya Sutskever. 2011. Learning recurrent neural networks with hessian-free optimization. In ICML. 1033–1040

2011
[37]

Yunwen Lei. 2023. Stability and generalization of stochastic optimization with nonconvex and nonsmooth problems. In Annual Conference on Learning Theory

2023
[38]

Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. 2020. The Break-Even Point on Optimization Trajectories of Deep Neural Networks. In ICLR

2020
[39]

Borjan Geshkovski, Hugo Koubbi, Yury Polyanskiy, and Philippe Rigollet. 2024. Dynamic Metastability in the Self-Attention Model. (2024). doi:10.48550/arXiv. 2410.06833 arXiv:arXiv:2410.06833

2024 doi arXiv
[40]

Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio

Yann N. Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. 2014. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In NeurIPS

2014
[41]

Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. 2019. Understanding and improving layer normalization

2019
[42]

Suraj Srinivas, Kyle Matoba, Himabindu Lakkaraju, and François Fleuret. 2022. Efficient Training of Low-Curvature Neural Networks. In NeurIPS

2022
[43]

Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. 2018. Towards better understanding of gradient-based attribution methods for Deep Neural Networks. In ICLR

2018
[44]

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. In NeurIPS

2018
[45]

Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. 2020. Pyhes- sian: Neural networks through the lens of the hessian. InInternational Conference on Big Data. 581–590

2020
[46]

Roberto Battiti. 1992. First-and second-order methods for learning: between steepest descent and Newton’s method. Neural Computation 4 (1992), 141–166

1992
[47]

Amirata Ghorbani, Abubakar Abid, and James Zou. 2019. Interpretation of neural networks is fragile. In AAAI Conference on Artificial Intelligence , Vol. 33. 3681–3688

2019
[48]

Pranava Madhyastha and Rishabh Jain. 2019. On Model Stability as a Function of Random Seed. In Conference on Computational Natural Language Learning

2019
[49]

Michael S Pratte, Sam Ling, Jascha D Swisher, and Frank Tong. 2013. How attention extracts objects from noise. Journal of Neurophysiology 110 (2013), 1346–1356

2013
[50]

Maksims Ivanovs, Roberts Kadikis, and Kaspars Ozols. 2021. Perturbation-based methods for explaining deep neural networks: A survey. Pattern Recognition Letters 150 (2021), 228–234

2021
[51]

Diego García-Martín, Martín Larocca, and M. Cerezo. 2024. Effects of noise on the overparametrization of quantum neural networks. Physical Review Research 6 (2024), 013295

2024
[52]

Julian Büchel, Fynn Firouz Faber, and Dylan Richard Muir. 2022. Network insen- sitivity to parameter noise via parameter attack during training. In ICLR

2022
[53]

Guillaume Alain, Nicolas Le Roux, and Pierre-Antoine Manzagol. 2019. Negative Eigenvalues of the Hessian in Deep Neural Networks. In ICLR

2019
[54]

Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. 2023. Studying Large Language Model Generalization with Influence Functions. (2023). doi:10. 48550/arXiv.2308.03296 arXiv:arXiv:2308.03296

2023 doi arXiv
[2024]

No wrong turns: the simple geometry of neural networks optimization paths. In ICML

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies . 4171–4186

work page 2019

[2] [2]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS

2017

[3] [3]

Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based rec- ommender system: A survey and new perspectives. Comput. Surveys 52 (2019), 1–38

work page 2019

[4] [4]

Yanming Yang, Xin Xia, David Lo, and John Grundy. 2022. A survey on deep learning for software engineering. Comput. Surveys 54 (2022), 1–73

work page 2022

[5] [5]

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2024. GPT understands, too. AI Open 5 (2024), 208–215

work page 2024

[6] [6]

Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016. Deep learning. Vol. 1. MIT press Cambridge

2016

[7] [7]

Ian Goodfellow and Oriol Vinyals. 2015. Qualitatively characterizing neural network optimization problems. In ICLR

work page 2015

[8] [8]

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. In NeurIPS

work page 2018