Pith. sign in

REVIEW 5 major objections 7 minor 72 references

Navigating the Latent Space Dynamics of Neural Models

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Trained autoencoders implicitly define a latent vector field, obtained by iterating the encode–decode map, whose attractors encode what the network memorized, what it generalizes, and what falls outside its learned distribution.

desk verdict Interesting empirical results on latent-space attractors, but the formal theory has a load-bearing contradiction: the contraction assumption forces a single fixed point while the experiments rely on many. read the letter →

arxiv 2505.22785 v4 pith:7BWVYFQD submitted 2025-05-28 cs.LG

Marco Fumero , Luca Moschella , Emanuele Rodolà , Francesco Locatello This is my paper
classification cs.LG MSC 68T07
keywords latentvectorfieldautoencodersattractordynamicsmemorizationversusgeneralizationout-of-distributiondetectiondata-freeprobingcontractivemappingsscorefunction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that every trained autoencoder carries a hidden dynamical system: repeatedly applying the encode–decode map $f(z) = E(D(z))$ in latent space produces a vector field whose trajectories converge to attractors. Standard training choices — bottleneck dimensions, weight decay, and data augmentation — make the map locally contractive, so the attractors exist with no extra training. The field is not a byproduct but a representation of the model: its attractors mark the memorization–generalization boundary, its noise-computed attractors recover knowledge stored in the weights of foundation models without any input data, and its trajectories separate in-distribution from out-of-distribution samples. The paper's theoretical core identifies the field with the score, $f(z) - z \propto \nabla_z \log q(z)$, so the dynamics point toward modes of the learned latent distribution. If correct, the claim turns any autoencoder into its own probe.

What carries the argument

The central object is the latent vector field $v(z) = f(z) - z$ defined by the iterated latent map $z_{t+1} = f(z_t)$, with $f = E \circ D$ the encode–decode composition; the iteration is a discrete dynamical system that discretizes $\partial z/\partial t = f(z) - z$. The argument is carried by contractivity: bottlenecks cap the rank of the encoder Jacobian, weight decay shrinks weight norms, and augmentation penalizes sensitivity, so the spectral norm $\|J_f(z)\|_\sigma$ stays below one near the data, and the Banach fixed-point theorem turns each stable fixed point into an attractor summarizing the dynamics. Theorem 1 aligns the field with the score $\nabla_z \log q(z)$ of the latent marginal, Proposition 3.2 decomposes reconstruction error into a prototype term and a coverage term measured against the nearest attractor, and the memorization coefficient — the cosine similarity of a decoded attractor to its closest training point — reads the memorization–generalization spectrum off the attractor set.

What would settle it

Take a trained autoencoder, estimate the density of latent codes of held-in data (for instance with a kernel density estimate), and compute the attractors by iterating the map from Gaussian noise. The central claim predicts that typical in-distribution latent codes lie close to some attractor and that the estimated log-density rises along every trajectory toward its attractor; a model whose latent codes sit far from its own attractors, or along whose trajectories the density falls, would break the identification of attractors with modes of the learned distribution. The experiment needs no new architecture — only the model's own iteration and the estimated latent density.

Watch

Extended reading notes

Core claim

The central claim is that the composite map $f = E \circ D$ of any autoencoder induces a latent vector field $v(z) = f(z) - z$, obtained by iterating $z_{t+1} = f(z_t)$, and that this field is a faithful representation of both model and data. Because the inductive biases of standard training — the bottleneck's rank bound on the Jacobian, weight decay, and augmentation — make $f$ locally contractive, the fixed points $z^*$ with $f(z^*) = z^*$ act as attractors, and the field pushes points toward high-probability regions: locally $f(z) - z$ is proportional to the score $\nabla_z \log q(z)$ of the latent marginal. The paper formalizes a dictionary view of generalization in which reconstruction error splits into a prototype error and a coverage error against the nearest attractor, so memorization shows up as attractors pinned to training points with narrow coverage and generalization as attractors that span and cover the latent support. Empirically, attractors computed from Gaussian noise in the Stable Diffusion autoencoder reconstruct images across six datasets better than a random orthogonal basis of the latent space, and trajectory distances to training attractors outperform KNN, Mahalanobis, and reconstruction baselines at out-of-distribution detection on masked autoencoders. The evolution of the field during training mirrors the model's passage from memorization to generalization.

Load-bearing premise

The formal theory assumes that the latent codes of the data are already concentrated on the fixed-point manifold of the iterated map, meaning the high-density region of the latent distribution is exactly where the attractors sit — which is essentially the relation the paper aims to prove, and which its experiments never check directly.

Editorial extensions

If this is right

  • Every trained autoencoder becomes its own probe: iterating $f$ computes the latent vector field with no additional training, so the architecture's parameters, not a separate analysis tool, reveal its behavior.
  • The attractor set reads the memorization–generalization balance: strongly regularized models store training points as attractors with narrow coverage, generalizing models form attractors that cover the latent support, and the transition from one regime to the other is visible epoch by epoch during training.
  • Weights of foundation models can be queried without any input data: attractors computed from Gaussian noise form a dictionary that reconstructs images from general, medical, and satellite domains better than a random orthogonal basis of the latent space.
  • Out-of-distribution samples are detectable from their dynamics: the mean distance from a sample's trajectory to the training attractors separates in- from out-of-distribution data with lower FPR95 than KNN, Mahalanobis, and reconstruction-error baselines on the tested benchmarks.
  • Attractor structure tracks generalization quality: the rank of the decoded-attractor matrix rises as models generalize better, giving a geometric signature of interpolation that holds both across bottleneck sizes and across training time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If trajectories really track the latent score, then the iteration is a data-free generative device: flowing noise forward to an attractor draws a representative of the learned distribution. The paper uses attractors only as a reconstruction dictionary and never evaluates the generated samples, which is a direct testable extension.
  • The memorization coefficient and attractor-rank curves are natural training diagnostics — for example early signals of double descent or of a model beginning to copy its training set — but the paper computes them only on its own architectures and does not compare them against standard generalization metrics.
  • Because the machinery requires only a locally contractive self-map, the same probing recipe should transfer to encoder-only self-supervised models fitted with decoders and to LLM residual streams, which the appendix treats only preliminarily; attractor dictionaries could then serve as a basis for comparing models, including representation-alignment questions.
  • The OOD score could be sharpened by combining how far a trajectory stays from training attractors with how quickly it converges — the paper identifies both as informative but uses only the mean distance along the trajectory.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes to view the encoder-decoder composition f = E∘D of an autoencoder as a dynamical system in latent space, with the associated vector field f(z) - z obtained by iterating the map without additional training. The central claims are that (i) training biases make such maps locally contractive; (ii) the attractors of the induced latent dynamics correspond to modes of the learned latent distribution q(z); (iii) the vector field and its attractors encode the model's memorization/generalization regime; and (iv) attractors computed from noise form a data-free dictionary that can be used for probing foundation models and for out-of-distribution detection. The formal theory is developed in Appendix A, where Theorem A.6 attempts to show convergence to a unique fixed point that is a stationary point and local maximum of q under Assumptions A.1-A.4. The empirical sections demonstrate latent fields for small convolutional autoencoders, memorization-generalization transitions during training, data-free dictionary reconstruction with the Stable Diffusion autoencoder, and OOD detection with a ViT-MAE backbone.

Significance. If the theoretical characterization were established, the framework would provide a unified, data-free lens on autoencoder internals and a practical tool for probing and OOD detection. The paper has notable empirical strengths: the contraction-at-initialization measurements across twelve vision backbones, the attractor-count and memorization-coefficient trajectories during training, the dictionary-reconstruction results on Stable Diffusion AEs, and the OOD detection gains over KNN, Mahalanobis, and reconstruction baselines. However, the formal theory as written does not support the headline claim that multiple attractors are modes of the learned distribution. The global contraction assumption forces a single fixed point, while the experiments rely on many attractors; additionally, the key concentration assumption largely assumes the desired conclusion. The empirical results are suggestive and potentially valuable, but they currently lack a rigorous supporting theory.

major comments (5)
  1. [Appendix A.1, Assumption A.3 and Theorem A.6] Assumption A.3 requires an open convex Ω with M ⊆ Ω and sup_{z∈Ω} ∥J_f(z)∥_σ ≤ L < 1. On a convex domain this global bound makes f a contraction in the norm, so by the mean value theorem f can have at most one fixed point in Ω; since M ⊆ Ω, M is a singleton. This directly contradicts the paper's central multi-attractor evidence, including the many attractors reported in Figure 3c, the 'dictionary of attractors' in Section 3.2, and the attractor sets used in the OOD experiments. Theorem A.6 therefore cannot justify the interpretation of multiple attractors as modes of q(z). Please replace the global contraction condition with a genuinely local contraction assumption on each basin of attraction, or explicitly restrict the formal claims to a single-attractor setting and re-frame the multi-attractor discussion as purely empirical.
  2. [Appendix A.1, Assumption A.4] Assumption A.4 states that the latent marginal q(z) is 'supported and concentrated on the fixed-point manifold M.' This is essentially the conclusion the paper aims to establish: that the high-probability regions of the latent distribution coincide with the attractor set. The assumption is never verified empirically; the paper does not estimate q(z), nor does it measure the distance between the support of q and M. As written, Theorem A.6 derives the mode-attractor correspondence from this assumption rather than proving it. Please either derive the concentration property from the training objective, or validate it directly in the experiments, or weaken the theoretical claims accordingly.
  3. [Appendix A.1, Lemma A.5] The proof of Lemma A.5 asserts without justification that 'by contractiveness and training optimality, f(z) is closer to the fixed-point set M than z is,' and then concludes q(f(z)) > q(z). Contractiveness alone does not imply movement toward M, and being closer to M does not imply higher density under an arbitrary smooth q. The additional claim that the Taylor remainder R = o(∥v(z)∥) requires a bound on ∥v(z)∥ = ∥f(z) - z∥, which global contractiveness does not provide for points far from the fixed point. The strict positivity ⟨∇log q(z), v(z)⟩ > 0 is therefore not established by the argument given.
  4. [Appendix A.1, Theorem A.6, Step 3] The proof that z* is a strict local maximum of q is incomplete. From the fact that q(z_t) increases along the specific orbit z_t = f^t(z_0) and converges to q(z*), it does not follow that every directional second derivative at z* is negative; directions not traversed by that orbit could be saddle or flat directions. The conclusion ∇²log q(z*) ≺ 0 is thus unsupported and needs a genuine second-order analysis.
  5. [Section 3.1, Theorem 1 (informal) vs Appendix A.1, Theorem A.6] The informal Theorem 1 in the main text claims that the latent dynamics f(z) - z is 'locally proportional to the score function ∇ log q(z).' The formal Theorem A.6, even if its proof were repaired, only attempts to establish convergence to a stationary point that is a local maximum of q; it does not state or prove proportionality to the score. The main-text claim is therefore strictly stronger than anything proved in the appendix. Please either prove the proportionality statement under appropriate local assumptions or weaken the main-text theorem to match the formal result.
minor comments (7)
  1. [Section 2, Eq. (3)] The phrase 'discrete ODE' is inaccurate; Eq. (3) defines a discrete-time dynamical system. The following sentence claiming that 'By the Banach fixed-point theorem, this holds if and only if f is Lipschitz-continuous with C < 1' is also not correct for existence and uniqueness of ODE solutions; local Lipschitz continuity, not contraction, is the standard sufficient condition.
  2. [Section 2, Definition 2] The definition of an attractor as a fixed point whose Jacobian eigenvalues are 'strictly less than one in absolute value' should state explicitly that all eigenvalues lie inside the unit circle; the current wording could be read as a per-eigenvalue condition without stating the spectral-radius requirement for discrete maps.
  3. [Figure 2 caption] The caption contains the stray token '4 16 32 k=256 ?', which appears to be a leftover placeholder; please remove or replace it with the actual axis information.
  4. [Appendix B.3] The word 'Mahanabolis' is a typo and should read 'Mahalanobis'.
  5. [Appendix B.4.1] The phrase 'the number od the iteration' should read 'the number of iterations'.
  6. [References] The citation for the Places365 dataset appears to be incorrect: the manuscript cites Zhou and Paffenroth (2017), which is an anomaly-detection paper, rather than the Places365 dataset reference; please correct this citation.
  7. [Section 5, Related Work] The sentence '(Alain and Bengio, 2014; Vincent, 2011), A similar phenomenon...' contains a misplaced comma and period; please fix the citation punctuation.

Circularity Check

1 steps flagged · score 7.0 of 10

The formal mode-attractor theorem is circular: Assumption A.4 places q's mass on the fixed-point manifold and Lemma A.5's proof simply asserts q(f(z))>q(z), so the conclusion that attractors are latent modes is assumed rather than derived.

  1. self definitional [Appendix A.1, Assumption A.4 and Lemma A.5 proof]
    "Then q(z) is supported and concentrated on the fixed-point manifold M, and f is locally contractive around M. ... By contractiveness and training optimality, f(z) is closer to the fixed-point set M than z is. Since q(z) is concentrated on M, we have: q(f(z)) > q(z)"

    Theorem A.6's advertised conclusion is that iterating f converges to a point z* in M that is a local maximum (mode) of the latent marginal q. Assumption A.4 pre-installs this result by declaring q to be supported and concentrated on M, the fixed-point set of f. Lemma A.5's proof then uses exactly that concentration to assert q(f(z)) > q(z), the inequality that drives the directional-ascent and local-maximality steps of Theorem A.6. Contractivity alone only gives ||f(z)-f(w)|| <= L||z-w||, and the training objective only bounds Jacobians at encoded training points; neither implies that f increases q along trajectories. Thus the central claim that attractors are modes of the learned distribution is assumed by construction rather than derived.

full rationale

The formal theory's key step is circular: Assumption A.4 places the latent marginal's mass on the fixed-point manifold M, and Lemma A.5's proof asserts q(f(z)) > q(z) from that placement, which is the target relation used to conclude that attractors are local maxima of q. The theorem therefore does not derive the mode-attractor link from the reconstruction objective; it builds it into the assumptions. Separately (a correctness risk, not a circularity), Assumption A.3's global contraction on a convex Omega with L<1 implies by Banach a unique fixed point, so Theorem A.6 cannot formally support the multi-attractor phenomena in Fig. 3c and the 'dictionary of attractors' in Fig. 2; the empirical sections illustrate the dynamics but do not verify Assumption A.4's concentration condition. The data-free probing and OOD experiments are honest external benchmarks and do not add circularity, but they do not repair the formal circularity. Score 7 reflects a central theoretical claim that reduces by construction, with independent empirical content elsewhere.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces the 'latent vector field' as an analytical representation, but this is a derived construct from the existing encoder/decoder maps, not a new physical entity. No new particles, forces, or dimensions are postulated.

free parameters (2)
  • attractor equality threshold = 0.99 (cosine similarity)
    Used to define distinct attractors in Figure 3(c); the reported attractor counts depend on this threshold.
  • convergence tolerances = 1e-6 (Figs 2,3), 1e-5 (Sec 4)
    Stop criteria for iterating f; influence the computed attractors and trajectory scores.
assumptions (4)
  • domain assumption Neural autoencoders are locally contractive with Lipschitz constant L < 1 on a domain Omega (Assumption A.3).
    The entire fixed-point/attractor analysis depends on this; the paper provides empirical evidence at initialization and for MAE, but no proof for arbitrary AE.
  • ad hoc to paper The latent marginal q(z) is supported and concentrated on the fixed-point manifold M (Assumption A.4).
    This is close to the conclusion of Theorem A.6: it presumes attractors align with high-probability regions, which is the main claim.
  • domain assumption The decoder D is L_D-Lipschitz on Omega (Proposition A.8 assumption A2).
    Used to prove the prototype-coverage error bound; not verified for the models used.
  • domain assumption The data distribution p(x) is smooth and the encoder is deterministic (delta) to define q(z) (Assumptions A.1 and A.4).
    Simplifies the theory but is not true for stochastic VAEs or real data; the paper later claims the framework extends beyond AEs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Navigating the Latent Space Dynamics of Neural Models." pith.science (2026). https://pith.science/paper/7BWVYFQD

@misc{pith2026250522785,
  author       = {Pith},
  title        = {Pith review of: Navigating the Latent Space Dynamics of Neural Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7BWVYFQD}},
  note         = {Machine review of arXiv:2505.22785}
}
read the original abstract

Neural networks transform high-dimensional data into compact, structured representations, often modeled as elements of a lower dimensional latent space. In this paper, we present an alternative interpretation of neural models as dynamical systems acting on the latent manifold. Specifically, we show that autoencoder models implicitly define a latent vector field on the manifold, derived by iteratively applying the encoding-decoding map, without any additional training. We observe that standard training procedures introduce inductive biases that lead to the emergence of attractor points within this vector field. Drawing on this insight, we propose to leverage the vector field as a representation for the network, providing a novel tool to analyze the properties of the model and the data. This representation enables to: (i) analyze the generalization and memorization regimes of neural models, even throughout training; (ii) extract prior knowledge encoded in the network's parameters from the attractors, without requiring any input data; (iii) identify out-of-distribution samples from their trajectories in the vector field. We further validate our approach on vision foundation models, showcasing the applicability and effectiveness of our method in real-world scenarios.

Figures

Figures reproduced from arXiv: 2505.22785 by the authors.

Figure 1
Figure 1. Latent dynamics of AEs. Latent vector fields induced by autoencoders with bottleneck k = 2, trained on MNIST, with z0 ∼ U[−8, 8]. Models with different initializations are shown. Colors (viridis colormap) represent vector norms ranging from violet (low) to yellow (high). The shape of the latent manifold identifies with the encoder’s support. White regions indicate where the vector field vanishes, revealing attractor… view at source ↗
Figure 2
Figure 2. Memorization vs Generalization. Attractors memorize the training data as a function of the rank of Jf (z) by adjusting the bottleneck dimension k (left) which is inversely proportional to the amount of generalization attained by the model (center); On the right we show example of decoded attractors transitioning from a strong memorization model (first row) to good generalization (last row). of the regularization ter… view at source ↗
Figure 3
Figure 3. Latent vector field dynamics. (a) The 2D vector field (k = 2) expands from a single attractor, eventually stabilizing and over-fitting because of capacity limits. Bottom: Evolution of larger capacity AEs (k = 128) across training. (b) Throughout training, the network first memorizes the data with a high memorization coefficient (in blue) and then generalizes, achieving a low test error (red). (c): Evolution of attra… view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Data-free weight information probing of Stable Diffusion model. We plot the error (MSE) vs sparsity (number of atoms) used to reconstruct samples from diverse dataset respectively from (i) an orthonormal random basis of the latent space (blue); (ii) attractors computed…
Figure 5
Figure 5. Figure 5: Trajectories in the latent vector field characterize distribution shifts We measure out-of￾distribution detection performance on ViTMAE: On the left we report scores for 4 different datasets, highly outperforming the KNN baseline. On the right, histograms of scores on …
Figure 6
Figure 6. Figure 6: Contraction at initialization Variance preserving ratio at initialization of torchvision models: all models considered have a ratio < 1, indicating that the map at initialization is contractive. (a) 256 atoms (b) 1024 atoms (c) 2048 atoms [PITH_FULL_IMAGE:figures/full…
Figure 7
Figure 7. Figure 7: Visualization of reconstructions from the data-free sample recovery experiment of Figure [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Visualization of reconstructions from the data-free sample recovery experiment of Figure [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Visualization of reconstructions from the data-free sample recovery experiment of Figure [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Histograms from OOD detections B.4 ANALYSIS SPEED OF CONVERGENCE In figure 11 we report an empirical validation of proposition A.9, for a 2 dimensional bottleneck convolution autoencoder trained on MNIST, by sampling points in the training set and measuring their conv…
Figure 11
Figure 11. Figure 11: Converge speed analysis We plot the convergence speed in terms of number of iterations vs the error (distance to the attractors) of the initial condition on a convolutional autoencoders trained on MNIST. The estimated convergence has a Pearson correlation coefficient …
Figure 12
Figure 12. Figure 12: Latent trajectories in masked autoencoders are contractive.: we plot norms of the latent residuals across iterations in a ViTMAE model (a) and norms of embeddings across iterations (b). B.5 RANK OF ATTRACTOR MATRIX In [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Generalization increase ranks of attractors matrix: entropy rank as a function of the bottleneck dimension (left) and during training, as a function of the number of epochs (right). Transitioning from memorization to generalization as underfitting (over-regularized) r…
Figure 14
Figure 14. Figure 14: Strong overparametrization favors memorization: Memorization coefficient as a function of the dataset size on CIFAR,MNIST,FashionMNIST, showing that networks trained on less data (strongly overparametrized) tend to memorize more the training data. B.7 DATA-FREE WEIGHT…
Figure 15
Figure 15. Figure 15: Data-free weight probing Stable diffusion XL AE : the results on the larger version of the table diffusion autoencoder confirm that attractors from noise form an informative dictionary of signals to reconstruct different datasets. To assess how the latent vector field…
Figure 16
Figure 16. Figure 16: SSL models induce well-behaved latent vector fields: For latent space iterations in DINOv2 and SigLIP2, we plot norm of residuals in output space (a);latent residuals norms (b); and norms of embeddings across iterations (c). B.8.2 LATENT VECTORS FIELDS IN LLMS We now …
Figure 17
Figure 17. Figure 17: LLM models can induce well-behaved latent vector fields: For latent space iterations in Qwen3-0.6B and SmolLM2-1.7b, we plot the latent residuals norms(a); and norms of embeddings across iterations (b). B.9 QUALITATIVE VISUALIZATION OF ATTRACTORS In this section we pr…
Figure 18
Figure 18. Figure 18: Visualization of decoded attractors from Stable Diffusion AE. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 25 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gated autoencoders with tied input weights

    Droniou Alain and Sigaud Olivier. Gated autoencoders with tied input weights. In International Conference on Machine Learning, pages 154--162. PMLR, 2013

    work page 2013

  3. [3]

    What regularized auto-encoders learn from the data-generating distribution

    Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15 0 (1): 0 3563--3593, 2014

    2014

  4. [4]

    Smollm2: When smol goes big--data-centric training of a small language model

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Mart \' n Bl \'a zquez, Guilherme Penedo, Lewis Tunstall, Andr \'e s Marafioti, Hynek Kydl \' c ek, Agust \' n Piqueres Lajar \' n, Vaibhav Srivastav, et al. Smollm2: When smol goes big--data-centric training of a small language model. arXiv preprint arXiv:2502.02737, 2025

    arXiv 2025

  5. [5]

    A closer look at memorization in deep networks

    Devansh Arpit, Stanis aw Jastrz e bski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233--242. PMLR, 2017

    work page 2017

  6. [6]

    Deep equilibrium models

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in neural information processing systems, 32, 2019

    2019

  7. [7]

    Representation learning: A review and new perspectives

    Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35 0 (8): 0 1798--1828, 2013

    2013

  8. [8]

    Nonlinear power method for computing eigenvectors of proximal operators and neural networks

    Leon Bungert, Ester Hait-Fraenkel, Nicolas Papadakis, and Guy Gilboa. Nonlinear power method for computing eigenvectors of proximal operators and neural networks. SIAM Journal on Imaging Sciences, 14 0 (3): 0 1114--1148, 2021

    work page 2021

Show all 72 references
  1. [9]

    Neural ordinary differential equations

    Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018

    2018

  2. [10]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606--3613, 2014

    2014

  3. [11]

    Why do we need weight decay in modern deep learning? Advances in Neural Information Processing Systems, 37: 0 23191--23223, 2024

    Francesco D'Angelo, Maksym Andriushchenko, Aditya Vardhan Varre, and Nicolas Flammarion. Why do we need weight decay in modern deep learning? Advances in Neural Information Processing Systems, 37: 0 23191--23223, 2024

    2024

  4. [12]

    Investigating data memorization in 3d latent diffusion models for medical image synthesis

    Salman Ul Hassan Dar, Arman Ghanaat, Jannik Kahmann, Isabelle Ayx, Theano Papavassiliu, Stefan O Schoenberg, and Sandy Engelhardt. Investigating data memorization in 3d latent diffusion models for medical image synthesis. In International Conference on Medical Image Computing ...

    2023

  5. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248--255. Ieee, 2009

    2009

  6. [14]

    Nonlinear spectral geometry processing via the tv transform

    Marco Fumero, Michael M \"o ller, and Emanuele Rodol \`a . Nonlinear spectral geometry processing via the tv transform. ACM Transactions on Graphics (TOG), 39 0 (6): 0 1--16, 2020

    2020

  7. [15]

    Latent functional maps: a spectral framework for representation alignment

    Marco Fumero, Marco Pegoraro, Valentino Maiorca, Francesco Locatello, and Emanuele Rodol\` a . Latent functional maps: a spectral framework for representation alignment. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in ...

    2024

  8. [16]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  9. [17]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupr \'e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024

    2024 arXiv

  10. [18]

    Nonlinear spectral analysis via one-homogeneous functionals: overview and future prospects

    Guy Gilboa, Michael Moeller, and Martin Burger. Nonlinear spectral analysis via one-homogeneous functionals: overview and future prospects. Journal of Mathematical Imaging and Vision, 56: 0 300--319, 2016

    2016

  11. [19]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249--256. JMLR Workshop and Conference Proceedings, 2010

    2010

  12. [20]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026--1034, 2015

    2015

  13. [21]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    2016

  14. [22]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll \'a r, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000--16009, 2022

    2022

  15. [23]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12 0 (7): 0 2217--2226, 2019

    2019

  16. [24]

    Neural networks and physical systems with emergent collective computational abilities

    John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79 0 (8): 0 2554--2558, 1982

    1982

  17. [25]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 0 (2): 0 3, 2022

    2022

  18. [26]

    The platonic representation hypothesis

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024

    2024 arXiv

  19. [27]

    Associative memory in iterated overparameterized sigmoid autoencoders

    Yibo Jiang and Cengiz Pehlevan. Associative memory in iterated overparameterized sigmoid autoencoders. In International conference on machine learning, pages 4828--4838. PMLR, 2020

    2020

  20. [28]

    Highly accurate protein structure prediction with alphafold

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Z \' dek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596 0 (7873): 0 583--589, 2021

    2021

  21. [29]

    Generalization in diffusion models arises from geometry-adaptive harmonic representations

    Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and St \'e phane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations. arXiv preprint arXiv:2310.02557, 2023

    2023 arXiv

  22. [30]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  23. [31]

    Auto-encoding variational bayes, 2013

    Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013

    2013

  24. [32]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. UofT, 2009

    2009

  25. [33]

    A simple weight decay can improve generalization

    Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991

    1991

  26. [34]

    Efficient backprop

    Yann LeCun, L \'e on Bottou, Genevieve B Orr, and Klaus-Robert M \"u ller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9--50. Springer, 2002

    2002

  27. [35]

    1399 h&e-stained sentinel lymph node sections of breast cancer patients: the camelyon dataset

    Geert Litjens, Peter Bandi, Babak Ehteshami Bejnordi, Oscar Geessink, Maschenka Balkenhol, Peter Bult, Altuna Halilovic, Meyke Hermsen, Rob Van de Loo, Rob Vogels, et al. 1399 h&e-stained sentinel lymph node sections of breast cancer patients: the camelyon dataset. GigaScience...

    2018

  28. [36]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  29. [37]

    Mallat and Zhifeng Zhang

    S.G. Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. IEEE Transactions on Signal Processing, 41 0 (12): 0 3397--3415, 1993. doi:10.1109/78.258082

    1993 doi

  30. [38]

    An empirical bayes estimator of the mean of a normal population

    Koichi Miyasawa et al. An empirical bayes estimator of the mean of a normal population. Bull. Inst. Internat. Statist, 38 0 (181-188): 0 1--2, 1961

    1961

  31. [39]

    Relative representations enable zero-shot latent space communication

    Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodol \`a . Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430, 2022

    2022 arXiv

  32. [40]

    Deep double descent: Where bigger models and more data hurt

    Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 0 (12): 0 124003, 2021

    2021

  33. [41]

    Sparse autoencoder

    Andrew Ng et al. Sparse autoencoder. CS294A Lecture notes, 72 0 (2011): 0 1--19, 2011

    2011

  34. [42]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth \'e e Darcet, Th \'e o Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  35. [43]

    Pytorch: An imperative style, high-performance deep learning library, 2019

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fan...

    2019 arXiv

  36. [44]

    Exponential expressivity in deep neural networks through transient chaos

    Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016

    2016

  37. [45]

    Grokking: Generalization beyond overfitting on small algorithmic datasets

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022

    2022 arXiv

  38. [46]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  39. [47]

    Overparameterized neural networks implement associative memory

    Adityanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Overparameterized neural networks implement associative memory. Proceedings of the National Academy of Sciences, 117 0 (44): 0 27162--27170, 2020

    2020

  40. [48]

    Hopfield networks is all you need

    Hubert Ramsauer, Bernhard Sch \"a fl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlovi \'c , Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.02217, 2020

    2008 arXiv

  41. [49]

    Contractive auto-encoders: Explicit invariance during feature extraction

    Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th international conference on international conference on machine learning, pages 833--840, 2011

    2011

  42. [50]

    An empirical bayes approach to statistics

    Herbert E Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pages 388--394. Springer, 1992

    1992

  43. [51]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684--10695, 2022

    2022

  44. [52]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Informa...

    2022

  45. [53]

    Diffusion art or digital forgery? investigating data replication in diffusion models

    Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6048--6058, 2023

    2023

  46. [54]

    How to train your energy-based models

    Yang Song and Diederik P Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288, 2021

    2021 arXiv

  47. [55]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  48. [56]

    Out-of-distribution detection with deep nearest neighbors

    Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International conference on machine learning, pages 20827--20840. PMLR, 2022

    2022

  49. [57]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and ...

    2025 arXiv

  50. [58]

    The inaturalist species classification and detection dataset

    Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8...

    2018

  51. [59]

    A connection between score matching and denoising autoencoders

    Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23 0 (7): 0 1661--1674, 2011

    2011

  52. [60]

    Extracting and composing robust features with denoising autoencoders

    Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096--1103, 2008

    2008

  53. [61]

    Sun database: Exploring a large collection of scene categories

    Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision, 119: 0 3--22, 2016

    2016

  54. [62]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  55. [63]

    Generalized out-of-distribution detection: A survey

    Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. International Journal of Computer Vision, 132 0 (12): 0 5635--5662, 2024

    2024

  56. [64]

    Deep structured energy based models for anomaly detection

    Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. In International conference on machine learning, pages 1100--1109. PMLR, 2016

    2016

  57. [65]

    Identity crisis: Memorization and generalization under extreme overparameterization

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C Mozer, and Yoram Singer. Identity crisis: Memorization and generalization under extreme overparameterization. arXiv preprint arXiv:1902.04698, 2019

    1902 arXiv

  58. [66]

    Understanding deep learning (still) requires rethinking generalization

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64 0 (3): 0 107--115, 2021

    2021

  59. [67]

    Diffusion transformers with representation autoencoders

    Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders. arXiv preprint arXiv:2510.11690, 2025

    2025 arXiv

  60. [68]

    Anomaly detection with robust deep autoencoders

    Chong Zhou and Randy C Paffenroth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 665--674, 2017

    2017

  61. [69]

    A survey on latent reasoning

    Rui-Jie Zhu, Tianhao Peng, Tianhao Cheng, Xingwei Qu, Jinfa Huang, Dawei Zhu, Hao Wang, Kaiwen Xue, Xuanliang Zhang, Yong Shan, et al. A survey on latent reasoning. arXiv preprint arXiv:2507.06203, 2025

    2025 arXiv

  62. [70]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  63. [71]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  64. [72]

    The encoder E_ _1 maps inputs x p(x) supported on X R ^m to a typically lower-dimensional space Z R ^k , and the decoder D_ _2 reconstructs the input

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2000

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.