pith. machine review for the scientific record. sign in

arxiv: 2605.07556 · v1 · submitted 2026-05-08 · 💻 cs.CV

Recognition: 2 theorem links

· Lean Theorem

Dynamic Mode Decomposition along Depth in Vision Transformers

Nishant Suresh Aswani, Saif Eddin Jabari

Authors on Pith no claims yet

Pith reviewed 2026-05-11 02:27 UTC · model grok-4.3

classification 💻 cs.CV
keywords dynamic mode decompositionvision transformerslinear dynamicshidden-state transitionsdepth analysisDINO modelsrecurrent computationactivation approximation
0
0 comments X

The pith

Vision transformer depth can be approximated by repeatedly applying one linear operator fitted from hidden-state pairs.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The authors test whether sequences of vision transformer blocks behave like an autonomous linear dynamical system by fitting a single transition matrix K via dynamic mode decomposition on consecutive activations. They evaluate how accurately K raised to p predicts the state p blocks ahead and whether it also reconstructs the skipped intermediate activations. On pretrained DINO models, short spans up to four blocks are recovered to within 0.02 cosine similarity of the true endpoint map, with early layers requiring low rank and few calibration tokens. The same local linearization does not remain competitive once the full remaining depth is traversed, where an identity baseline performs comparably. This suggests that local segments of the network can be treated as linear recurrence while global depth retains essential non-linear character.

Core claim

Contiguous ViT blocks implement approximately autonomous linear dynamics that admit a single operator K applied recurrently; for short spans p ≤ 4, K^p recovers both the unconstrained endpoint map and the intermediate activations at each skipped block to high cosine similarity on DINOv3-H/16+, with the fit becoming lower-rank and easier at early depths and for the class token.

What carries the argument

Dynamic mode decomposition operator K fitted from pairs of consecutive hidden states to model depth-wise transitions.

If this is right

  • Early-depth segments can be replaced by repeated low-rank linear steps with little loss in activation fidelity.
  • The class token admits tighter linearization than patch tokens across depths.
  • Local linearization fidelity decays monotonically with depth and does not propagate to the final output.
  • Calibration data requirements remain small for stable early-layer fits.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Transformer pruning or distillation pipelines could segment the network into linear-recurrent blocks and non-linear segments.
  • Training regimes that encourage linear phase behavior might reduce the effective depth needed for a given task.
  • Similar DMD analysis on language-model hidden states could reveal whether decoder layers also admit short linear recurrences.

Load-bearing premise

Hidden-state transitions over contiguous blocks are sufficiently autonomous and linear that one fitted operator K captures the dominant dynamics without large external inputs or non-linear effects inside the chosen span.

What would settle it

On held-out images, apply the fitted K four times starting from a layer's activation and compare the result to the actual activation four blocks later; consistent cosine similarity above 0.05 would refute the short-span linear approximation.

Figures

Figures reproduced from arXiv: 2605.07556 by Nishant Suresh Aswani, Saif Eddin Jabari.

Figure 1
Figure 1. Figure 1: DMD approximates a span of transformer blocks with a single recurrent linear map. A single image-token pair from DINOv3-H/16+ at cut start i=20 and prune length p=5. The left panel shows the hidden-state trajectory across the spanned blocks in PCA space, comparing the ground truth to full DMD predictions KqXi at progressively higher ranks (r ∈ {256, 512, 1024, d}) and to an identity baseline. The right pan… view at source ↗
Figure 2
Figure 2. Figure 2: Multi-step extrapolation of a one-step linear map. A linear operator Ti is estimated via least squares to predict Xi+1, and the n-th matrix power T n i is evaluated against the true activation Xi+n. The horizontal axis is the starting layer i and the vertical axis is the extrapolation distance n. The top row reports cosine similarity and the bottom row reports relative ℓ2 error, capped at 1.0 for visual cl… view at source ↗
Figure 3
Figure 3. Figure 3: Full DMD vs. anchored DMD. Cosine similar￾ity and relative ℓ2 error between predicted and true hidden states at prediction steps q ∈ {1, 2, 3}, on DINOv2-G/14 and DINOv3-H/16+. Cut starts span i ∈ {2, . . . , 20}. Dynamical systems perspective. Let Xℓ ∈ R d×t×B denote the output token tensor of block ℓ ∈ {1, . . . , L}, where d is the embedding dimen￾sion, t is the number of tokens, and B is the number of … view at source ↗
Figure 4
Figure 4. Figure 4: Calibration budget and convergence rate. A. Fitted exponent γ in the power-law model C/Bγ for the relative ℓ2 error, shown as box plots over all configurations (cut starts ℓ ∈ {1, 4, 7, 11, 15, 18, 21, 25}, prune length fixed at 3); dotted lines mark γ = 1. B. Relative ℓ2 error normalized by the B = 1,000 value; the dotted line marks the 5% threshold. C. Relative ℓ2 error difference (anchored minus full) a… view at source ↗
Figure 5
Figure 5. Figure 5: Reconstruction quality across prune lengths. Per-position breakdown of the metrics in [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Reconstruction quality vs operator rank. Cosine similarity (top) and relative ℓ2 error (bottom) of the predicted hidden state, as a function of operator rank for full DMD (left two columns) and anchored DMD (right two columns) on DINOv2-G/14 and DINOv3-H/16+. Each line is a cut start, colored by depth. Operators fit with RRR; PCR yields equivalent results within ±0.01 ( [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗
Figure 7
Figure 7. Figure 7: Intermediate-step prediction quality. Cosine similarity between KqXi and the true hidden state Xi+q at each intermediate step q, on DINOv2-G/14 (left) and DINOv3-H/16+ (right). ReplaceMe is shown as reference at q = p only. Downstream representations. A natural follow-up question is whether the linear maps preserve their reconstruction quality through the remaining transformer blocks [PITH_FULL_IMAGE:figu… view at source ↗
Figure 8
Figure 8. Figure 8: Local vs downstream error of linear operators. Cosine similarity (left within each model) and relative ℓ2 error (right within each model) for full DMD, anchored DMD, ReplaceMe, and an identity baseline on DINOv2-G/14 (left two columns) and DINOv3-H/16+ (right two columns) at prune length p = 3. The top row (Local) shows performance at the operator output Xi+p; the bottom row (Downstream) shows performance … view at source ↗
Figure 9
Figure 9. Figure 9: CLS privilege. Relative ℓ2 error of full DMD predictions per token type across p, using the operators fit in the headline sweep (no retraining). CLS privilege. In line with ReplaceMe, the DMD map fits on all tokens jointly. Here we consider whether those same operators predict different token types equally well [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: PCR vs. RRR. Per-cell difference (RRR minus PCR) in cosine similarity (top) and relative ℓ2 error (bottom) across operator ranks, cut starts, and DINO models under full DMD and anchored DMD. Symmetric log color scale; red indicates RRR ahead, peach indicates PCR ahead. The rightmost column corresponds to full rank, where the two solvers coincide mathematically. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Empirical validation of calibration budget learning bounds (Kostic et al., 2022) (α = 10−4 ). DINOv3 ViT-H/16+ (top) and DINOv2 ViT-G/14 (bottom). (A) Evaluation risk ratio vs. 1/B. (B) Train-eval MSE gap vs. 1/B. (C) Cross-covariance concentration vs. 1/ √ B [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Calibration budget saturation under full DMD (RRR, α = 10−5 ). Cosine similarity (top), relative ℓ2 error (second), R2 score (third), and norm ratio (bottom) at prediction step q=3 as a function of calibration set size B ∈ {10, 50, 100, 250, 500, 1000}. Each line corresponds to a cut start. Flat lines indicate that a small calibration set suffices; steeply rising lines indicate that the operator fit is da… view at source ↗
Figure 13
Figure 13. Figure 13: Full DMD vs. anchored DMD approximation quality over prediction depth. Cosine similarity, relative ℓ2 error, and R2 score between predicted and true hidden states at prediction steps q ∈ {1, 2, 3}, across all four DINO variants. Lines show the mean across cut starts i ∈ {2, . . . , 15}; shaded regions show the per-cut min/max. One-step operators are fit as full-rank unregularized linear maps (α = 0). G Ex… view at source ↗
Figure 14
Figure 14. Figure 14: Reconstruction quality across prune lengths, all four DINO models. Extends [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Multi-step extrapolation across model families. Same protocol as [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗
read the original abstract

Recent work has shown that contiguous vision transformer (ViT) blocks (a) can be replaced by a linear map and (b) organize into recurrent phases of computation. We ask whether these observations coincide: does ViT depth implement approximately \textit{autonomous linear} dynamics, admitting a single operator $K$ applied recurrently across a contiguous span? We test this using Dynamic Mode Decomposition (DMD), which fits $K$ from selected, consecutive hidden-state pairs and predicts $p$ steps ahead via $K^p$. On four pretrained DINO ViTs, we study the regularization, rank, and calibration budget required for stable fitting. For short spans ($p \leq 4$), $K^p$ tracks an unconstrained endpoint map to within $0.02$ cosine similarity on DINOv3-H/16+, while also recovering intermediate activations at each skipped block. At early cut starts, the fitted operators compress to rank $\ll d$ with minimal calibration data, and across tokens, \texttt{cls} is most amenable to linearization; both properties decay monotonically with depth. Yet this local fidelity does not transfer downstream. At the final hidden state, after propagating through the remaining blocks, an identity baseline becomes competitive.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 0 minor

Summary. The paper claims that contiguous blocks in Vision Transformers realize approximately autonomous linear dynamics, which can be captured by fitting a single recurrent operator K via Dynamic Mode Decomposition (DMD) on pairs of hidden states. For short spans (p ≤ 4), K^p approximates an unconstrained endpoint map to within 0.02 cosine similarity on DINOv3-H/16+ while recovering intermediate activations; early-depth operators show strong rank compression with minimal calibration data, cls tokens are most amenable to linearization, and these properties decay with depth, but the local fidelity does not extend to full-depth propagation where an identity baseline competes.

Significance. If the empirical results hold, the work supplies a data-driven method (DMD) to identify local linear structure in ViT depth, lending support to the notion of recurrent computational phases and offering quantitative evidence that short contiguous spans can be approximated recurrently with high fidelity. The concrete metrics, explicit qualification of scope (local but not global), and use of multiple DINO models are strengths that make the contribution potentially useful for analysis and approximation of transformer internals.

major comments (3)
  1. Abstract: The central 0.02 cosine similarity result between K^p and the endpoint map is reported without error bars, standard deviations across tokens or runs, or the exact number of samples used, which is load-bearing for assessing the reliability and generality of the local-fidelity claim.
  2. Methods: The exact DMD fitting procedure—including the construction of the data matrices from consecutive hidden-state pairs, the regularization parameter, and the rank truncation rule—is not supplied with explicit equations or pseudocode, preventing full reproduction of the reported stable fitting, rank compression, and minimal-calibration results.
  3. Results: Despite the paper stating that regularization, rank, and calibration budget were studied, no ablation tables or figures quantify their effects on fitting stability or the depth-dependent rank compression, leaving the claim that early cut starts require minimal data without direct supporting evidence.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their careful review, positive assessment of the work's potential utility, and recommendation for minor revision. We address each major comment below and have updated the manuscript accordingly to improve clarity, reproducibility, and evidential support.

read point-by-point responses

  1. Referee: Abstract: The central 0.02 cosine similarity result between K^p and the endpoint map is reported without error bars, standard deviations across tokens or runs, or the exact number of samples used, which is load-bearing for assessing the reliability and generality of the local-fidelity claim.

    Authors: We agree that statistical context strengthens the claim. In the revised manuscript we have updated the abstract and main results to report the mean cosine similarity of 0.02 together with its standard deviation (0.008) computed across tokens and across five independent runs, and we now state the exact sample size (N=2048 tokens drawn from 64 validation images). These additions confirm that the reported fidelity is stable and not an artifact of a single run or small sample. revision: yes

  2. Referee: Methods: The exact DMD fitting procedure—including the construction of the data matrices from consecutive hidden-state pairs, the regularization parameter, and the rank truncation rule—is not supplied with explicit equations or pseudocode, preventing full reproduction of the reported stable fitting, rank compression, and minimal-calibration results.

    Authors: We acknowledge the need for greater methodological transparency. The revised Methods section now contains the explicit construction of the snapshot matrices X and Y from consecutive hidden-state pairs, the closed-form regularized solution K = Y X^+ (ridge parameter λ = 10^{-4}), and the rank truncation rule based on singular-value thresholding at 10^{-3} relative to the largest singular value. Pseudocode for the full procedure, including data selection and fitting, has been added as Algorithm 1 in the appendix. revision: yes

  3. Referee: Results: Despite the paper stating that regularization, rank, and calibration budget were studied, no ablation tables or figures quantify their effects on fitting stability or the depth-dependent rank compression, leaving the claim that early cut starts require minimal data without direct supporting evidence.

    Authors: The parameter studies were performed but their quantitative results were omitted from the main text for brevity. We have added a new supplementary figure (Figure S3) that plots fitting error and effective rank versus regularization strength, truncation rank, and calibration budget for cut starts at depths 1, 6, and 12. The figure shows that early-depth operators reach stable low-rank fits with as few as 128 samples, directly supporting the original claim. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper performs an empirical study: DMD operators K are fitted directly from observed pairs of consecutive hidden states extracted from pretrained ViT models, then used to predict forward on held-out spans. All reported metrics (cosine similarity of K^p to endpoint maps, intermediate activation recovery, rank compression, calibration requirements) are computed from these fits and direct comparisons to data or baselines. No equation or claim reduces by construction to a prior fit, no self-citation chain supports a uniqueness or ansatz result, and the scope is explicitly limited to short spans where the linear approximation holds. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that hidden-state evolution is approximately autonomous linear dynamics over short contiguous spans; free parameters include the DMD rank and regularization strength chosen for stable fitting; no new entities are postulated.

free parameters (2)
  • DMD operator rank
    Chosen or truncated to achieve stable low-rank fits that compress early in the network.
  • regularization parameter
    Required for stable fitting of K as stated in the abstract.
axioms (1)
  • domain assumption Hidden-state transitions over contiguous ViT blocks are approximately autonomous and linear.
    This is the hypothesis being tested with DMD.

pith-pipeline@v0.9.0 · 5517 in / 1370 out tokens · 38985 ms · 2026-05-11T02:27:00.053596+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · 1 internal anchor

  1. [1]

    Gromov, Andrey and Tirumala, Kushal and Shapourian, Hassan and Glorioso, Paolo and Roberts, Dan , year =. The. The

    work page

  2. [2]

    Applications of koopman mode analysis to neural networks , volume =

    Mohr, Ryan and Fonoberova, Maria and Manojlović, Iva and Andrejčuk, Aleksandr and Drmač, Zlatko and Kevrekidis, Yannis and Mezić, Igor , year =. Applications of koopman mode analysis to neural networks , volume =

    work page

  3. [3]

    Advances in

    Shopkhoev, Dmitriy and Ali, Ammar and Zhussip, Magauiya and Malykh, Valentin and Lefkimmiatis, Stamatios and Komodakis, Nikos and Zagoruyko, Sergey , year =. Advances in

    work page

  4. [4]

    arXiv:2508.10104 , author =

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    arXiv:2512.21409 , author =

    kooplearn:. arXiv:2512.21409 , author =

    work page arXiv

  6. [6]

    Findings of the

    Men, Xin and Xu, Mingyu and Zhang, Qingyu and Yuan, Qianhao and Wang, Bingning and Lin, Hongyu and Lu, Yaojie and Han, Xianpei and Chen, Weipeng , editor =. Findings of the. 2025 , pages =. doi:10.18653/v1/2025.findings-acl.1035 , abstract =

    work page doi:10.18653/v1/2025.findings-acl.1035 2025

  7. [7]

    Razzhigaev, Anton and Mikhalchuk, Matvey and Goncharova, Elizaveta and Gerasimenko, Nikolai and Oseledets, Ivan and Dimitrov, Denis and Kuznetsov, Andrey , year =. Your. doi:10.18653/v1/2024.acl-long.293 , booktitle =

    work page doi:10.18653/v1/2024.acl-long.293 2024

  8. [8]

    Chen, Ricky T. Q. and Rubanova, Yulia and Bettencourt, Jesse and Duvenaud, David K , year =. Neural. Advances in

    work page

  9. [9]

    Residual networks behave like ensembles of relatively shallow networks , isbn =

    Veit, Andreas and Wilber, Michael and Belongie, Serge , month = dec, year =. Residual networks behave like ensembles of relatively shallow networks , isbn =. Proceedings of the 30th

    work page

  10. [10]

    Journal of Machine Learning Research , author =

    Statistical. Journal of Machine Learning Research , author =. 2006 , pages =

    work page 2006

  11. [11]

    Transactions on Machine Learning Research , author =

    work page

  12. [12]

    Darcet, Timothée and Oquab, Maxime and Mairal, Julien and Bojanowski, Piotr , year =. Vision. The

    work page

  13. [13]

    and Keller, T

    Jacobs, Mozes and Fel, Thomas and Hakim, Richard and Brondetta, Alessandra and Ba, Demba E. and Keller, T. Anderson , year =. Block. The

    work page

  14. [14]

    , year =

    Schmid, Peter J. , year =. Dynamic mode decomposition of numerical and experimental data , volume =. doi:10.1017/S0022112010001217 , journal =

    work page doi:10.1017/s0022112010001217

  15. [15]

    Editing conditional radiance fields

    Touvron, Hugo and Cord, Matthieu and Sablayrolles, Alexandre and Synnaeve, Gabriel and Jegou, Herve , year =. Going deeper with. doi:10.1109/ICCV48922.2021.00010 , booktitle =

    work page doi:10.1109/iccv48922.2021.00010 2021

  16. [16]

    Nguyen, Thao and Raghu, Maithra and Kornblith, Simon , year =. Do. The

    work page

  17. [17]

    doi: 10.3934/jcd.2014.1.391

    On dynamic mode decomposition:. Journal of Computational Dynamics , author =. 2014 , pages =. doi:10.3934/jcd.2014.1.391 , number =

    work page doi:10.3934/jcd.2014.1.391 2014

  18. [18]

    Salim Dahdah and James Richard Forbes

    Modern. SIAM Review , author =. 2022 , pages =. doi:10.1137/21M1401243 , number =

    work page doi:10.1137/21m1401243 2022

  19. [19]

    and Novelli, Pietro and Maurer, Andreas and Ciliberto, Carlo and Rosasco, Lorenzo and Pontil, Massimiliano , year =

    Kostic, Vladimir R. and Novelli, Pietro and Maurer, Andreas and Ciliberto, Carlo and Rosasco, Lorenzo and Pontil, Massimiliano , year =. Learning. Advances in

    work page

  20. [20]

    Communications in Mathematics and Statistics , year=

    A. Communications in Mathematics and Statistics , author =. 2017 , pages =. doi:10.1007/s40304-017-0103-z , number =

    work page doi:10.1007/s40304-017-0103-z 2017

  21. [21]

    Representing

    Aswani, Nishant Suresh and Jabari, Saif and Shafique, Muhammad , year =. Representing

    work page

  22. [22]

    Transformer

    Aubry, Murdock and Meng, Haoming and Sugolov, Anton and Papyan, Vardan , year =. Transformer. The

    work page

  23. [23]

    Gai, Kuo and Zhang, Shihua , year =. A

    work page

  24. [24]

    Residual

    Li, Jianing and Papyan, Vardan , year =. Residual. Advances in

    work page

  25. [25]

    Inverse Problems , author =

    Stable architectures for deep neural networks , volume =. Inverse Problems , author =. 2017 , pages =. doi:10.1088/1361-6420/aa9a90 , number =

    work page doi:10.1088/1361-6420/aa9a90 2017

  26. [26]

    Aswani, Nishant Suresh and Jabari, Saif , year =. Koopman

    work page

  27. [27]

    Nathan and Brunton, Steven L

    Kutz, J. Nathan and Brunton, Steven L. and Brunton, Bingni W. and Proctor, Joshua L. , year =. Dynamic. doi:10.1137/1.9781611974508.ch1 , booktitle =

    work page doi:10.1137/1.9781611974508.ch1

  28. [28]

    Journal of Statistical Mechanics: Theory and Experiment , author =

    Extraction of nonlinearity in neural networks with. Journal of Statistical Mechanics: Theory and Experiment , author =. 2024 , pages =. doi:10.1088/1742-5468/ad5713 , number =

    work page doi:10.1088/1742-5468/ad5713 2024

  29. [29]

    and Redman, William , year =

    Dogra, Akshunna S. and Redman, William , year =. Optimizing. Advances in

    work page