Pith. sign in

REVIEW 4 major objections 4 minor 35 references

Attention-Only White-Box Transformer via LeJEPA-Based Self-Supervised Pretraining

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Attention-only Transformer matches CRATE with 31% fewer parameters.

desk verdict A clean new equivalence theorem and a promising attention-only architecture, but the white-box interpretation hinges on an unverified gradient approximation that should be checked experimentally before publication. read the letter →

arxiv 2608.04213 v1 pith:YYUNF4CX submitted 2026-08-04 cs.LG

Yang Bai , Linyuan Wang , Haoyang Jiang , Nuolin Sun , Libin Hou , Bin Yan This is my paper · ORCID
classification cs.LG
keywords attention-onlytransformerwhite-boxself-supervisedlearningsparseratereductionADMMunrollingLeJEPASIGReginterpretablearchitecture
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that a fully attention-only Transformer can be derived, not hand-designed, from a single white-box optimization principle—a white-box network being one whose layers are unrolled from a mathematical objective. The key move is to split the sparse rate reduction objective: the expansion term that prevents representation collapse is handed to LeJEPA's isotropic Gaussian regularizer, while the compression and sparsity terms are solved by ADMM and unrolled into layers. The resulting AoT-ADMM block contains only multi-head subspace self-attention, a ReLU proximal step, and a dual-state update, with no ISTA dictionary or MLP. At Base scale it reaches 88.88% on CIFAR-10 and 63.54% on CIFAR-100, within 0.30 and 0.02 points of CRATE, while using about 31% fewer parameters. If the derivation holds, it provides a principled way to build simpler Transformers whose operations still carry optimization meaning.

What carries the argument

The load-bearing mechanism is the three-state ADMM recursion unrolled into a layer: $Z^{\ell+1} = (1-\eta\gamma-\eta\rho)Z^{\ell} + \eta\gamma\,\mathrm{MSSA}(Z^{\ell}|U^{\ell}_{[K]}) + \eta\rho(V^{\ell}-W^{\ell})$, $V^{\ell+1}=\mathrm{ReLU}(Z^{\ell+1}+W^{\ell}-\lambda/\rho)$, and $W^{\ell+1}=W^{\ell}+Z^{\ell+1}-V^{\ell+1}$. Here MSSA (multi-head subspace self-attention) is the operator that compresses representations onto learned subspaces, ReLU is the proximal operator of the sparsity penalty under non-negativity, and $W$ is the scaled ADMM multiplier that enforces consistency between the representation and its sparse proxy. The loss-level half is Theorem 1: with fixed normalized energy, the covariance component of SIGReg and the coding rate have the same isotropic optimum and proportional leading-order gaps, which justifies replacing the expansion term by the LeJEPA regularizer.

What would settle it

Compute the true gradient $\nabla_Z R^c(Z|U_{[K]})$ by automatic differentiation on representations from a trained or mid-training AoT-ADMM, and compare its direction and norm with $\gamma(Z - \mathrm{MSSA}(Z|U_{[K]}))$. If the cosine similarity is near zero or negative, or the norm ratio drifts far from $\gamma$, then Eq. (11) does not hold and the ADMM-to-architecture derivation is not the mechanism carrying the reported performance.

Watch

Extended reading notes

Core claim

The paper's central claim is that the sparse rate reduction objective, split at the right place, yields an attention-only white-box encoder. The split uses a proved correspondence: around the isotropic optimum, SIGReg's covariance component and the coding rate $R(Z)$ share the same maximizer, and their gaps are proportional to leading order, so LeJEPA can absorb the expansion term. The remaining objective $R^c(Z|U_{[K]}) + \lambda\lVert Z\rVert_0$ is minimized by scaled-form ADMM, and unrolling the three updates makes one network layer: MSSA performs the compression step, ReLU performs nonnegative soft-thresholding, and the dual variable tracks the consistency residual. The paper reports that this network matches CRATE's linear-probe accuracy on CIFAR-10 and CIFAR-100 at roughly 31% fewer parameters, outperforms the prior attention-only AoT by 4.68–6.55 points at matched parameter counts, and transfers from ImageNet-1K pretraining with about 32% parameter savings at a small accuracy cost. A control experiment on standard ViTs replaces every MLP with ReLU and removes about 66% of parameters while preserving accuracy under knowledge distillation.

Load-bearing premise

The load-bearing premise is the unproven approximation that the gradient of the compression rate equals $\gamma$ times $(Z - \mathrm{MSSA}(Z|U_{[K]}))$; if that identity fails on representations seen during training, the attention module is not provably executing the compression step and the white-box interpretation loses its mathematical grounding.

Editorial extensions

If this is right

  • A white-box Transformer can remove both the ISTA dictionary and MLP layers without sacrificing representation quality: 31% fewer parameters keeps Base-scale CIFAR accuracy within 0.30 points on CIFAR-10 and 0.02 points on CIFAR-100.
  • The training loss and the forward architecture can originate from the same optimization objective, so the self-supervised paradigm and the white-box derivation are no longer decoupled.
  • Not all attention-only Transformers are equal: at matched parameter counts, the ADMM-derived version beats the subspace-denoising-derived AoT by 4.68–6.55 points, indicating that the optimization principle behind the derivation matters.
  • MLP blocks in standard Vision Transformers may be largely redundant under knowledge distillation: replacing them with ReLU removes about 66% of parameters with accuracy changing by −0.82 points (Tiny) and +0.09 points (Small).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves unexamined whether the learned branch coefficients stay near their ADMM initialization; if they drift far during training, the white-box interpretation would hold at initialization only, and the model would be an attention-only network whose performance comes from learned mixing rather than from unrolled optimization.
  • The same loss-level/architecture-level split could be applied to other white-box objectives (for example, token-statistics or variational rate reduction) and to other covariance-based self-supervised regularizers, since any regularizer that penalizes anisotropic covariance should be able to play the role of the expansion term.
  • A direct test of the derivation's bridge is available: measure the alignment between the true gradient of the compression rate and the attention residual on real representations, which would separate the mathematical grounding of Eq. (11) from the empirical success of the architecture.
  • The ViT control hints at a broader, unproven claim: if MLP blocks can be replaced by a proximal nonlinearity plus distillation in standard architectures, then much of the MLP's learned function may be reproducible by a fixed nonlinearity, which would extend the redundancy observation beyond white-box models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes AoT-ADMM, an attention-only Transformer derived from the sparse rate reduction objective. The global expansion term R(Z) is transferred to the LeJEPA training loss via SIGReg, and the remaining compression and sparsity terms are optimized with ADMM and unrolled into a layer-wise update consisting of a multi-head subspace self-attention (MSSA) compression step, a ReLU proximal sparsity step, and a dual-state recursion. The authors prove in Theorem 1 that, at the covariance level and under a fixed trace/energy constraint, the coding-rate objective and the second-order component of SIGReg share the same isotropic optimum and have proportional leading-order gaps. They report linear-probe accuracies close to CRATE on CIFAR-10 and CIFAR-100 at Tiny/Small/Base scales with about 31% fewer parameters, ImageNet-pretrained transfer results, and comparisons with the prior attention-only AoT model.

Significance. If the claimed derivation is fully supported, the paper would be a valuable contribution: it jointly grounds a self-supervised loss and an attention-only architecture in a common optimization principle, removes the ISTA dictionary and MLP from CRATE, and demonstrates competitive accuracy with substantially fewer parameters. The proof of Theorem 1 in Appendix A is careful and the shared-optimum/local-expansion statement is established rigorously. The empirical comparisons use identical LeJEPA pretraining and report a consistent parameter reduction. The main weakness is that the central architectural derivation relies on an unproven and unverified gradient approximation, Eq. (11), and the deployed model learns branch coefficients rather than running the derived ADMM iteration; these gaps currently separate the empirical architecture from the claimed white-box optimization interpretation.

major comments (4)
  1. [Method, 'Optimizing the Compression and Sparsity Terms', Eq. (11)] The entire derivation of the attention-only architecture rests on the approximation ∇_Z R^c(Z|U_[K]) ≈ γ(Z − MSSA(Z|U_[K])). This identity is imported from CRATE but is not proved and, crucially, is not verified on the representations encountered during training. If the approximation fails on trained representations, then Eq. (14) is not a compression-rate update and the AoT-ADMM layer has no established connection to Eq. (5). The paper should either prove this identity under stated conditions or provide a numerical gradient-alignment check (e.g., comparing ∇_Z R^c(Z|U) against γ(Z − MSSA(Z|U)) across layers and training epochs), and the white-box claim should be tempered until such evidence is supplied.
  2. [Method, Eq. (14) and Eq. (10)] The Z-step in Eq. (14) is a single linearized gradient step, not the solution of the ADMM subproblem in Eq. (10). While this is a standard unrolling heuristic, the paper sometimes states the architecture 'solves' Eq. (5) or 'optimizes' the remaining terms. The one-step approximation means the network can only approximately implement the compression step even under the validity of Eq. (11). The authors should explicitly characterize the approximation error, or at minimum state that the derived network is an approximate unrolled solver rather than an exact ADMM iteration.
  3. [Experiments/Algorithm 1, lines 7-9 and Eq. (18)] The trained model is not the ADMM iteration whose derivation is claimed: the branch coefficients (a_l, b_l, c_l) are learned under a softmax parametrization and the states are RMS-rescaled per layer, as stated in Appendix B. This contradicts the assertion that 'The inner loop is exactly one unrolled ADMM iteration per network layer.' At minimum, the authors should ablate the learned coefficients by fixing them to their ADMM-initialized values and report whether competitive performance survives; without such an ablation, the empirical architecture cannot be said to validate the optimization derivation rather than merely using the ADMM derivation as a parameterization prior.
  4. [Appendix A, Theorem 1 and Eq. (3)] The consistency result in Theorem 1 is established only for the covariance-level component L^(2)_SIGReg under fixed normalized energy, while SIGReg's full statistic also contains L_>2 involving higher-order cumulants. The paper acknowledges this separation is exact only on the Gaussian slice. This is a reasonable motivation for using SIGReg as an expansion surrogate, but the wording 'conceptually equivalent' and the claim that LeJEPA optimizes the expansion term should be softened: the theoretical guarantee concerns the second-order component at the common optimum, not equivalence of the full objectives on non-Gaussian embeddings encountered in training.
minor comments (4)
  1. [Eq. (12)] The definition of MSSA is missing a closing parenthesis in the softmax argument; as written, the expression for MSSA(Z|U_[K]) is incomplete.
  2. [Figure 3 caption] The caption text contains garbled unicode tokens that appear to be placeholders from a rendering pipeline; the figure caption should be cleaned up before publication.
  3. [Abstract/Introduction] The abstract states that LeJEPA assumes an isotropic Gaussian as the optimal embedding distribution; this is paraphrased from Balestriero and LeCun, but the paper should clarify that this is an assumption of the LeJEPA framework rather than a proven property of downstream tasks.
  4. [Experiments, Table 4] The ViT control replaces MLP blocks with ReLU, which is not equivalent to the ADMM-proximal update derived in Eq. (17); the discussion in the conclusion should not imply that this control directly validates the white-box derivation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core equivalence (Theorem 1) is proven independently, the ADMM derivation is a standard unrolling, and the empirical results are genuine; the main dependency, the CRATE gradient approximation in Eq. (11), is an acknowledged external approximation rather than a self-referential reduction.

full rationale

The paper's central claim is that AoT-ADMM is derived from the sparse rate reduction objective by moving the expansion term into the LeJEPA/SIGReg loss and unrolling ADMM on the remaining compression and sparsity terms. The bridge between R(Z) and SIGReg is Theorem 1, which is proved in Appendix A from the definitions of the coding rate and the Epps-Pulley statistic; the shared isotropic optimum and the local proportionality are independent mathematical facts, not assumptions of the conclusion. The ADMM derivation in Eqs. (8)-(17) is a standard linearized ADMM unrolling: the Z-step uses the explicitly labeled approximation grad R^c approx gamma(Z - MSSA(Z|U)) imported from CRATE, the V-step is the proximal operator of the ell_1 norm, and the W-step is the standard dual update. This is an acknowledged external approximation, not a definition that makes the attention operator identical to the objective by construction, so it is a correctness risk rather than circularity. The branch coefficients in Eq. (18) are initialized from the ADMM step and then learned, but the functional form of the architecture is still derived from the unrolled update; learning coefficients is standard unrolling practice, not a fitted parameter renamed as a prediction. The reported accuracies come from genuine experiments against CRATE under the same LeJEPA protocol, so the empirical contribution is self-contained. No load-bearing self-citation or uniqueness argument is present: the cited CRATE and AoT works are external baselines, not the authors' own prior results. Overall, no circular step can be exhibited from the paper's own equations, and the derivation is best assessed on correctness grounds rather than circularity grounds.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central derivation rests on the sparse rate reduction principle, the CRATE gradient approximation, the covariance-level SIGReg correspondence, and the choice to learn the unrolled coefficients. These supporting assumptions are either unproven imports from prior work or modeling choices that weaken the claim that the final network is a faithful unrolling of the stated objective.

free parameters (5)
  • Branch coefficients (a_l, b_l, c_l) = learned per layer, softmax-normalized to unit sum
    Initialized from the ADMM-derived values (1 - eta*gamma - eta*rho, eta*gamma, eta*rho) but freely learned during pretraining, so the deployed forward pass is not strictly the unrolled ADMM update.
  • Subspace bases U_l_[K] = learned per layer
    The MSSA compression subspaces are learned from data at each depth, following CRATE; they are network weights rather than quantities fixed by the derivation.
  • Sparsity threshold tau (lambda/rho) = not reported in main text
    The threshold in the ReLU proximal step is a free hyperparameter with no value given in the paper.
  • SIGReg weight alpha_sig = not reported in main text
    Balances the predictive consistency loss and the SIGReg expansion loss; no value or schedule is reported.
  • ADMM initialization coefficients (eta, gamma, rho) = not reported
    These set the initial branch coefficients through softmax-normalized products, but the underlying scalar values are not disclosed.
assumptions (5)
  • domain assumption Sparse rate reduction with expansion, compression, and sparse coding is the correct principle for representation learning
    The derivation starts from Eq. (1) as the ideal objective, imported from Yu et al. (2023); the paper does not independently justify this principle beyond the prior literature.
  • domain assumption The compression-rate gradient can be approximated by gamma(Z - MSSA(Z|U))
    Eq. (11) is taken from CRATE without proof; it is the bridge that turns the ADMM Z-step into a residual attention update, and its accuracy is not assessed.
  • ad hoc to paper Covariance-level equivalence between SIGReg and R(Z) is sufficient to treat SIGReg as the expansion term
    Theorem 1 shows a shared optimum and local quadratic agreement, but the full SIGReg includes higher-order statistics and the training loss also contains L_pred, neither of which is shown to behave like R(Z).
  • ad hoc to paper One unrolled ADMM iteration per layer with learned coefficients preserves the optimization interpretation
    The forward pass uses a single linearized step per layer and learns the branch coefficients, so after training the network does not actually run ADMM; the white-box interpretation holds only at initialization.
  • standard math L1 relaxation and nonnegativity constraint are valid surrogates for the L0 sparsity term
    Standard sparse recovery practice, cited to Chen, Donoho, and Saunders (1998) and Daubechies et al. (2004).
invented entities (2)
  • Dual state W (ADMM multiplier state)
    purpose: Tracks the consistency residual between the main representation Z and the auxiliary sparse variable V in each unrolled layer
    The dual state is a new recurrent variable in the architecture. It has no falsifiable prediction outside the model; its only evidence is the model's accuracy on the reported benchmarks.
  • AoT-ADMM architecture
    purpose: Attention-only white-box Transformer encoder derived from ADMM unrolling, with MSSA, ReLU proximal step, and dual state
    The architecture itself is the paper's contribution; its empirical support is the reported benchmark accuracy, and no released code or checkpoints provide external verification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Attention-Only White-Box Transformer via LeJEPA-Based Self-Supervised Pretraining." pith.science (2026). https://pith.science/paper/YYUNF4CX

@misc{pith2026260804213,
  author       = {Pith},
  title        = {Pith review of: Attention-Only White-Box Transformer via LeJEPA-Based Self-Supervised Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YYUNF4CX}},
  note         = {Machine review of arXiv:2608.04213}
}
abstract

Existing studies on self-supervised learning for white-box networks typically decouple the derivation of white-box networks via optimization algorithms from self-supervised learning paradigms. In this work, we instead revisit the two components from a joint perspective. The LeJEPA-based self-supervised framework assumes an isotropic Gaussian distribution as the optimal embedding distribution for downstream tasks, which is conceptually equivalent to the expansion term $R(Z)$ in the sparse rate reduction objective guiding white-box Transformer optimization. Building on this observation, we use the LeJEPA self-supervised paradigm to optimize $R(Z)$, and derive the remaining terms $R^{c}(Z\mid U_{[K]})+\lambda\lVert Z\rVert_{0}$ via the alternating direction method of multipliers (ADMM) into an attention-only Transformer that dispenses with the ISTA structure or MLP layers of the original design. Experimental results demonstrate that our attention-only white-box Transformer achieves classification accuracies of $88.88\%$ on CIFAR-10 and $63.54\%$ on CIFAR-100 at the Base scale under the LeJEPA self-supervised paradigm, while the original white-box Transformer CRATE achieves classification accuracies of $89.18\%$ on CIFAR-10 and $63.56\%$ on CIFAR-100. Our model achieves competitive performance while reducing the parameter count by roughly $31\%$. Beyond the white-box setting, we further investigate standard ViTs and find that replacing all MLP blocks with ReLU activations under knowledge distillation removes approximately 66\% of the parameters while preserving competitive accuracy, motivating further investigation into the potential redundancy of MLP modules in standard ViT architectures.

Figures

Figures reproduced from arXiv: 2608.04213 by the authors.

Figure 1
Figure 1. Overview of the proposed white-box self-supervised framework. LeJEPA optimizes the global expansion term, while [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. One AoT-ADMM layer as an unrolled ADMM iteration. MSSA realizes the compression update, the ReLU proximal [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparison of LeJEPA pretraining loss between [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 23 canonical work pages

  1. [1]

    Advances in Neural Information Processing Systems , volume =

    Attention Is All You Need , author =. Advances in Neural Information Processing Systems , volume =

  2. [2]

    Advances in Neural Information Processing Systems , volume =

    Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction , author =. Advances in Neural Information Processing Systems , volume =

    work page

  3. [3]

    Proceedings of the 37th International Conference on Machine Learning , series =

    A Simple Framework for Contrastive Learning of Visual Representations , author =. Proceedings of the 37th International Conference on Machine Learning , series =. 2020 , publisher =

    work page 2020

  4. [4]

    Advances in Neural Information Processing Systems , volume =

    Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning , author =. Advances in Neural Information Processing Systems , volume =

  5. [5]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =

    Emerging Properties in Self-Supervised Vision Transformers , author =. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =

  6. [6]

    Proceedings of the 38th International Conference on Machine Learning , series =

    Barlow Twins: Self-Supervised Learning via Redundancy Reduction , author =. Proceedings of the 38th International Conference on Machine Learning , series =. 2021 , publisher =

    work page 2021

  7. [7]

    Bardes, Adrien and Ponce, Jean and LeCun, Yann , booktitle =

  8. [8]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

    Masked Autoencoders Are Scalable Vision Learners , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

Show all 35 references
  1. [9]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

    Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

  2. [10]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =

    An Empirical Study of Training Self-Supervised Vision Transformers , author =. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =

  3. [11]

    Proceedings of the 38th International Conference on Machine Learning , series =

    Attention Is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth , author =. Proceedings of the 38th International Conference on Machine Learning , series =. 2021 , publisher =

    2021

  4. [12]

    International Conference on Learning Representations , year =

    Simplifying Transformer Blocks , author =. International Conference on Learning Representations , year =

  5. [13]

    Proceedings of the 27th International Conference on Machine Learning , pages =

    Learning Fast Approximations of Sparse Coding , author =. Proceedings of the 27th International Conference on Machine Learning , pages =

  6. [14]

    SIAM Journal on Imaging Sciences , volume =

    A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , author =. SIAM Journal on Imaging Sciences , volume =. 2009 , doi =

    2009

  7. [15]

    SIAM Journal on Scientific Computing , volume =

    Atomic Decomposition by Basis Pursuit , author =. SIAM Journal on Scientific Computing , volume =. 1998 , doi =

    1998

  8. [16]

    Foundations and Trends in Optimization , volume =

    Proximal Algorithms , author =. Foundations and Trends in Optimization , volume =. 2014 , doi =

    2014

  9. [17]

    Communications on Pure and Applied Mathematics , volume =

    An Iterative Thresholding Algorithm for Linear Inverse Problems with a Sparsity Constraint , author =. Communications on Pure and Applied Mathematics , volume =. 2004 , doi =

    2004

  10. [18]

    SIAM Journal on Imaging Sciences , volume =

    An Accelerated Linearized Alternating Direction Method of Multipliers , author =. SIAM Journal on Imaging Sciences , volume =. 2015 , doi =

    2015

  11. [19]

    Mathematical Programming , volume =

    On the Douglas--Rachford Splitting Method and the Proximal Point Algorithm for Maximal Monotone Operators , author =. Mathematical Programming , volume =. 1992 , doi =

    1992

  12. [20]

    arXiv preprint arXiv:1607.06450 , year =

    Layer Normalization , author =. arXiv preprint arXiv:1607.06450 , year =

    arXiv

  13. [21]

    Advances in Neural Information Processing Systems , volume=

    White-Box Transformers via Sparse Rate Reduction , author=. Advances in Neural Information Processing Systems , volume=

  14. [22]

    arXiv preprint arXiv:2405.20299 , year=

    Scaling White-Box Transformers for Vision , author=. arXiv preprint arXiv:2405.20299 , year=

    arXiv

  15. [23]

    arXiv preprint arXiv:2412.17810 , year=

    Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction , author=. arXiv preprint arXiv:2412.17810 , year=

    arXiv

  16. [24]

    arXiv preprint arXiv:2308.16271 , year=

    Emergence of Segmentation with Minimalistic White-Box Transformers , author=. arXiv preprint arXiv:2308.16271 , year=

    arXiv

  17. [25]

    International Conference on Learning Representations , year=

    Masked Completion via Structured Diffusion with White-Box Transformers , author=. International Conference on Learning Representations , year=

  18. [26]

    Tong, Shengbang and Chen, Yubei and Ma, Yi and LeCun, Yann , journal=

  19. [27]

    Balestriero, Randall and LeCun, Yann , journal=

  20. [28]

    Proceedings of the 42nd International Conference on Machine Learning , series=

    Attention-Only Transformers via Unrolled Subspace Denoising , author=. Proceedings of the 42nd International Conference on Machine Learning , series=

  21. [29]

    Foundations and Trends in Machine Learning , volume=

    Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , author=. Foundations and Trends in Machine Learning , volume=

  22. [30]

    Learning Multiple Layers of Features from Tiny Images , author=

  23. [31]

    IEEE Conference on Computer Vision and Pattern Recognition , pages=

    ImageNet: A Large-Scale Hierarchical Image Database , author=. IEEE Conference on Computer Vision and Pattern Recognition , pages=

  24. [32]

    International Conference on Learning Representations , year=

    Decoupled Weight Decay Regularization , author=. International Conference on Learning Representations , year=

  25. [33]

    International Conference on Learning Representations , year=

    An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale , author=. International Conference on Learning Representations , year=

  26. [34]

    International Conference on Machine Learning , pages=

    Training Data-Efficient Image Transformers and Distillation through Attention , author=. International Conference on Machine Learning , pages=

  27. [35]

    SIAM Journal on Scientific Computing , volume =

    Yang, Junfeng and Zhang, Yin , title =. SIAM Journal on Scientific Computing , volume =

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.