Pith. sign in

REVIEW 5 major objections 5 minor 56 references

Reducing Per-Sample Harm in Stochastic Optimization

T0 review · 5 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Mini-batch updates can hurt individual samples; the paper shows a cheap last-layer reweighting that reduces this harm and improves test accuracy.

desk verdict Novel per-sample harm-reduction framework with a sound vanilla-SGD core and a clever last-layer implicit computation; the headline gain is real but the central proportionality approximation is rough and the experimental reporting lacks variance, so this is promising, not settled. read the letter →

arxiv 2607.16261 v1 pith:GILW2YNB submitted 2026-06-28 cs.LG cs.AIstat.ML

Apostolos Avranas This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords per-sampleharmstochasticoptimizationgradientconflictslast-layerapproximationGrammatrixKhatri-RaofactorizationAdamWgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard optimizers average gradients over a mini-batch and blend them with momentum or adaptive moments; this can actively increase the loss of some samples in the batch. The paper formalizes that per-sample damage as a convex objective—minimize the total positive overlap between the update and each sample's gradient—while staying within a trust region of the optimizer's own update. Because the exact problem is intractable, the paper derives a surrogate in batch space that uses only the final linear layer's per-sample gradients, based on the empirical finding that this layer alone captures the second-order statistics of the full per-sample gradients. The resulting per-sample reweighting of the loss integrates into SGD-momentum and AdamW with small overhead, and improves test accuracy on CIFAR, Tiny-ImageNet, and ImageNet-100, with gains growing as batch size increases.

What carries the argument

The load-bearing object is the per-sample gradient Gram matrix Q = G G^T in batch space, and the argument's hinge is the proportionality Q ≈ α Q_L with the last-layer Gram matrix, which kills the unknown scale α. For a linear layer, the Khatri-Rao (row-wise Kronecker) factorization Q_L = (x x^T + I) ⊙ (g_y g_y^T) computes the surrogate without building per-sample gradients, aided for AdamW by a rank-r SVD of the adaptive scaling. The surrogate is a convex non-smooth program—an ellipsoidal trust region plus a hinge-like harm objective—solved by a first-order primal-dual method whose output weights are folded back into the loss as a per-sample reweighting.

What would settle it

On a model whose last layer is small (e.g., 10 outputs, 64-dim penultimate), compute the full per-sample Gram matrix Q and the last-layer Q_L at batch 1024, then solve the surrogate with Q_L and measure the true full-network harm H(G dθ*) of the resulting update. If the relative effective objective reduction is negative on average—or if replacing Q_L with a random PSD matrix of the same size leaves test accuracy unchanged—the last-layer proportionality is not the causal ingredient the paper claims.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the last linear layer alone reliably captures the second-order statistics of the full per-sample gradients: the full Gram matrix Q is approximately a scalar multiple of the last-layer Gram matrix Q_L (Q ≈ α Q_L), and under vanilla SGD the unknown scale α cancels out of the resulting surrogate problem. This makes it possible to reweight the current batch's losses so that the effective update stays near the standard optimizer update while minimizing the total harm to individual samples, all without ever materializing per-sample gradients. The paper argues this last-layer fidelity is what makes the whole pipeline cheap and representative.

Load-bearing premise

The entire method rests on the assumption that the full per-sample gradient Gram matrix is a scalar multiple of the last-layer Gram matrix; the paper's measurements show only rough agreement (relative error around 0.5), and in the small-head/large-batch regime (ResNet-44, batch 1024) the approximation fails visibly, requiring a shrunken trust region to avoid accuracy loss.

Editorial extensions

If this is right

  • If the last-layer proportionality holds, harm reduction is almost free: the per-step overhead is O(B^2(d_in + d_out)) and does not grow with model depth, so relative cost shrinks as backbones scale.
  • Larger batches, where per-sample gradient conflicts are more frequent, show the largest accuracy gains (e.g., +2.24% for ViT on CIFAR-100 at batch 1024), pointing to a concrete fix for the large-batch generalization gap.
  • Because the reweighted loss is a standard backward pass, the method plugs into SGD-momentum and AdamW without changing the optimizer's state machinery, and composes with distributed training at the cost of two extra synchronization steps.
  • The method accelerates convergence: gains are larger under short schedules (100 vs 300 epochs) and stronger augmentation, suggesting harm reduction is most valuable when the training budget is tight.
  • The same surrogate machinery could be adapted to other optimizers, since the derivation only needs the update rule to be expressed as a sum of a current-batch term and history.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to apply the same last-layer surrogate to language models, where the final linear head is often the largest parameter block; the proportionality assumption may hold even more strongly there.
  • The paper's own failure case (tiny head, huge batch) suggests a natural diagnostic: track the relative Frobenius error ∥Q − αQ_L∥/∥Q∥ online and fall back to including a second layer when it exceeds a threshold.
  • Because the method is equivalent to an adaptive per-sample loss weighting, it could be composed with techniques that already compute per-sample gradients, such as differential privacy clipping, where the reweighting might offset the harm caused by clipped gradients.
  • The REOR distribution (the paper reports 80–95% positive) implies that in a non-trivial fraction of steps the surrogate 'fix' actually increases harm; a variant that only applies the reweighting when REOR is predicted positive could improve robustness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a method to reduce per-sample harm in mini-batch stochastic optimizers. It formulates the parameter update as a constrained convex problem that penalizes per-sample loss increases, proves a representer-type result (Prop. 4.1) that reduces the problem from parameter space to batch space, and then approximates the full per-sample gradient Gram matrix by the last linear layer's Gram matrix under a proportionality assumption (Eq. 6). The surrogate is solved with a Chambolle–Pock primal-dual solver, using Khatri-Rao factorizations to avoid materializing per-sample gradients, and is extended heuristically to SGD with momentum and AdamW. Experiments on CIFAR-10/100, Tiny-ImageNet, and ImageNet-100 report modest accuracy gains with small runtime overhead and include open-source code.

Significance. If the last-layer proportionality finding were robust, the paper would be a useful contribution: it offers a practical, memory-efficient way to incorporate per-sample conflict information into standard optimizers, with a correct vanilla-SGD derivation, a clean complexity analysis, and a REOR sanity check computed against the full per-sample objective. The Khatri-Rao factorization and the α-invariance observation are genuinely elegant. However, the central approximation is only weakly validated, the AdamW/momentum extensions rest on acknowledged heuristics, and one key configuration requires a special-case hyperparameter change. The significance is therefore conditional on substantially stronger empirical and theoretical support for the surrogate.

major comments (5)
  1. [§4.3, Eq. (6), Fig. 1] The paper's central premise is Q≈αQ_{ℓ:L} with exact proportionality, which lets α cancel from the surrogate. Figure 1 reports a best-fit relative Frobenius error of roughly 0.5 for the last layer alone on ResNet-20 and ViT-small; this is a rough approximation, and no error bound or regime characterization is given. The failure in §5.2 (ResNet-44, CIFAR-10, BS=1024), where Q_L is rank-deficient and the authors resort to a setting-specific c=0.1, shows that the approximation can break in a plausible configuration. The method is then applied to ResNet-50 and ConvNeXt-T without any approximation-quality measurement. I ask for (i) a systematic study of the approximation error as a function of batch size, head dimension, and training stage, and (ii) either a theoretical justification or an explicit statement of the regime in which the surrogate is trustworthy.
  2. [§4.3–§4.4, Eqs. (10), (14)] The α-cancellation argument is derived for vanilla SGD with dg=-lr G^T1. For SGD with momentum and for AdamW, dg contains historical terms, so G dg and ||dg||^2 are not proportional to their last-layer counterparts under Eq. (6); replacing them by last-layer quantities is an additional, unstated approximation. For AdamW, Proposition 4.1 does not apply because the update lies in Dκ G^T span rather than G^T span, and the w≥0 and Q_{2,L} constraint choices are ad hoc. Since most of the positive test-accuracy results use AdamW, this gap is load-bearing. The REOR checks in Fig. 1 are only on small models; please provide an end-to-end validation of the surrogate on the architectures actually used in Tables 1–2, or narrow the theoretical claims accordingly.
  3. [§5.2, Table 1] The only configuration that required changing the global hyperparameter c (ResNet-44, CIFAR-10, BS=1024, c=0.1) is also the one with a negative result (-0.22%). This suggests a sensitivity to rank deficiency of the last-layer Gram matrix that is not captured by the proposed safeguards. The paper should either give a principled rule for adapting c when Q_L is singular or ill-conditioned, or report results under a fixed protocol; otherwise the claim of consistent improvements is overstated.
  4. [§5.2, Table 2] BatchNorm and Mixup/Cutmix break the per-sample independence assumption underlying the last-layer quantities. The ResNet-50 BatchNorm row shows a degradation (56.37→56.20), which is acknowledged but not analyzed. Because BatchNorm is standard in modern CNNs, the method's coverage is materially limited. Please either provide a treatment for BatchNorm or state the restriction explicitly in the abstract and conclusion.
  5. [§5, Tables 1–2] The generalization claim rests on small absolute differences (often +0.2–0.3%), but no standard deviations, confidence intervals, or per-seed results are reported for the 5-seed averages, and the ImageNet-100 results are single-seed. The reader cannot assess whether the gains are significant. Please report seed-wise results or error bars.
minor comments (5)
  1. [Eq. (7)] The constraint is written as w^T Q w ≤ c^2||dg||^2, but after the last-layer approximation it should be w^T Q_{ℓ:L} w ≤ c^2||dg_{ℓ:L}||^2; as written it appears to use the full Gram matrix.
  2. [§4.6, §D.1] The Chambolle–Pock step-size condition has inconsistent notation: the text writes τσ||Q1||^2_σ < 1, which should be τσ||Q1||_2^2 < 1 with the spectral norm. Please make the norm notation uniform.
  3. [Appendix A] In the proof of Prop. 4.1, the displacement vector is declared as δ:=dθ−dg∈R^B; it should be in R^Θ (the parameter dimension), not R^B.
  4. [Appendix E, Table 1] The naming 'ViT-c' vs 'ViT-small' is confusing; clarify which architecture appears in Table 1 and which in Figure 1.
  5. [§4.6, Algorithm 5] The approximate projection in Algorithm 5 is called Π_C but is not the true Euclidean projection onto C; this should be stated explicitly in the main text because it means the CP convergence guarantee is not directly inherited.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the surrogate derivation rests on an explicit approximation (Eq. 6) that is empirically validated against the full objective, and the main claims are supported by independent test accuracy.

full rationale

The paper's derivation chain is not circular. The harm objective (Eq. 3) is a first-order, convex formulation; Proposition 4.1 reduces it to batch space (Eq. 5); the layer-wise decomposition is additive; and Eq. (6), Q ≈ αQ_{ℓ:L}, is introduced explicitly as a 'simple approximation' rather than as a consequence of the target. The α-cancellation is algebraic: under vanilla SGD, Gdg = -lr Q1 and ∥dg∥² = lr²1ᵀQ1, so a single scalar α scaling Q also scales all three defining quantities and cancels from objective and constraint. This is not fitting a parameter to the target and then calling it a prediction. The approximation's quality is checked against the full Q computed from materialized per-sample gradients (Figure 1), and the resulting dθ* is evaluated on the original full-G harm objective via REOR, not on the surrogate. The final claims of improved generalization rest on external test accuracy on CIFAR, Tiny-ImageNet, and ImageNet-100, which is independent of the surrogate objective. There are no self-citations or imported uniqueness theorems. The acknowledged limitations—Eq. (6) holds only roughly (relative error ~0.5), the special-case c=0.1 for ResNet-44/CIFAR-10 at B=1024, and the statement that applying Eq. (7) to momentum/AdamW 'introduces a theoretical inconsistency'—are robustness and validity concerns, not circularity. A failed or rough approximation is an empirical weakness, not a reduction of the output to the input.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced; 'harm' is a definition (H(Gdθ)), not a postulated mechanism. The free parameters are algorithm choices (trust-region radius, weight cap, solver settings, SVD rank, safeguards); the axioms are the linearization, the last-layer proportionality, the AdamW β2≈1 approximation, the approximate projection, and the domain assumption that the last-layer Gram matrix is representative.

free parameters (6)
  • c (trust-region radius) = 0.2 (0.1 in one special case)
    Controls the maximum deviation of the harm-reduced update from the standard optimizer update; fixed across most experiments but reduced post hoc for ResNet-44, CIFAR-10, BS=1024 after observing degradation.
  • Sample weight cap t (w≤2) = 2
    Bounds the per-sample reweighting factors; chosen to prevent unreasonably large weights.
  • Solver iterations and warm-up (IPGD, Isolver) = 10, 50
    Number of CP iterations and PGD warm-up steps; chosen as a practical trade-off between runtime and convergence.
  • SVD rank r (AdamW) = 5
    Rank of the low-rank SVD of κL; keeps relative Frobenius error of Q1,L under ~2%.
  • Safeguard EMAs (mom_dg, max_raise_dg, mom_k, max_raise_k) = 0.98, 1.25, 0.98, 1.25
    Hyperparameters for the spike-clipping and condition-number-preconditioning safeguards; chosen by hand.
  • Solver step-size safety factors = 0.5, 0.9
    CP step sizes τ,σ and relaxation θ are scaled by these factors to damp oscillations.
assumptions (5)
  • domain assumption First-order Taylor approximation Li(θ+dθ) ≈ Li(θ) + G_i^T dθ (Eq. 1) is accurate for the allowed updates
    The harm definition and REOR metric both rely on this linearization; the trust region c=0.2 is intended to keep updates small, but no bound is proven.
  • ad hoc to paper Q ≈ α Q_{ℓ:L} for the last layer (Eq. 6) with α cancelling in the surrogate
    Empirically asserted for two small models (Figure 1) with relative Frobenius error ~0.5 for last-layer-only; not derived and no architecture-independent guarantee.
  • domain assumption For AdamW, ṽ_t ≈ v_t and ~κ ≈ κ because β2=0.999 is close to 1
    Standard approximation for Adam-family algorithms; reasonable but not exact, and used to justify the weighted-loss trick in Eq. 13.
  • ad hoc to paper The approximate projection onto C (box clip then radial scaling, Algorithm 5) is a sufficient stand-in for the true Euclidean projection in the CP solver
    Convergence of CP is guaranteed for the exact proximal operator; the paper uses an approximation and relies on empirical stability.
  • domain assumption The last-layer per-sample gradient Gram matrix captures the core second-order statistics of the full network
    This is the paper's central empirical claim; it is validated on two small models only, and the paper's own AdamW/ResNet-44 failure mode shows its limits.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reducing Per-Sample Harm in Stochastic Optimization." pith.science (2026). https://pith.science/paper/GILW2YNB

@misc{pith2026260716261,
  author       = {Pith},
  title        = {Pith review of: Reducing Per-Sample Harm in Stochastic Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GILW2YNB}},
  note         = {Machine review of arXiv:2607.16261}
}
read the original abstract

Modern optimizers combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments. While highly effective, aggregating across the batch and incorporating this history can produce parameter updates that increase the loss of individual samples. We term this effect harm and formalize the parameter update as an optimization problem that explicitly minimizes the conflicting impact of both batch averaging and past optimization state on current data. Because the exact formulation is intractable, we introduce a highly efficient proxy. We first reduce the problem's dimensionality to the batch size, and then drastically cut memory and speed bottlenecks by successfully restricting the optimization to the last linear layer. This hinges on the unexpected finding that this layer alone reliably captures the second-order statistics of the per-sample gradients. The resulting surrogate problem integrates readily into standard optimizers like SGD and AdamW, and can be solved using a small number of GPU-friendly iterations. Crucially, the method exhibits favorable scaling properties, as the relative computational overhead shrinks as the model size or input grows. Experiments on image classification benchmarks confirm reduced per-sample interference and improved generalization.

Figures

Figures reproduced from arXiv: 2607.16261 by the authors.

Figure 1
Figure 1. Approximation quality at epoch 100. Top Row: Relative L2 error of the Gram matrix approximation, ∥Q−αQℓ:L∥F ∥Q∥F , as a function of included layers (lower is better). The solid curve shows the mean, and the two dotted curves show one standard deviation above and below the mean at each point. Bottom Row: Distribution of the relative effective objective reduction (REOR), J(dg)−J(dθ ⋆) J(dg) , on the original problem (… view at source ↗
Figure 2
Figure 2. Solver convergence behavior. Top Row: Relative objec￾tive reduction achieved by the solver (higher is better). Bottom Row: Number of iterations required to reach convergence. 4.7. Full Training Algorithm and Computational Complexity Algorithm 1 details the full proposed training iteration for SGD. The network is decomposed as F(x; θ) = FL(F1:L−1(x; θ1:L−1); θL), where FL represents a stan￾dard linear layer (whose ou… view at source ↗
Figure 3
Figure 3. Approximation quality for ResNet-20 on CIFAR-10 for Batch Size = 128 at different training epochs. Top Row: Relative L2 error of the Gram matrix approximation Q. Bottom Row: Distribution of the relative effective objective reduction (REOR). all layers 10 20 30 last layer 0 0.2 0.4 0.6 0.8 Layer index ℓ Relative L2 Error −0.5 0 0.5 1 0 1 2 3 Rel. Effective Obj. Reduction (REOR) Prob. Density P(REOR > 0) = 87.7% (a) E… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Approximation quality for ResNet-20 on CIFAR-100 for Batch Size = 128 at different training epochs. Top Row: Relative L2 error of the Gram matrix approximation Q. Bottom Row: Distribution of the relative effective objective reduction (REOR). 23 [PITH_FULL_IMAGE:figure…
Figure 5
Figure 5. Figure 5: Approximation quality for ViT-small on CIFAR-10 for Batch Size = 128 at different training epochs. Top Row: Relative L2 error of the Gram matrix approximation Q1. Bottom Row: Distribution of the relative effective objective reduction (REOR). all layers 10 20 30 40 50la…
Figure 6
Figure 6. Figure 6: Approximation quality for ViT-small on CIFAR-100 for Batch Size = 128 at different training epochs. Top Row: Relative L2 error of the Gram matrix approximation Q1. Bottom Row: Distribution of the relative effective objective reduction (REOR). 24 [PITH_FULL_IMAGE:figur…
Figure 7
Figure 7. Figure 7: Relative objective reduction (top) and number of solver iterations (bottom) across different batch sizes for ResNet-44 on CIFAR-10. 0 20k 40k 60k 0 0.2 0.4 0.6 0.8 1 Rel. Solver Obj. Reduction 0 20k 40k 60k 0 10 20 30 40 50 Training Step Solver Iterations (a) Batch Siz…
Figure 8
Figure 8. Figure 8: Relative objective reduction (top) and number of solver iterations (bottom) across different batch sizes for ResNet-44 on CIFAR-100. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: Relative objective reduction (top) and number of solver iterations (bottom) across different batch sizes for ViT-c on CIFAR-10. 0 20k 40k 60k 0 0.2 0.4 0.6 0.8 1 Rel. Solver Obj. Reduction 0 20k 40k 60k 0 10 20 30 40 50 Training Step Solver Iterations (a) Batch Size = …
Figure 10
Figure 10. Figure 10: Relative objective reduction (top) and number of solver iterations (bottom) across different batch sizes for ViT-c on CIFAR-100. H. Generalized Multi-Layer Algorithm and Precision Analysis Algorithm 8 details the generalized training iteration using SGD when expanding…
Figure 11
Figure 11. Figure 11: Efficiency comparison for ResNet-20-gn (accuracy, time per batch, and peak memory across number of blocks). I. Ablation on the Trust-Region Radius In our main experiments, we fixed the trust-region radius hyperparameter to c = 0.2. This choice, along with other algori…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 5 linked inside Pith

  1. [1]

    SIAM Journal on Imaging Sciences , volume=

    A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , author=. SIAM Journal on Imaging Sciences , volume=. 2009 , publisher=

    2009

  2. [2]

    International conference on computational learning theory , pages=

    A generalized representer theorem , author=. International conference on computational learning theory , pages=. 2001 , organization=

    2001

  3. [3]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  4. [4]

    Advances in neural information processing systems , volume=

    Gradient episodic memory for continual learning , author=. Advances in neural information processing systems , volume=

  5. [5]

    Proceedings of the national academy of sciences , volume=

    Overcoming catastrophic forgetting in neural networks , author=. Proceedings of the national academy of sciences , volume=. 2017 , publisher=

    2017

  6. [6]

    Workshop on Multi-Task and Lifelong Reinforcement Learning , year=

    Continual learning with tiny episodic memories , author=. Workshop on Multi-Task and Lifelong Reinforcement Learning , year=

  7. [7]

    Efficient Lifelong Learning with A-

    Arslan Chaudhry and Marc’Aurelio Ranzato and Marcus Rohrbach and Mohamed Elhoseiny , booktitle=. Efficient Lifelong Learning with A-. 2019 , url=

    2019

  8. [8]

    arXiv preprint arXiv:2011.07801 , year=

    Gradient episodic memory with a soft constraint for continual learning , author=. arXiv preprint arXiv:2011.07801 , year=

    arXiv 2011

Show all 56 references
  1. [9]

    Advances in neural information processing systems , volume=

    Gradient surgery for multi-task learning , author=. Advances in neural information processing systems , volume=

  2. [10]

    International conference on artificial intelligence and statistics , pages=

    Orthogonal gradient descent for continual learning , author=. International conference on artificial intelligence and statistics , pages=. 2020 , organization=

    2020

  3. [11]

    International Conference on Learning Representations , year=

    Gradient Projection Memory for Continual Learning , author=. International Conference on Learning Representations , year=

  4. [12]

    Nature Machine Intelligence , volume=

    Continual learning of context-dependent processing in neural networks , author=. Nature Machine Intelligence , volume=. 2019 , publisher=

    2019

  5. [13]

    IEEE Transactions on Audio, Speech and Language Processing , year=

    An empirical study of catastrophic forgetting in large language models during continual fine-tuning , author=. IEEE Transactions on Audio, Speech and Language Processing , year=

  6. [14]

    Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages=

    Gradient similarity surgery in multi-task deep learning , author=. Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages=. 2025 , organization=

    2025

  7. [15]

    Proceedings of the 39th International Conference on Machine Learning , pages =

    Multi-Task Learning as a Bargaining Game , author =. Proceedings of the 39th International Conference on Machine Learning , pages =. 2022 , volume =

    2022

  8. [16]

    Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages=

    Gdod: Effective gradient descent using orthogonal decomposition for multi-task learning , author=. Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages=

  9. [17]

    Advances in Neural Information Processing Systems , volume=

    Conflict-averse gradient descent for multi-task learning , author=. Advances in Neural Information Processing Systems , volume=

  10. [18]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Independent component alignment for multi-task learning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  11. [19]

    International conference on learning representations , year=

    Towards impartial multi-task learning , author=. International conference on learning representations , year=

  12. [20]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Adatask: A task-aware adaptive learning rate approach to multi-task learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  13. [21]

    Advances in Neural Information Processing Systems , volume=

    Famo: Fast adaptive multitask optimization , author=. Advances in Neural Information Processing Systems , volume=

  14. [22]

    arXiv preprint arXiv:1412.6980 , year=

    Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=

    arXiv

  15. [23]

    International Conference on Learning Representations , year=

    Large Batch Optimization for Deep Learning: Training BERT in 76 minutes , author=. International Conference on Learning Representations , year=

  16. [24]

    Advances in neural information processing systems , volume=

    Symbolic discovery of optimization algorithms , author=. Advances in neural information processing systems , volume=

  17. [25]

    URL https://kellerjordan

    Muon: An optimizer for hidden layers in neural networks, 2024 , author=. URL https://kellerjordan. github. io/posts/muon , volume=

    2024

  18. [26]

    International Conference on Machine Learning , pages=

    Adafactor: Adaptive learning rates with sublinear memory cost , author=. International Conference on Machine Learning , pages=. 2018 , organization=

    2018

  19. [27]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Scaling vision transformers , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  20. [28]

    , author=

    Adaptive subgradient methods for online learning and stochastic optimization. , author=. Journal of machine learning research , volume=

  21. [29]

    International Conference on Machine Learning , pages=

    Shampoo: Preconditioned stochastic tensor optimization , author=. International Conference on Machine Learning , pages=. 2018 , organization=

    2018

  22. [30]

    arXiv preprint arXiv:2409.11321 , year=

    Soap: Improving and stabilizing shampoo using adam , author=. arXiv preprint arXiv:2409.11321 , year=

    arXiv

  23. [31]

    Proceedings of the 30th International Conference on Machine Learning , year=

    On the importance of initialization and momentum in deep learning , author=. Proceedings of the 30th International Conference on Machine Learning , year=

  24. [32]

    Doklady Akademii Nauk SSSR , volume=

    A method for solving the convex programming problem with convergence rate O(1/k^2) , author=. Doklady Akademii Nauk SSSR , volume=

  25. [33]

    International Conference on Learning Representations , year=

    On the Variance of the Adaptive Learning Rate and Beyond , author=. International Conference on Learning Representations , year=

  26. [34]

    International Conference on Learning Representations , year=

    Decoupled Weight Decay Regularization , author=. International Conference on Learning Representations , year=

  27. [35]

    arXiv preprint arXiv:2510.00236 , year=

    Per-example gradients: a new frontier for understanding and improving optimizers , author=. arXiv preprint arXiv:2510.00236 , year=

  28. [36]

    Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages=

    Deep learning with differential privacy , author=. Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages=

    2016

  29. [37]

    arXiv preprint arXiv:2109.12298 , year=

    Opacus: User-friendly differential privacy library in PyTorch , author=. arXiv preprint arXiv:2109.12298 , year=

    arXiv

  30. [38]

    International Conference on Learning Representations , year=

    BackPACK: Packing more into Backprop , author=. International Conference on Learning Representations , year=

  31. [39]

    International Conference on Learning Representations , year=

    Large Language Models Can Be Strong Differentially Private Learners , author=. International Conference on Learning Representations , year=

  32. [40]

    ArXiv , year=

    Efficient Per-Example Gradient Computations in Convolutional Neural Networks , author=. ArXiv , year=

  33. [41]

    International Conference on Artificial Intelligence and Statistics , pages=

    Gradient diversity: a key ingredient for scalable distributed learning , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2018 , organization=

    2018

  34. [42]

    International Conference on Machine Learning , pages=

    Differentially private optimization on large model at small cost , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  35. [43]

    Journal of mathematical imaging and vision , volume=

    A first-order primal-dual algorithm for convex problems with applications to imaging , author=. Journal of mathematical imaging and vision , volume=. 2011 , publisher=

    2011

  36. [44]

    Advances in Order Restricted Statistical Inference: Proceedings of the Symposium on Order Restricted Statistical Inference held in Iowa City, Iowa, September 11--13, 1985 , pages=

    A method for finding projections onto the intersection of convex sets in Hilbert spaces , author=. Advances in Order Restricted Statistical Inference: Proceedings of the Symposium on Order Restricted Statistical Inference held in Iowa City, Iowa, September 11--13, 1985 , pages...

    1985

  37. [45]

    Fixed-point algorithms for inverse problems in science and engineering , pages=

    Proximal splitting methods in signal processing , author=. Fixed-point algorithms for inverse problems in science and engineering , pages=. 2011 , publisher=

    2011

  38. [46]

    SIAM review , volume=

    On projection algorithms for solving convex feasibility problems , author=. SIAM review , volume=. 1996 , publisher=

    1996

  39. [47]

    International Conference on Learning Representations , year=

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author=. International Conference on Learning Representations , year=

  40. [48]

    Proceedings of the European conference on computer vision (ECCV) , pages=

    Group normalization , author=. Proceedings of the European conference on computer vision (ECCV) , pages=

  41. [49]

    International conference on machine learning , pages=

    Batch normalization: Accelerating deep network training by reducing internal covariate shift , author=. International conference on machine learning , pages=. 2015 , organization=

    2015

  42. [50]

    2021 , journal =

    omihub777 , title =. 2021 , journal =

    2021

  43. [51]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Deep residual learning for image recognition , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  44. [52]

    Advances in neural information processing systems , volume=

    Train longer, generalize better: closing the generalization gap in large batch training of neural networks , author=. Advances in neural information processing systems , volume=

  45. [53]

    arXiv preprint arXiv:1706.02677 , year=

    Accurate, large minibatch sgd: Training imagenet in 1 hour , author=. arXiv preprint arXiv:1706.02677 , year=

    arXiv

  46. [54]

    Journal of research of the National Bureau of Standards , volume=

    An iteration method for the solution of the eigenvalue problem of linear differential and integral operators , author=. Journal of research of the National Bureau of Standards , volume=

  47. [55]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Liu, Zhuang and Mao, Hanzi and Wu, Chao-Yuan and Feichtenhofer, Christoph and Darrell, Trevor and Xie, Saining , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2022 , pages =

    2022

  48. [56]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages=

    Randaugment: Practical automated data augmentation with a reduced search space , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages=

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.