Pith. sign in

REVIEW 4 cited by

LaProp: Separating Momentum and Adaptivity in Adam

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2002.04839 v3 pith:WAQA4WLR submitted 2020-02-12 cs.LG stat.ML

Liu Ziyin , Zhikang T.Wang , Masahito Ueda This is my paper · ORCID
classification cs.LGstat.ML
keywords adaptivitylapropmomentumadammethodsadam-styleboundcoupling
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We identity a by-far-unrecognized problem of Adam-style optimizers which results from unnecessary coupling between momentum and adaptivity. The coupling leads to instability and divergence when the momentum and adaptivity parameters are mismatched. In this work, we propose a method, Laprop, which decouples momentum and adaptivity in the Adam-style methods. We show that the decoupling leads to greater flexibility in the hyperparameters and allows for a straightforward interpolation between the signed gradient methods and the adaptive gradient methods. We experimentally show that Laprop has consistently improved speed and stability over Adam on a variety of tasks. We also bound the regret of Laprop on a convex problem and show that our bound differs from that of Adam by a key factor, which demonstrates its advantage.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization

    stat.ML 2026-01 conditional novelty 7.0 of 10

    Momentum SGD pays a provable tracking penalty under distribution shift — lag growing as (1−β)⁻¹ and tracking floors worse than vanilla SGD in drift-dominated regimes.

  2. What makes a good feedforward computational graph?

    cs.LG 2025-02 conditional novelty 7.0 of 10

    The authors define mixing time and minimax fidelity for feedforward graphs, use them to design a recursive sparse graph (FS) with polylogarithmic mixing time, and show it matches dense attention on parity and retrieval tasks.

  3. UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction

    cs.IR 2026-07 conditional novelty 6.0 of 10

    UniRank is an open benchmark that standardizes chronological autoregressive supervision, multi-task evaluation, and capacity controls for 15 unified ranking models on five large datasets.

  4. SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

    cs.LG 2026-07 conditional novelty 6.0 of 10

    SOAP and Muon, stabilized by per-step QR eigenbasis updates and KL-Shampoo covariance accumulation, beat AdamW on large-batch LLM pretraining up to 100M-token batches.

Pith tools