REVIEW 3 cited by
Big Batch SGD: Automated Inference using Adaptive Batch Sizes
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We propose alternative "big batch" SGD schemes that adaptively grow the batch size over time to maintain a nearly constant signal-to-noise ratio in the gradient approximation. The resulting methods have similar convergence rates to classical SGD, and do not require convexity of the objective. The high fidelity gradients enable automated learning rate selection and do not require stepsize decay. Big batch methods are thus easily automated and can run with little or no oversight.
Forward citations
Cited by 3 Pith papers
-
Multi-Iteration Stochastic Optimizers
MICE is a multi-iteration control variate estimator for stochastic gradients that exploits correlations between iterates to achieve O(tol^{-1}) complexity in smooth strongly convex problems, outperforming adaptive batch SGD.
-
Scaling Laws for Transfer
Effective data transferred from pre-training to fine-tuning is described by a power law in model parameter count and fine-tuning dataset size, acting like a multiplier on the fine-tuning data.
-
Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism
Adaptive batch size schedules based on the norm test can run with FSDP-distributed Llama-style pretraining, but the reported runs do not consistently beat constant batch sizes and the convergence theorem does not cove...
Discussion (0). Continue with ORCID to comment.