A two-stage CNN reconstructs pseudo 6D phase space from 16 x-y images taken at varying rotation angles in the KEK-ATF injector.
Cyclical Learning Rates for Training Neural Networks
5 Pith papers cite this work. Polarity classification is still indexing.
5
Pith papers citing it
abstract
It is known that the learning rate is the most important hyper-parameter to tune for training deep neural networks. This paper describes a new method for setting the learning rate, named cyclical learning rates, which practically eliminates the need to experimentally find the best values and schedule for the global learning rates. Instead of monotonically decreasing the learning rate, this method lets the learning rate cyclically vary between reasonable boundary values. Training with cyclical learning rates instead of fixed values achieves improved classification accuracy without a need to tune and often in fewer iterations. This paper also describes a simple way to estimate "reasonable bounds" -- linearly increasing the learning rate of the network for a few epochs. In addition, cyclical learning rates are demonstrated on the CIFAR-10 and CIFAR-100 datasets with ResNets, Stochastic Depth networks, and DenseNets, and the ImageNet dataset with the AlexNet and GoogLeNet architectures. These are practical tools for everyone who trains neural networks.
representative citing papers
A momentum schedule from critical damping speeds convergence and yields an optimizer-invariant diagnostic for locating and correcting specific underperforming layers in trained networks.
SGDR uses periodic warm restarts of the learning rate in SGD to reach new state-of-the-art error rates of 3.14% on CIFAR-10 and 16.21% on CIFAR-100.
New CAMELS simulations in larger (50 Mpc/h)^3 boxes with 35 varied parameters produce tighter neural-network constraints on model parameters than prior smaller-volume runs, with public data release.
Staged factorial screening recovers stable early penalties from total batch, depth, and width in 2-10 minute pretraining runs and supports a bridge-centered recommendation through 24-hour continuations on two hosts.
citing papers explorer
-
Two-stage Convolutional Neural Network for pseudo six-dimensional phase space reconstruction
A two-stage CNN reconstructs pseudo 6D phase space from 16 x-y images taken at varying rotation angles in the KEK-ATF injector.
-
Beta-Scheduling: Momentum from Critical Damping as a Diagnostic and Correction Tool for Neural Network Training
A momentum schedule from critical damping speeds convergence and yields an optimizer-invariant diagnostic for locating and correcting specific underperforming layers in trained networks.
-
SGDR: Stochastic Gradient Descent with Warm Restarts
SGDR uses periodic warm restarts of the learning rate in SGD to reach new state-of-the-art error rates of 3.14% on CIFAR-10 and 16.21% on CIFAR-100.
-
Learning the Universe with the 2nd Generation of CAMELS: Varying 35 parameters of the IllustrisTNG model in (50Mpc/h)^3 boxes
New CAMELS simulations in larger (50 Mpc/h)^3 boxes with 35 varied parameters produce tighter neural-network constraints on model parameters than prior smaller-volume runs, with public data release.
-
Staged Factorial Screening for Budget-Constrained Micro-Pretraining
Staged factorial screening recovers stable early penalties from total batch, depth, and width in 2-10 minute pretraining runs and supports a bridge-centered recommendation through 24-hour continuations on two hosts.