REVIEW 2 cited by
Information Geometry of Dropout Training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Dropout is one of the most popular regularization techniques in neural network training. Because of its power and simplicity of idea, dropout has been analyzed extensively and many variants have been proposed. In this paper, several properties of dropout are discussed in a unified manner from the viewpoint of information geometry. We showed that dropout flattens the model manifold and that their regularization performance depends on the amount of the curvature. Then, we showed that dropout essentially corresponds to a regularization that depends on the Fisher information, and support this result from numerical experiments. Such a theoretical analysis of the technique from a different perspective is expected to greatly assist in the understanding of neural networks, which are still in their infancy.
Forward citations
Cited by 2 Pith papers
-
NAE: Normalizing AutoEncoder
A conditional surrogate loss that always picks the gradient estimate aligned with the reconstruction loss improves flow autoencoder training and reaches state-of-the-art generative performance on molecules, tabular da...
-
MID-L: Matrix-Interpolated Dropout Layer with Layer-wise Neuron Selection
MID-L is an input-dependent gating layer that interpolates between two transformations via a learned Top-k mask, with claimed efficiency and accuracy benefits.
Discussion (0). Continue with ORCID to comment.