Improving neural networks by preventing co-adaptation of feature detectors
read the original abstract
When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition.
This paper has not been read by Pith yet.
Forward citations
Cited by 29 Pith papers
-
Generative Adversarial Networks
A generative model is trained to match a data distribution by competing in a minimax game against a discriminator, reaching an equilibrium where the generator recovers the true distribution and the discriminator outpu...
-
Deep Residual Learning for Image Recognition
Residual networks reformulate layers to learn residual functions, enabling effective training of up to 152-layer models that achieve 3.57% error on ImageNet and win ILSVRC 2015.
-
Conditional Generative Adversarial Nets
Conditional GANs generate samples matching a given condition by supplying the condition to both generator and discriminator.
-
Adam: A Method for Stochastic Optimization
A first-order stochastic optimizer that maintains bias-corrected exponential moving averages of the gradient and its square, dividing the former by the square root of the latter to set per-parameter step sizes.
-
Simultaneous measurements of $N$-subjettiness observables in jets from gluons and light-flavour quarks, and in decays of boosted W bosons and top quarks
CMS reports a simultaneous measurement of 25 N-subjettiness observables in 1-, 2-, and 3-prong jets, unfolded to stable particles with particle-level correlations for QCD modeling.
-
A Spectral Approach for Learning Spatiotemporal Neural Differential Equations
A spectral neural differential equation learning method is proposed that handles nonlocal spatial interactions on unbounded domains without discretization.
-
FIESTA: Fast IdEntification of State-of-The-Art models using adaptive bandit algorithms
FIESTA uses bandit algorithms to adaptively decide how many seeds and splits to run for each candidate model, focusing effort on promising ones while providing guarantees on selecting the optimal model.
-
Improved Regularization of Convolutional Neural Networks with Cutout
Randomly masking square regions of input images during CNN training yields new state-of-the-art test errors of 2.56% on CIFAR-10, 15.20% on CIFAR-100, and 1.30% on SVHN.
-
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
The paper introduces and compares gradient estimators for stochastic binary neurons, notably a decomposition approach and the straight-through estimator, to support sparse conditional computation in deep networks.
-
OmniISR: A Unified Framework for Centralized and Federated Learning via Intermediate Supervision and Regularization
OmniISR unifies centralized, federated, and hybrid learning by injecting mutual-information supervision and negative-entropy regularization at multiple hidden layers, with supporting convergence and drift bounds.
-
Explicit Dropout: Deterministic Regularization for Transformer Architectures
Explicit dropout reformulates stochastic dropout as deterministic loss penalties for Transformers, matching or exceeding standard performance with independent control per component.
-
Language models recognize dropout and Gaussian noise applied to their activations
Language models detect, localize, and distinguish dropout from Gaussian noise applied to their activations, often with high accuracy.
-
Rotary Masked Autoencoders are Versatile Learners
RoMAE applies rotary positional embeddings to masked autoencoders to enable representation learning and interpolation on continuous positional data across irregular time-series, images, and audio without modality-spec...
-
Open DNN Box by Power Side-Channel Attack
Power side-channel analysis recovers DNN architecture and parameters at 96.5% average accuracy on real embedded devices.
-
Adaptive Weighting Depth-variant Deconvolution of Fluorescence Microscopy Images with Convolutional Neural Network
A CNN predicts depth-variant PSFs for patch-wise deconvolution of fluorescence microscopy images, with adaptive weighting to reduce artifacts, claiming 98.2% accuracy and up to 6.6 dB PSNR gain.
-
Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
Introduces a margin-adaptive confidence ranking method that learns an estimator from simulated diversity and derives margin-dependent generalization bounds for use in fixed-sequence testing of LLM-human agreement.
-
Improving Neutrino Oscillation Measurements through Event Classification
Supervised ML classification of neutrino events by interaction channel prior to energy reconstruction improves accuracy and sensitivity by 10-20% in simulated DUNE analyses while remaining robust to generator mismodeling.
-
Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning
Graph Laplacian interpolating activation replaces softmax in DNNs and improves natural accuracy, robust accuracy, and data efficiency.
-
Defending Adversarial Attacks by Correcting logits
A two-layer network trained on mixed clean and perturbed logits recovers original predictions for a range of adversarial attacks without needing image data.
-
Robust Deepfake Detection: Mitigating Spatial Attention Drift via Calibrated Complementary Ensembles
A multi-stream ensemble using DINOv2 and CLIP backbones trained with extreme degradations achieves stable deepfake detection and fourth place in the NTIRE 2026 challenge.
-
Quantum memory and scrambling from the perspective of a classical neural network
Time-dependent quantum memory oscillates faster than OTOC, does not equilibrate, and is more sensitive to symmetry breaking, as shown by neural-network predictions on helical spin chains.
-
Optimized Sharing of Coefficients in Parallel Filter Banks
A two-stage coefficient grouping algorithm for parallel filter banks that increases sharing and reduces registers, LUTs, and DSP48s by up to 50% on FPGAs.
-
Simple vs complex temporal recurrences for video saliency prediction
Both ConvLSTM and exponential moving average modifications to a static saliency model achieve state-of-the-art video saliency prediction on DHF1K after SALICON pre-training and yield similar maps.
-
Enhancing Gravitational Lens Study with Deep Learning: A Study on Effects of Dropout Regularization
Dropout in an AlexNet-based CNN raises R² to ~0.96 and reduces SIE parameter errors by 60-76% on simulated galaxy-galaxy lens images.
-
Delving Deep into Liver Focal Lesion Detection: A Preliminary Study
Authors outline a preliminary CNN framework for liver lesion detection in CT images that adds image processing, region proposal, registration and classification steps to handle 3D medical data.
-
A Unified Deep Framework for Joint 3D Pose Estimation and Action Recognition from a Single RGB Camera
A multitask framework lifts 2D keypoints to 3D poses via a two-stream network then applies ENAS to model spatio-temporal pose evolution for action recognition on Human3.6M, MSR Action3D and SBU datasets.
-
Voice Pathology Detection Using Deep Learning: a Preliminary Study
A CNN plus LSTM model on 64 ms raw audio segments from SVD achieves 68% test accuracy for voice pathology detection, comparable to earlier work with different features.
-
Fashion Retail: Forecasting Demand for New Items
Generalized ML models trained on past sales data forecast demand for new fashion items from their attributes, with experiments across neural architectures and loss functions showing robust performance.
-
Online Hand Gesture Recognition Using 3D Convolutional Neural Networks
Proposes an online hand gesture recognition system using 3D CNNs achieving 98%+ detector accuracy and 90%+ classifier accuracy on Jester, with 37.5% Levenshtein accuracy on a homemade dataset.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.