REVIEW 3 cited by
Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural networks have been introduced to deal with different types of problems. However, the main goal of any neural network is to transform the non-linearly separable input data into more linearly separable abstract features using a hierarchy of layers. These layers are combinations of linear and nonlinear functions. The most popular and common non-linearity layers are activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish and Mish. In this paper, a comprehensive overview and survey is presented for AFs in neural networks for deep learning. Different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based are covered. Several characteristics of AFs such as output range, monotonicity, and smoothness are also pointed out. A performance comparison is also performed among 18 state-of-the-art AFs with different networks on different types of data. The insights of AFs are presented to benefit the researchers for doing further research and practitioners to select among different choices. The code used for experimental comparison is released at: \url{https://github.com/shivram1987/ActivationFunctions}.
Forward citations
Cited by 3 Pith papers
-
Enhancing Reinforcement Learning in 3D Environments through Semantic Segmentation: A Case Study in ViZDoom
Semantic-segmentation masks can replace RGB input to ViZDoom RL agents with comparable performance and far lower buffer memory, and adding them as an extra channel improves frag scores.
-
Criticality analysis of nuclear binding energy neural networks
On a two-input nuclear binding energy network, the paper validates ANNFT predictions for variance, kurtosis, and an optimal depth-to-width ratio r*=0.034 under SGD, while adaptive optimizers obscure criticality.
-
FlexAct: Why Learn when you can Pick?
A Gumbel-Softmax router that discretely selects among five fixed activation functions, plus a gradient-norm regularizer, recovers the generating activation on toy regression tasks but never beats the matching fixed ac...
Discussion (0). Continue with ORCID to comment.