Pith. sign in

REVIEW 3 major objections 5 minor 25 references

Rethinking Neural Nonlinearity as Gating

T0 review · 3 major / 5 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Neural nonlinearity is not a post-sum activation but input-conditioned threshold gating through a few affine branches; every standard activation and softmax is an instance of one Threshold Gating primitive.

desk verdict Clean algebraic unification of standard activations (and softmax) as one threshold-gating primitive, with real lossless conversion tables; theorem and hardware claims are secondary framing. read the letter →

arxiv 2607.03148 v1 pith:VIIDYGKL submitted 2026-07-03 cs.LG cs.AI

Muhammad Sabih , Frank Hannig , Jürgen Teich This is my paper
classification cs.LGcs.AI
keywords thresholdgatingactivationfunctionsneuralnonlinearityanalogin-memorycomputingsoftmaxconversionminimalbranchtheoremmodeluniversalapproximators
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the familiar pattern of a linear map followed by an activation is a convention, not a necessity. The same nonlinearity can be realized by inspecting each input and softly (or hard) selecting among a handful of affine branches before the sum. Under this view, ReLU, SiLU, GELU, Sigmoid, Tanh, Hardtanh and even softmax are simply special parameter settings of a single Threshold Gating (TG) primitive. Because the conversion is algebraic or nearly so, pretrained CNNs, transformers, recurrent nets and language models can be rewritten in TG form while preserving accuracy with no fine-tuning. The same primitive also lets networks train from scratch with reduced width or depth, better performance at fixed size, and fewer training steps, and it maps onto a single tunable analog front-end that eases the ADC/DAC bottleneck in in-memory computing.

What carries the argument

The Threshold Gating (TG) primitive yi = ∑_k g_k(x_i)(s_k x_i + c_k), where soft (or hard) gates form a partition of unity and each branch is affine. It carries the argument by supplying closed-form parameters that recover every common activation and an exact per-element Sigmoid form of softmax, plus the Minimal Branch Theorem that links K ∈ {2,3} to trainability.

What would settle it

Take a production-scale model whose activation has several strong inflections, convert it under K=2 TG, and measure a clear accuracy or perplexity collapse that cannot be recovered by calibration; or train a deep net with deliberately multi-inflection activations that remains stable and outperforms its K=2 TG counterpart.

Watch

Extended reading notes

Core claim

Standard neural activations—piecewise-linear or smooth—and softmax are instances of one Threshold Gating primitive in which each input is gated across a small number of affine branches. The conversion leaves the original weights untouched, so pretrained networks across CNNs, transformers, RNNs and LLMs retain their performance without retraining. A Minimal Branch Theorem further asserts that activations useful for deep training need only two or three such branches.

Load-bearing premise

Activations that work for deep training are assumed to be well-behaved—essentially one dominant transition and bounded derivative—so two or three branches always suffice and anything more complex is uninteresting for general use.

Editorial extensions

If this is right

  • Pretrained CNNs, transformers, RNNs and LLMs convert to TG form without retraining while preserving accuracy or perplexity.
  • Training with TG can match or beat baseline accuracy at reduced width or depth, improve metrics at fixed parameter count, and reach the baseline’s converged level in fewer steps.
  • Analog in-memory hardware can implement every standard activation with one tunable Sigmoid front-end, removing the need for bespoke nonlinear circuits and easing ADC/DAC cost.
  • Softmax attention itself becomes a TG instance with data-dependent thresholds, so the entire forward pass of a transformer can sit under the same primitive.
  • Activation choice stops being a discrete architectural decision and becomes continuous per-neuron (or per-channel) parameters inside one family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Digital accelerators could likewise collapse separate kernels and tables for exp, Tanh and the Gaussian CDF into a single parameterized gate evaluator.
  • The distributions of learned gate sharpness, thresholds and branch slopes across layers may serve as a new interpretability signal for how networks route information during training.
  • If the well-behaved restriction is real, architecture search could stop cataloguing exotic multi-inflection activations and optimize only the continuous TG parameters.
  • Hardware clamping of saturation regions can drop K=3 activations to K=2 for free, further simplifying both analog and digital implementations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Threshold Gating (TG) as a universal primitive for neural nonlinearity: each input is softly (or hard) gated into K affine branches, yi = ∑k gk(xi)(sk xi + ck), with the conventional post-MAC activation recovered as a special case. Closed-form identities show that ReLU, LeakyReLU, SiLU, Sigmoid, Tanh, and Hardtanh are exact TG instances (Propositions 1–5, Table 2); GELU is approximate at K=2 and recovered by calibration or K=3; softmax is rewritten via the Bridle identity as constant-branch TG with data-dependent threshold (Eq. 11). Pretrained CNNs, ViTs, LSTMs/GRUs, and LLMs convert without retraining at Δ≈0 for exact activations (Tables 4–5). The authors further introduce TG tiers (TGS/TGW/TGA), a Minimal Branch Theorem linking K∈{2,3} to “well-behaved” activations and trainability (Theorem 8, Corollaries 10–11), training-from-scratch gains in width/depth, WER, and step speedup (Fig. 3), and a conceptual mapping of TG onto analog in-memory circuits.

Significance. If the equivalences hold as stated, the paper offers a clean algebraic unification of the standard activation zoo and of softmax under one parameterized gate-plus-branch form. The lossless (or near-lossless) conversion of pretrained models without weight changes is a concrete, falsifiable result and is supported by exact identities plus end-to-end PPL/accuracy tables. Framing activations as input-conditioned branch selection is useful for interpretability (multiplicative vs constant families) and for hardware: a single tunable Sigmoid front-end is a plausible path toward reducing bespoke activation circuitry in analog IMC. The algebraic conversions and the empirical no-retrain tables are the paper’s strongest assets; the Minimal Branch Theorem and the training/hardware claims are secondary and more provisional.

major comments (3)
  1. [§2.4, Definition 7, Theorem 8, Corollaries 10–11] Definition 7 and Theorem 8 (Minimal Branch Theorem, §2.4): “well-behaved” is defined essentially as continuous, piecewise C², bounded derivative, and one dominant transition—the structural property that makes K=2 TG match asymptotics and a single transition. The claim that anything K=2 misses is “uninteresting for general use” and that Lmax ∝ 1/κ(f)² (Corollary 11) therefore follows for general deep nets is stronger than the supporting analysis. Appendices B–C only approximate E[φ''²] under Hayou et al.’s random-network edge-of-chaos setting via local cubic expansion; this is not a characterization of all trainable architectures. Either narrow the theorem to “TG can represent the standard well-behaved activations with K∈{2,3}” or supply a non-circular criterion and stronger evidence that multi-inflection activations are systematically untrainable in practice.
  2. [§3, Fig. 3, Appendix D] Section 3 / Fig. 3 (expressiveness and training): TGW multiplies branch weights by K. The width/depth reductions are the right control, but the reported gains rest on small-scale settings (ResMLP/ResNet on CIFAR-10; DeepSpeech2 on AN4; limited Mamba medical run) with fixed seeds or 3-seed means and recipe-specific activation LRs / c2 init. Parameter-matched baselines at equal wall-clock and equal optimizer budget, plus at least one larger-scale vision or language run, are needed before claiming general compression, performance, or step-speedup advantages. The family-preservation ablation (Appendix D) is useful but should appear in the main text with error bars.
  3. [Abstract, §5, §7] Abstract and §5 claim that TG “addresses the bottleneck of ADC/DAC” in analog in-memory systems. Section 5 only sketches a differential-pair Sigmoid and a resistive MAC and explicitly defers circuit integration and silicon validation; the Limitations section concedes ADC/DAC may still be required for system integration. Soften the abstract and introduction to “motivates a unified analog front-end” and reserve “addresses the bottleneck” until quantitative area/power estimates or measurements are provided.
minor comments (5)
  1. [Table 2, Proposition 4] Table 2 GELU row lists θ=2 for the K=2 case; the surrounding text and Proposition 4 use θ=0. Clarify which is intended (likely a typesetting error).
  2. [§4] Notation switches between xi (feedforward) and zi (softmax) without a short reminder in §4; a one-line restatement would help.
  3. [Fig. 3(d)] Fig. 3(d) caption is dense; spell out the four model–dataset pairs and the exact definition of “first smoothed crossing” in the main text, not only Appendix D.
  4. [§6] Related Work could more explicitly contrast TG with Maxout and MoE on the algebraic invertibility to a pretrained weight matrix (the paper mentions this for Maxout but could sharpen the distinction).
  5. [Abstract, Table 2] Typos / polish: “Preprint.” left in the header; occasional missing spaces around citations; “PReLU” appears in the abstract but is not given a TG row in Table 2 (LeakyReLU is).

Circularity Check

1 steps flagged · score 2.0 of 10

Mild self-definitional framing in the secondary Minimal Branch Theorem; core TG equivalences and conversions are algebraic identities, not circular.

  1. self definitional [Section 2.4, Definition 7 + Theorem 8 + “Why K=2 suffices” paragraph]
    "We observe empirically that all activations deployed in production architectures (Table 3) share a common structural property: f′ is bounded and has at most one local extremum. ... We call functions satisfying the single-extremum property well-behaved. ... Theorem 8 (Minimal Branch Theorem). Let f be well-behaved ... There exist parameters ... such that K=2 TG matches the asymptotic slopes of f ... Why K=2 suffices. Setting K=2 enforces well-behavedness architecturally ... so anything K=2 misses is uninteresting for general use."

    “Well-behaved” is defined as the structural property (one dominant transition / bounded derivative with at most one local extremum) that makes the K=2 matching construction of Theorem 8 hold by design. The theorem then asserts that K=2 TG is sufficient for such functions, and the surrounding text dismisses non-well-behaved activations as uninteresting for deep training. The class of interest is therefore partly coextensive with the theorem’s hypothesis; the trainability corollaries inherit this definitional alignment rather than independently characterizing all trainable nets.

full rationale

The paper's load-bearing claims—that standard activations (ReLU, SiLU, Sigmoid, Tanh, Hardtanh, approx. GELU) and softmax are exact or near-exact instances of the TG primitive yi = ∑k gk(xi)(sk xi + ck), enabling lossless or near-lossless conversion of pretrained models without retraining—are established by closed-form algebraic reparameterizations (Propositions 1–5, Eq. 11, Table 2) and direct empirical substitution (Tables 4–5). These hold independently of any fitting or self-reference: ReLU/SiLU/Tanh/Sigmoid/Hardtanh convert by identity; softmax is the standard Bridle identity rewritten with data-dependent threshold; GELU residual is quantified (≤0.3% generic, recovered at K=3 or by optional per-layer calibration on held-out pre-activations). No self-citations are load-bearing; Hayou et al. is external. The only mild circularity is secondary: Definition 7 defines “well-behaved” activations precisely by the single-dominant-transition property that makes Theorem 8 (K=2 TG matches asymptotics) true by construction, then dismisses anything K=2 misses as “uninteresting for general use.” This slightly tautological framing of the trainability corollaries does not infect the unification/conversion core, which remains self-contained against external benchmarks. Score 2 reflects that single non-central definitional alignment.

Assumptions & free parameters 4 free parameters · 6 assumptions · 4 invented entities

Central conversion claims rest on standard calculus identities and the Bridle softmax form; the theorem rests on an author-defined well-behaved class plus Hayou-style random-network trainability; hardware claims rest on known analog circuit motifs without new measured entities. Free parameters appear mainly in approximate GELU fits and training recipes, not in exact ReLU/SiLU/softmax identities.

free parameters (4)
  • GELU gate sharpness τ (Hendrycks–Gimpel) = 1.702 (global); per-layer calibrated for ViT
    Conventional global L2 fit τ=1.702 of σ(τx) to Φ(x); Lemma 9 local match gives ≈1.596. Residual drives the only nonzero conversion drift until K=3 calibration.
  • Per-layer GELU K=3 parameters (θ1,θ2,τ,s,c) = per-layer; init (−1,+1,1.8), s=(0,1,1)
    Nine parameters fit by Nelder–Mead on 640 ImageNet val pre-activations to reduce residual to ~0.04% (Appendix F).
  • TGA/TGW training hyperparameters (activation LR scales, c2 init) = recipe-specific
    Appendix D: TGA-LR scale 0.25 on DS2; TG activation-LR scale 4 and c2=−0.05 on Mamba; affect reported speedups and WER.
  • Branch count K and sharing tier (TGS/TGW/TGA) = K∈{2,3}; tier per experiment
    Architectural choices that set expressiveness vs parameter cost; K=2 default, K=3 for saturating/GELU cases.
assumptions (6)
  • standard math Bridle identity: softmax(z)_i = σ(z_i − LSE(z_−i))
    Used for exact TG-softmax conversion (Eq. 11); classical result cited as [2].
  • standard math lim τ→∞ σ(τ(x−θ)) = 1[x>θ] almost everywhere
    Appendix A; justifies hard-gate limit for ReLU/Hardtanh.
  • domain assumption Hayou, Doucet, Rousseau edge-of-chaos trainable depth bound involving E[φ′²]/E[φ″²]
    Corollaries 10–11 map κ(f) to step size and L_max; assumes their random-network Gaussian preactivation setting.
  • ad hoc to paper Well-behaved activation (Def. 7): continuous, piecewise C², bounded derivative, one dominant transition
    Defines the class for which Minimal Branch Theorem asserts K=2 sufficiency; secondary tail inflections and compositions handled by remarks.
  • domain assumption Attributing σ to pre-MAC inputs is network-equivalent to post-MAC activations in multi-layer pipelines
    Section 1–2 notational convention; true for standard feedforward stacking, needs care for residual/branching graphs.
  • domain assumption MOSFET differential pair realizes tunable sigmoid; RRAM crossbar realizes MAC
    Section 5 hardware mapping; standard analog IMC motifs, not measured in this work.
invented entities (4)
  • Threshold Gating (TG) primitive / Threshold-Gated Layers (TGLs) independent evidence
    purpose: Universal elementwise form yi=∑_k g_k(xi)(s_k xi+c_k) that subsumes standard activations and enables conversion and training tiers.
    Core postulated primitive; independent evidence is algebraic match to known activations and empirical conversion tables, not a new physical object.
  • Minimal Branch Theorem (Theorem 8) and activation complexity κ(f)
    purpose: Argue K=2 (or 3) suffices for trainable deep nets and link residual/trainability to |f‴(θ_f)|.
    New theoretical packaging; depends on Def. 7 and Hayou-style analysis rather than external measurement.
  • TG tiers TGS / TGW / TGA
    purpose: Parameter-sharing ladder from per-synapse untied branches to act-only drop-in replacement.
    Engineering taxonomy for experiments in Section 3; not independently evidenced outside this paper’s ablations.
  • Structural families (multiplicative vs constant vs saturating) from (s_k,c_k) independent evidence
    purpose: Interpretability taxonomy and RNN gate family-preservation claim.
    Derived from TG parameters; supported by DS2 ablation where multiplicative gates hurt WER.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Neural Nonlinearity as Gating." pith.science (2026). https://pith.science/paper/VIIDYGKL

@misc{pith2026260703148,
  author       = {Pith},
  title        = {Pith review of: Rethinking Neural Nonlinearity as Gating},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VIIDYGKL}},
  note         = {Machine review of arXiv:2607.03148}
}
read the original abstract

Activation functions are considered an essential primitive for neural nonlinearity, i.e., they enable neural networks to serve as universal approximators. In this paper, we show that this nonlinearity can also be achieved by input-conditioned threshold gating through branches as a universal primitive. We demonstrate that standard activations -- whether piecewise-linear (ReLU, PReLU, Hardtanh) or smooth (SiLU, Sigmoid, Tanh, GELU) -- are in fact instances of a single Threshold Gating (TG) primitive. For softmax, we show that it admits an exact TG conversion via its equivalent per-element Sigmoid form. We then validate these equivalences by converting pretrained networks across CNNs, transformer-based models, and recurrent architectures, preserving model performance without requiring retraining. Threshold Gating also enables training from scratch that goes beyond replacing existing activations, enabling gains in model compression, performance, and shorter training. We also propose a 'Minimal Branch Theorem' which relates the minimum number of required branches in our primitive to the trainability of general deep neural networks. In terms of hardware implementation, TG maps to a unified implementation in the case of analog in-memory systems, addressing the bottleneck of analog-to-digital and digital-to-analog converters (ADC/DAC) that is known to significantly impact power consumption and on-chip area.

Figures

Figures reproduced from arXiv: 2607.03148 by the authors.

Figure 1
Figure 1. Gate sharpness (τ) continuum. where θ is the threshold and τ > 0 controls the soft￾ness: large τ approaches a hard step, small τ blends the branches smoothly, enabling exact replacement of smooth activations like SiLU that the hard indicator cannot. Generalizing to K regions with K−1 thresh￾olds θ1 < ··· < θK−1, each region k carries a soft gate gk(xi) ∈ [0,1] and an affine branch sk · xi + ck . The gates form a par… view at source ↗
Figure 2
Figure 2. Conventional (post-summation σ) vs. gating TG (general primitive, per-synapse branches); the primitive applies to any layer. Eqs. (3) and (5) serve two roles. As a decomposition, the TG primitive reveals the gating structure already im￾plicit in any pretrained acti￾vation σ (Section 2.2; con￾version in Section 2.5). As a learnable activation, the parameters (τ,θ,sk ,ck) be￾come trainable degrees of freedom. 2.2 Stan… view at source ↗
Figure 3
Figure 3. Complexity–expressiveness tradeoff. (a) ResMLP width [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: (a) diff-pair Sigmoid; (b) RRAM MAC. Each operation in the TG primitive has a di￾rect analog counterpart. The tunable Sigmoid στ (x − θ) is realized by a MOSFET differ￾ential pair (Fig. 4a) whose transfer function Vo ∝ tanh gm 2Ib (V + −V −)  produces a Sigmoid with s…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 1 canonical work pages

  1. [1]

    Learning Activation Functions to Improve Deep Neural Networks

    Forest Agostinelli et al. “Learning Activation Functions to Improve Deep Neural Networks”. In: Workshop Track Proceedings of the 3rd International Conference on Learning Representations (ICLR)(San Diego, CA, USA). Ed. by Yoshua Bengio and Yann LeCun. May 7–9, 2015. arXiv: 1412.6830 [cs.NE]

    arXiv 2015

  2. [2]

    Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters

    John S. Bridle. “Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters”. In:Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS)(Denver, CO, USA). Ed. by David S. Touretzky. V ol. 2. Morgan Kaufmann, Nov. 27–30, 1989, pp. 211–217.URL: https : / / pr...

    1989

  3. [3]

    PRIME: A Novel Processing-in-Memory Architecture for Neural Network Computation in ReRAM-Based Main Memory

    Ping Chi et al. “PRIME: A Novel Processing-in-Memory Architecture for Neural Network Computation in ReRAM-Based Main Memory”. In:Proceedings of the 43rd ACM/IEEE Annual International Symposium on Computer Architecture (ISCA)(Seoul, South Korea). IEEE, June 18–22, 2016, pp. 27–39.DOI:10.1109/ISCA.2016.13

    work page doi:10.1109/isca.2016.13 2016

  4. [4]

    Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

    Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. “Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)”. In:Proceedings 4th International Conference on Learning Representations (ICLR)(San Juan, Puerto Rico). Ed. by Yoshua Bengio and Yann LeCun. May 2–4, 2016. arXiv:1511.07289 [cs.LG]

    arXiv 2016

  5. [5]

    Maxout Networks

    Ian J. Goodfellow et al. “Maxout Networks”. In:Proceedings of the 30th International Con- ference on Machine Learning (ICML)(Atlanta, GA, USA). JMLR.org, June 16–21, 2013, pp. 1319–1327.URL:http://proceedings.mlr.press/v28/goodfellow13.html

    2013

  6. [6]

    On the Impact of the Activation func- tion on Deep Neural Networks Training

    Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. “On the Impact of the Activation func- tion on Deep Neural Networks Training”. In:Proceedings of the 36th International Conference on Machine Learning (ICML)(Long Beach, CA, USA). Ed. by Kamalika Chaudhuri and Ruslan Salakhutdinov. PMLR, June 9–15, 2019, pp. 2672–2680.URL: http://proceedings. mlr.press/v...

    2019

  7. [7]

    Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

    Kaiming He et al. “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification”. In:Proceedings of the IEEE International Conference on Computer Vision (ICCV)(Santiago, Chile). IEEE, Dec. 7–13, 2015, pp. 1026–1034.DOI: 10.1109/ ICCV.2015.123

    2015

  8. [8]

    Gaussian Error Linear Units (GELUs)

    Dan Hendrycks and Kevin Gimpel. “Gaussian Error Linear Units (GELUs)”. In:The Comput- ing Research Repository (CoRR)(2016). arXiv:1606.08415 [cs.LG]

    arXiv 2016

Show all 25 references
  1. [9]

    Curvature Tuning: Provable Training- free Model Steering From a Single Parameter

    Leyang Hu, Matteo Gamba, and Randall Balestriero. “Curvature Tuning: Provable Training- free Model Steering From a Single Parameter”. In:The Computing Research Repository (CoRR)(2025). arXiv:2502.07783 [cs.LG]

    2025

  2. [10]

    In-Memory Computing with Resistive Switching Devices

    Daniele Ielmini and H.-S. Philip Wong. “In-Memory Computing with Resistive Switching Devices”. In:Nature Electronics1.6 (2018), pp. 333–343.DOI: 10.1038/s41928- 018- 0092-2

    2018 doi

  3. [11]

    Yann LeCun, Corinna Cortes, and Christopher J. C. Burges.The MNIST Database of Hand- written Digits. 1998.URL:http://yann.lecun.com/exdb/mnist/

    1998

  4. [12]

    Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Models

    Nathan Leroux et al. “Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Models”. In:Nature Computational Science5.9 (2025), pp. 813–824.DOI:10.1038/s43588-025-00854-1

    2025 doi

  5. [13]

    KAN: Kolmogorov-Arnold Networks

    Ziming Liu et al. “KAN: Kolmogorov-Arnold Networks”. In:Proceedings of the Thirteenth International Conference on Learning Representations (ICLR)(Singapore). OpenReview.net, Apr. 24–28, 2025.URL:https://openreview.net/forum?id=Ozo7qJ5vZi

    2025

  6. [14]

    Rectifier Nonlinearities Improve Neural Network Acoustic Models

    Andrew L. Maas, Awni Y . Hannun, and Andrew Y . Ng. “Rectifier Nonlinearities Improve Neural Network Acoustic Models”. In:Proceedings of the ICML Workshop on Deep Learning for Audio, Speech and Language Processing (WDLASL)(Atlanta, GA, USA). June 16, 2013, pp. 1–6

    2013

  7. [15]

    Mish: A Self Regularized Non-Monotonic Activation Function

    Diganta Misra. “Mish: A Self Regularized Non-Monotonic Activation Function”. In:Pro- ceedings of the 31st British Machine Vision Conference (BMVC). BMV A Press, 2020. arXiv: 1908.08681 [cs.LG]. 10

    2020 arXiv

  8. [16]

    Padé Activation Units: End- to-end Learning of Flexible Activation Functions in Deep Networks

    Alejandro Molina, Patrick Schramowski, and Kristian Kersting. “Padé Activation Units: End- to-end Learning of Flexible Activation Functions in Deep Networks”. In:Proceedings of the 8th International Conference on Learning Representations (ICLR)(Addis Ababa, Ethiopia). Open- Re...

    2020

  9. [17]

    Rectified Linear Units Improve Restricted Boltzmann Machines

    Vinod Nair and Geoffrey E. Hinton. “Rectified Linear Units Improve Restricted Boltzmann Machines”. In:Proceedings of the 27th International Conference on Machine Learning (ICML) (Haifa, Israel). Ed. by Johannes Fürnkranz and Thorsten Joachims. Omnipress, June 21–24, 2010, pp. ...

    2010

  10. [18]

    Training and Operation of an Integrated Neuromorphic Network Based on Metal-Oxide Memristors

    Mirko Prezioso et al. “Training and Operation of an Integrated Neuromorphic Network Based on Metal-Oxide Memristors”. In:Nature521.7550 (2015), pp. 61–64.DOI:10.1038/ nature14441

    2015

  11. [19]

    Searching for Activation Functions

    Prajit Ramachandran, Barret Zoph, and Quoc V . Le. “Searching for Activation Functions”. In: Workshop Track Proceedings of the 6th International Conference on Learning Representations (ICLR)(Vancouver, BC, Canada). OpenReview.net, Apr. 30–May 3, 2018.URL: https : //openreview....

    2018

  12. [20]

    ImageNet Large Scale Visual Recognition Challenge

    Olga Russakovsky et al. “ImageNet Large Scale Visual Recognition Challenge”. In:Interna- tional Journal of Computer Vision115.3 (2015), pp. 211–252.DOI: 10.1007/S11263-015- 0816-Y

    2015 doi

  13. [21]

    Memory Devices and Applications for In-Memory Computing

    Abu Sebastian et al. “Memory Devices and Applications for In-Memory Computing”. In: Nature Nanotechnology15.7 (2020), pp. 529–544.DOI:10.1038/s41565-020-0655-z

    2020 doi

  14. [22]

    ISAAC: A Convolutional Neural Network Accelerator with In-Situ Ana- log Arithmetic in Crossbars

    Ali Shafiee et al. “ISAAC: A Convolutional Neural Network Accelerator with In-Situ Ana- log Arithmetic in Crossbars”. In:Proceedings of the 43rd ACM/IEEE Annual International Symposium on Computer Architecture (ISCA)(Seoul, South Korea). IEEE, June 18–22, 2016, pp. 14–26.DOI:1...

    2016 doi

  15. [23]

    GLU Variants Improve Transformer

    Noam Shazeer. “GLU Variants Improve Transformer”. In:The Computing Research Repository (CoRR)(2020). arXiv:2002.05202 [cs.LG]

    2020 arXiv

  16. [24]

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture- of-Experts Layer

    Noam Shazeer et al. “Outrageously Large Neural Networks: The Sparsely-Gated Mixture- of-Experts Layer”. In:Proceedings of the 5th International Conference on Learning Rep- resentations (ICLR)(Toulon, France). OpenReview.net, Apr. 24–26, 2017.URL: https : //openreview.net/forum...

    2017

  17. [25]

    Training Very Deep Net- works

    Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. “Training Very Deep Net- works”. In:Proceedings of the Annual Conference on Advances in Neural Information Pro- cessing Systems (NIPS)(Montreal, Quebec, Canada). Ed. by Corinna Cortes et al. Dec. 7–12, 2015, pp. 237...

    2015

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.