REVIEW 5 cited by
ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Sparse computation offers a compelling solution for the inference of Large Language Models (LLMs) in low-resource scenarios by dynamically skipping the computation of inactive neurons. While traditional approaches focus on ReLU-based LLMs, leveraging zeros in activation values, we broaden the scope of sparse LLMs beyond zero activation values. We introduce a general method that defines neuron activation through neuron output magnitudes and a tailored magnitude threshold, demonstrating that non-ReLU LLMs also exhibit sparse activation. To find the most efficient activation function for sparse computation, we propose a systematic framework to examine the sparsity of LLMs from three aspects: the trade-off between sparsity and performance, the predictivity of sparsity, and the hardware affinity. We conduct thorough experiments on LLMs utilizing different activation functions, including ReLU, SwiGLU, ReGLU, and ReLU$^2$. The results indicate that models employing ReLU$^2$ excel across all three evaluation aspects, highlighting its potential as an efficient activation function for sparse LLMs. We will release the code to facilitate future research.
Forward citations
Cited by 5 Pith papers
-
SparSEEty: Extracting Tokens from Sparsity-Exploiting LLM Serving Systems via Deterministic Side Channels
Using page-fault side channels, an attacker can observe which FFN neurons a sparsity-exploiting LLM activates and invert those binary traces to recover prompt and response tokens with BLEU above 0.95.
-
Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
Canon layers—residual 1-d causal convolutions over adjacent tokens—boost synthetic reasoning depth 2-4x, lift NoPE to RoPE level, and bring GLA up to Mamba2/GDN, with qualitative real-world confirmation.
-
BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.
-
Why Neural Network Can Discover Symbolic Structures with Gradient-based Training: An Algebraic and Geometric Foundation for Neurosymbolic Reasoning
This paper proves that under O(d)-equivariant gradient flow, neural network training on reasoning tasks decouples into independent monomial potentials and reduces effective dimensionality, yielding algebraic compositi...
-
SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment
Two new MoE language models and a co-designed inference engine claim 20+ tokens/s CPU decoding under 1-8 GB memory with benchmark scores comparable to much larger models.
Discussion (0). Continue with ORCID to comment.