Replacing softmax with ReLU in Vision Transformers

Jaehoon Lee; Justin Gilmer; Mitchell Wortsman; Simon Kornblith

Replacing softmax with ReLU in Vision Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.08586 v2 pith:LINTUUTQ submitted 2023-09-15 cs.CV cs.LG

Mitchell Wortsman , Jaehoon Lee , Justin Gilmer , Simon Kornblith This is my paper · ORCID

classification cs.CVcs.LG

keywords transformersvisiondegradationrelureplacingsoftmaxwhenaccuracy

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

Previous research observed accuracy degradation when replacing the attention softmax with a point-wise activation such as ReLU. In the context of vision transformers, we find that this degradation is mitigated when dividing by sequence length. Our experiments training small to large vision transformers on ImageNet-21k indicate that ReLU-attention can approach or match the performance of softmax-attention in terms of scaling behavior as a function of compute.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Lightweight Backbone Networks Only Require Adaptive Lightweight Self-Attention Mechanisms
cs.CV 2025-08 conditional novelty 5.0 of 10

LOLViT, a GhostNet-based lightweight backbone using adaptive window attention, reports CNN-like CPU speed with MobileViT-level accuracy.

Pith tools