Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers

Anand Raghunathan; Brucek Khailany; Jacob R. Stevens; Rangharajan Venkatesan; Steve Dai

Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2103.09301 v1 pith:BYVSFC2H submitted 2021-03-16 cs.AR

Jacob R. Stevens , Rangharajan Venkatesan , Steve Dai , Brucek Khailany , Anand Raghunathan This is my paper

classification cs.AR

keywords softmaxsoftermaxtransformersconsistsaccountsaccuracyaddressattributed

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

Transformers have transformed the field of natural language processing. This performance is largely attributed to the use of stacked self-attention layers, each of which consists of matrix multiplies as well as softmax operations. As a result, unlike other neural networks, the softmax operation accounts for a significant fraction of the total run-time of Transformers. To address this, we propose Softermax, a hardware-friendly softmax design. Softermax consists of base replacement, low-precision softmax computations, and an online normalization calculation. We show Softermax results in 2.35x the energy efficiency at 0.90x the size of a comparable baseline, with negligible impact on network accuracy.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

SpeLLM: Character-Level Multi-Head Decoding
cs.CL 2025-07 conditional novelty 6.0 of 10

SpeLLM converts a standard token-based LLM into a character-spelling model with multiple parallel output heads, achieving competitive downstream performance with a 5.1% average decoding speedup.

Pith tools