Selective Self-Attention adds query- and value-dependent temperature scaling to transformer attention, improving language modeling accuracy and passkey retrieval with under 0.5% extra parameters.
Generalization on the unseen, logic reasoning and degree curriculum
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Selective Attention: Enhancing Transformer through Principled Context Control
Selective Self-Attention adds query- and value-dependent temperature scaling to transformer attention, improving language modeling accuracy and passkey retrieval with under 0.5% extra parameters.