Derives closed-form optimal attention temperature minimizing ICL generalization error under distribution shift, linked to pre-softmax score moments, with LLM validation.
Note that our inputs ¯xi are centered, i.e., ¯xi =x i − 1 l P i≤l xi, so their distribution isN(0,Σ x)as l→ ∞
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
stat.ML 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Optimal Attention Temperature Improves the Robustness of In-Context Learning under Distribution Shift in High Dimensions
Derives closed-form optimal attention temperature minimizing ICL generalization error under distribution shift, linked to pre-softmax score moments, with LLM validation.