{"id":"35803a78-5bec-49d7-896c-d5af8af68aaa","arxiv_id":"2411.12892","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Selective Self-Attention adds query- and value-dependent temperature scaling to transformer attention, improving language modeling accuracy and passkey retrieval with under 0.5% extra parameters.","lead":"This paper adds a lightweight, learned temperature scaling to the query and value vectors inside transformer attention, letting each token control how sharply it focuses on other tokens. The method, called Selective Self-Attention, claims small but consistent accuracy gains on language benchmarks and a large boost on a synthetic passkey retrieval task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The learned temperature module is too restricted to realize the optimal temperatures used in the theory, so the claimed principled link between theory and SSA implementation is not established.","rationale":"The reader's weakest assumption is exactly the gap I find most load-bearing: the theory constructs ideal temperatures without analyzing whether the implemented temperature module can represent them. The central claim is that SSA provides a principled temperature scaling strategy that improves accuracy and retrieval. For that claim to hold, the parameterization must be able to realize temperatures that produce the theoretical benefits. The concrete range calculation shows that the implemented τ is strictly positive and grows with log n, whereas Proposition 2's optimal τ_n can be zero or negative in the very regime the theorem analyzes. This is not a disagreement with the empirical trend; the synthetic experiments and consistent average gains in Table 3 are suggestive. But they do not close the expressivity gap, because the empirical success could come from generic gating rather than from the specific principled scaling the theory describes. I also note that Proposition 1's proof in Appendix C.1 has a suspect step where a softmax approximation error is converted into a logit lower bound; that is a separate correctness risk, but the temperature-range issue is more directly tied to the abstract's central claim. The reader's conditional verdict remains appropriate: the paper should either restrict its theoretical claims to the attainable temperature range or modify the parameterization to match the theory, and the proposed synthetic test would settle whether the gap is real.","tokens_in":19719,"tokens_out":14348,"duration_ms":155835,"concrete_test":"Train the real SSA parameterization (tanh MLP plus 1+σ(α) log n) on the noiseless imbalanced-token task from Proposition 2 with α=1/2 and sequences that include positions with κ_n=1 and positions with κ_n≥4, and record the converged loss and learned τ_n values. If the loss does not approach zero while the unconstrained optimum τ_n=log κ_n is realizable, or if the learned τ_n avoids the negative/zero region, the theory-to-implementation bridge fails. Separately, compute the attainable temperature interval [σ(α) log n, 2+σ(α) log n] for the actual context lengths used in Table 3 and check whether it contains the values required by Lemma 1 and Proposition 2; report the overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 2 (Section 4.2) constructs an optimal position-dependent inverse temperature τ_n = log κ_n + log(α/(1−α)) that drives the loss to zero. The implemented parameterization in Section 3 is τ(x) = tanh(f(x)) + 1 + σ(α) log n. Because tanh takes values in (−1,1) and σ(α) ∈ (0,1), the total temperature is always strictly positive and, for each n, is confined to [σ(α) log n, 2+σ(α) log n]. Thus the module cannot express the negative (or near-zero) temperatures that Proposition 2 requires when κ_n < (1−α)/α; for α=1/2 and κ_n=1 the optimal τ is 0, while the implementation is bounded below by σ(α) log n > 0 for n > 1. The same range restriction applies to Lemma 1's construction τ(a)=L_a, τ(b)=L_b whenever the desired specificity levels lie outside this interval. The paper never analyzes what temperatures the tanh-plus-log parameterization can attain, nor does it show that learned temperatures approximate the theoretically optimal curve. Without such an expressivity argument, the theory in Section 4 does not carry over to the actual SSA layer, and the claimed principled advantage rests on an unverified bridge.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Selective Self-Attention (SSA) layer that applies input-dependent inverse temperatures to query and value embeddings, together with a position-dependent term tau_pos(x) = 1 + sigma(alpha) log n. Theoretical arguments (Lemma 1, Propositions 1 and 2, Lemma 2) claim that such scaling controls attention spikiness, mitigates attention dilution, and improves expressivity and denoising. The method is evaluated on language modeling benchmarks with GPT-2, Pythia, Llama, and Llama-3, under both fine-tuning and pre-training, reporting perplexity/accuracy gains and improved passkey retrieval, with weight-sharing and feature-based variants that add less than 0.5% parameters.","tokens_in":20020,"tokens_out":5631,"duration_ms":55451,"significance":"If the link between the theory and the implemented temperature module were established, SSA would be an attractive lightweight, architecture-agnostic improvement: it is simple, parameter-efficient, and shows broad gains across model families and training regimes. The paper also offers a formal derivation of a YaRN-like logarithmic temperature rule, which is a useful conceptual contribution. However, the theory is an existence argument that sets temperatures by construction (tau(a)=La, tau(b)=Lb, and tau_n=log kappa_n + log(alpha/(1-alpha))), while the implemented parameterization is a bounded tanh-plus-log function whose expressivity is never characterized. Several synthetic experiments replace the learned module with hand-set temperatures or hard thresholding proxies. The empirical results are broad but all single-run, so the 'consistent improvement' claim is not yet statistically supported.","major_comments":[{"comment":"The implemented temperature function tau(x) = tanh(f(x)) + 1 + sigma(alpha) log n is, for every n > 1, bounded below by sigma(alpha) log n > 0 and above by 2 + sigma(alpha) log n. It therefore cannot realize the optimal inverse temperatures of Proposition 2, tau_n = log kappa_n + log(alpha/(1-alpha)), which are zero or negative for natural parameter choices (e.g., alpha = 1/2, kappa_n = 1 gives tau_n = 0; alpha < 1/2 and kappa_n < (1-alpha)/alpha gives negative tau_n). The same issue applies to the constructions in Lemma 1 and Proposition 1 when the desired temperatures La, Lb lie outside the attainable interval. This directly contradicts the statement in Section 3 that 'we don't restrict tau_{k/q/v} to be non-negative.' Since no expressivity analysis of the learned module is provided, the theoretical advantages derived in Section 4 do not carry over to the actual SSA layer as implemented, and the claimed 'principled' bridge is not established.","section":"Section 3 and Section 4.2 (Proposition 2)"},{"comment":"All empirical results are reported from single runs with no seeds, confidence intervals, or significance tests. Several reported gains are small (e.g., fine-tuned Pythia-160m Wikitext perplexity 26.681 vs. 26.514, and Lambada_std perplexity 47.996 vs. 47.945), and at least one weight-sharing entry is worse than the vanilla baseline (Pythia-160m fine-tune Wikitext perplexity 26.780 vs. 26.681). Given the abstract's claim of a 'noticeable and consistent accuracy improvement,' the absence of repeated runs is a load-bearing gap: the aggregate pattern is suggestive, but the consistency claim cannot be evaluated statistically.","section":"Section 5.1, Table 3"},{"comment":"The proof of Proposition 1 is not rigorous. It writes ||P* - S(EWE^T)||_infty as ||1/(1+e^{-EWE^T}) - P*||_infty, which incorrectly treats the row-wise softmax as an elementwise sigmoid. The subsequent line '||EWE^T||_infty >= 1/(4 epsilon) - Gamma' does not follow from the preceding expression, and the final lower bound on ||W|| is left unexplained. As written, this proof does not establish the proposition, which is one of the paper's theoretical contributions.","section":"Appendix C.1 (proof of Proposition 1)"},{"comment":"The denoising experiment does not evaluate the proposed SSA value-temperature module. The paper states that, as a proxy for token-aware temperature scaling, it uses the hard thresholding function max_j x_{ij} >= 1/2, rather than the learned tanh-based temperature function. Similarly, the token-generation experiment in Section 4.1 (Table 1) assigns temperatures by hand to groups of nodes with equal degree. These experiments therefore demonstrate the utility of input-dependent scaling in principle, but they do not provide evidence that the actual SSA temperature module realizes these behaviors.","section":"Section 4.3 and Table 2"}],"minor_comments":[{"comment":"The bullet list says SSA 'introduces 5% additional parameters to the model,' which contradicts the Abstract's 'less than 0.5% new parameters' and the weight-sharing discussion in Section 3. This should be corrected.","section":"Section 1 (Introduction)"},{"comment":"The caption states 'the norm of the vanilla attention layer is approximately three times larger than that of SSA (dashed red line compare to green line),' but the figure legend and color references are unclear; the dashed-line convention should be explained precisely.","section":"Figure 2 caption"},{"comment":"The implementation details state that pre-training takes about 2 hours on 4 A40 GPUs for 15B tokens, while fine-tuning takes about 2 days. This appears to be a typographical inconsistency and should be clarified.","section":"Appendix A"},{"comment":"There are numerous typographical errors, including 'perfromance,' 'di fferent,' 'trainig,' and 'funtion'; a careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuinely useful empirical observation and a plausible design idea, but the advertised theoretical grounding is not yet connected to the actual implementation. The missing seeds/error bars are standard for this venue but should be addressed given the strength of the consistency claim. I would be willing to consider a revised version that either extends the theory to the real parameterization or substantially softens the 'principled' language, and that adds multi-seed experiments for at least the main benchmark table."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the SSA layer is a reasonable idea — per-token temperature scaling on queries and values plus a positional term, with a weight-sharing trick that keeps parameter overhead under 0.5%. The empirical body is wide: five model families, fine-tuning and pre-training, standard benchmarks and passkey retrieval. Accuracy gains are small but consistent, and passkey retrieval jumps from about 57% to 74% on Pythia-160m. If those numbers replicate, this is a useful drop-in.\n\nThe weight-sharing design is the most practical contribution: reuse attention weights in the temperature MLP and add only a few vectors per head, so no extra matrix multiplies. That is genuinely neat.\n\nNow the soft spots, in order of severity.\n\nFirst, the theory overclaims. The paper says the approach is principled, but the constructed optimal temperatures — τ_n = log κ_n + log(α/(1−α)) in Proposition 2, or τ(a)=L_a in Lemma 1 — are not in the range the implementation can produce. The implemented τ(x) = tanh(f(x)) + 1 + σ(α) log n is always positive and, for each n, lies in [σ(α) log n, 2+σ(α) log n]. It cannot realize zero, negative, or large power-law temperatures, which the theory requires. The stress-test is correct: there is an expressivity gap that is never analyzed. The theory is existence, not a description of what SSA actually learns. The authors should either characterize the representable temperature range or tone down the 'principled' language.\n\nSecond, the empirical reporting is statistically thin. Single runs, no seeds, no error bars. That is a real problem when the headline accuracy gains are around 1–2 points. The passkey result is more robust-looking, but still only a few iterations.\n\nThird, Proposition 1's proof in Appendix C.1 is sloppy: the sigmoid-style manipulation is not correct softmax, and the lower bound does not follow from the stated argument. The reader flagged this and I agree.\n\nMinor: the compute budget in A.1 ('pre-training takes about 2 hours using 4 A40') looks optimistic for 15B tokens, and the paper states 5% and 0.5% parameter overhead in different places; the latter is the honest figure.\n\nBottom line: this is a solid, incremental empirical paper with a clean idea and a real practical trick, but the theory section is not as load-bearing as presented. I would send it to review with major revisions required — fix the expressivity gap, add error bars, clean up the proofs. It deserves referee time.","headline":"A plausible parameter-efficient attention tweak with broad but thin experiments; the theory overclaims because the implemented temperature module cannot express the optimal temperatures the proofs construct.","tokens_in":20571,"tokens_out":5231,"would_cite":false,"duration_ms":47945,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a Selective Self-Attention layer that gives each query its own softmax temperature, letting a transformer control how sharply it focuses without growing its weight matrices.","keywords":["selective attention","temperature scaling","softmax spikiness","attention dilution","contextual sparsity","transformer language modeling","long-context retrieval"],"falsifier":"Train an SSA-equipped language model while clamping every learned temperature to a narrow interval such as [0.9, 1.1] or freezing the temperature module to a constant; if perplexity and passkey accuracy stay at the SSA level instead of falling back to the vanilla baseline, then the reported gains do not come from query-dependent temperature control.","tokens_in":19525,"feed_emoji":"🌡️","tokens_out":6220,"duration_ms":59167,"temperature":0.7,"pith_summary":"Standard self-attention applies the same softmax map to every query, so a transformer cannot independently decide how spiky its attention should be once the key-query weights are fixed. This paper argues that this uniformity forces weight matrices to grow large when different queries need different levels of focus, which slows optimization and dilutes attention over long contexts. As a remedy, it introduces Selective Self-Attention (SSA): a learnable inverse-temperature per query and value, plus a positional term that grows logarithmically with context length, applied as elementwise scaling of the embeddings. The paper claims this decouples semantic similarity from contextual sparsity, and it backs the claim with lower bounds showing when temperature helps, a synthetic denoising task, and experiments where SSA improves perplexity and accuracy on several transformer families with less than 0.5% added parameters under weight sharing. The payoff, if correct, is a general, nearly free modification that makes existing transformers focus better and retrieve better in long contexts.","feed_headline":"Per-token temperatures sharpen transformer attention for nearly free","feed_subtitle":"Selective Self-Attention tunes each query's softmax focus, cutting attention dilution and lifting language-model accuracy.","key_machinery":"The load-bearing object is the learned inverse-temperature field $\\tau(x)=\\tanh(f(x))+1+\\sigma(\\alpha)\\log n$, applied elementwise to query and value embeddings, with $f(x)=W_{\\mathrm{tmp}}\\,\\mathrm{GeLU}(W_{k/q/v}x)$ reusing existing attention projections plus one small output vector per head. It acts as a scalar gate on each token: positive values sharpen the softmax, negative values can de-emphasize or suppress a token, and the $\\log n$ term compensates for attention flattening as context grows. The mechanism's work is to decouple the two roles of $W_qW_k^\\top$: semantic similarity stays in the projection matrices while $\\tau$ carries contextual sparsity, so the required spectral norm no longer grows with specificity differences or with $\\log L$.","core_discovery":"The central claim is that softmax temperature is not a nuisance hyperparameter but a missing degree of freedom in attention. In canonical self-attention $V^\\top\\mathrm{softmax}(Kq)$, the same weights $W=W_qW_k^\\top$ must simultaneously encode which tokens are semantically similar and how concentrated the attention map should be; the paper proves (Lemma 1 and Proposition 1) that when two queries are semantically close yet need very different spikiness, standard attention needs a weight norm that blows up as the inverse of their embedding distance, whereas query-temperature $\\tau(q)$ lets the same weights serve both queries with bounded norm. For position, the paper derives (Proposition 2) that a flat temperature cannot solve even a two-token imbalanced-context retrieval problem, while $\\tau_n=\\log\\kappa_n+\\log(\\alpha/(1-\\alpha))$ solves it exactly, motivating a positional term of the form $1+\\sigma(\\alpha)\\log n$. For values, a linear value projection provably fails a synthetic denoising task that nonlinear value scaling nearly matches at the Bayes-optimal level. Empirically, SSA-equipped models achieve consistent accuracy gains over baselines on standard language modeling benchmarks and a large jump on passkey retrieval (for example, Pythia-160m from 56.9% to 74.4%).","pith_inferences":["The same temperature field can be read as an inductive bias toward input-dependent gating inside attention; if correct, comparisons with other selective mechanisms (gates, sparse top-k attention) should be made at the level of the effective temperature they induce, not just parameter count.","Proposition 2's logarithmic rule is likely a low-order correction that matters most for very long contexts; a testable extension is whether SSA reduces the need for additional position-embedding scaling when the context window is extended during fine-tuning.","Because the theory constructs temperatures that the implemented $\\tanh$ MLP may not reach, the empirical gains could be improved or matched by a more expressive closed-form temperature rule; a direct test would clamp or span-check the learned $\\tau$ values.","The decoupling story suggests a diagnostic: attention-head entropy should vary more across queries with SSA than without, which can be measured on real corpora to verify the mechanism is actually being used."],"forward_implications":["Transformer weights no longer have to grow like $1/\\|a-b\\|_2$ to give similar tokens different focus; SSA keeps effective weights bounded, which should reduce vanishing-softmax slowdowns and sensitivity to small context perturbations.","The positional term $1+\\sigma(\\alpha)\\log n$ is a differentiable, principled replacement for hand-tuned length-scaling rules, and it provably solves the imbalanced-context task that flat temperature cannot.","Value-temperature gives attention a nonlinear denoising channel, so irrelevant or noisy tokens can be suppressed even when a linear value projection cannot separate them.","Because temperature adapts per query and per position, long-context retrieval improves, with the paper reporting passkey retrieval rising from 56.9% to 74.4% for Pythia-160m under SSA.","Weight sharing makes the module nearly parameter-free and compatible with fast attention implementations, so the gains transfer to existing large language models by fine-tuning."],"supporting_citations":[{"why":"supplies the premise that attention maps are sparse and ties specificity to query-norm growth, which Lemma 1 and Proposition 1 build on.","marker":"[43]"},{"why":"provides the logarithmic length-scaling attention rule that Proposition 2 formally justifies and that the positional temperature term is designed to recover.","marker":"[33]"},{"why":"documents the attention dilution problem in long contexts that motivates position-aware temperature scaling.","marker":"[7]"},{"why":"supplies the Markov-chain next-token-prediction framework used in the synthetic expressivity experiment with the latent transition matrix.","marker":"[20]"},{"why":"inspires the feature-based temperature variant that sets temperature from token statistics such as corpus frequency.","marker":"[26]"},{"why":"provides the GPT-2 architecture and pretrained checkpoints used as a baseline and fine-tuning testbed.","marker":"[34]"},{"why":"provides the Pythia model family and pretrained checkpoints used for both fine-tuning and pretraining evaluations.","marker":"[3]"},{"why":"provides the Llama model used to show SSA transfers to modern large language models.","marker":"[44]"}],"fun_headline_variants":["Per-token temperature sharpens attention, cuts dilution","Learned softmax temperature boosts LLM accuracy at <0.5% cost","Selective Self-Attention: principled temperature control","Adaptive temperature scaling improves transformer focus","Query-wise temperature: a lightweight fix for attention dilution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benefits hinge on the small learned temperature module being able to produce the very different per-token and per-position temperature values the theory requires, which the paper does not directly verify.","fun_headline_variants_meta":{"raw":{"variants":["Per-token temperature sharpens attention, cuts dilution","Learned softmax temperature boosts LLM accuracy at <0.5% cost","Selective Self-Attention: principled temperature control","Adaptive temperature scaling improves transformer focus","Query-wise temperature: a lightweight fix for attention dilution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1481,"prompt_tokens":1067,"completion_tokens":414,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":334}},"tokens_in":683,"tokens_out":414,"duration_ms":4601,"temperature":1.0,"reasoning_tokens":334,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:05:34.913931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an SSA-equipped language model while clamping every learned temperature to a narrow interval such as [0.9, 1.1] or freezing the temperature module to a constant; if perplexity and passkey accuracy stay at the SSA level instead of falling back to the vanilla baseline, then the reported gains do not come from query-dependent temperature control.","supporting_citations":[{"cited_title":"Max-margin token selection in attention mechanism","cited_arxiv_id":null,"evidence_quote":"supplies the premise that attention maps are sparse and ties specificity to query-norm growth, which Lemma 1 and Proposition 1 build on."},{"cited_title":"Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation","cited_arxiv_id":"2311.00684","evidence_quote":"documents the attention dilution problem in long contexts that motivates position-aware temperature scaling."},{"cited_title":"From self-attention to markov models: Unveiling the dynamics of generative transformers","cited_arxiv_id":null,"evidence_quote":"supplies the Markov-chain next-token-prediction framework used in the synthetic expressivity experiment with the latent transition matrix."},{"cited_title":"Long-tail learning via logit adjustment","cited_arxiv_id":null,"evidence_quote":"inspires the feature-based temperature variant that sets temperature from token statistics such as corpus frequency."},{"cited_title":"Language models are unsupervised multitask learners","cited_arxiv_id":null,"evidence_quote":"provides the GPT-2 architecture and pretrained checkpoints used as a baseline and fine-tuning testbed."}],"review_version":1}