{"id":"40b78e69-69e9-43b0-9e47-2a6d0ae845c4","arxiv_id":"2608.06776","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Faster query-key learning relative to output-value learning sharpens attention onto task-relevant tokens at comparable prediction performance, derived from gradient-flow dynamics and shown on synthetic and real tasks.","lead":"This paper shows that in a single-layer self-attention model, making the query-key parameters learn faster than the output-value parameters makes attention concentrate on relevant tokens without hurting prediction accuracy. The authors derive closed-form training dynamics that explain the effect and demonstrate it on synthetic and real language tasks.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The theory's log^2 sharpening is derived for collapsed circuits; the paper never verifies that factorized circuits follow the same closed-form dynamics, leaving the headline factorization claim unsupported.","rationale":"The reader identified the idealized data assumptions as the weakest point; I agree that those assumptions limit quantitative transfer. However, I find a more specific and more load-bearing gap: the paper's closed-form dynamics are derived for collapsed circuits, but the headline claim concerns factorized parameterizations and their implicit learning-rate rescaling. Lemma 2 provides a preconditioning identity, but the paper never shows that this identity preserves the scalar ODE structure used in Lemma 3. In the orthogonal setting the factorized OV dynamics acquire an extra factor of mu in the effective learning rate, so the closed-form log vs log^2 growth is not automatically inherited. This is an internal-theory gap, not merely a question of whether the assumptions match real data. The real-data experiments do not close the gap because they manipulate the explicit LR ratio rather than the factorization-induced rescaling. Because the concern is addressable with a targeted simulation, it does not reject the paper; it keeps the verdict conditional on the factorized trajectories actually following the predicted rescaled collapsed dynamics. The reader's CONDITIONAL verdict is therefore unchanged.","tokens_in":25614,"tokens_out":28441,"duration_ms":249498,"concrete_test":"Run the synthetic orthogonal experiment of Theorem 1 in both collapsed and factorized (FAFO) versions, using balanced initializations with equal collapsed initial matrices. Measure the effective scalars mu_OV(t) and mu_QK(t) of the products W_OV = W_O W_V and W_QK = W_K W_Q^T. Then compare the factorized trajectories to the collapsed ODEs of Theorem 1 with the time-varying learning rate eta_eff(t) = eta_F * c * mu(t) predicted by Lemma 2. If the factorized mu_OV, mu_QK, or attention mass alpha(t) do not match this rescaled collapsed prediction, or if the relative growth rates diverge from log t and log^2 t, then the factorization claim in the paper is not supported by its own theory.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 1 and Lemma 3 analyze collapsed W_OV and W_QK with W_OV(0)=W_QK(0)=0, yielding mu_OV = Theta(log(1+t)) and mu_QK = Theta((eta_QK/eta_OV) log^2(1+t)). Lemma 2 shows factorized OV follows a preconditioned flow with a degree-1 homogeneous preconditioner Omega(C). In the orthogonal setting, Omega(W_OV) acts on vec(W_OV) as a scalar c mu(t), so the factorized scalar dynamics become dmu/dt = eta_F * c * mu * G(mu), not eta * G(mu). This is not the same closed form as Theorem 1: the effective OV learning rate is itself proportional to mu and therefore time-dependent. The paper never integrates this mu-dependent rescaling into the coupled ODEs, so the log^2 sharpening mechanism and the claim that the ratio eta_QK/eta_OV controls attention are not proven for the factorized parameterizations emphasized in the abstract and used in real transformers. The Section 7 experiments vary only the explicit query-key learning rate; they do not test the parameterization-induced rescaling that is a headline contribution. Thus the central theoretical claim is established for collapsed circuits only, and the bridge to factorized circuits is asserted via Lemma 2 but never verified quantitatively.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies how the relative optimization speeds of the query-key and output-value circuits in a single-layer self-attention transformer shape attention concentration during next-token prediction. It compares collapsed (W_QK = W_K W_Q^T, W_OV = W_O W_V) and factorized parameterizations and proposes that factorization induces a state-dependent preconditioning that rescales the circuits' effective learning rates. In a synthetic orthogonal data model, the authors derive closed-form population gradient-flow ODEs (Theorem 1) and prove growth bounds (Lemma 3): mu_OV(t) = Theta(ln(1+t)) and mu_QK(t) = Theta((eta_QK/eta_OV)(ln(1+t))^2) in a pre-saturation regime, yielding attention mass on relevant tokens that increases with r = eta_QK/eta_OV. Experiments on synthetic data, SQuAD, HateXplain, and subject-verb agreement show that increasing the query-key learning-rate multiplier sharpens attention-based interpretability proxies while preserving predictive performance.","tokens_in":25874,"tokens_out":12342,"duration_ms":105941,"significance":"If the mechanism holds as stated, the paper offers a clean, falsifiable account of how training dynamics, rather than only the loss landscape, determine attention sharpening, and it gives practitioners a simple intervention (the QK/OV learning-rate ratio) for controlling attention concentration. The closed-form bounds in Lemma 3 are a useful addition to the dynamics-of-attention literature, and the paper ships code and reports confidence intervals. However, the quantitative theory is established for collapsed parameterizations only; the bridge to factorized parameterizations, which is a headline contribution, is asserted via Lemma 2 but never carried through the ODE analysis. This gap tempers the significance of the main theoretical claim.","major_comments":[{"comment":"The proof that attention scores take exactly two values, u_j = mu_QK(t) for distinct tokens and u_j = 0 for common tokens, requires the query token x_T to satisfy <sum_tau s^tau_y, x_T> = 1 for the true class y and to be orthogonal to the other class sums. Assumptions 1-3 state nothing about the query-token distribution or normalization. Without such a condition, the formula alpha(t) = m exp(mu_QK(t)) / (m exp(mu_QK(t)) + n), which is used throughout Lemma 3, is not justified. Please state the required assumption on x_T explicitly or carry the factor <sum_tau s^tau_y, x_T> through the derivation.","section":"Section 6.1, Theorem 1; Appendix B"},{"comment":"Theorem 1 and Lemma 3 analyze plain gradient flow on the collapsed parameters W_OV and W_QK. Lemma 2 (and Lemma 5 for QK) shows that factorized training induces the preconditioned flow d/dt vec(W_OV) = -Omega(W_OV) vec(nabla L_1(W_OV)) with a degree-1 homogeneous Omega. For the rank-one trajectories of Theorem 1, Omega(mu C_0) scales linearly in mu, so the factorized scalar ODE becomes d mu/dt = eta_F c mu G(mu), not d mu/dt = eta G(mu). The paper never integrates this mu-dependent rescaling into the coupled ODEs, so the explicit Theta((eta_QK/eta_OV) (ln(1+t))^2) growth and the ratio-r mechanism are not proven for the factorized parameterizations emphasized in the abstract and used in the Section 7 experiments. Either extend the derivation to the preconditioned flow or restrict the theoretical claims to collapsed circuits and present the factorized behavior as empirical.","section":"Section 6, Lemmas 1-2 vs. Theorem 1 and Lemma 3"},{"comment":"The real-data experiments vary only the explicit query-key learning-rate multiplier (20x/30x/100x) in factorized models trained with AdamW, whereas Lemma 3 is a gradient-flow statement about the ratio eta_QK/eta_OV for collapsed parameters. No experiment isolates the parameterization-induced rescaling (for example, factorized versus collapsed with matched effective preconditioning), and no SGD comparison is reported for the real data. In addition, Appendix A.2.2 states that the SVA model is trained with SGD, while Section 7 states that all models are trained with AdamW. The paper should either add explicit validation of the factorization-induced rescaling or label the real-data results as qualitative evidence for the collapsed-circuit mechanism only.","section":"Section 7, Table 1; Appendix A.2.2"}],"minor_comments":[{"comment":"Assumption 1 should be strengthened to state that all bM class-token vectors {s^tau_y} are mutually orthogonal and unit-norm; the proof of Theorem 1 uses orthogonality across both tau and classes simultaneously (Appendix B, Eq. (7)).","section":"Section 6.1, Assumption 1"},{"comment":"The notation \"direct sum\" (Kroneycker sum) for Omega(C) is used without a definition in the main text; the explicit definition (C^T C)^{1/2} \\otimes I_M + I_d \\otimes (C C^T)^{1/2} should appear where Omega is first introduced.","section":"Lemma 2 / Appendix D.3"},{"comment":"The Figure 3 captions contain apparent missing symbols (for example, \"Lt( )\"), and the Figure 4(b) caption contains a typo in \"mu_OV (t)evolution\".","section":"Figure 3 and Figure 4 captions"},{"comment":"The statement that adaptive optimizers \"treat all parameter groups equally\" is imprecise: Adam's per-parameter normalization interacts with a learning-rate multiplier, so the effective ratio r under AdamW is not guaranteed to equal the nominal multiplier.","section":"Section 7"},{"comment":"The stated vocabulary size of 50 tokens appears inconsistent with 4 classes x 10 distinct tokens + 10 common tokens + 4 query/next tokens = 54 tokens; please clarify whether query and next tokens are drawn from the distinct or common sets.","section":"Section 4 / Appendix A.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable contribution but the abstract overclaims. The collapsed-circuit theory is internally coherent and the experiments are carefully reported, so I do not see grounds for rejection. The main work for revision is to either carry the preconditioned flow through the orthogonal-data ODE analysis or substantially soften the factorized-parameterization claims, and to fix the missing query-token assumption in Theorem 1. If the authors can do that, I would support publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The log versus log-squared growth rate result is a real, clean contribution, and the paper is honest about its idealized setting. But the headline story about factorization is not actually proven: the closed-form dynamics are derived for collapsed circuits, and the authors never integrate the preconditioning induced by factorization into the coupled ODEs.\n\nWhat's new: for a single-layer attention model on an orthogonal synthetic data distribution, with zero initialization, they show that mu_OV(t) = Theta(log(1+t)) and mu_QK(t) = Theta((eta_QK/eta_OV) log^2(1+t)). That dependence on the learning-rate ratio is non-obvious and mechanistically appealing. Lemma 2, which derives a homogeneous preconditioner for factorized OV/QK circuits from deep linear network theory, is also a useful observation. The real-data experiments consistently show that multiplying the QK learning rate sharpens attention metrics (AC, ACMC, MRTA, sufficiency/comprehensiveness) with roughly stable accuracy. The limitations section is candid about the single-layer setting and the meaning of attention metrics.\n\nSoft spots. The factorization claim is the weak point. Theorem 1 and Lemma 3 analyze collapsed W_OV and W_QK. For factorized parameters, Lemma 2 says the effective matrix follows a preconditioned flow, and the preconditioner is state-dependent—roughly, the effective learning rate is multiplied by a function of the current scale. The paper states that factorization 'induces implicit rescaling' but never plugs this rescaling into the scalar ODEs or checks whether the log-squared rates survive. So the central mechanism is established for collapsed circuits and for explicit learning-rate interventions, but not for the factorization that real transformers use. The experiments only vary the explicit QK multiplier; they don't test the parameterization-induced rescaling. That's a real mismatch with the abstract's framing, though it doesn't sink the empirical claim about LR ratios. Also minor: QK multipliers are validation-tuned per dataset, and code is not verifiable from the manuscript alone.\n\nWho it's for: people working on attention dynamics or mechanistic interpretability will find the LR-ratio mechanism worth engaging with. It deserves a serious referee, but the referee should push hard on the factorization story—either prove the factorized analogue or cut the claim down to what is actually shown.","headline":"A clean collapsed-circuit result with an overreaching factorization story: the log^2 sharpening is real under strong assumptions, but the bridge to factorized transformers is asserted, not proved.","tokens_in":26396,"tokens_out":6703,"would_cite":true,"duration_ms":54715,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Raising the query-key learning rate relative to the output-value rate sharpens self-attention even when predictive loss stays flat.","keywords":["self-attention","attention sharpening","training dynamics","gradient flow","query-key circuit","output-value circuit","learning-rate ratio","next-token prediction"],"falsifier":"Under the paper's orthogonal synthetic setting, train with $r = 100$ and with $r = 0.01$, stopping each run at the same loss threshold: the relevant-token attention mass $\\alpha(t)$ should be near one in the first case and near uniform in the second. If attention at matched loss is independent of $r$, the claimed decoupling between loss decay and attention sharpening is contradicted. A second check: with fixed large $r$, $\\alpha(t)$ should keep growing roughly as $\\log^2$ of time after the loss is already near zero, so extending training should show attention still concentrating while loss stays flat.","tokens_in":25369,"feed_emoji":"🎯","tokens_out":11155,"duration_ms":76527,"temperature":0.7,"pith_summary":"This paper claims that attention sharpening in a self-attention model is governed by the relative learning speeds of its two circuits, not by the final loss alone. For a single-layer transformer trained for next-token prediction, the query-key circuit (which allocates attention) and the output-value circuit (which maps attended tokens to predictions) evolve at different effective rates, and the ratio of their learning rates, $r = \\eta_{QK}/\\eta_{OV}$, controls how much attention mass lands on task-relevant tokens. Under population gradient flow in an orthogonal synthetic setting, the paper derives closed-form scalar dynamics in which the output-value scale grows like log of time while the query-key scale grows like $(\\eta_{QK}/\\eta_{OV}) \\log^2$ of time; a larger ratio pushes attention toward relevant tokens even as training loss decays at the same rate. The paper argues this mechanism explains why factorized versus collapsed parameterizations of the two circuits produce qualitatively different attention patterns at similar predictive performance, and shows experimentally that raising the query-key learning rate sharpens attention on real datasets.","feed_headline":"Faster query-key learning concentrates attention at equal loss","feed_subtitle":"Raising the QK-to-OV learning-rate ratio sharpens attention on relevant tokens without hurting test accuracy.","key_machinery":"The argument is carried by decomposing the self-attention layer into two linear circuits — the query-key circuit $W_K W_Q^\\top$, which sets the attention pattern, and the output-value circuit $W_O W_V$, which maps attended representations to logits — and tracking their population gradient-flow trajectories under factorization. Lemma 2 shows that factorized training induces a preconditioned gradient flow on collapsed parameters with the state-dependent, degree-1 homogeneous operator $\\Omega(C) = (C^\\top C)^{1/2} \\oplus (C C^\\top)^{1/2}$, so factorization behaves like a learning-rate rescale that grows over time. Under orthogonality assumptions on the token vectors and zero initialization, Theorem 1 collapses the full training dynamics to two coupled scalar ordinary differential equations for the output-value scale $\\mu_{OV}$ and the query-key scale $\\mu_{QK}$, and Lemma 3 derives their relative growth rates — log versus log-squared in time — which drive the attention mass $\\alpha(t) = m \\exp(\\mu_{QK}(t)) / (m \\exp(\\mu_{QK}(t)) + n)$ through the softmax nonlinearity.","core_discovery":"The central claim is that faster query-key learning relative to output-value learning produces sharper attention, because the model compensates for slower output-value learning by concentrating attention mass on relevant tokens. Concretely, Lemma 3 states that in the pre-saturation regime the output-value scale obeys $\\mu_{OV}(t) = \\Theta(\\ln(1+t))$ while the query-key scale obeys $\\mu_{QK}(t) = \\Theta((\\eta_{QK}/\\eta_{OV})(\\ln(1+t))^2)$, and since the attention mass on relevant tokens is $\\alpha(t) = m \\exp(\\mu_{QK}(t)) / (m \\exp(\\mu_{QK}(t)) + n)$, that mass grows faster when the learning-rate ratio $r = \\eta_{QK}/\\eta_{OV}$ is large. Because $\\mu_{OV}$ drives loss decay and $\\mu_{QK}$ drives attention concentration, the two quantities decouple: training loss can reach zero while attention remains near uniform (small $r$) or concentrates to near one (large $r$). The paper further proves that factorized training of either circuit induces a state-dependent preconditioned gradient flow on the collapsed parameters, so factorization acts as an implicit, time-varying rescale of that circuit's learning rate, and this is why collapsed and factorized models can achieve comparable loss with different attention trajectories.","pith_inferences":["If relative circuit speed is the true driver, then any intervention that slows output-value learning — weight decay on $W_O$ and $W_V$, a smaller OV learning rate, or unbalanced initialization — should sharpen attention in the same way; this is a directly testable corollary the paper does not run.","The log versus log-squared growth asymmetry implies sharpening compounds with training time: attention should keep concentrating long after the loss has visually converged, so early stopping that watches only loss would silently discard attention structure.","Adaptive optimizers partially normalize away raw per-group learning-rate differences, which may be why the experiments need large QK multipliers (20x-100x); a principled alternative would be to scale the QK learning rate relative to the OV gradient scale rather than to a shared base rate.","The compensation story suggests a possible origin for the familiar diversity of attention patterns among equally performant transformers: not architecture alone, but the accidental effective ratio of query-key to output-value learning speeds during training."],"forward_implications":["With a large ratio $r = \\eta_{QK}/\\eta_{OV}$, attention mass on relevant tokens approaches one even though training loss decays at essentially the same rate as for small $r$, so final loss alone cannot reveal whether attention is concentrated.","Factorized and collapsed parameterizations of the same circuit are not equivalent under optimization: factorization acts as a state-dependent learning-rate rescale, so models with equal predictive performance can follow different attention trajectories.","Raising the query-key learning rate improves attention-based interpretability proxies — relevant-token attention, sufficiency, comprehensiveness, DTAP summary metrics — without architectural change and without degrading test performance.","The qualitative effect persists in deeper, multi-layer transformers, even though the closed-form dynamics are derived for a single layer."],"supporting_citations":[{"why":"Supplies the synthetic data construction of class-specific distinct tokens and class-agnostic common tokens that the closed-form analysis is built on.","marker":"(Tian et al., 2023)"},{"why":"Provides the transformer-circuit decomposition into query-key and output-value circuits that frames the whole analysis.","marker":"(Elhage et al., 2021)"},{"why":"Basis for the conservation-law argument that factorized gradient flow preserves balancedness of factor matrices.","marker":"(Arora et al., 2018)"},{"why":"Supplies the automatic-balancing result for deep homogeneous models used to justify the induced dynamics on collapsed parameters.","marker":"(Du et al., 2018)"},{"why":"Defines attention rollout, the aggregation used to measure relevant-token attention in the real-data experiments.","marker":"(Abnar & Zuidema, 2020)"},{"why":"Defines the sufficiency and comprehensiveness faithfulness metrics reported in the experiments.","marker":"(DeYoung et al., 2020)"},{"why":"Provides the HateXplain dataset with token-level rationales used to test attention sharpening on real text.","marker":"(Mathew et al., 2020)"},{"why":"Provides the subject-verb agreement dataset whose syntax-sensitive tokens test whether faster QK learning sharpens attention.","marker":"(Linzen et al., 2016)"},{"why":"Provides the SQuAD question-answering benchmark used for the 6-layer model experiment.","marker":"(Rajpurkar et al., 2016)"}],"fun_headline_variants":["Faster query-key learning sharpens attention at equal loss","Learning rate ratio controls attention sharpness","Quick QK, slow OV: sharper attention, same accuracy","Attention sharpens when QK outpaces OV learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative theory depends on a synthetic idealization — class tokens mutually orthogonal, zero-mean background tokens orthogonal to all class tokens, attention scores forced to exactly two values, and both parameter matrices started at zero — conditions that real embeddings, layer normalization, and random initialization violate, so the exact log-squared law is guaranteed only in this simplified setting.","fun_headline_variants_meta":{"raw":{"variants":["Faster query-key learning sharpens attention at equal loss","Learning rate ratio controls attention sharpness","Quick QK, slow OV: sharper attention, same accuracy","Attention sharpens when QK outpaces OV learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1387,"prompt_tokens":1014,"completion_tokens":373,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":320}},"tokens_in":630,"tokens_out":373,"duration_ms":3703,"temperature":1.0,"reasoning_tokens":320,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:30:17.476889+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Under the paper's orthogonal synthetic setting, train with $r = 100$ and with $r = 0.01$, stopping each run at the same loss threshold: the relevant-token attention mass $\\alpha(t)$ should be near one in the first case and near uniform in the second. If attention at matched loss is independent of $r$, the claimed decoupling between loss decay and attention sharpening is contradicted. A second check: with fixed large $r$, $\\alpha(t)$ should keep growing roughly as $\\log^2$ of time after the loss is already near zero, so extending training should show attention still concentrating while loss stays flat.","supporting_citations":[{"cited_title":"M., Biemann, C., Goyal, P., and Mukherjee, A","cited_arxiv_id":null,"evidence_quote":"Provides the HateXplain dataset with token-level rationales used to test attention sharpening on real text."}],"review_version":2}