{"id":"2d467fb9-afab-41b5-98ea-5445ea9328eb","arxiv_id":"2608.06849","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Effective rank of the frozen query-key kernel W_K^T W_Q identifies retrieval heads, enabling training-free sparse attention at 50% sparsity with small accuracy loss.","lead":"A new data-free method, Autonomy-of-Heads, sorts attention heads into retrieval and streaming types using only the singular-value spectrum of frozen query-key weight products, with no runtime scores or calibration prompts. It preserves around 96.5% of full-attention LongBench accuracy at 50% sparsity while cutting prefill and decode latency substantially, which could simplify long-context LLM serving.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Head-role stability across inputs is the load-bearing assumption; aggregate LongBench/passkey evidence does not rule out per-input role switching, and Table 9's 9.4-point Synthetic drop on Qwen2.5-7B suggests the fixed 50% label set can be insufficient.","rationale":"The reader's weakest assumption—that head roles are stable across inputs and that input-free effective rank remains predictive—is exactly the load-bearing point for the paper's central claim. If a head classified as streaming (high ER) occasionally needs global context, the 50% sparse policy will lose accuracy on those inputs; if a low-ER head is sometimes only local, the method is merely conservative. The paper's evidence for stability is indirect: average attention distance, passkey accuracy under progressive conversion, and LongBench averages. These are aggregate metrics that can mask per-input switching. The Qwen2.5-7B Synthetic drop in Table 9 is a concrete warning that the pre-chosen 50% budget is not universally safe, but it does not by itself prove the spectral label is wrong—it could be a budget problem. The proposed greedy ablation on diverse prompts directly tests whether the set of heads that must stay global varies across inputs and whether it aligns with the low-ER set. Until such a test is run, the central claim remains plausible but not fully secured. Since the reader already conditioned acceptance on similar generalization evidence, my read does not move the verdict.","tokens_in":25753,"tokens_out":12050,"duration_ms":119959,"concrete_test":"On a held-out model (e.g., Mistral-7B or Llama-3.1-70B), construct 200 prompts spanning retrieval, multi-hop QA, summarization, and code from RULER and LongBench-v2. For each prompt, start from Full Attention and greedily restrict heads to sink+recent in order of least accuracy loss until 50% of heads are restricted, recording the per-prompt order. Then compute the IoU between the per-prompt 'dispensable' head set (the 50% restricted last) and AoH's high-ER streaming set, along with variance across prompts. If high-ER heads are required on some prompts (low or highly variable IoU), the frozen-label assumption fails; if the dispensable set is consistently a superset of the high-ER set, AoH's data-free labels are validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"AoH's central claim is that a head's retrieval/streaming role is encoded in the frozen spectrum of M_h and is stable across inputs, so labels computed once from weights remain valid at deployment. The supporting evidence—Figure 3's attention-distance correlation, Figure 4's passkey ablation, and LongBench averages—is aggregate. It does not establish that the same head never needs global access on some inputs and local-only on others. If roles are input-dependent, the fixed low-ER/high-ER split will misassign heads on out-of-distribution prompts. Table 9 already shows a 9.4-point drop on Qwen2.5-7B's Synthetic category at 50% sparsity (28.51 to 19.11), suggesting that on some retrieval-heavy tasks the pre-selected low-ER set is not sufficient; this is consistent with head-role instability or with a budget that is too aggressive, and the paper does not distinguish the two. The Limitations section concedes the budget is manually chosen, but the deeper issue is whether the frozen label is input-independent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Autonomy-of-Heads (AoH), a training-free and data-free method that classifies attention heads into retrieval heads (served with full attention) and streaming heads (served with sink-plus-recent-window attention) using the effective rank of the frozen kernel matrix M_h = W_K^{hT} W_Q^h. The authors derive an efficient d_head-dimensional proxy for the singular spectrum, present LongBench evaluations on Qwen2.5-7B, Qwen3-8B, Llama3.1-8B-Instruct, and a MoE model, report passkey and attention-distance evidence for the retrieval/streaming distinction, and measure latency and KV-cache savings. The central claim is that at 50% sparsity AoH retains 96.5% of Full Attention performance on average while outperforming random/reversed head selection and sparse baselines.","tokens_in":25940,"tokens_out":9302,"duration_ms":92166,"significance":"If the central assumption holds—that a head's retrieval or streaming role is encoded in the frozen query-key spectrum and is stable across inputs—AoH would be a practically valuable contribution, since it removes runtime score computation, calibration prompts, and learned gates from head diagnosis. The paper has several genuine strengths: the trace-reduction and AB-BA eigenvalue arguments in Appendix L are mathematically sound; the d_head-dimensional proxy (Eq. 5) is a clean and useful reduction; the passkey conversion experiment (Figure 4) is a well-designed causal-style check that low-effective-rank heads matter for retrieval; the Random and Reverse ablations are appropriately controlled; and the RoPE-aware ranking stability analysis (Table 6) is a thoughtful robustness check. However, the load-bearing empirical generalization (input-independent head roles) is supported only by aggregate, single-seed evidence, and there are notable per-category failures that the paper does not discuss.","major_comments":[{"comment":"The paper reports no error bars, no multiple seeds, and no statistical tests. The headline comparisons against Full Attention are small (e.g., Qwen3-8B: 38.78 vs. 39.10, a 0.32-point gap; Llama3.1-8B: 47.55 vs. 48.65, a 1.10-point gap), and the differences against DuoAttention and other baselines are also of single-digit size. Without variance estimates, it is impossible to tell whether these gaps reflect method behavior or run-to-run noise. The authors should report standard deviations or confidence intervals over at least several seeds, or, if this is too expensive, on a representative subset of tasks and models.","section":"§6.1, §6.2, Tables 2 and 9"},{"comment":"The central assumption that head roles are stable across inputs is only tested through aggregate statistics. Figure 3 and Figure 7 report per-head average attention distances or correlations, which average over many prompts and contexts; Figure 4 is a single-model (Llama3.1-8B-Instruct) passkey study. These do not rule out per-input role switching. A concrete warning sign is the Synthetic category on Qwen2.5-7B in Table 9: Full Attention scores 28.51, AoH at 50% sparsity scores 19.11 (a 9.4-point drop), and AoH is far below SnapKV (29.47) and Quest (20.37) on this category. This is consistent either with head-role instability or with a budget that is too aggressive for retrieval-heavy inputs, but the paper does not distinguish these possibilities. The authors should provide per-task or per-input evidence of label stability—for example, per-task correlations between effective rank and attention distance, or an analysis of how the low-effective-rank head set performs on individual prompts—or explicitly temper the claim that the frozen label is input-independent.","section":"§3, §6.2, Table 9, Figure 4"},{"comment":"The 50% sparsity budget is a manually chosen hyperparameter, as the Limitations section admits, and Figure 6 shows that this value is selected from the LongBench accuracy-sparsity trade-off curves. This means that although the head ordering is data-free, the method's overall configuration is calibrated on the evaluation benchmark. The paper should clearly separate these two aspects and either (a) justify the budget as a fixed prior or deployment constraint, or (b) provide a sensitivity analysis that does not use the test benchmark to choose the budget. Without this, the 'data-free' claim in the title and abstract is overstated.","section":"§6.4, Figure 6, Limitations"},{"comment":"The abstract states that AoH reduces prefill and decode latency by up to 41.4% and 66.0%, respectively. This is inconsistent with the reported efficiency tables: for Llama3.1-8B at 75% sparsity and 256K context, Table 11 reports a 3.24x prefill speedup (about 69.1% reduction) and a 9.14x decode speedup (about 89.1% reduction), and Table 12 reports a 14.5x decode speedup on Qwen2.5-7B. The authors should clarify the exact configuration used for the abstract's numbers or update them to match the maximum speedups reported in the tables.","section":"Abstract, §6.3, Table 11"}],"minor_comments":[{"comment":"The Conclusion contains a typo: 'trainging-free baselines' should be 'training-free baselines'.","section":"Conclusion"},{"comment":"The heading 'Reults for MoE models' contains a typo ('Reults' should be 'Results').","section":"Appendix F"},{"comment":"The label 'Effecive Rank' in Figure 3 contains a typo; it should be 'Effective Rank'. Additionally, the color scale on the heatmap is not described, making it difficult to interpret the magnitude of the effective-rank values.","section":"Figure 3"},{"comment":"The MMLU accuracy values in Table 10 are suspiciously constant (0.600, 0.595, 0.605) across all sink-size, recent-window, and sparsity combinations. The authors should explain whether this is finite-precision rounding or a genuine lack of sensitivity, and report standard deviations.","section":"Table 10"},{"comment":"In Eq. (7), the notation is dense and the normalization of the singular values is implicit. It would clarify the presentation to explicitly define the normalized singular values \\hat{\\sigma}_k before writing the effective-rank formula, as is done in Eq. (4).","section":"§5, Eq. (7)"}],"recommendation":"major_revision","confidential_remarks":"This is a clever and potentially useful method, and the mathematical core is sound. My main reservation is empirical: the paper's strongest selling point—input-free head diagnosis—rests on a stability assumption that the aggregate evidence does not yet establish. The Synthetic category failure on Qwen2.5-7B is a concrete case where the method underperforms token-level baselines, and it should be addressed head-on in revision. I would be willing to accept a revised version that provides per-input/per-task stability evidence, error bars, and a clearer treatment of the sparsity-budget tuning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first head-diagnosis method I've seen that labels retrieval/streaming heads entirely from frozen weights, and the core claim holds up better than I expected. The effective-rank of W_K^T W_Q is a real signal: low-ER heads attend farther and matter more for passkey retrieval; high-ER heads are safely streamed. The d_head-dimensional spectrum proxy is a correct and useful trick (Sylvester/AB-BA), and the RoPE-consistency analysis (Spearman >0.98 out to 128K) addresses the obvious objection that vanilla AoH ignores positional rotations. The controls are the right ones. Random and reversed ER selection are much worse, and the spectral-metric ablation (Frobenius, spectral norm, stable rank) shows effective rank specifically is what carries the signal. The GQA group-level mean aggregation, the MoE pass, and the passkey depth/length sweep all add evidence. The authors also admit the budget is manually chosen, which is honest. Soft spots, in order of real weight. No error bars or seeds anywhere; LongBench averages over 21 tasks with one run each are hard to interpret. The headline 50% sparsity and sink/recent window sizes were selected on LongBench itself, so the 0.32-point gap on Qwen3-8B is mildly optimistic; I'd want a held-out setting. No code or harness is public, which for a serving-oriented method is a real reproducibility gap. The stress-test concern about per-input head-role instability is worth stating but not established: Table 9's 9.4-point drop on Qwen2.5-7B Synthetic (28.51 to 19.11) is consistent with the 50% budget being too aggressive on retrieval-heavy tasks, and the paper doesn't distinguish budget failure from role switching. That's a missing experiment, not a contradiction of the frozen-geometry thesis. The correlations in Figure 7 and the passkey ordering are aggregate, so a direct per-input role-stability test would be the natural refereeing ask. Verdict: send it to review. The idea is clean, the math is correct, and the empirical case is strong enough to merit a serious referee even though the evaluation needs tightening. I'd suggest asking for seeds, a held-out budget, and code release.","headline":"A weight-only, data-free head classifier that mostly works; the caveats are evaluation rigor, not the core idea.","tokens_in":26511,"tokens_out":2027,"would_cite":true,"duration_ms":20410,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frozen query-key geometry alone can classify attention heads as retrieval or streaming, enabling data-free sparse attention.","keywords":["long-context LLM inference","sparse attention","effective rank","retrieval heads","streaming heads","KV cache compression","data-free head classification","query-key geometry"],"falsifier":"On a held-out long-context model, compute the effective-rank ordering of heads and then run the passkey-conversion experiment: restrict the lowest-effective-rank half of heads to sink-plus-recent attention and measure exact-match retrieval. If converting low-ER heads loses no more accuracy than converting the high-ER half, or if the Spearman correlation between effective rank and average attention distance becomes positive on a substantial task suite, the paper's central association is false.","tokens_in":1736,"feed_emoji":"⚡","tokens_out":3605,"duration_ms":109133,"temperature":0.7,"pith_summary":"This paper tries to establish that the functional role of an attention head—whether it searches the full context for specific content or mainly streams over nearby and sink tokens—is written into the model's frozen query and key projection weights, before any input is seen. It proposes Autonomy-of-Heads (AoH), a data-free and training-free classifier that labels low-effective-rank heads as retrieval heads needing global attention and high-effective-rank heads as streaming heads served by sink-plus-recent attention. Running long-context inference with this split is claimed to retain roughly 96.5% of full-attention performance at 50% sparsity while cutting prefill and decode latency and halving KV-cache memory at 256K tokens. If the paper is right, head diagnosis for sparse attention no longer needs runtime scores, calibration prompts, or learned gates.","feed_headline":"Frozen weights alone find which attention heads need full context","feed_subtitle":"No data or training: a frozen-weight spectral score halves the KV cache at ~96.5% of full-attention accuracy.","key_machinery":"The load-bearing object is the kernel attention matrix $M_h = W_K^{h\\top} W_Q^h \\in \\mathbb{R}^{d_{\\text{model}}\\times d_{\\text{model}}}$, which is exactly the head-specific middle operator in the attention-score map $X_{\\text{ctx}} W_K^{h\\top} W_Q^h x_i$. Its nonzero singular spectrum is computed efficiently from the $d_{\\text{head}}\\times d_{\\text{head}}$ proxy $C_h = (W_Q^h W_Q^{h\\top})(W_K^h W_K^{h\\top})$, whose eigenvalues are the squared singular values of $M_h$ by Sylvester's determinant theorem. From that spectrum AoH computes the effective rank, the exponential of the Shannon entropy of the normalized singular values, and uses it as a per-layer ranking: the lowest-$k$ heads (or, under GQA, the KV groups whose member heads have the lowest mean effective rank) are labeled retrieval heads and the rest streaming heads. This ranking is what carries the entire argument: it turns a qualitative retrieval-versus-streaming distinction into a one-time, input-free ordering of heads.","core_discovery":"The central claim is that the effective rank of the kernel attention operator $M_h = W_K^{h\\top} W_Q^h$, formed from frozen weights, predicts each head's contribution to long-context behavior. A concentrated singular spectrum (low effective rank) means the head has a few dominant query-key matching directions; AoH identifies these as retrieval heads, which should keep full global attention. A diffuse spectrum (high effective rank) means the head has no dominant global matching direction; AoH identifies these as streaming heads, which can safely attend only to sink tokens and a recent window. The paper supports the label with three kinds of evidence: per-head attention-distance patterns are stable across context lengths and negatively correlated with effective rank; converting low-effective-rank heads to streaming attention collapses passkey retrieval while converting high-effective-rank heads does not; and on LongBench, the effective-rank ordering beats random and reversed head selection and several alternative spectral metrics. As a direct corollary, sparse attention can begin at prefill, before any input is observed.","pith_inferences":["The same frozen-weight criterion may transfer to encoder-decoder or multimodal models, but only if their query-key projections carry the same low-rank structure; the paper's limitation section explicitly says this was not tested.","An adaptive per-layer budget could be chosen from the distribution of effective ranks within each layer—layers with many low-ER heads keep more retrieval heads—rather than the fixed 50% split, which the paper's limitations section concedes is manually chosen.","The low-ER ordering could also seed token selection for neighboring layers: top-k positions attended by a retrieval head in one layer could serve as lazy indices for reuse layers, an idea the paper's H2Share appendix starts to develop but does not claim as a general method."],"forward_implications":["At 50% sparsity AoH retains about 96.5% of Full Attention performance on average across the tested Qwen2.5-7B, Qwen3-8B, and Llama3.1-8B-Instruct models, while cutting the KV-cache budget roughly in half.","Prefill and decode latency decrease by up to 41.4% and 66.0% at 50% sparsity, with larger gains reported at 75% sparsity (up to 3.24$\\times$ and 9.14$\\times$ at 256K context on Llama3.1-8B).","Because labels come from frozen weights, AoH applies sparse attention from prefill, with no calibration prompts, runtime attention traces, or trained head gates.","The effective-rank ordering is behaviorally meaningful: random head selection and reversed ordering (high effective rank as retrieval) degrade LongBench scores substantially, and effective rank outperforms Frobenius norm, spectral norm, and stable rank as a selection metric.","As a data-free head prior, AoH can be composed with other sparse-attention techniques; the paper demonstrates this by adding a cross-layer index-reuse variant (H2Share)."],"supporting_citations":[{"why":"Defines effective rank, the spectral concentration measure AoH uses to classify heads.","marker":"Roy and Vetterli, 2007"},{"why":"Introduces attention sinks plus a recent window, the streaming-head policy AoH adopts.","marker":"Xiao et al., 2024"},{"why":"Establishes the retrieval/streaming head taxonomy and a trained-gate head-wise sparse baseline AoH compares against.","marker":"Xiao et al., 2025"},{"why":"Provides LongBench, the 21-task benchmark used for all accuracy comparisons.","marker":"Bai et al., 2024"},{"why":"Defines GQA, which forces AoH to aggregate per-head effective ranks into KV-group-level labels.","marker":"Ainslie et al., 2023"},{"why":"Provides H2O, a runtime observation-based sparse baseline that AoH must match without runtime scores.","marker":"Zhang et al., 2023"}],"fun_headline_variants":["No data, no training: frozen weights pick attention heads to prune","Spectral geometry of frozen weights exposes which heads to sparsify","KV cache halved by ranking attention heads with frozen weights","Rank frozen query-key spectra to cut KV cache without any data","Data-free head sparsity: frozen weights predict which heads to keep"],"cache_read_input_tokens":28672,"weakest_assumption_plain":"Head role is a stable, input-independent property: the effective rank of the frozen query-key product, computed without positional rotations and without the value/output projections, keeps predicting whether a head needs global attention at the 32K–256K contexts where AoH is deployed.","fun_headline_variants_meta":{"raw":{"variants":["No data, no training: frozen weights pick attention heads to prune","Spectral geometry of frozen weights exposes which heads to sparsify","KV cache halved by ranking attention heads with frozen weights","Rank frozen query-key spectra to cut KV cache without any data","Data-free head sparsity: frozen weights predict which heads to keep"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000882,"raw_usage":{"total_tokens":3839,"prompt_tokens":1000,"completion_tokens":2839,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":2751}},"tokens_in":616,"tokens_out":2839,"duration_ms":19269,"temperature":1.0,"reasoning_tokens":2751,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:37:22.879001+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out long-context model, compute the effective-rank ordering of heads and then run the passkey-conversion experiment: restrict the lowest-effective-rank half of heads to sink-plus-recent attention and measure exact-match retrieval. If converting low-ER heads loses no more accuracy than converting the high-ER half, or if the Spearman correlation between effective rank and average attention distance becomes positive on a substantial task suite, the paper's central association is false.","supporting_citations":[{"cited_title":"2024 , cdate=","cited_arxiv_id":null,"evidence_quote":"Introduces attention sinks plus a recent window, the streaming-head policy AoH adopts."}],"review_version":1}