{"id":"210396f7-878f-45f0-998a-9118f983fba1","arxiv_id":"2601.08131","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ExoFormer shows that externally computed anchor projections, mixed with normalized Q/K/V/G projections, improve LLM perplexity and downstream accuracy over internal first-layer anchors.","lead":"This paper introduces ExoFormer, a Transformer design that projects the input embedding through separate \"anchor\" matrices and mixes those projections into every layer's attention alongside the layer's own queries, keys, values, and gates. The authors report that this external-anchor design beats first-layer-anchor reuse on perplexity and gives about 1.5 points higher downstream accuracy than gated attention at 450M–1B scale.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central external-vs-internal claim is untested at 1B and already inconsistent at 450M: H-ExoFormer underperforms H-NuResFormer and no 1B internal-anchor control exists.","rationale":"Reader's conditional verdict is appropriately cautious. The architecture is well-specified, and the 450M results in Table 1 are internally plausible: E-ExoFormer and Dynamic E-ExoFormer achieve the best perplexity and best average accuracy among the compared variants, and the normalization story (Section 4.2) has at least correlational support. However, the central thesis—that externalizing the anchor is the cause of the gains—requires comparing ExoFormer against NuResFormer at the same scale. Table 4 provides only Dynamic ExoFormer vs Gated Attention at 1B; no internal-anchor or static external variant is run. Since the abstract advertises '450M–1B scale' and the conclusion claims 'this decoupling consistently improves performance,' the missing 1B internal control is a load-bearing gap. It is possible the Dynamic Mixing MLP, the extra 4d² parameters, or better optimization conditions explain the 1B gain. Additionally, Table 1 itself weakens the word 'consistently': H-ExoFormer's 49.23 average accuracy is below H-NuResFormer's 49.42, and differences of 0.1–0.2 points without error bars are likely noise. The separate '1.5× fewer tokens' claim is also not directly tabulated; both 1B models in Table 4 were trained on 20B tokens, so the data-efficiency statement must be reconstructed from loss curves and needs a dedicated token-ablation. These issues make the paper suitable for conditional acceptance contingent on the missing controls, not rejection: there is no apparent internal inconsistency in the mechanism, and the 450M data provide some positive evidence.","tokens_in":15438,"tokens_out":9748,"duration_ms":89951,"concrete_test":"Run Dynamic NuResFormer—the internal-anchor analogue with the same Dynamic Mixing MLP, base-λ initialization, and 4d²-equivalent capacity—at 1B/20B tokens under Table 4 hyperparameters, with 3 seeds. Compare final validation loss and average 5-shot accuracy to Dynamic ExoFormer. If Dynamic NuResFormer matches or beats Dynamic ExoFormer, the claim that exogenous anchoring drives the 1B gain fails; if Dynamic ExoFormer is clearly better across seeds, the scale-generalization gap is closed. Also report seed-wise 450M results for H-ExoFormer vs H-NuResFormer to test the consistency claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim requires that exogenous anchors are the cause of the observed gains, not merely that a particular ExoFormer variant beats Gated Attention. That causal attribution is untested at 1B. Table 4 reports only Gated Attention and Dynamic ExoFormer at 1B; there is no NuResFormer (internal anchor) control, nor a static ExoFormer, at that scale. Consequently the 1B result cannot distinguish the external-anchor hypothesis from the Dynamic Mixing MLP, the extra 4d² anchor parameters, or the elementwise mixing itself. The 450M table also undercuts the word 'consistently': H-ExoFormer averages 49.23 vs 49.42 for H-NuResFormer (full residual norms), and S-ExoFormer is only +0.05 over its counterpart; with no seeds or error bars these differences are within noise. The paper's own abstract claims 'ExoFormer variants consistently outperform their internal-anchor counterparts,' but at the only scale where that comparison exists, one variant goes the wrong way.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ExoFormer, a Transformer modification in which the query, key, value, and gate-logit projections of every layer are replaced by a normalized, learnable mixture of the current layer's projections and a fixed set of 'anchor' projections. Two anchor instantiations are compared: NuResFormer, which reuses the first layer's projections as anchors, and ExoFormer, which uses a dedicated external projection module applied to the input embeddings. The authors train models around 450M and 1B parameters on FineWeb-Edu and evaluate validation perplexity plus six multiple-choice tasks. The central claims are that ExoFormer variants consistently outperform their internal-anchor counterparts, that the Dynamic E-ExoFormer variant beats Gated Attention by roughly 1.5 downstream accuracy points while matching validation loss with about 1.5x fewer tokens, and that these results support an 'Offloading Hypothesis' in which external anchors preserve token identity while layers specialize in refinement.","tokens_in":15784,"tokens_out":6991,"duration_ms":69240,"significance":"If the central claims held, the decoupling of anchor projections from the first layer would be a simple and transferable architectural idea with modest parameter and FLOP overhead, and the paper's systematic comparison of scalar/headwise/elementwise mixing and anchor normalization would be a useful reference for future work. The authors also release code and models and provide a transparent complexity analysis in Appendix C. However, the evidence reported in the current version is too thin and partly contradicts the headline claim: the 450M table shows one internal-anchor variant outperforming its external counterpart, and the 1B experiment lacks the internal-anchor control needed to attribute the gains to exogenous anchors. The contribution is promising but requires substantially more experimental support and a recalibration of the claims.","major_comments":[{"comment":"The assertion that 'ExoFormer variants consistently outperform their internal-anchor counterparts' is not supported by the reported 450M results. H-ExoFormer averages 49.23 accuracy vs. 49.42 for H-NuResFormer, S-ExoFormer is only +0.05 over S-NuResFormer, and only E-ExoFormer shows a nontrivial gain (+0.17). With a single run per configuration and no error bars, differences of this size are indistinguishable from noise. Please report multiple seeds or otherwise provide uncertainty information, and revise the 'consistently' / 'every ExoFormer variant outperforms' language to match the data.","section":"Table 1 / Abstract / §1"},{"comment":"The 1B experiment compares only Gated Attention and Dynamic E-ExoFormer. There is no 1B NuResFormer (internal-anchor) control, no static E-ExoFormer, and no dynamic internal-anchor control. Consequently the 1B gains cannot be attributed specifically to the exogenous anchor; they could come from the Dynamic Mixing MLP, elementwise mixing, the gating mechanism, or the extra 4d^2 anchor parameters. This missing control is load-bearing for the paper's central external-vs-internal claim.","section":"§4.1 / Eq. (11)-(12) / Fig. 2"},{"comment":"The paper reports no seeds, error bars, or statistical significance measures. The headline comparison against Gated Attention is reasonably large (+1.47 average accuracy, PPL 14.09 vs. 14.64), but the internal-anchor comparisons are around 0.1-0.2 accuracy points and 0.01-0.02 perplexity, which are within ordinary run-to-run variation for this scale and budget. The manuscript should either provide variance estimates or explicitly state that these small differences are not statistically resolved.","section":"Table 1 / §4.1"},{"comment":"The data-efficiency claim ('matching validation loss using ~1.5x fewer tokens than Gated Attention') is not backed by any reported token-matched measurement. If this is inferred from Fig. 2, the manuscript should define the token threshold, show how it is computed, and report the associated uncertainty. As written, the claim appears without experimental support in the text.","section":"Abstract / §1 / Fig. 2"},{"comment":"The Offloading Hypothesis is presented as validated, but the evidence is qualitative and partly circular. The anchor-removal ablation in §4.5 is expected: a model trained with a persistent anchor will fail when that anchor is removed, and this does not by itself prove that the sequential layers 'specialize in refinement' rather than merely relying on a trained input. The hypothesis may be a useful interpretation, but the conclusion should not assert validation without direct tests or a clear falsification condition.","section":"§4.5 / Conclusion"}],"minor_comments":[{"comment":"Section 3.4 states that all lambda parameters are initialized to 0.5, while Section 3.6 says the base lambda parameters must be initialized to 1.0 for the dynamic variant. Please clarify this apparent inconsistency explicitly.","section":"§3.4 / §3.6"},{"comment":"The column names 'No Gate', 'With Gate', and 'Decoupled (ExoFormer)' do not map transparently to the Table 1 rows (e.g., NuResFormer models include gating logits; are they in the 'With Gate' column?). Add an explicit mapping or an anchor-type column.","section":"Table 3 / Table 1"},{"comment":"The phrase 'yields 1.5x downstream accuracy points' is imprecise; it should be '~1.5 percentage points of average downstream accuracy relative to Gated Attention' to avoid confusion with a multiplicative factor.","section":"Abstract"},{"comment":"The figure caption refers to validation loss for ~450M and ~1B models, but the curves are not labeled in the reproduced text. Please ensure all curves (Gated Attention, NuResFormer, ExoFormer, Dynamic ExoFormer) are legibly labeled and described, especially in the right panel.","section":"Fig. 2"}],"recommendation":"major_revision","confidential_remarks":"The proposed framework is worth pursuing, but the current evidence is underpowered for the paper's central claims. I would encourage the authors to add multi-seed runs, an internal-anchor control at 1B, and a direct token-matched data-efficiency experiment. The code and model release is a positive factor that makes these additional experiments feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The bottom line: ExoFormer is a sensible variation on the ResFormer/MUDDFormer line, and the headline result — a dynamic variant that beats Gated Attention by ~1.5 downstream points and ~1.5x data efficiency at 450M–1B — is worth testing. But the paper's strongest claim, that external anchors consistently beat internal anchors, is not actually supported by the table the authors show.\n\nWhat's new: the unified normalized mixing of Q, K, V, and gate logits with learnable coefficients at three granularities is a clean generalization of value-only residuals, and the idea of computing anchors from H0 with dedicated projection matrices is not in the cited prior work. The analysis of normalization (RMSNorm on anchor sources) is useful, and the mix-compress-refine story connecting gated attention and residual mixing to attention sinks is a nice interpretive contribution, even if it is post-hoc. The complexity analysis is honest, including the 8–15% measured latency overhead versus the 1.3% FLOP estimate. Code and models are promised.\n\nWhere it's soft: the abstract says \"consistently outperform their internal-anchor counterparts,\" but in Table 1, H-ExoFormer has lower average accuracy (49.23) than H-NuResFormer (49.42), and S-ExoFormer is only +0.05. The perplexity advantage is consistent, but the accuracy advantage is not. The 1B experiment compares only Dynamic ExoFormer against Gated Attention; there is no NuResFormer or static ExoFormer control at that scale, so the 1B gain cannot be attributed to the exogenous anchor rather than to the dynamic mixing MLP, the extra 4d² anchor parameters, or the elementwise mixing itself. The data-efficiency claim (~1.5x fewer tokens) is not directly tabulated — no token-ablation table appears. There are no seeds or error bars, and with 15 variants the selection of the best one inflates the expected gain. These are real limitations, but they are limitations of evidence, not of the underlying idea.\n\nWho this is for: People working on efficient pretraining or architectural tweaks for Transformers will find the framework useful. It deserves a serious referee — the experiment cost is nontrivial and the idea is falsifiable. My recommendation: send it to review, but request a 1B internal-anchor control, a token-ablation table, and multiple seeds for the key comparisons.","headline":"The external-anchor idea is clean and the dynamic variant's gains over Gated Attention are worth a serious look, but the paper's own table undercuts the 'consistently outperform' claim, and the 1B experiment lacks the necessary internal-anchor control.","tokens_in":16221,"tokens_out":3071,"would_cite":false,"duration_ms":28658,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ExoFormer: a Transformer that feeds every layer a fixed set of attention projections computed once from the input embeddings, outperforming internal-anchor reuse.","keywords":["ExoFormer","attention projection mixing","exogenous anchors","cross-layer reuse","token identity","normalized mixing","gated attention","data efficiency"],"falsifier":"Train a 1B-parameter ExoFormer on 50B+ tokens and compare against gated attention with matched compute; if ExoFormer's validation loss does not stay ahead or its downstream accuracy advantage narrows below one point, the offloading benefit is not monotonic with data. Alternatively, replace the H0-derived anchors with a fixed random projection that is frozen after initialization; if the gains persist, the benefit is not from learned identity preservation.","tokens_in":15298,"feed_emoji":"⚙️","tokens_out":3517,"duration_ms":29107,"temperature":0.7,"pith_summary":"The paper argues that reusing early attention projections in deep Transformers creates a conflict: the first layer must simultaneously act as a stable reusable anchor and as an effective computational layer. To resolve this, it introduces ExoFormer, which computes a dedicated set of anchor projections once from the input embeddings and mixes them into every layer's queries, keys, values, and gate logits using learned, normalized coefficients. Across 450M- and 1B-parameter models trained on 10-20B tokens, ExoFormer variants consistently beat their internal-anchor counterparts, and a dynamic variant achieves roughly 1.5 higher downstream accuracy points while matching validation loss with about 1.5x fewer training tokens than gated attention. The paper proposes the Offloading Hypothesis: the external anchor preserves token identity, freeing sequential layers to specialize in feature transformation.","feed_headline":"ExoFormer beats gated attention with fixed attention anchors","feed_subtitle":"A Transformer that reuses one set of projections computed from the input saves ~1.5x training tokens at matched validation loss.","key_machinery":"Exogenous anchor projections: a set of four projection matrices (Q, K, V, G) applied once to the input embeddings H0, producing anchor tensors that are reused at every layer. They are mixed with current-layer projections via learnable coefficients lambda (scalar, headwise, or elementwise) after RMS-normalizing the anchor source; a dynamic variant modulates these coefficients via a per-layer two-layer MLP with sigmoid output. This mechanism is what offloads token-identity preservation from the residual stream.","core_discovery":"The paper's central claim is that decoupling the anchor role from the first layer, by learning a fixed set of exogenous projections W^Q_anc, W^K_anc, W^V_anc, W^G_anc applied to the input embeddings H0, resolves a structural tension in cross-layer attention reuse. In the internal-anchor design (NuResFormer), the first layer must both act as a reusable reference and perform its own computation, a compromise the paper shows empirically through permissive first-layer gating. ExoFormer instead mixes the exogenous anchor with each layer's own projections, bS_n = lambda_{n,1} * RMSNorm(S_anc) + lambda_{n,2} * S_n, with optional context-dependent scaling from a small MLP. The paper reports that eve","pith_inferences":["A natural next test is whether the exogenous anchor remains effective at 10B+ parameter scale, or whether a single H0-based projection becomes a bottleneck as depth grows; the paper itself notes this is untested beyond 1B.","The mixing framework could be reinterpreted as a learned, layer-independent bias in the attention logits; comparing ExoFormer to a learned per-layer additive bias in Q/K might isolate what the anchor contributes beyond a simple regularizer.","The offloading story suggests an architectural family: any mechanism that injects a fixed, token-discriminative signal at every layer (e.g., input-embedding residuals, learned identity embeddings) should produce similar gains, a testable prediction.","The runtime overhead (8-15% in the reference implementation vs. theoretical 1.3%) suggests that with a fused kernel the data-efficiency gains could translate into wall-clock savings, a practical engineering implication."],"forward_implications":["If ExoFormer's gains hold, training LLMs to a given validation loss requires roughly 1.5x fewer tokens than gated attention, directly cutting compute and energy costs.","The Offloading Hypothesis gives a mechanistic picture: deep layers can devote capacity to feature transformation because identity information is re-injected from a fixed source; this should be testable in other architectures, such as deeper or wider models.","Elementwise mixing granularity becomes the best choice when anchors are exogenous, while scalar is best for internal anchors, a design lesson for future residual mixing.","Anchor normalization (RMSNorm) is necessary; unnormalized mixing suppresses the anchor pathway, suggesting that distributional mismatch is a key obstacle to projection reuse.","Removing the anchor at inference causes catastrophic loss of token distinctiveness, meaning the architecture deliberately trades a fixed external memory for internal capacity."],"fun_headline_variants":["Exogenous anchors resolve attention reuse tension","ExoFormer: Offloading anchors cuts training tokens 1.5x","Fixed projections outside layers beat internal anchors","Decoupling anchor role boosts Transformer efficiency","External anchors preserve token identity, speed training"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method assumes that a single set of projections computed from the input embeddings carries enough token-identity information to remain useful at every depth, and that RMSNorm plus linear mixing prevents distributional drift; if this fails at larger scale or with other anchor sources, the reported gains may be specific to this training setup.","fun_headline_variants_meta":{"raw":{"variants":["Exogenous anchors resolve attention reuse tension","ExoFormer: Offloading anchors cuts training tokens 1.5x","Fixed projections outside layers beat internal anchors","Decoupling anchor role boosts Transformer efficiency","External anchors preserve token identity, speed training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1393,"prompt_tokens":735,"completion_tokens":658,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":588}},"tokens_in":479,"tokens_out":658,"duration_ms":7318,"temperature":1.0,"reasoning_tokens":588,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T10:52:55.211678+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a 1B-parameter ExoFormer on 50B+ tokens and compare against gated attention with matched compute; if ExoFormer's validation loss does not stay ahead or its downstream accuracy advantage narrows below one point, the offloading benefit is not monotonic with data. Alternatively, replace the H0-derived anchors with a fixed random projection that is frozen after initialization; if the gains persist, the benefit is not from learned identity preservation.","supporting_citations":[],"review_version":1}