{"id":"bdcd1435-4412-4a51-b101-5bd197e9864f","arxiv_id":"2608.08888","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fusing the previous top-layer hidden state into the next input token through a gated linear unit improves transformer pretraining data efficiency and decoding accuracy at 1B scale.","lead":"This paper adds a feedback channel to transformer language models, feeding the previous step's top-layer hidden state back into the model along with the sampled token. The method improves accuracy and reasoning efficiency at 1B scale, matching models trained on substantially more data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Contraction evidence (Figs. 3/10) is obtained on fixed-token prefill passes, whereas decoding iterates the feedback map over the model's own sampled tokens; the central inference-time equivalence claim depends on this untested transfer.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern I find: the stability diagnostics validate the wrong iteration. Training and the repeated-prefill-pass diagnostic iterate the feedback map over fixed ground-truth token embeddings, whereas decoding couples the state update to the model's own sampled tokens. The paper's own Sec. 6 concedes that the feedback-pass schedule is a heuristic, and Figs. 3/10 do not report sampled-token rollout stability. I agree with the CONDITIONAL verdict: the empirical results are plausible and internally consistent, the synthetic probes and the 3%-three-pass contraction result are genuine supporting evidence, and the paper is honest about the heuristic scheduling. But the central efficiency claim rests on a distribution transfer that has not been demonstrated. Secondary issues—single training runs, no released code or checkpoints, no error bars on Fig. 5/Table 1—amplify the risk but are not the core weakness. Because the reader already identified this concern, no verdict change is needed; the proposed rollout-length diagnostic would either confirm or retire the concern.","tokens_in":19665,"tokens_out":8619,"duration_ms":88780,"concrete_test":"On the released (or independently reproduced) 200B/400B checkpoints, run SOFT decoding on held-out prompts from GSM8K, Math500, HumanEval, MBPP, and open-domain text for unrolled lengths of 100, 500, and 2000 sampled tokens. At every step record ||h_t − h_{t−1}|| and ||h_t||, and compare these sampled-token trajectories with the repeated-prefill-pass iterates of Fig. 3/Fig. 10 computed on the same prefixes. If on sampled-token rollouts the hidden-state update does not decay to the same plateau (e.g., remains above 2x the prefill-pass plateau or grows with length), the contraction claim does not transfer to decoding and the headline equivalence should be conditioned on rollout length. A secondary check isolates teacher forcing from exposure bias: feed each sampled sequence through one extra fused prefill pass and verify that validation loss stays flat.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim—latent feedback during decoding matches standard baselines trained on roughly 2x more tokens—requires the learned feedback map to remain stable and useful over long autoregressive rollouts. Training (Eqs. 9–11) and the scheduled multi-pass objective expose the model only to teacher-forced ground-truth tokens: pass k fuses shifted states h^{(k-1)}_{t-1} with actual embeddings e_t and builds contexts from actual data. Decoding (Eqs. 3 and 8) instead feeds back h_{t-1} produced from the model's own sampled tokens a_t, so the input distribution changes every step as a function of the state. The stability evidence in Fig. 3 and Fig. 10 iterates a different map: it repeatedly applies fused prefill passes over fixed tokens, i.e., iterates k on the same positions while the token inputs stay constant. A contraction on that fixed-token iteration does not imply contraction of the stochastic sampled-token dynamics that decoding actually runs. The 3% three-pass stabilization is explicitly acknowledged as heuristic in Sec. 6, and no decoding-side hidden-state stability diagnostic is reported. Because the matched-2x-tokens results (Fig. 4, Fig. 5, Table 1) and the shorter-trace results (Fig. 6, Fig. 8) are measured on finite, single-run decoded outputs, any divergence or quality collapse on longer or out-of-distribution rollouts would invalidate the efficiency transfer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the full-bandwidth transformer, which widens the inter-step feedback channel of an autoregressive transformer by fusing the previous top-layer hidden state with the sampled token embedding via a gated linear unit (Eq. 4) and feeding the result back as the next input. To train this recurrence without sacrificing parallel teacher forcing, the authors propose a scheduled multi-pass objective that starts from a standard checkpoint, spends most of training on one-pass batches, and mixes small fractions of two- and three-pass batches; a prefix mixin and lightweight stabilization recipes (RMSNorm on fused inputs, tied embeddings, jitter noise) are added to keep the feedback map stable. At 1B scale and up to 400B tokens, the paper reports improvements in validation loss, 5-shot LM Eval, GSM8K, MATH-500, HumanEval, MBPP, and instruction-tuned settings, with fused prefilling letting a 100B-token model match a 200B-token standard baseline and a 200B-token model match a 400B-token standard baseline. The paper also reports shorter reasoning traces under latent-feedback decoding and controlled state-tracking experiments showing that recurrent prefilling exposes global state to shallow layers.","tokens_in":19867,"tokens_out":6311,"duration_ms":61300,"significance":"If the empirical claims hold, this is a significant result: it proposes a new axis of scaling—spending extra compute per token through a parallelizable latent-feedback recurrence—while preserving the standard transformer architecture, KV cache, and language-modeling objective. The paper is unusually concrete on engineering: it provides full training and inference pseudo-code, a vLLM-compatible implementation strategy, an explicit token-equivalent-compute accounting, and controlled synthetic probes that mechanistically verify the claimed bandwidth increase. The central falsifiable predictions—shorter reasoning traces and shallow-layer accessibility—are clean and well motivated. The main risk is not circularity but transfer of the stability evidence: the contraction diagnostics are run on repeated fused prefill passes over fixed tokens, whereas decoding iterates the feedback map over the model's own sampled tokens. This is a load-bearing gap that the revision should close before the 2x-token equivalence claim can be considered established.","major_comments":[{"comment":"The stability evidence in Fig. 3 and Fig. 10 is obtained by repeatedly applying fused prefill passes over fixed token sequences, i.e., iterating k in Eqs. (10)-(11) while the token embeddings remain constant. Decoding, by contrast, iterates Eq. (8) where the token a_t is sampled from the model's own distribution conditioned on the fed-back hidden state, so the input distribution changes at every step as a function of the state. A contraction on the fixed-token iteration does not by itself imply stability of the stochastic sampled-token dynamics, and the sentence in Sec. 3.3 claiming that 'hundred-token rollouts show no sign of breakdown' is not supported by a diagnostic on the actual decoding distribution. Because the central equivalence claim in Secs. 4.1-4.2 depends on long, stable latent-feedback decoding, please add decoding-side diagnostics: hidden-state norm and step-to-step change over 100-1000 generated tokens, task quality as a function of rollout length, and ideally multiple decoding runs or seeds to check for divergence or collapse.","section":"Sec. 3.3, Eqs. (8)-(11), Figs. 3 and 10"},{"comment":"The headline results that latent feedback 'matches or approaches' standard baselines trained with roughly 2x more tokens are reported from single runs without error bars or repeated decoding seeds. The math and coding numbers are Pass@1 and Pass@3 estimates from 10 rollouts per problem with temperature selected separately per method, so sampling noise is nontrivial. Please report confidence intervals, ideally across multiple decoding seeds, for SOFT and FUSED versus STANDARD decoding, and state whether the observed ordering is stable. Without this, the 2x-token equivalence claim is not fully supported by the evidence as presented.","section":"Sec. 4.2, Fig. 5, Table 1, Fig. 6"},{"comment":"The stability recipe is centered on a specific unablated configuration: 75% one-pass, 22% two-pass, 3% three-pass batches, with jitter noise sigma=0.02 and feedback objective weight lambda=1. The paper acknowledges in Sec. 6 that the schedule is heuristic, but because Fig. 3 shows that the 3% three-pass fraction is precisely what changes a diverging map into a contraction, the reader cannot tell how robust the recipe is to model scale, data mixture, or training length. Please include a small sensitivity study of the three-pass fraction and sigma, or replace the heuristic with a principled fixed-point convergence check that determines the schedule.","section":"Sec. 3.3, Fig. 3, Sec. 6"}],"minor_comments":[{"comment":"There are several typos: 'veritical' should be 'vertical' in Sec. 2, 'perfix' should be 'prefix' in the Fig. 7 caption, 'comparision' should be 'comparison' in the Table 2 caption, and 'bandwith' should be 'bandwidth' in Fig. 4.","section":"Sec. 2, Fig. 7, Table 2"},{"comment":"Equation (4) uses the notation sigma and circled-dot without defining them in the text; please state explicitly that sigma is the sigmoid function and circled-dot is elementwise multiplication.","section":"Eq. (4)"},{"comment":"The 10B run is trained with 100% three-pass batches, which is inconsistent with the schedule used for the 100B-400B runs; please explain why this run is included in the same figure or move it to an ablation.","section":"Fig. 4"},{"comment":"The abstract says 'roughly 1.5x more tokens' while Sec. 4.1 concludes 'roughly 2x pretraining data efficiency'; please reconcile the claimed factor or specify the different regimes to which each number refers.","section":"Abstract and Sec. 4.1"},{"comment":"The comment 'standard decoding: x = embed(tok)' is placed on a line that is commented out in a way that may confuse readers; please clarify that the active fused line is the only change relative to standard decoding.","section":"Listing 2"},{"comment":"The temperature values selected for each decoding regime are not reported; please list the chosen temperatures for each task and method, since the comparison is conditioned on this selection.","section":"Sec. 4.2"},{"comment":"The claim that latency-feedback decoding costs 'under 1% per token' should be quantified relative to the full forward pass, and the overhead of the vLLM buffer copy described in Appendix D should be included in that estimate.","section":"Sec. 3.1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about stability transfer is genuine and not a circularity issue; it is a missing experiment on the actual decoding distribution. The paper's own Sec. 6 admission that the schedule is heuristic strengthens the need for the decoding-side diagnostics and schedule sensitivity analysis I requested. The related-work discussion is thorough and I have no concerns about novelty disclosure or citation patterns. I would be willing to look at a revised version that closes this gap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a serious empirical paper about a simple, clean idea — at each decoding step, fuse the previous top-layer hidden state with the sampled token embedding through a gated linear unit and feed that back as the next input. They train 1B models up to 400B tokens with a scheduled multi-pass objective, and the results are consistently positive: validation loss, LM Eval, GSM8K, Math500, HumanEval, MBPP, and instruction tuning all improve relative to self-trained standard baselines. The 100B recurrent model matching the 200B baseline, and 200B matching 400B, is the kind of claim that makes you pay attention.\n\nWhat's actually new: the externally gated injection (only 2D² new parameters, no architecture change, KV cache and serving stack untouched), and the specific finding that mixing just 3% of three-pass batches turns a feedback map that diverges past its trained depth into one that stays stable for hundreds of passes. That stabilization result is the most interesting thing here. The synthetic state-tracking probes are also a good mechanistic check — one recurrent prefill step makes the target linearly decodable at layer 0, which is what the bandwidth story predicts. The paper also does right by prior work: Feedback Transformer, Coconut, Soft Thinking, T2MLR, and LRT are cited and discussed honestly, with a candid concession that other injection schemes might perform similarly.\n\nSoft spots, in proportion. The headline free-form generation comparisons (Fig. 5, Table 1) are single runs without error bars, while the validation-loss plots have them. The 75/22/3 schedule is a post-hoc heuristic — the paper says so in Sec. 6 — and we don't know how robust it is to scale or data changes. And the stability evidence deserves a closer look: the contraction diagnostic in Figs. 3/10 iterates over fixed token embeddings, whereas decoding iterates over the model's own sampled tokens. That is a genuine mismatch between diagnostic and deployment. I'd push back, though, on calling it a load-bearing flaw: the free-form generation results are themselves decoded on sampled tokens for hundreds of steps, and they improve rather than collapse. So it's a gap in explanation, not evidence of failure — but a decoding-side hidden-state stability measurement would make the 2x-data claim much more convincing. Minor notes: the 1T baseline uses a larger batch size, and no code or data are released.\n\nWho it's for: anyone working on recurrent or feedback transformers, latent reasoning, or spending extra training compute to buy inference efficiency. It deserves a serious referee; I'd send it out and expect heavy but productive revision. If I were reviewing, my main asks would be error bars on the decoding results, some robustness check on the schedule, and ideally released code. I'd cite it.","headline":"A clean, well-executed 1B-scale study of feeding the top-layer state back through a gate — consistent gains and a genuinely interesting stabilization finding, but the headline 2x data-efficiency claim needs error bars and a decoding-side stability check before I'd fully trust it.","tokens_in":20519,"tokens_out":7227,"would_cite":true,"duration_ms":64012,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Autoregressive transformers waste a second feedback channel between decoding steps; fusing the previous top-layer hidden state with the sampled token embedding through a gated linear unit converts negligible per-token compute into gains…","keywords":["latent feedback","full-bandwidth transformer","recurrent decoding","multi-pass pretraining","data efficiency","gated linear unit","reasoning trace length","temporal parallelism"],"falsifier":"Hold the weights of a 200B full-bandwidth model fixed and run SOFT decoding past the trained 3-pass horizon on held-out, out-of-distribution prompts while logging $\\lVert h_t^{L} - h_{t-1}^{L}\\rVert$: if the per-step state change grows or oscillates instead of decaying to a small plateau, the contraction premise fails; if SOFT does not beat STANDARD decoding on GSM8K or HumanEval across the same rollouts, the data-efficiency claim is overturned.","tokens_in":19364,"feed_emoji":"🔁","tokens_out":11201,"duration_ms":110486,"temperature":0.7,"pith_summary":"The paper claims that autoregressive transformers waste a second feedback channel: when decoding, only the sampled token re-enters the bottom of the stack, while the top-layer hidden state, a full $D$-dimensional vector, is discarded. It proposes latent feedback decoding, which fuses that previous top-layer state with the sampled token embedding through a gated linear unit and feeds the fusion back as the next input, at a per-token cost below one percent of the forward pass. To train this recurrence without giving up parallel teacher forcing, it uses a scheduled multi-pass objective: extra fused passes are introduced late in pretraining, with a small fraction of three-pass batches that empirically turns the learned feedback map into a contraction stable far beyond the trained depth. At 1B scale, latent-feedback models match or approach standard transformers trained on roughly $1.5\\times$ more tokens, improve math and coding generation, and produce shorter reasoning traces at equal or better accuracy on base models. The reason to care is that it trades a negligible amount of inference-time compute for what would otherwise require substantially more pretraining data.","feed_headline":"One decode tweak matches a model trained on twice the data","feed_subtitle":"Fusing the last hidden state into the next input matches 2x-token baselines at 1B scale.","key_machinery":"The central object is latent feedback decoding, a gated linear unit fusion $e_t \\otimes h_{t-1}^{L} = W_U h_{t-1}^{L} \\odot \\sigma(W_G e_t)$ that turns the previous top-layer hidden state into the next input, with the sampled token embedding acting only as a multiplicative gate. Around it sits the scheduled multi-pass training objective: each pass shifts the previous pass's hidden states one position right, fuses them with token embeddings, and re-runs the stack in parallel across positions, so a length-$T$ recurrence is trained over $k$ instead of $T$ sequential steps. The load-bearing stability ingredient is the feedback-pass schedule, mostly single-pass, with about 22\\% two-pass and 3\\% three-pass batches late in training, which empirically converts the learned feedback map into a contraction toward a fixed point; prefix mixin matches the prompt-then-generate input distribution, and depth scaling plus a shared input basis and jitter noise keep long rollouts stable.","core_discovery":"On the paper's own terms, the discovery is that the vertical feedback channel between decoding steps can be widened to its full width without changing the transformer architecture or the language-modeling objective. The mechanism is the gated fusion $e_t \\otimes h_{t-1}^{L} = W_U h_{t-1}^{L} \\odot \\sigma(W_G e_t)$: the sampled token embedding acts as a gate on the previous top-layer state, which occupies the value pathway, so discarding the state would discard the input itself and the channel cannot be bypassed. Trained with a scheduled multi-pass objective (temporal parallelism), latent feedback yields better validation loss and 5-shot accuracy, with the 100B-token recurrent model matching the 200B-token standard baseline and the 200B model matching the 400B standard baseline under fused prefilling. On free-form generation, the same weights under latent-feedback decoding beat standard decoding on GSM8K, MATH-500, HumanEval, and MBPP, sometimes approaching baselines trained with up to $5\\times$ the tokens, and base models often produce much shorter reasoning traces at equal or better accuracy. A secondary finding is that the training signal itself improves representations: even evaluated with standard decoding, models trained with the latent-feedback objective beat next-token-only models on the same data.","pith_inferences":["The stabilization result implies a transferable recipe the paper does not test: a 3\\% dose of deeper unrolls may contractivize other shallow-trained recurrent maps, such as looped transformers, at little extra training cost.","The disappearance of concise traces after instruction tuning suggests an untested fix: on-policy post-training whose rollouts are decoded with latent feedback should keep both the accuracy and the conciseness.","Should the token-equivalence hold at larger scales, the practical scaling axis shifts from acquiring more unique tokens to spending more compute per unique token, which would matter most for data-constrained training runs.","A more direct test of the mechanism than any downstream benchmark would be to measure contraction on long sampled-token rollouts, since the paper's stability diagnostic repeatedly applies fused prefilling on the training distribution rather than the actual decoding distribution."],"forward_implications":["Under fused prefilling, the 100B-token full-bandwidth model matches the 200B-token standard baseline and the 200B-token model matches the 400B-token standard baseline in validation loss and 5-shot LM Eval accuracy, so a small amount of prefill-time compute buys roughly $2\\times$ pretraining data efficiency.","With weights held fixed, latent-feedback decoding (SOFT) improves over standard decoding on every free-form task tested; on MATH-500 the 200B model rises from 0.27 to 0.37 Pass@1, surpassing the 1T-token no-recurrence baseline.","The gains survive long-context extension and instruction tuning: on GSM8K the instructed model improves from 64.5 to 67.9 and on HumanEval from 42.5 to 45.9, beating the matched 200B standard baseline on all tasks.","On base models, latent feedback often produces markedly shorter reasoning traces at equal or better accuracy, consistent with intermediate computation riding the hidden state instead of being verbalized; this conciseness disappears after instruction tuning because the tuning traces imitate standard verbose reasoning.","Training with the latent-feedback objective improves downstream 5-shot accuracy even when the model is evaluated as an ordinary transformer with standard decoding, so the multi-pass objective acts as an auxiliary state-supervision signal independent of the decoding change."],"supporting_citations":[{"why":"Establishes the feedback-memory idea in transformers; its ablation showing a top-layer memory nearly matches a full-layer mixture motivates the choice to feed back the top-layer state.","marker":"Fan et al., 2020"},{"why":"Supplies the multi-forward-pass parallel training strategy with interleaved embedding and hidden-state inputs that the paper's temporal parallelism schedule adapts, along with Jacobi-style convergence diagnostics for choosing recurrence depth.","marker":"Zeng et al., 2025"},{"why":"Provides the depth scaling that keeps the top-layer state norm $O(1)$, a load-bearing part of the stability recipe for long feedback horizons.","marker":"Yang et al., 2024"},{"why":"Defines chain-of-thought decoding, the verbalized feedback channel against which latent feedback is compared as a wider, non-verbal channel that carries intermediate state without spending tokens.","marker":"Wei et al., 2022"},{"why":"A temporal middle-layer recurrence with a similar training motivation; the paper contrasts its extra MLP parameters and small-scale evaluation with the external, input-level reinjection used here.","marker":"Cai et al., 2026"},{"why":"Latent Recurrent Transformer, the closest method in motivation; the paper argues its contributions are external reinjection, minimal added parameters, and validation at up to 400B training tokens.","marker":"Huang et al., 2026"}],"fun_headline_variants":["Widen the feedback channel: match 2x-token baselines","Gated hidden-state reuse halves training tokens","Full-bandwidth transformer: 1.5x token savings","Latent feedback: 1B model matches 2x-token training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All downstream gains depend on one thing: the feedback map learned from at most three training passes must keep behaving stably over hundreds or thousands of generated tokens; the paper verifies this with repeated fused-prefill passes on the training distribution, not with long sampled decoding, and admits the schedule is a heuristic.","fun_headline_variants_meta":{"raw":{"variants":["Widen the feedback channel: match 2x-token baselines","Gated hidden-state reuse halves training tokens","Full-bandwidth transformer: 1.5x token savings","Latent feedback: 1B model matches 2x-token training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1529,"prompt_tokens":1065,"completion_tokens":464,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":392}},"tokens_in":681,"tokens_out":464,"duration_ms":5456,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:21:59.482253+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold the weights of a 200B full-bandwidth model fixed and run SOFT decoding past the trained 3-pass horizon on held-out, out-of-distribution prompts while logging $\\lVert h_t^{L} - h_{t-1}^{L}\\rVert$: if the per-step state change grows or oscillates instead of decaying to a small plateau, the contraction premise fails; if SOFT does not beat STANDARD decoding on GSM8K or HumanEval across the same rollouts, the data-efficiency claim is overturned.","supporting_citations":[],"review_version":1}