{"id":"2d09bc07-e678-4395-a593-df2b19db5e01","arxiv_id":"2608.02870","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using a full-context prefiller to write memory targets, Maglev trains a sliding-window recurrent transformer that keeps bounded memory at inference and improves language-modeling benchmarks.","lead":"Maglev trains a sliding-window Transformer to keep a compact recurrent memory by having a stronger full-context model write good memory states, then teaching the smaller model to reproduce them. At inference the strong model is removed, and the small model runs with fixed-size memory, beating sliding-window and recurrent baselines in 435M-parameter tests.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on an unverified closed-loop assumption: training feeds P the prefiller memories, but inference feeds P its own memories, and no drift measurement shows these trajectories remain close.","rationale":"The paper's central claim is that the deployed decoder P, running recurrently with its own memories, outperforms matched baselines. The training procedure optimizes a consistency loss under teacher forcing, but the deployed model is closed-loop. The paper provides no evidence that the closed-loop trajectory stays near the teacher-forced trajectory, no measurement of memory drift, and no evaluation at sequence lengths beyond the 2048-token training context. This is the most load-bearing concern because it directly threatens the validity of the reported numbers in the deployment setting: if the loop is unstable, the architecture's promised benefit over sliding-window attention may disappear. The reader's weakest_assumption identified exactly this issue, and I agree. I considered other concerns such as single-run evaluation, missing knowledge-distillation baseline, and no released code; these affect confidence in the magnitude and attribution of the gains but do not undermine the mechanism as directly as the closed-loop gap. The paper is honest about its preliminary status, and the method is clearly described, but the central claim cannot be accepted without addressing the loop-closure question. Since the reader already set a CONDITIONAL verdict on these grounds, my recommendation is UNCHANGED.","tokens_in":10754,"tokens_out":5058,"duration_ms":48476,"concrete_test":"On a held-out set of sequences longer than the 2048-token training context (e.g., 8192 tokens), run the trained decoder P recurrently (Eq. 10) and record its memories m_t. Separately run the teacher-forced path (Eq. 3) with prefiller memories m'_t. Report the per-step normalized L2 divergence ||m_t - m'_t||_2/sqrt(d) as a function of t, and report FW BPB under both protocols. If the divergence grows with t or the closed-loop BPB differs materially from the teacher-forced BPB (e.g., by more than 0.01), the loop-closure assumption fails and the reported recurrent-inference numbers are not supported. Also compare with the sliding-window SSSS baseline at the same long context; if Maglev's advantage vanishes, the recurrent memory is not stable outside the training distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step in Maglev is the replacement of the prefiller memory trajectory m'_t by the decoder's own recurrent memories m_t at inference. Training minimizes L_cons (Eq. 5), which aligns m_t with m'_t when P is teacher-forced with shifted prefiller memories m'_{0:T-1} (Eq. 3). At inference, P instead conditions on its own previous memories (Eq. 10); this changes the input distribution for every recurrent K/V injection (Eqs. 6-8) and for the attention cache (Eq. 9). Pointwise L2 alignment of m_t to m'_t at each position is neither necessary nor sufficient for the closed-loop trajectory to remain close to the teacher-forced trajectory: small per-step mismatches can compound through the nonlinear transformer recurrence, and the cache contains W prior memories whose joint distribution also shifts. The paper reports no measurement of ||m_t - m'_t|| during inference, no evaluation beyond the 2048-token training context, and no comparison between open-loop (teacher-forced) and closed-loop performance. If the closed loop drifts, the reported FW BPB 0.7251 and average downstream 56.4 (Section 4) may not reflect the deployed model, and the central claim of bounded recurrent inference outperforming sliding-window baselines would not hold.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Maglev, a recurrent Transformer with fixed-size memory. A prefiller Q with full attention processes the whole sequence in parallel to produce memory targets m'_t; a decoder P with sliding-window attention and recurrent K/V injection consumes shifted targets, predicts next tokens, and produces its own memories m_t. Training minimizes a next-token loss plus a consistency loss aligning m_t with m'_t, so that at inference Q can be discarded and P runs recurrently. The authors evaluate on the nanochat d20 setup with 43.52B tokens, reporting FineWeb-Edu BPB and several downstream benchmarks. The best separate-parameter Maglev reaches 0.7251 BPB and 56.4 average downstream accuracy, improving over sliding-window (SSSS) and latent recurrent transformer (LRT) baselines. The paper is explicitly preliminary and identifies scaling and sharing patterns as future work.","tokens_in":10992,"tokens_out":8267,"duration_ms":68824,"significance":"If the closed-loop consistency assumption holds, Maglev offers a practical way to train a fixed-memory nonlinear recurrent transformer with parallel pretraining, with potential advantages in bounded inference cost. The experiments are unusually controlled for a pretraining paper: all models share the nanochat stack, tokenizer, data, sequence length, and optimizer, and the architectural delta is precisely specified. The paper also honestly discusses limitations and acknowledges it is preliminary. However, the central empirical claim depends on an unverified train/inference consistency, and no long-context evaluation is provided. The strengths (controlled comparison, explicit architecture) are significant, but the missing evidence is currently load-bearing.","major_comments":[{"comment":"The training objective aligns decoder memories m_t with prefiller targets m'_t under teacher forcing, where P conditions on the shifted prefiller trajectory m'_{0:T-1}. At inference, P instead conditions on its own preceding memories m_{0:T-1}. This changes the input distribution at every recurrent K/V injection (Eqs. 6-8) and for the attention cache (Eq. 9). Pointwise L2 alignment is neither necessary nor sufficient to guarantee that the closed-loop trajectory remains close to the teacher-forced trajectory; small per-step mismatches can compound through the nonlinear recurrence. The paper reports no measurement of ||m_t - m'_t|| during inference, no open-loop versus closed-loop comparison, and no evaluation beyond the 2048-token training context. Because the headline numbers in Table 1 are claimed for P alone, this missing stability evidence is load-bearing.","section":"Section 2.1/2.3, Eqs. (3), (5), (10)"},{"comment":"All evaluations operate on contexts within or below the 2048-token training sequence (LAMBADA, HellaSwag, PIQA, etc.). The architectural motivation of Maglev is to extend memory beyond the fixed window via recurrence, but no experiment shows behavior on sequences longer than 2048 tokens. Without such an experiment, the observed improvement over SSSS could be due solely to the added recurrent K/V pathway and consistency objective, not to genuine long-range memory. At minimum, the authors should add a long-context language modeling evaluation (e.g., 8k or 32k contexts) comparing Maglev against baselines.","section":"Section 4, Evaluation"},{"comment":"The experimental section does not state the inference protocol used to obtain the numbers in Table 1. In particular, it is not specified whether the reported BPB and downstream accuracies are produced by P alone under the recurrent inference of Eq. (10) or by the two-pass training procedure of Eq. (3). This distinction is central to the paper's contribution, and the paper must report which procedure was used and, ideally, report both open-loop and closed-loop numbers.","section":"Section 4, Experimental setup"},{"comment":"Maglev training requires two sequence-parallel passes (one for Q, one for P) and a consistency loss, while the baselines are presumably trained with a single forward/backward per token. The paper states that all models use 'the same token budget,' but it does not report training FLOPs, wall-clock time, or parameter counts for the separate-parameter variant. Since the best result (0.7251 BPB, 56.4 average) comes from the separate-parameter model, the improvement may in part reflect additional training compute and capacity. The comparison should be placed on a compute-matched footing, or FLOPs and parameter counts should be reported so the reader can judge the trade-off.","section":"Section 4, Training setup"}],"minor_comments":[{"comment":"All configurations are single runs with no error bars or significance tests. Several adjacent scores (e.g., 56.2 vs 56.4, 56.0 vs 56.2) are within a few tenths of a percent, so the ranking among the better models may not be robust.","section":"Table 1"},{"comment":"The parameter-sharing description ('share Transformer blocks, with separate residual-scaling parameters') is vague; the number of shared layers and the resulting parameter count for each variant should be stated.","section":"Section 2.2"},{"comment":"The paper does not report key training hyperparameters (learning rate, warmup, weight decay, dropout) beyond the optimizer name, and no code is linked. This reduces reproducibility.","section":"Section 4, Training setup"},{"comment":"Figure 2 is referenced with a caption describing training dynamics, but no figure image is included in the submitted text; the figure should be restored or removed.","section":"Figure 2"},{"comment":"The two Liu et al. entries marked 2024a and 2024b appear to refer to the same paper (Longhorn) and should be merged or disambiguated.","section":"References"},{"comment":"The temperature-scaled softmax and tanh clipping in Eq. (4) are not explained; please state whether this follows nanochat and what effect it has.","section":"Section 2.1, Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a lightly edited preprint with at least one missing figure and some unresolved formatting issues. The novelty relative to very recent concurrent work (LRT, Supervised Memory Training, Rec2PM) should be checked carefully by the editor; the related-work section does cite them, but the boundary is not sharply drawn. The paper may be better suited to a venue that accepts preliminary empirical studies, unless the authors add the requested stability and long-context evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bo, quick take on arXiv:2608.02870. It's an honest, well-controlled empirical paper that proposes a training trick for token-wise recurrent Transformers. The new bit is using a full-attention prefiller to produce memory targets and a sliding-window decoder that learns to reproduce them via a consistency loss, while keeping inference bounded at sliding-window cost. The comparisons are clean: same nanochat stack, same 43.52B tokens, same data, matched LRT baselines. Credit where due: the authors cite concurrent work (SMT, Rec2PM, LRT) and don't oversell.\n\nThe central claim holds at 2048 context: the decoder alone, with no full attention at inference, beats the SSSS sliding-window baseline and the LRT baseline on FW BPB (0.7251 vs 0.7413 and 0.7331) and on average downstream accuracy (56.4 vs 54.1 and 55.0). The separate-parameter λ=1 variant is the best; the shared variant still improves over baselines, which is useful. The fact that larger λ helps the separate model but hurts the shared model is a nice, non-obvious observation.\n\nThe soft spots are real but proportionate. The biggest is the loop-closure assumption. Training feeds P the prefiller memories; inference feeds P its own memories. The stress-test worry is that the consistency loss only aligns states under teacher forcing, and drift could accumulate. That is a legitimate concern, but note the reported FW BPB and downstream numbers are themselves from the closed-loop model at 2048 tokens, so the concern is not that those numbers are fake; it's that we have no idea how the recurrence behaves at longer horizons. The paper has no measurement of ||m_t - m'_t|| during inference and no evaluation beyond the training context. I'd want both before trusting the claim beyond 2048.\n\nOther gaps: single runs, no error bars, no released code, no distillation baseline to isolate the consistency loss contribution. These are minor in the nanochat regime, and the authors explicitly call the study preliminary.\n\nWho's this for? People working on efficient long-context transformers and recurrent memory. It's a reasonable step, not a breakthrough. I'd send it to peer review with a request for drift analysis and longer-context evaluation; the architecture description is precise enough that a referee can reproduce the setup. I'd probably bring it to reading group to discuss the closed-loop drift question, though I wouldn't cite it in my own work yet.","headline":"A well-run, honest study of a modest architecture trick; the closed-loop drift gap is real but mainly a longer-context question, and the paper deserves a serious referee.","tokens_in":11540,"tokens_out":3886,"would_cite":false,"duration_ms":35335,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A sliding-window decoder, taught by a stronger prefiller's memory states, beats sliding-window and latent-recurrent baselines on language modeling.","keywords":["recurrent transformer","sliding-window attention","memory consistency","parallel training","language modeling","fixed-size memory","K/V injection","prefiller"],"falsifier":"Run the trained decoder in closed loop on a sequence much longer than the 2048-token training window, e.g., 10,000 tokens, and measure both the memory drift $\\|m_t - m'_t\\|$ (available by also running the prefiller as an oracle) and the language-modeling perplexity over the tail. If the drift grows without bound and the accuracy degrades toward the sliding-window baseline, the loop-closure claim fails.","tokens_in":10499,"feed_emoji":"🧠","tokens_out":10293,"duration_ms":72692,"temperature":0.7,"pith_summary":"Maglev tries to give Transformers a fixed-size, token-wise recurrent memory without giving up parallel pretraining. The trick is to train two coupled models: a stronger prefiller that sees the full context and, in one parallel pass, writes a target memory at every token; and a decoder that sees only a sliding window plus the previous token's memory, and is trained to both predict the next token and reproduce the prefiller's memory. A consistency loss aligns the two memories, so at inference the prefiller is discarded and the decoder runs recurrently on its own memories at the same cache cost as sliding-window attention. The paper reports that this decoder, at 435M parameters and 43.52B training tokens, improves FineWeb-Edu validation bits-per-byte from 0.7413 to 0.7251 and average downstream accuracy from 54.1 to 56.4 over a matched sliding-window transformer, while also beating a latent recurrent transformer. Sharing parameters between prefiller and decoder preserves most of the gain.","feed_headline":"Sliding-window decoder beats full-attention baselines with memory","feed_subtitle":"A prefiller writes memory targets in parallel; a decoder reproduces them, cutting inference cost to a fixed window.","key_machinery":"The central mechanism is lifted parallel training: a prefiller $Q$ with full-attention access constructs an auxiliary memory trajectory $m'_{1:T}$ in one causal, sequence-parallel pass; a sliding-window decoder $P$ consumes the shifted trajectory, predicts tokens, and emits its own memories $m_t$; and a consistency loss $\\mathcal{L}_{\\mathrm{cons}} = \\lambda\\|m_t - m'_t\\|_2/\\sqrt{d}$ teaches $P$ to reproduce the prefiller's memory so that at inference its own recurrent memories can substitute. The recurrence is carried by gated K/V injection: shifted memories are projected into key/value features, gated with local token features, and fed into the decoder's sliding-window attention, so the bounded KV cache and attention cost match ordinary sliding-window attention while the memory carries information from beyond the window.","core_discovery":"The central claim is that a decoder $P$ that uses only sliding-window attention with recurrent K/V injection—no full attention at inference—can outperform both a matched sliding-window transformer and a latent recurrent transformer on the same training budget. Training lifts the sequence-parallel problem by having a stronger prefiller $Q$ produce memory targets $m'_t$ for every position in one causal pass; $P$ then predicts tokens from the shifted targets while being penalized by $\\lambda\\|m_t - m'_t\\|_2/\\sqrt{d}$, aligning its own memories with the prefiller's. At inference $Q$ is removed and $P$ feeds its own memories back through the same K/V pathway, so the deployed model is a bounded-memory recurrent transformer. The best separate-parameter variant reaches $0.7251$ FineWeb-Edu BPB and $56.4$ average downstream accuracy, versus $0.7413$ and $54.1$ for the sliding-window baseline; the shared-parameter variant keeps most of the improvement.","pith_inferences":["If the closed-loop alignment remains stable far beyond the 2048-token training context, the same scheme could offer a practical route to unbounded-context language modeling at fixed memory cost; a direct drift measurement would settle that.","The prefiller need not be trained from scratch: a pretrained model could supply memory targets, recasting Maglev as a distillation method that compresses a strong model's representations into a compact recurrent decoder.","Alternative memory-injection schemes (residual-stream injection, recurrent tokens, or cross-attention) and stronger consistency objectives (e.g., contrastive or adversarial) might reduce the teacher-forcing gap further; the paper leaves those comparisons open."],"forward_implications":["Inference stays at fixed cost: the KV cache and attention pattern are those of ordinary sliding-window attention, independent of sequence length, because the recurrent memory is injected through existing K/V features.","The decoder trains with two parallel Transformer passes instead of a sequential unroll, preserving large-scale pretraining throughput while still yielding a recurrent model at deployment.","On the same token budget, the consistency-trained decoder improves both validation loss and several downstream commonsense benchmarks over matched sliding-window and latent-recurrent baselines.","Parameter sharing between the prefiller and decoder removes much of the extra parameter cost while keeping most of the gain, so the method can be implemented without doubling the model."],"supporting_citations":[{"why":"Supplies the recurrent K/V injection mechanism and the latent recurrent Transformer baseline that Maglev is compared against.","marker":"[Huang et al., 2026]"},{"why":"Establishes sliding-window attention as the bounded-attention baseline whose cache profile Maglev preserves.","marker":"[Beltagy et al., 2020]"},{"why":"Defines the pretraining stack and evaluation scripts used for all models in the comparison.","marker":"[Karpathy, 2025]"},{"why":"Provides the compute-optimal scaling rule that sets the 43.52B-token training budget.","marker":"[Hoffmann et al., 2022]"},{"why":"Supplies the FineWeb-Edu validation set that yields the reported bits-per-byte results.","marker":"[Lozhkov et al., 2024]"}],"fun_headline_variants":["Sliding recurrent memory beats sliding and RNN baselines","Memory-aligned sliding decoder outperforms recurrent baselines","Fixed-window memory tops sliding-window and RNN peers","Sliding recurrent memory outdoes full-sliding and latent RNNs","Parallel prefiller, sliding decoder: beats window and RNN baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is training–inference loop closure: the consistency loss $\\lambda\\|m_t - m'_t\\|_2/\\sqrt{d}$ aligns decoder memories with prefiller memories only under teacher forcing, and the paper assumes this alignment keeps the decoder's own recurrent trajectory stable in closed-loop inference; it does not measure the actual drift or test beyond the 2048-token training context.","fun_headline_variants_meta":{"raw":{"variants":["Sliding recurrent memory beats sliding and RNN baselines","Memory-aligned sliding decoder outperforms recurrent baselines","Fixed-window memory tops sliding-window and RNN peers","Sliding recurrent memory outdoes full-sliding and latent RNNs","Parallel prefiller, sliding decoder: beats window and RNN baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001339,"raw_usage":{"total_tokens":5440,"prompt_tokens":942,"completion_tokens":4498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":4411}},"tokens_in":558,"tokens_out":4498,"duration_ms":31854,"temperature":1.0,"reasoning_tokens":4411,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:57:00.905909+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained decoder in closed loop on a sequence much longer than the 2048-token training window, e.g., 10,000 tokens, and measure both the memory drift $\\|m_t - m'_t\\|$ (available by also running the prefiller as an oracle) and the language-modeling perplexity over the tail. If the drift grows without bound and the accuracy degrades toward the sliding-window baseline, the loop-closure claim fails.","supporting_citations":[],"review_version":2}