{"id":"67f50dc8-2da0-4b63-af14-d6cf1a6d8b5f","arxiv_id":"2608.02050","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A hierarchical local-attention NCA for language modeling shows that the narrow-to-wide window schedule explains most of its behavior, with iteration adding a small bounded benefit, and the model underperforms transformers.","lead":"TextNCA applies a local, iterated, weight-shared cellular automaton update to language modeling, and finds that the staged narrow-to-wide window schedule, not iteration or locality, drives most of its behavior. The model still trails transformers, so the paper is an honest ablation study rather than a performance claim.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SWin-TF-Staged control does not match Hier-TextNCA's receptive field (331 vs 661) or sequential depth (6 vs 12), so the +4.1 PPL gap does not cleanly isolate iteration/weight sharing.","rationale":"The reader's weakest_assumption correctly flags the parameter mismatch between SWin-TF-Staged (35.5M) and Hier-TextNCA (30.8M) and the single-seed PPL. Our analysis identifies a more fundamental confound: the control also fails to match receptive field (331 vs 661) and sequential depth (6 vs 12). This undermines the isolation of iteration/weight sharing from schedule and depth, which is the linchpin of the paper's quantitative decomposition. The paper's own data support the concern: a non-staged, non-iterating uniform-w=128 control (SWin-TF-6L, R=763) already achieves 63.2 PPL with only 6 layers and more parameters, leaving only 2.9 PPL below Hier-TextNCA; a 12-layer non-iterating model could plausibly close or reverse the gap. This is not an internal inconsistency in the paper's ablations—the schedule perturbations and U-shaped Ts sweep are internally valid—but it means the headline claim about iteration being a 'secondary contributor adding ~4 PPL' is not established by the reported experiments. The paper is otherwise careful, honest about limitations, and provides a wealth of interpretability analysis, so the appropriate verdict remains CONDITIONAL, contingent on a properly matched non-iterating control. The reader's condition about parameter matching should be broadened to include receptive-field and depth matching. No ad hominem is intended; this is an argument about experimental design.","tokens_in":21673,"tokens_out":7882,"duration_ms":63417,"concrete_test":"Train a 12-layer non-iterating sliding-window Transformer with layer windows [8,8,8,8,32,32,32,32,128,128,128,128], matching Hier-TextNCA's receptive field (R=661) and sequential depth (12 attention applications), and tune hidden size to match ~30.8M parameters. Evaluate WikiText-103 PPL at matched 60k steps under the same protocol. If this control's PPL is ≤60.3, the claimed iteration benefit is an artifact of unmatched depth/receptive field; if it is substantially worse (e.g., >64.5), the +4.1 PPL is more likely attributable to iteration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that iterated weight sharing contributes only ~4 PPL rests on the SWin-TF-Staged control, but this control is not matched on two architectural dimensions that independently affect PPL. Using the paper's own receptive-field formula (§3.2, R_K = 1 + Σ T_s(w_k−1)), Hier-TextNCA with [8,32,128] and T_s=4 achieves R = 1 + 4·7 + 4·31 + 4·127 = 661. SWin-TF-Staged, a 6-layer transformer with layer windows [8,8,32,32,128,128], has R = 1 + 7 + 7 + 31 + 31 + 127 + 127 = 331 — only half the receptive field. The paper's claim in §5.1 that this control 'matches Hier-TextNCA's receptive field' is therefore incorrect. Additionally, the control applies 6 sequential attention operations versus the 12 NCA steps of Hier-TextNCA, so the +4.1 PPL gap could be explained by reduced depth and less context, rather than by the absence of iteration/weight sharing. The paper's own uniform-w=128 control (SWin-TF-6L, R=763) reaches 63.2 PPL — better than the staged control and only 2.9 PPL behind the full model — suggesting that a properly matched non-iterating model might match or beat Hier-TextNCA. Without a control that matches both receptive field and depth, the quantitative attribution of the +4.1 PPL to iteration is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TextNCA, a 1D causal windowed-attention realization of the Neural Cellular Automaton primitive, and studies a hierarchical variant with three stages of window sizes [8,32,128] and T_s=4 shared-weight iterations per stage. Training on SlimPajama and evaluating on WikiText-103 at ~30M parameters and 60k steps, the paper reports that Hier-TextNCA reaches 60.3 PPL, behind Transformer-6L (52.8) and Transformer-12L (44.7). The central claim is a decomposition: the staged narrow-to-wide window schedule accounts for most of the architecture's behavior, while iterated weight sharing adds only a bounded ~4.1 PPL benefit on top. Supporting evidence includes schedule perturbations (reversed, uniform, non-monotone costs of +16.7 to +70.8 PPL), a U-shaped T_s sweep with optimum at 4, gating and step-embedding ablations, stochastic-T_s training for an inference-time iteration knob, and extensive interpretability analyses. The paper is positioned as an analytical probe rather than a performance bid.","tokens_in":22042,"tokens_out":5402,"duration_ms":47482,"significance":"If the central attribution is correct, the paper provides a valuable controlled decomposition of NCA-style computation for language modeling: the dominant factor would be the staged context schedule, not iterative weight sharing. The paper is commendably honest about compute-matched gaps, out-of-domain evaluation, and downstream limitations, and it reports extensive ablations and mechanistic analyses with a planned code release. However, the key quantitative claim that iteration adds only ~4.1 PPL rests on a non-iterating control that does not actually match the iterated model's receptive field or sequential depth. Because the flagship configuration is also selected on the test set, the main quantitative conclusions need additional support before they can be accepted.","major_comments":[{"comment":"The central claim that iterated weight sharing contributes only ~4.1 PPL relies on the SWin-TF-Staged control, which is described as matching Hier-TextNCA's receptive field. This is incorrect under the paper's own formula R_K = 1 + Σ_k T_s(w_k−1). Hier-TextNCA with [8,32,128] and T_s=4 has R = 1 + 4·7 + 4·31 + 4·127 = 661, while the 6-layer SWin-TF-Staged with layer windows [8,8,32,32,128,128] has R = 1 + 7 + 7 + 31 + 31 + 127 + 127 = 331. The control also has only 6 sequential attention layers versus 12 NCA steps. The +4.1 PPL gap can therefore be attributed to reduced context and reduced sequential depth, not solely to the absence of iteration/weight sharing. Moreover, the uniformly wide SWin-TF-6L control (R=763) reaches 63.2 PPL, within 2.9 of the flagship, suggesting that a properly matched non-iterating model could match or beat Hier-TextNCA. A 12-layer non-iterating sliding-window","section":"§5.1, Table 2; §3.2 Eq. (R_K)"},{"comment":"The claim that the staged schedule's benefit is 'conditional on the iterated weight-shared body' is also confounded. The comparison SWin-TF-Staged (64.5 PPL) vs SWin-TF-6L uniform w=128 (63.2 PPL) varies both the schedule and the receptive field (331 vs 763). The better performance of the uniform control may simply reflect its larger receptive field, not a negative interaction between staging and non-iterated transformers. The paper reads the uniform control's advantage as evidence that the schedule's benefit requires iteration, but this requires a non-iterating control that matches both the scheduled windows and the receptive field. Without such a control, the 'schedule dominates, iteration is secondary' conclusion is not quantitatively established.","section":"§5.1, Table 2"},{"comment":"The flagship configuration (K=3, T_s=4) is selected using WikiText-103 test perplexity: the paper states 'We fix K=3 and T_s=4 based on the Ts sweep in §5.3... and the stage-count ablation in §5.1,' and all reported PPL numbers are on the test split. No held-out validation set or selection protocol is described. This selection-on-test procedure inflates the flagship's PPL and biases the magnitude of the schedule and iteration effects. At minimum, the authors should report a validation-based selection or show that the qualitative conclusions are unchanged under a proper validation split.","section":"§5.1, §5.3, Tables 2 and 7"}],"minor_comments":[{"comment":"The abstract and §4.2 describe the comparison as 'parameter-matched,' but Table 3 shows Hier-TextNCA at 30.8M parameters vs Transformer-6L at 35.5M and Transformer-12L at 54.5M. Matching here is on training steps/compute, not parameter count. Please reword to 'matched-step' or 'matched-compute' to avoid ambiguity.","section":"Abstract; §5.2, Table 3"},{"comment":"The caption states 'at matched architecture and compute,' while §4.1 refers to both matched-parameter and matched-compute comparisons. These are different protocols; the table caption should specify which one applies to each row or clarify the comparison basis.","section":"Table 2 caption"},{"comment":"The matched-FLOP and wall-clock PPL values for Hier-TextNCA at 31k and 23k steps are interpolated from the eval-loss curve. This is reasonable, but it should be stated in the main text (§5.2) when citing these numbers, not only in the appendix.","section":"Appendix F, Table 10"},{"comment":"The stochastic-T_s model is called 'STOCHASTICT' in §7 and 'Hier-StochasticT' in Appendix E. Please use one consistent name throughout.","section":"§7 vs Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper is an honest, well-executed empirical study, and the schedule-effect result is large and robust. The main weakness is the control for isolating iteration: SWin-TF-Staged does not match the receptive field or depth of Hier-TextNCA, and the uniform w=128 control already comes close to the flagship. This is a fixable issue with additional experiments, and the selection-on-test issue should also be addressed. The paper's analytical framing and the quality of the ablations make it worth further consideration, provided the authors can supply the missing matched controls and a validation-based selection protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an honest, carefully executed ablation study—the first NCA-style language model—and it does a genuinely useful decomposition at 30M scale. The qualitative finding that the narrow-to-wide window schedule dominates, and that iterated weight sharing adds only a small bounded benefit, is plausible and mostly supported. But the specific number the paper sells, the +4.1 PPL contribution of iteration, is not actually supported by the control as described. SWin-TF-Staged is called \"matching Hier-TextNCA's receptive field,\" but by the paper's own formula it reaches 331 tokens versus 661 for the full model, and it applies 6 sequential attention operations versus 12. So the +4.1 gap could just be missing depth and context, not the absence of iteration. Worse, the uniform-w=128 non-iterating control gets 63.2 PPL, better than the staged control's 64.5, which muddies the claim that the schedule alone does the work without iteration. That section needs a properly matched control—say, 12 layers with a schedule that actually reaches 661, or a FLOP-matched run—before the quantitative attribution lands.\n\nWhat's genuinely good: the paper does not oversell. It positions the model as a probe, reports matched-compute numbers that look bad for the model, gives honest limitations, and plans to release code and checkpoints. The ablation axes are extensive: gate, step embeddings, T_s U-shape, kernel interactions, downstream with seeds. The negative result on the inference-time iteration knob is interesting and carefully scoped. The interpretability analyses are thorough and avoid overclaiming.\n\nThe softer but real issue: the flagship configuration (T_s=4, K=3) is selected from sweeps evaluated on the WikiText-103 test split, which inflates the headline. A validation split for model selection would fix that. The key comparison is also single-seed, with the control at 35.5M parameters versus the model's 30.8M; multi-seed error bars and closer parameter matching would help.\n\nWho this is for: people working on iterative or weight-shared language models, local attention, and anyone who wants a template for honest ablation reporting. It deserves a serious referee, and the control mismatch is fixable with a supplementary experiment.","headline":"The staged-schedule result is probably real, but the paper's flagship quantitative claim about iteration rests on a control that doesn't match receptive field or depth; still worth refereeing.","tokens_in":22562,"tokens_out":4382,"would_cite":true,"duration_ms":35544,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A staged narrow-to-wide attention schedule, not iteration, drives TextNCA's language-modeling behavior.","keywords":["neural cellular automata","language modeling","local attention","iterated weight sharing","hierarchical window schedule","perplexity ablation","WikiText-103","GRU gating"],"falsifier":"Retrain SWin-TF-Staged and Hier-TextNCA with exactly matched parameter counts and per-step FLOPs (for instance by adjusting the control's width) under the same 60k-step protocol, and check whether the 4.1-perplexity gap remains, shrinks, or inverts; also test whether the gap is sensitive to the uniform-w=128 control's parameter count.","tokens_in":21567,"feed_emoji":"🤖","tokens_out":7448,"duration_ms":57584,"temperature":0.7,"pith_summary":"TextNCA asks whether a neural cellular automaton primitive—strictly local attention, iterated weight-shared updates, and a gated in-place state—can model language, and which of those properties actually does the work. The paper's central claim is that the staged narrow-to-wide window schedule, starting at 8 tokens and widening to 128, accounts for most of the model's behavior on WikiText-103. A non-iterating sliding-window transformer that reuses the same schedule comes within +4.1 perplexity of the full iterated model, while reversing, flattening, or breaking the schedule costs between +16.7 and +70.8 perplexity. Iteration acts as a bounded effective-depth knob with a sharp optimum at four steps per stage, and it only helps when a GRU gate and learned per-step embeddings are present. The model does not beat transformers at this scale, so the paper frames itself as an analytical probe rather than a performance proposal.","feed_headline":"Narrow-to-wide schedule, not iteration, drives TextNCA's gains","feed_subtitle":"A non-iterating control comes within 4.1 perplexity of the iterated model; schedule changes cost up to 70.8.","key_machinery":"The load-bearing object is the NCA cell: a shared causal windowed-attention module followed by LayerNorm, an FFN, and a GRU gate that combines the perception output with the previous token state; a learned step embedding is added at each iteration so the shared weights can specialise across steps. Three such stages, each with its own parameters and windows w=8, 32, and 128, are applied T_s=4 times each, giving a receptive field that grows from 29 to 153 to 661 tokens. That receptive-field growth is the mechanism that makes the narrow-to-wide schedule matter: dependencies beyond 152 tokens cannot be resolved until the final wide stage. The key control, SWin-TF-Staged, is a non-iterating trans","core_discovery":"The paper defines TextNCA, a 1D causal windowed-attention realization of the NCA primitive: a token state is updated in place by a shared local rule, iterated T_s times per stage, across three cascaded stages with windows 8, 32, and 128. On WikiText-103 at roughly 30M parameters and 60k training steps, Hier-TextNCA reaches 60.3 perplexity. The central discovery is that the staged narrow-to-wide schedule carries most of the behavior: a non-iterating 6-layer sliding-window transformer with the same scheduled windows reaches 64.5 perplexity, only +4.1 worse, while reversing the schedule costs +70.8, flattening it costs +59.4, and breaking monotonicity costs +16.7. Iteration is a bounded effecti","pith_inferences":["As an editorial extension: the schedule-dominance result suggests a cheap recipe for Transformer training—warm up the context window from narrow to wide during pretraining—that the paper did not test directly but that its control experiment implies.","As an editorial extension: since the non-iterating control is 15% larger than the iterated model (35.5M vs 30.8M) and matching is on training steps rather than parameters or FLOPs, an exact parameter-matched comparison could shrink or enlarge the +4.1 PPL gap; that experiment is outside the paper's current design.","As an editorial extension: the prepare-then-decode interpretation—early stages build linear separability, the final wide stage converts it to next-token probability—could be tested as a design principle for hierarchical language models, e.g. by making only the final stage use global attention while early stages remain local.","As an editorial extension: the inverse result for GLA kernels (better at single scale, worse in the hierarchy) suggests that kernel choice and window schedule interact; testing other kernels such as linear attention or state-space layers inside the staged hierarchy could map where the interaction comes from."],"forward_implications":["If the narrow-to-wide schedule is the dominant factor, then NCA-style iteration and weight sharing are not what make local language models work at this scale; the context-width curriculum is.","Iteration functions as an effective-depth knob: perplexity is U-shaped in T_s, with a clear optimum at 4, so adding more shared updates eventually hurts rather than converging to a fixed point.","The iteration benefit is conditional: it appears only when the GRU gate and learned per-step embeddings are present, so un-gated or unconditioned iterated transformers are expected to behave qualitatively differently.","Attention locality, not iteration, limits extractive QA: local-attention variants cluster at SQuAD F1 19–26 while global-attention models reach 27–33, and adding a global readout block does not close the gap.","An inference-time iteration-count knob is achievable with stochastic-T_s training and sinusoidal step embeddings, but it costs about 41 perplexity points on this benchmark."],"fun_headline_variants":["Schedule, not iteration, drives TextNCA's language gains","Narrow-to-wide window order, not iteration, powers TextNCA","TextNCA's gains come from schedule; iteration adds little","Breaking the window schedule costs TextNCA up to 70 PPL","In TextNCA, the schedule matters: iteration is secondary"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the non-iterating control SWin-TF-Staged (35.5M parameters) is a fair comparison for Hier-TextNCA (30.8M) even though matching is on training steps rather than on parameter count or per-step FLOPs, so the +4.1 PPL gap attributed to iteration could change under exact parameter matching.","fun_headline_variants_meta":{"raw":{"variants":["Schedule, not iteration, drives TextNCA's language gains","Narrow-to-wide window order, not iteration, powers TextNCA","TextNCA's gains come from schedule; iteration adds little","Breaking the window schedule costs TextNCA up to 70 PPL","In TextNCA, the schedule matters: iteration is secondary"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1562,"prompt_tokens":903,"completion_tokens":659,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":569}},"tokens_in":647,"tokens_out":659,"duration_ms":5902,"temperature":1.0,"reasoning_tokens":569,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T16:00:30.064983+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain SWin-TF-Staged and Hier-TextNCA with exactly matched parameter counts and per-step FLOPs (for instance by adjusting the control's width) under the same 60k-step protocol, and check whether the 4.1-perplexity gap remains, shrinks, or inverts; also test whether the gap is sensitive to the uniform-w=128 control's parameter count.","supporting_citations":[],"review_version":1}