{"id":"f337a706-8fa2-4ab8-bce7-b22ebf586b54","arxiv_id":"2501.15570","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A three-stage distillation pipeline converts Qwen 2.5 into a pure RWKV-7 RNN, but the resulting 7B models lose substantial benchmark accuracy.","lead":"The authors describe a three-stage process to convert a Transformer language model (Qwen 2.5) into a pure recurrent neural network (RWKV-7) using attention alignment and knowledge distillation. Reported models score far below the teacher on standard benchmarks, so the promise of matching performance is not yet met.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that distillation preserves Qwen 2.5's performance is contradicted by Table 1 on every benchmark, and the paper never evaluates the full three-stage pipeline; the 20M-token Stage-1 alignment assumption is therefore untested as the cause.","rationale":"The paper is an 'ongoing work' memo, not a complete study. Its strongest claim is that a distilled pure-RNN model born from Qwen maintains the teacher's performance. That claim requires both that Stage-1's 20M-token hidden-state alignment is sufficient for the RWKV-7 time-mixing module to compress Qwen's attention without a capacity bottleneck, and that Stage-3 restores any remaining gap. Neither is shown. Table 1, measured after Stage 2 only, shows double-digit drops on GSM8K and IFEval and single-digit drops on MMLU; the paper does not report the same benchmarks after Stage 3. The reader's weakest assumption is well placed: the failure could be due to token budget, fixed-state capacity, the loss function, or simply the unreported Stage-3 result. The paper additionally contains an apparent inconsistency between the Figure 5 caption (4B tokens) and Section 3.3 (20M tokens), and it claims state-tracking ability beyond transformers without evaluating state tracking on these distilled models. I do not see a way to read the internal evidence as supporting the headline claim; the existing REJECT verdict is therefore unchanged. This is not an attack on RWKV-7 architecture, and I am not relying on any disagreement with the broader research program.","tokens_in":4482,"tokens_out":4417,"duration_ms":39689,"concrete_test":"Run the released 'ARWKV-7B-Preview-0.1' checkpoint, which should include Stage-3 SFT/DPO, on the same six benchmarks and evaluation harness used for Table 1 (GSM8K, MMLU, GPQA, WinoGrande, IFEval, Arc-c) and compare directly with Qwen2.5-7B-Instruct. If the Stage-3 model remains near the Stage-2 values (e.g., GSM8K ≈ 40-52), the claim that the pipeline maintains Qwen 2.5's performance is empirically false; if it recovers to within a few points of the teacher, the missing Stage-3 evaluation is the key omission rather than a fundamental state-capacity failure. To isolate the Stage-1 budget, additionally repeat Stage 2 with 20M, 200M, and 2B Stage-1 tokens and check whether downstream scores saturate; saturation would show the 20M budget is not the binding constraint.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is that the three-stage pipeline converts Qwen 2.5 into a pure RWKV-7 RNN while maintaining teacher-level capability. For this to hold, the Stage-1 hidden-state alignment (Eq. 5) with only 20M tokens must let the fixed-size RWKV-7 state compress Qwen's attention without losing the information needed downstream, and Stage-3 must restore any residual losses.\n\nTable 1, measured after Stage 2 only, contradicts the abstract's 'maintaining Qwen 2.5's performance' on every reported benchmark: GSM8K falls from 82.34 to 39.95-51.93, IFEval from 73.62 to 44.12-52.16, MMLU from 71.72 to 58.22-64.77, and even WinoGrande and Arc-c drop by 2-5 points. The paper does not report results after Stage 3 (SFT/DPO and context extension, Section 3.3), so the reader cannot tell whether the full pipeline recovers the teacher's capabilities. The apparent conflict between the Figure 5 caption ('4B tokens') and the Section 3.3 text ('20M tokens in stage 1') further weakens the reproducibility of the central budget claim.\n\nThe paper itself acknowledges the fixed-size recurrent state is a compression process (Section 3.1), but it provides no curve of Eq. 5 loss versus token count and no ablation varying the Stage-1 budget. Thus the observed degradation could be caused by insufficient alignment time, inadequate state capacity, the KL loss in Stage 2, or the unreported Stage-3 gap. These have very different fixes, and the paper's causal claim is not established by its own evidence. This is an internal evidence gap, not a disagreement with RWKV-7 as an architecture.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ARWKV, a pure RNN language model obtained by replacing the self-attention layers of Qwen 2.5 with RWKV-7 time-mixing modules and then distilling the teacher's hidden states (Stage 1, Eq. 5), output distributions (Stage 2, KL divergence), and finally applying SFT and DPO (Stage 3). The abstract claims that this process maintains Qwen 2.5's performance while converting the architecture to a native RWKV-7 RNN, and that the resulting model demonstrates state-tracking ability beyond transformers. The paper reports only Stage-2 benchmark results in Table 1, notes that the work is ongoing, and provides code and checkpoint links.","tokens_in":4907,"tokens_out":3130,"duration_ms":28481,"significance":"If the stated claims held, the paper would demonstrate a practical recipe for converting a pretrained transformer into a pure linear-RNN architecture with minimal capability loss, which would be valuable for efficient inference and further RNN research. The idea of combining hidden-state alignment with KL distillation is sensible and the authors are transparent in sharing code, checkpoints, and training budgets. However, the central claim is contradicted by the paper's own evidence: every benchmark in Table 1 shows substantial degradation relative to Qwen2.5-7B-Instruct, and the full three-stage pipeline is never evaluated. The paper's significance is therefore currently negative as a validation of the method, though the method itself might merit interest if the unsupported claims were revised and the full pipeline were evaluated.","major_comments":[{"comment":"The abstract's claim that the distillation 'maintains Qwen 2.5's performance' is directly contradicted by the paper's own Table 1. For example, GSM8K drops from 82.34 to 39.95–51.93, MMLU from 71.72 to 58.22–64.77, and IFEval from 73.62 to 44.12–52.16. This is not a minor deviation but a systematic across-the-board degradation that negates the load-bearing claim. The paper must either provide results showing recovery after Stage 3 or substantially weaken the stated claim.","section":"Abstract and Table 1"},{"comment":"The paper's central claim about the final ARWKV model is untested because Table 1 explicitly reports benchmarks 'based on stage-2' only. Stage 3 (SFT, DPO, and context extension) is described but no post-Stage-3 evaluation is given. Without evaluating the full pipeline, the paper cannot support the assertion that the three-stage distillation preserves teacher capabilities.","section":"Section 3.3 and Table 1"},{"comment":"There is a direct inconsistency in the reported Stage-1 token budget: Section 3.3 states '20M tokens in stage 1', while the caption of Figure 5 states the Stage-1 loss was computed on '4B tokens'. This discrepancy affects reproducibility and the paper's own claims of efficiency. The authors must clarify the actual budget and ensure the text and figures agree.","section":"Section 3.3 vs. Figure 5"},{"comment":"The paper asserts that Stage-1 hidden-state alignment with a small token budget is sufficient for RWKV-7 to capture the teacher's attention behavior, but it provides no loss-vs-token curve, no ablation varying the Stage-1 budget, and no comparison of Eq. (5) convergence across configurations. Given the large benchmark drops, the observed degradation could stem from insufficient alignment, insufficient state capacity, the effect of Stage-2 KL loss, or the unreported Stage-3 gap. The causal role of Stage-1 alignment is not established by the presented evidence.","section":"Section 3.1, Eq. (5)"},{"comment":"The paper claims that RWKV-7 demonstrates 'stronger state tracking capabilities than transformers' and that ARWKV inherits this ability, but no state-tracking evaluation is performed on the ARWKV models themselves. The benchmarks in Table 1 (e.g., GSM8K, GPQA) suggest the opposite on reasoning tasks. A direct passkey retrieval or state-tracking test on ARWKV is necessary before such a claim can be taken seriously.","section":"Introduction and Section 4"}],"minor_comments":[{"comment":"The phrase 'with more fewer tokens' is ungrammatical; it should be 'with fewer tokens'.","section":"Abstract"},{"comment":"The word 'retrivel' in the first paragraph is a typo for 'retrieval'.","section":"Introduction"},{"comment":"The text 'relpace self-attention in every layer' contains a typo: 'relpace' should be 'replace'.","section":"Section 2"},{"comment":"The abstract mentions 'QRWK 32B' while the footnote refers to 'QRWKV6-32B-Instruct-Preview-v0.1'. The naming is inconsistent and should be unified.","section":"Abstract and Footnote 2"},{"comment":"The GPQA(Diamond) row has missing values for several models, and the column alignment makes the table difficult to read. The authors should include all measurements or explicitly mark unreported entries as '—'.","section":"Table 1"}],"recommendation":"reject","confidential_remarks":"This manuscript is an initial technical report that openly bills itself as ongoing work. Its central claim is refuted by its own experimental table, the full pipeline is not evaluated, and key training figures are internally inconsistent. These are not presentation issues but fundamental gaps in the evidence supporting the abstract's promises. A revision that added Stage-3 results, corrected the budget discrepancy, and moderated the performance claims might become a valid workshop-quality paper, but as it stands it does not meet the bar for publication in a serious journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the first reported distillation of Qwen 2.5 into a pure RWKV-7 RNN, with ablations on gate-free and frozen-MLP variants. Second, the paper's headline claim that this preserves teacher performance is contradicted by its own Table 1: GSM8K falls from 82.34 to 39.95–51.93, MMLU from 71.72 to 58.22–64.77, and IFEval from 73.62 to 44.12–52.16. That is not a minor drop.\n\nCredit where due: the authors are transparent that this is ongoing work; they release checkpoints and code; they cite the relevant distillation and RWKV literature. The combination of hidden-state alignment (Bick et al.) with word-level KL (Xu et al.) applied to RWKV-7 is a legitimate extension. The Stage-1-only results are useful for people working on architecture conversion.\n\nThe load-bearing flaw is internal: those benchmark numbers are after Stage 2, and Stage 3 (SFT/DPO and context extension) is never evaluated, so we have no evidence the full pipeline recovers teacher capability. Also, Figure 5 says 4B tokens for Stage 1 while Section 3.3 says 20M tokens. That is a two-orders-of-magnitude discrepancy and needs correcting. No error bars, no hyperparameter table, and no ablation varying the Stage-1 budget means the paper's implicit claim—that the fixed-size RNN state is the bottleneck—is untested.\n\nWho is this for? Practitioners and researchers working on converting Transformers to RNNs, and anyone tracking RWKV-7. As a finished paper it fails to substantiate its central claim; as a technical preview it is honest and worth a look. I would send it to peer review rather than desk reject, because the method is novel and the community needs referees to push for the missing evidence. But in current form it is a reject, not a minor revision.","headline":"A useful architecture-conversion preview whose own benchmarks undercut its central claim—send it to review, expect a reject unless Stage 3 and budget inconsistencies are addressed.","tokens_in":5444,"tokens_out":2931,"would_cite":false,"duration_ms":25968,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretraining is unnecessary: a Transformer's attention can be distilled into a pure RNN via three-stage alignment and KL distillation.","keywords":["linear RNN","RWKV-7","knowledge distillation","transformer-to-RNN conversion","hidden-state alignment","state tracking","Qwen 2.5","efficient language models"],"falsifier":"Run the identical pipeline but scale Stage 1's token budget from 20M to several billion tokens (the paper's own figure shows a 4B-token stage-1 run); if downstream benchmarks (e.g., GSM8K, MMLU) still fall far short of the teacher, then the fixed-size recurrent state, not the alignment budget, is the bottleneck.","tokens_in":4300,"feed_emoji":"⚡","tokens_out":8529,"duration_ms":64071,"temperature":0.7,"pith_summary":"The paper proposes that a pretrained Transformer can be converted into a pure recurrent language model without pretraining from scratch, by replacing each self-attention block with RWKV-7's time-mixing module and then running three stages of training: hidden-state alignment, word-level KL-divergence distillation, and SFT/DPO post-training. The authors claim this attention-alignment-plus-distillation recipe preserves the teacher's knowledge and makes the RNN more expressive, giving it state-tracking ability that goes beyond Transformers. They demonstrate it by distilling from Qwen 2.5 to a 7B pure-RNN model, and by a 32B RWKV-6-based run that finished in eight hours on 16 GPUs, arguing that pretraining is not a necessary step for building capable foundation models.","feed_headline":"Pretraining not needed: distill a Transformer into a pure RNN","feed_subtitle":"Three-stage alignment plus KL distillation swaps Qwen 2.5 attention for RWKV-7 time-mixing.","key_machinery":"The central object is the RWKV-7 time-mixing module, a recurrent layer whose hidden state is a matrix updated by $S_t = S_{t-1}\\left(\\mathrm{diag}(w_t) - \\hat{\\kappa}_t^T (a_t \\odot \\hat{\\kappa}_t)\\right) + v_t^T \\tilde{k}_t$, with $a_t$ an in-context learning rate. It replaces the Transformer's self-attention, turning attention into a dynamic recurrent state that acts like a meta-learner. The second machinery is the staged distillation pipeline: Stage 1 aligns hidden states between the teacher's attention block and the student's time-mixing module using $\\mathcal{L}_{\\mathrm{special}} = \\|h_{\\mathrm{teacher}} - h_{\\mathrm{student}}\\|_2 \\cdot d_{\\mathrm{model}}^{-0.5}$; Stage 2 uses word-level KL divergence (with a data distribution balanced to stage-1 training); Stage 3 applies SFT for context-extension and DPO for preference alignment. The paper's argument rests on these two pieces working together: a sufficiently expressive recurrent state plus a training signal that transfers the teacher's attention behavior.","core_discovery":"The central claim is that attention expressiveness is the key to RNN-based LLMs, and that this expressiveness can be implanted into a recurrent architecture by direct alignment. Concretely, the paper shows that replacing self-attention with the RWKV-7 time-mixing module—a linear recurrent layer with a matrix-valued state—and then minimizing the distance between teacher and student hidden states (Eq. 5), followed by KL distillation, turns a Transformer into an RNN that retains the teacher's behavior. The paper presents this as a general method that works with any LLM teacher and requires far fewer tokens than pretraining; the reported numbers are framed as an ongoing, updateable baseline. The authors argue the matrix-valued time-mixing state gives the RNN state-tracking ability beyond Transformers, drawing on the transition matrix's eigenvalue structure.","pith_inferences":["The benchmark gaps in the paper (e.g., GSM8K 39.95 vs. the teacher's 82.34) suggest the 20M-token stage-1 budget or the fixed state size may be the binding constraint; a controlled experiment varying the stage-1 token budget while holding everything else fixed could isolate which.","A hybrid comparison—keeping a fraction of the Transformer's attention layers while converting the rest to time-mixing—would show whether the capability loss comes from pure recurrence or from the distillation signal, and could lead to a cheaper sweet spot.","The claimed state-tracking advantage beyond Transformers is testable on the Sudoku/Othello-style tasks the paper's footnotes point to; running those evals on the released 7B checkpoints would either substantiate or soften the claim."],"forward_implications":["If the method holds, pretraining is no longer a prerequisite: any pretrained Transformer can be converted to a pure RNN with a small token budget, cutting the compute needed for recurrent LLMs.","The recipe should transfer across teachers and student sizes, so larger teachers can be compressed into smaller RNNs with fewer tokens than standard distillation.","The RWKV-7 time-mixing state, with its matrix-valued dynamics and wider eigenvalues, would give RNN-based models a state-tracking capability that standard Transformers lack.","The success of the conversion points to attention expressiveness as the main design axis for RNN architectures; improving the time-mixing module should directly improve distilled model quality.","The three-stage pipeline (alignment, distillation, post-training) provides a template for future architecture conversions beyond Transformers-to-RNN."],"supporting_citations":[{"why":"Supplies the hidden-state alignment method used in Stage 1.","marker":"Bick et al. (2024)"},{"why":"Defines the RWKV family with matrix-valued states and dynamic recurrence, including the time-mixing module.","marker":"Peng et al. (2024)"},{"why":"Provides the eigenvalue analysis that supports the state-tracking advantage claim.","marker":"Grazzi et al. (2024)"},{"why":"The 'illusion of state' critique that the paper claims RWKV-7 surpasses.","marker":"Merrill et al. (2024)"},{"why":"The Mamba-in-the-Llama distillation approach that inspired replacing attention with RNN layers.","marker":"Wang et al. (2024)"},{"why":"The Qwen 2.5 technical report describing the teacher architecture (GQA, RMSNorm, SwiGLU).","marker":"Yang et al. (2024a)"},{"why":"The Magpie data synthesis used to balance the Stage-2 distillation data distribution.","marker":"Xu et al. (2024b)"},{"why":"Hymba, cited for hybrid-head architectures using attention as a memory component.","marker":"Dong et al. (2024)"}],"fun_headline_variants":["No pretraining: distill Transformer attention into a pure RNN","ARWKV turns Qwen into an RWKV-7 RNN via hidden-state distillation","Attention-level expressiveness from an RNN: ARWKV distills it","State-tracking RNN from Transformer distillation, no pretraining","Distill any LLM into a pure RNN: ARWKV's recipe"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that 20 million tokens of hidden-state alignment are enough for the RWKV-7 time-mixing module to fully capture the teacher's attention behavior, so that the fixed-size recurrent state never becomes the limiting factor.","fun_headline_variants_meta":{"raw":{"variants":["No pretraining: distill Transformer attention into a pure RNN","ARWKV turns Qwen into an RWKV-7 RNN via hidden-state distillation","Attention-level expressiveness from an RNN: ARWKV distills it","State-tracking RNN from Transformer distillation, no pretraining","Distill any LLM into a pure RNN: ARWKV's recipe"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000445,"raw_usage":{"total_tokens":2272,"prompt_tokens":992,"completion_tokens":1280,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":1181}},"tokens_in":608,"tokens_out":1280,"duration_ms":10604,"temperature":1.0,"reasoning_tokens":1181,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:08:58.704898+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical pipeline but scale Stage 1's token budget from 20M to several billion tokens (the paper's own figure shows a 4B-token stage-1 run); if downstream benchmarks (e.g., GSM8K, MMLU) still fall far short of the teacher, then the fixed-size recurrent state, not the alignment budget, is the bottleneck.","supporting_citations":[],"review_version":1}