{"id":"8b7afa13-84ed-4f7f-a7eb-f345872e3057","arxiv_id":"2504.21463","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"RWKV-X combines RWKV-7 with top-k chunk sparse attention and compressed KV caching, reporting near-perfect passkey retrieval at 64K tokens and stable decoding up to 1M tokens.","lead":"RWKV-X is a hybrid language model that mixes RWKV-7 recurrent blocks with a sparse, top-k chunk attention mechanism for long contexts. The paper reports near-perfect 64K-token passkey retrieval and constant-memory decoding up to 1M tokens, but its evidence contains internal inconsistencies and an underived complexity claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 4's per-query scoring of all N/B chunks makes training O(N^2/B), not O(N); the linear-training claim omits this cost.","rationale":"In good faith, the paper is an engineering contribution with released code and an honest limitations section, but the central complexity claim is the main reason to prefer RWKV-X over quadratic hybrids. If chunk scoring is O(N^2/B), that advantage disappears. This is an internal inconsistency rather than a disagreement with consensus: Eq. 4 and Sec. 3.2.2 cannot both be true with B constant. The reader's weakest assumption focuses on the decoding cache update, which may actually be constant-time if the cache is fixed-size; the more serious gap is the omitted chunk-scoring cost in training. A revision could fix this by adding a hierarchical or approximate chunk selector, or by scaling B with N and rederiving the complexity, but as written the claim is unsupported. I therefore keep the reader's rejection; the concrete test would tell us whether the released implementation happens to avoid the cost via an undisclosed approximation.","tokens_in":12244,"tokens_out":6355,"duration_ms":70537,"concrete_test":"Instrument the released code to time the chunk-scoring phase (Eq. 4) separately from the selected-chunk attention. Run the same training step with B and k held fixed and N = 8K, 16K, 32K, 64K. If the chunk-scoring time scales roughly 4x per doubling of N (slope ~2 on a log-log plot), Eq. 4 alone is quadratic and the O(N) training claim fails. Independently, count FLOPs: sum over N queries of N/B dot products of dimension d gives N^2 d/B; compare that with the paper's stated O(kBN) and report whether any approximation (e.g., ANN chunk retrieval) is used in the released code.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is in the training-complexity analysis, not the decoding cache update. Sec. 3.2 defines Top-k Chunk Sparse Attention with Eq. 4: for each query token q, a relevance score s_i = q · (mean-pooled key of chunk i) is computed for every chunk i = 1..n, then Eq. 5 selects the top-k chunks. With n = N/B chunks, this costs n dot products per query, or N · n = N^2/B total, before any attention is computed. Sec. 3.2.2's complexity statement 'O(kBN)' counts only the attention over the k selected chunks and omits the O(N^2/B) chunk-scoring term. Since B is described as a small constant, N^2/B is quadratic in N. No hierarchical, approximate, or shared chunk-selection scheme that would reduce this cost is described anywhere in the paper; Appendix D addresses cache compression, not training chunk selection. Therefore the headline 'linear-time complexity during training' is unsupported by the method as written. The reader's decoding concern is real but weaker: if the compressed cache is truly fixed size m, then Eq. 7 costs O(mLobs) per step, which is constant for fixed m,Lobs; the paper should still state these values.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"RWKV-X is a hybrid language model that interleaves frozen/expanded RWKV-7 blocks with newly added Top-k Chunk Sparse Attention blocks, together with a KV-cache compression scheme inspired by SnapKV. The paper claims O(N) training complexity, O(1) per-token decoding complexity and memory, and demonstrates near-perfect S-NIAH passkey retrieval after continual pretraining on 64K-token sequences, while retaining competitive short-context accuracy. The authors release checkpoints and code.","tokens_in":12424,"tokens_out":8274,"duration_ms":84418,"significance":"If the complexity claims were established, RWKV-X would be a practically valuable hybrid: it builds directly on RWKV-7 checkpoints, adds long-range retrieval capability, and maintains a bounded cache during decoding. The paper reports genuine S-NIAH measurements (Table 2) and informative ablations (LongCE loss, attention-layer ratio, model size, positional encoding), and it ships code and checkpoints. However, the analysis of both training and decoding complexity is incomplete, which leaves the two headline claims ('linear-time training' and 'constant-time decoding') unsupported as written. With the gaps fixed, the work could be a solid systems/empirical contribution.","major_comments":[{"comment":"The claimed training complexity O(kBN)=O(N) omits the cost of computing the chunk relevance scores in Eq. (4). For each query token q, the model computes a dot product with the mean-pooled key of every one of the n=N/B chunks, which costs n operations per query and O(N^2/B) total per layer. Since B is described as a small constant, this term is quadratic in N. The paper does not describe any hierarchical, approximate, or shared chunk-selection scheme that would reduce this cost; Appendix D addresses only KV-cache compression, not training-time chunk selection. Consequently, the abstract's 'linear-time complexity during training' is unsupported by the method as written.","section":"Sec. 3.2, Eq. (4); Sec. 3.2.2"},{"comment":"The constant-time decoding claim rests on the ability to compute the importance scores C in Eq. (7) over a bounded set. As written, Kpast is the 'earlier cached states' split from the past cache; if that cache grows with the generated sequence, the sum over Kpast costs O(N_past) per decoding step, making per-token decoding O(N) rather than O(1). The paper does not state that Kpast has already been truncated to a constant size before Eq. (7) is evaluated, nor does it provide an incremental update of the importance scores (the 'Update cluster' element in Figure 7 is not described in the text). Without one of these, the headline claim of constant-time, constant-memory decoding up to 1M tokens is not established.","section":"Sec. 3.2.1, Eq. (7); Sec. 3.2.2"},{"comment":"The cache management procedure is underspecified in a way that affects both correctness and memory. The text says that 'top-m keys and values are selected based on C' from Kpast, and then the cache is reconstructed by concatenating the selected entries with the observation window. If Kpast already has size m-Lobs (for a total budget m), selecting 'top-m' is ill-defined, and concatenating with Kobs would produce more than m entries. The paper should state explicitly the invariant on the cache size (e.g., select top-(m-Lobs) from Kpast and then append the observation window) and specify the sizes of Kpast and Kobs in terms of m and Lobs.","section":"Sec. 3.2.1 and Appendix D"}],"minor_comments":[{"comment":"The alignment-stage context length is inconsistent: Section 3.3.1 says 'short texts with a context length of 1024', while Appendix A lists 4,096 tokens for the 3.6B model's alignment phase. Please reconcile the two descriptions.","section":"Sec. 3.3.1 vs Appendix A, Table 7"},{"comment":"The headline claim of near-perfect accuracy on the 64K passkey retrieval benchmark is supported only by Figure 1(c); Table 2 reports S-NIAH results only up to 8K. Adding a 64K column (or a separate table) with the numerical values would make the central empirical claim directly verifiable.","section":"Table 2"},{"comment":"The description of Mamba as 'integrates retrieval-based and generative components' is inaccurate; Mamba is a selective state-space model, not a retrieval-augmented architecture.","section":"Sec. 2.2"},{"comment":"The decoding-latency comparison between RWKV-X-3.6B and RWKV-7-2.9B confounds architecture differences with parameter count and cache configuration; a footnote or sentence explaining that RWKV-7 is a recurrent model with no KV cache would clarify the comparison.","section":"Sec. 4.4, Figure 4"},{"comment":"The ablation on the percentage of attention layers does not state the training data or the number of training tokens used for the 126M models; please provide this information for reproducibility.","section":"Sec. 4.5.2, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is built on RWKV-7, and two of the authors (Haowen Hou and Kaifeng Tan) are also authors of the RWKV-7 paper. This is not in itself a problem, but the editor may want to ensure that the baseline comparison is not disadvantaged by the experimental setup (e.g., parameter counts differ by 0.7B). The code and checkpoint release is a positive factor. The main technical gaps are in the complexity analysis and the cache-update specification; these are fixable within the manuscript's scope, so I do not recommend rejection at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRWKV-X is a hybrid that slots top-k chunk sparse attention into RWKV-7 blocks, with a SnapKV-style compressed cache, and continually pretrains the result on 64K sequences. The headline numbers are real: near-perfect passkey retrieval at 64K, and credible S-NIAH-2/3 scores at 8K. The authors ship code and checkpoints and write a straight Limitations section. There is a genuine engineering result here.\n\nThe problem is the complexity analysis. Eq. 4 scores every query against every chunk, so training is O(N^2/B) before any attention is computed. Section 3.2.2 says the cost is O(kBN), counting only the selected chunks. No hierarchical or approximate chunk selection is described. That makes the central \"linear-time training\" claim unsupported by the method as written. The Figure 6 claim that RWKV-X trains faster than RWKV-7 at all lengths is unexplained and looks odd given the added scoring cost. This is load-bearing, not cosmetic.\n\nOther soft spots are smaller. The alignment-stage context length is 1,024 in Section 3.3.1 but 4,096 in Appendix A for the 3.6B model. The constants B, k, m, and Lobs are never reported, so the complexity claims are unfalsifiable at the hyperparameter level. The main long-context comparison is against a 2.9B RWKV-7 with a 3.6B RWKV-X, so parameter count is not matched. The abstract's \"constant-time decoding\" claim is actually defensible if the compressed cache is fixed-size; the paper should state m and the update rule, but that is a minor fix.\n\nThe decoding concern the reader raised about Eq. 7 being O(N_past) is not the real issue -- with a fixed-size cache it is O(m L_obs) per step. The real issue is training.\n\nWould I bring it to reading group? Maybe, as a cautionary example of how a complexity claim can slip through. For peer review: send it out, but any competent referee should flag the missing chunk-scoring cost. A revised version that fixes the analysis, reports constants, and adds a harder long-context benchmark (multi-needle) could become a solid systems paper. As written, the central claim fails.","headline":"Real retrieval numbers, but the training-complexity claim collapses under its own Eq. 4.","tokens_in":13110,"tokens_out":3483,"would_cite":false,"duration_ms":35296,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RWKV-X: a hybrid RWKV that retrieves passkeys at 64K with linear training cost.","keywords":["RWKV-X","hybrid language model","linear complexity","sparse attention","KV cache management","long-context modeling","passkey retrieval","continual pretraining"],"falsifier":"Run decoding on a fixed model with context lengths 64K, 128K, 256K, 512K, and 1M, and profile the cache-management step. If per-token latency or memory grows with context length, or if the importance-score computation in Equation (7) visits every past key at each step, the O(1) decoding claim is falsified.","tokens_in":1660,"feed_emoji":"⚡","tokens_out":5454,"duration_ms":108938,"temperature":0.7,"pith_summary":"RWKV-X is a hybrid language model that grafts a sparse-attention block onto the RWKV-7 linear-recurrent backbone, aiming to fix the long-range recall failures of linear models without paying the quadratic cost of full attention. The paper's central claim is that this hybrid trains in linear time in sequence length and decodes in constant time per token with a fixed-size cache, while a 3.6B version continually pretrained on 64K-token sequences reaches near-perfect accuracy on the 64K passkey retrieval benchmark. If true, long-context language modeling would no longer face the memory and time wall that full-attention hybrids inherit. The short-context scores stay close to the underlying RWKV-7 model, so the added long-range mechanism is not bought by sacrificing general language ability.","feed_headline":"Hybrid RWKV hits near-perfect 64K retrieval at linear cost","feed_subtitle":"Sparse attention plus a fixed-size cache keeps per-token decoding constant out to 1M tokens.","key_machinery":"The load-bearing mechanism is the Top-$k$ Chunk Sparse Attention block paired with a top-$m$ KV-cache manager. In the block, each query scores every chunk by $s_i = q \\cdot (1/B)\\sum_{j=1}^B k_j^{(i)}$, selects the $k$ highest-scoring chunks, and runs softmax attention only over those chunks. The cache manager splits the cache into the recent observation window and the earlier past, computes an importance score for each past key as the column sum of softmax attention from observation-window queries, keeps the top-$m$ past keys and values, and concatenates them with the window. This fixed-size cache is what converts decoding from growing memory into constant memory and, the paper argues, keeps per-token cost independent of context length.","core_discovery":"The paper establishes that a fixed RWKV-7 checkpoint can be widened with periodically inserted sparse-attention blocks through block expansion, then aligned on short text and continually pretrained on 64K sequences, producing a model that retrieves a hidden passkey almost perfectly at 64K context. The sparse attention divides the sequence into chunks, scores each chunk by the inner product of the query with its mean-pooled keys, attends only to the top-$k$ chunks, and compresses the past KV cache to a constant budget by keeping the top-$m$ past keys by cumulative observation-window attention. In the reported S-NIAH evaluation, the 3.6B RWKV-X holds near-perfect accuracy through 8K context and 95.6--99.8 accuracy on the harder UUID and number-in-haystack tasks at 8K, where RWKV-7 baselines collapse. The paper therefore claims that the long-context weakness of linear RNNs is fixable by a sparse-attention component that remains linear in training and constant-memory in decoding.","pith_inferences":["Beyond the paper: the heuristic chunk scorer is never stress-tested at chunk boundaries; placing the passkey near or across a chunk edge would show whether top-$k$ selection can miss retrievable information.","Beyond the paper: the same observation-window importance scoring should transfer to other sparse-attention hybrids with growing KV caches, converting them to constant-memory decoders if the top-$m$ update can be made incremental.","Beyond the paper: the authors' own limitation note says sparse-attention decoding is currently slower than vanilla RWKV in wall-clock time; the constant-time claim is asymptotic, and fused kernels are the concrete test of whether it becomes visible in practice."],"forward_implications":["Long-context training of a hybrid RWKV-style model stays linear in sequence length, so 64K--1M contexts can be handled without the quadratic memory blowup of full-attention hybrids.","Because the sparse blocks are inserted into an existing RWKV-7 checkpoint and aligned before long-context pretraining, the recipe is a candidate drop-in upgrade path for deployed RWKV models.","Near-perfect 64K passkey retrieval after only 1B tokens of continual pretraining indicates long-range recall can be regained cheaply relative to training from scratch.","Decoding latency and memory remain flat out to 1M tokens when the cache is fixed at 64K entries, provided the importance-scoring update can be made constant-time.","The fixed-size cache also makes the model a plausible backbone for generation over very long contexts, since released memory does not grow with the prompt."],"supporting_citations":[{"why":"Provides the RWKV-7 backbone, its recurrent state equations, and the checkpoint that RWKV-X expands.","marker":"Peng et al., 2025"},{"why":"Supplies the chunk-selection idea (Mixture of Block Attention) that Top-k Chunk Sparse Attention adapts, and the comparison showing MoBA's KV cache grows linearly.","marker":"Lu et al., 2025"},{"why":"Provides the observation-window importance-scoring method that the fixed-size KV cache management mimics.","marker":"Li et al., 2024"},{"why":"Supplies the interleaved block-expansion and zero-initialization method used to insert sparse-attention blocks into RWKV-7.","marker":"Wu et al., 2024"},{"why":"Gives the LongCE loss whose ablation is shown to be necessary for S-NIAH-2/3 performance at 8K.","marker":"Fang et al., 2025"},{"why":"Provides the ProLong-64K dataset used for the 64K continual-pretraining stage.","marker":"Gao et al., 2025a"},{"why":"Defines the S-NIAH benchmark suite from which the passkey and needle-in-haystack numbers are taken.","marker":"Hsieh et al., 2024"},{"why":"Supplies baseline S-NIAH numbers for Gated DeltaNet used in the comparison tables.","marker":"Yang et al., 2024a"},{"why":"Is the full-attention baseline for prefill latency comparison in the efficiency analysis.","marker":"Shah et al., 2024"}],"fun_headline_variants":["Linear-speed RWKV-X nails 64K passkey retrieval","RWKV-X: linear training, constant decode, 1M tokens","Sparse attention keeps RWKV-X linear to 1M tokens","RWKV-X beats RWKV-7 at 64K, stays short-context strong","RWKV-X: sparse attention, linear speed, 1M tokens"],"cache_read_input_tokens":15104,"weakest_assumption_plain":"The constant-time and constant-memory decoding claim rests on the assumption that the top-$m$ cache selection can be updated incrementally in constant time per generated token; the paper gives the importance-score formula as a sum over the whole past cache, and no incremental update rule is shown, so if that update is missing the decoding claim would fail.","fun_headline_variants_meta":{"raw":{"variants":["Linear-speed RWKV-X nails 64K passkey retrieval","RWKV-X: linear training, constant decode, 1M tokens","Sparse attention keeps RWKV-X linear to 1M tokens","RWKV-X beats RWKV-7 at 64K, stays short-context strong","RWKV-X: sparse attention, linear speed, 1M tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000932,"raw_usage":{"total_tokens":3982,"prompt_tokens":929,"completion_tokens":3053,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":2956}},"tokens_in":545,"tokens_out":3053,"duration_ms":23537,"temperature":1.0,"reasoning_tokens":2956,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:03:59.889997+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run decoding on a fixed model with context lengths 64K, 128K, 256K, 512K, and 1M, and profile the cache-management step. If per-token latency or memory grows with context length, or if the importance-score computation in Equation (7) visits every past key at each step, the O(1) decoding claim is falsified.","supporting_citations":[],"review_version":1}