REVIEW 3 major objections 5 minor 26 references
RWKV-X: A Linear Complexity Hybrid Language Model
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read RWKV-X: a hybrid RWKV that retrieves passkeys at 64K with linear training cost.
desk verdict Real retrieval numbers, but the training-complexity claim collapses under its own Eq. 4. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Top-$k$ Chunk Sparse Attention block paired with a top-$m$ KV-cache manager. In the block, each query scores every chunk by $s_i = q \cdot (1/B)\sum_{j=1}^B k_j^{(i)}$, selects the $k$ highest-scoring chunks, and runs softmax attention only over those chunks. The cache manager splits the cache into the recent observation window and the earlier past, computes an importance score for each past key as the column sum of softmax attention from observation-window queries, keeps the top-$m$ past keys and values, and concatenates them with the window. This fixed-size cache is what converts decoding from growing memory into constant memory and, the paper argues, keeps per-token cost independent of context length.
What would settle it
Run decoding on a fixed model with context lengths 64K, 128K, 256K, 512K, and 1M, and profile the cache-management step. If per-token latency or memory grows with context length, or if the importance-score computation in Equation (7) visits every past key at each step, the O(1) decoding claim is falsified.
Extended reading notes
Core claim
The paper establishes that a fixed RWKV-7 checkpoint can be widened with periodically inserted sparse-attention blocks through block expansion, then aligned on short text and continually pretrained on 64K sequences, producing a model that retrieves a hidden passkey almost perfectly at 64K context. The sparse attention divides the sequence into chunks, scores each chunk by the inner product of the query with its mean-pooled keys, attends only to the top-$k$ chunks, and compresses the past KV cache to a constant budget by keeping the top-$m$ past keys by cumulative observation-window attention. In the reported S-NIAH evaluation, the 3.6B RWKV-X holds near-perfect accuracy through 8K context and 95.6--99.8 accuracy on the harder UUID and number-in-haystack tasks at 8K, where RWKV-7 baselines collapse. The paper therefore claims that the long-context weakness of linear RNNs is fixable by a sparse-attention component that remains linear in training and constant-memory in decoding.
Load-bearing premise
The constant-time and constant-memory decoding claim rests on the assumption that the top-$m$ cache selection can be updated incrementally in constant time per generated token; the paper gives the importance-score formula as a sum over the whole past cache, and no incremental update rule is shown, so if that update is missing the decoding claim would fail.
Editorial extensions
If this is right
- Long-context training of a hybrid RWKV-style model stays linear in sequence length, so 64K--1M contexts can be handled without the quadratic memory blowup of full-attention hybrids.
- Because the sparse blocks are inserted into an existing RWKV-7 checkpoint and aligned before long-context pretraining, the recipe is a candidate drop-in upgrade path for deployed RWKV models.
- Near-perfect 64K passkey retrieval after only 1B tokens of continual pretraining indicates long-range recall can be regained cheaply relative to training from scratch.
- Decoding latency and memory remain flat out to 1M tokens when the cache is fixed at 64K entries, provided the importance-scoring update can be made constant-time.
- The fixed-size cache also makes the model a plausible backbone for generation over very long contexts, since released memory does not grow with the prompt.
Reading between the lines
- Beyond the paper: the heuristic chunk scorer is never stress-tested at chunk boundaries; placing the passkey near or across a chunk edge would show whether top-$k$ selection can miss retrievable information.
- Beyond the paper: the same observation-window importance scoring should transfer to other sparse-attention hybrids with growing KV caches, converting them to constant-memory decoders if the top-$m$ update can be made incremental.
- Beyond the paper: the authors' own limitation note says sparse-attention decoding is currently slower than vanilla RWKV in wall-clock time; the constant-time claim is asymptotic, and fused kernels are the concrete test of whether it becomes visible in practice.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. RWKV-X is a hybrid language model that interleaves frozen/expanded RWKV-7 blocks with newly added Top-k Chunk Sparse Attention blocks, together with a KV-cache compression scheme inspired by SnapKV. The paper claims O(N) training complexity, O(1) per-token decoding complexity and memory, and demonstrates near-perfect S-NIAH passkey retrieval after continual pretraining on 64K-token sequences, while retaining competitive short-context accuracy. The authors release checkpoints and code.
Significance. If the complexity claims were established, RWKV-X would be a practically valuable hybrid: it builds directly on RWKV-7 checkpoints, adds long-range retrieval capability, and maintains a bounded cache during decoding. The paper reports genuine S-NIAH measurements (Table 2) and informative ablations (LongCE loss, attention-layer ratio, model size, positional encoding), and it ships code and checkpoints. However, the analysis of both training and decoding complexity is incomplete, which leaves the two headline claims ('linear-time training' and 'constant-time decoding') unsupported as written. With the gaps fixed, the work could be a solid systems/empirical contribution.
major comments (3)
- [Sec. 3.2, Eq. (4); Sec. 3.2.2] The claimed training complexity O(kBN)=O(N) omits the cost of computing the chunk relevance scores in Eq. (4). For each query token q, the model computes a dot product with the mean-pooled key of every one of the n=N/B chunks, which costs n operations per query and O(N^2/B) total per layer. Since B is described as a small constant, this term is quadratic in N. The paper does not describe any hierarchical, approximate, or shared chunk-selection scheme that would reduce this cost; Appendix D addresses only KV-cache compression, not training-time chunk selection. Consequently, the abstract's 'linear-time complexity during training' is unsupported by the method as written.
- [Sec. 3.2.1, Eq. (7); Sec. 3.2.2] The constant-time decoding claim rests on the ability to compute the importance scores C in Eq. (7) over a bounded set. As written, Kpast is the 'earlier cached states' split from the past cache; if that cache grows with the generated sequence, the sum over Kpast costs O(N_past) per decoding step, making per-token decoding O(N) rather than O(1). The paper does not state that Kpast has already been truncated to a constant size before Eq. (7) is evaluated, nor does it provide an incremental update of the importance scores (the 'Update cluster' element in Figure 7 is not described in the text). Without one of these, the headline claim of constant-time, constant-memory decoding up to 1M tokens is not established.
- [Sec. 3.2.1 and Appendix D] The cache management procedure is underspecified in a way that affects both correctness and memory. The text says that 'top-m keys and values are selected based on C' from Kpast, and then the cache is reconstructed by concatenating the selected entries with the observation window. If Kpast already has size m-Lobs (for a total budget m), selecting 'top-m' is ill-defined, and concatenating with Kobs would produce more than m entries. The paper should state explicitly the invariant on the cache size (e.g., select top-(m-Lobs) from Kpast and then append the observation window) and specify the sizes of Kpast and Kobs in terms of m and Lobs.
minor comments (5)
- [Sec. 3.3.1 vs Appendix A, Table 7] The alignment-stage context length is inconsistent: Section 3.3.1 says 'short texts with a context length of 1024', while Appendix A lists 4,096 tokens for the 3.6B model's alignment phase. Please reconcile the two descriptions.
- [Table 2] The headline claim of near-perfect accuracy on the 64K passkey retrieval benchmark is supported only by Figure 1(c); Table 2 reports S-NIAH results only up to 8K. Adding a 64K column (or a separate table) with the numerical values would make the central empirical claim directly verifiable.
- [Sec. 2.2] The description of Mamba as 'integrates retrieval-based and generative components' is inaccurate; Mamba is a selective state-space model, not a retrieval-augmented architecture.
- [Sec. 4.4, Figure 4] The decoding-latency comparison between RWKV-X-3.6B and RWKV-7-2.9B confounds architecture differences with parameter count and cache configuration; a footnote or sentence explaining that RWKV-7 is a recurrent model with no KV cache would clarify the comparison.
- [Sec. 4.5.2, Figure 5] The ablation on the percentage of attention layers does not state the training data or the number of training tokens used for the 126M models; please provide this information for reproducibility.
Circularity Check
No circularity: empirical claims are benchmark-measured; the training-complexity gap is an omitted-cost error, not a circular derivation.
full rationale
The paper's derivation chain is not circular. RWKV-X is an explicit combination of RWKV-7 blocks, top-k chunk sparse attention, block expansion, and LongCE loss, with each component attributed to prior work and none defined in terms of the target results. The S-NIAH and short-context numbers are external benchmark measurements, and no fitted constant is relabeled as a prediction. The self-citations to RWKV-7 (Peng et al., 2025) involve overlapping authors, but the citation is used as a base-model reference, not as evidence for the new long-context claims, so it is not load-bearing. The main technical defect is a complexity-analysis error: Eq. (4) computes a relevance score for all n = N/B chunks per query before selection, which costs O(N^2/B) and is omitted from the stated O(kBN) training complexity. This makes the linear-training claim unsupported as written, but it is an omitted-cost mistake and not an equivalence between inputs and outputs by construction. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- chunk size B
- number of selected chunks k
- KV cache memory budget m =
fixed 64K cache (Sec. 4.4)
- observation window size Lobs
- sparse-attention layer ratio =
25% of layers
assumptions (4)
- domain assumption Eq. (2)-(3) from RWKV-7 correctly describe the state evolution and transition matrix of the base model.
- domain assumption RULER S-NIAH scores are a valid measure of long-context retrieval capability.
- domain assumption Mean-pooled chunk keys (Eq. 4) provide a sufficient relevance signal for top-k chunk selection in long-context retrieval.
- ad hoc to paper The top-m cache importance scores (Eq. 7) can be updated in constant time per decoding step without scanning the full past cache.
Cite this review
Pith. "Pith review of RWKV-X: A Linear Complexity Hybrid Language Model." pith.science (2026). https://pith.science/paper/B7A2URNV
@misc{pith2026250421463,
author = {Pith},
title = {Pith review of: RWKV-X: A Linear Complexity Hybrid Language Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/B7A2URNV}},
note = {Machine review of arXiv:2504.21463}
}
read the original abstract
In this paper, we introduce RWKV-X, a novel hybrid architecture that combines the efficiency of RWKV for short-range modeling with a sparse attention mechanism designed to capture long-range context. Unlike previous hybrid approaches that rely on full attention layers and retain quadratic complexity, RWKV-X achieves linear-time complexity in training and constant-time complexity in inference decoding. We demonstrate that RWKV-X, when continually pretrained on 64K-token sequences, achieves near-perfect accuracy on the 64K passkey retrieval benchmark. It consistently outperforms prior RWKV-7 models on long-context benchmarks, while maintaining strong performance on short-context tasks. These results highlight RWKV-X as a scalable and efficient backbone for general-purpose language modeling, capable of decoding sequences up to 1 million tokens with stable speed and memory usage. To facilitate further research and analysis, we have made the checkpoints and the associated code publicly accessible at: https://github.com/howard-hou/RWKV-X.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. 2023. https://arxiv.org/abs/2312.04927 Zoology: Measuring and improving recall in efficient language models . Preprint, arXiv:2312.04927
arXiv 2023
-
[2]
Yingfa Chen, Xinrong Zhang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun. 2024. Stuffed mamba: State collapse and state capacity of rnn-based long-context modeling. arXiv preprint arXiv:2410.07145
arXiv 2024
-
[3]
Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. 2025. https://arxiv.org/abs/2410.23771 What is wrong with perplexity for long-context language modeling? Preprint, arXiv:2410.23771
arXiv 2025
-
[4]
Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2025 a . https://arxiv.org/abs/2410.02660 How to train long-context language models (effectively) . Preprint, arXiv:2410.02660
arXiv 2025
-
[5]
Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi, Junjie Lai, Hayden Kwok-Hay So, Ting Cao, Fan Yang, and Mao Yang. 2025 b . https://arxiv.org/abs/2410.13276 Seerattention: Learning intrinsic sparse attention in your llms . Preprint, arXiv:2410.13276
arXiv 2025
-
[6]
Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. 2024. https://arxiv.org/abs/2405.16712 Zamba: A compact 7b ssm hybrid model . Preprint, arXiv:2405.16712
arXiv 2024
-
[7]
Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752
arXiv 2023
-
[8]
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. https://arxiv.org/abs/2404.06654 Ruler: What's the real context size of your long-context language models? Preprint, arXiv:2404.06654
arXiv 2024
Show all 26 references
-
[9]
Jean Kaddour. 2023. https://arxiv.org/abs/2304.08442 The minipile challenge for data-efficient language models . Preprint, arXiv:2304.08442
2023 arXiv
-
[10]
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156--5165. PMLR
2020
-
[11]
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. https://arxiv.org/abs/2404.14469 Snapkv: Llm knows what you are looking for before generation . Preprint, arXiv:2404.14469
2024 arXiv
-
[12]
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Haim Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avshalom Manevich, Nir Ratner, Noam Rozen,...
2024 arXiv
-
[13]
Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, and 6 others. 2025. https://arxiv.org/abs/2502.131...
2025 arXiv
-
[14]
MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, and 71 others. 2025. https://arxiv.org/abs/2501.0831...
2025 arXiv
-
[15]
Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. 2023. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pages 26670--26698. PMLR
2023
-
[16]
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, and 1 others. 2023. Rwkv: Reinventing rnns for the transformer era. In Findings of the Association for Computational Linguistics: EMN...
2023
-
[17]
Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, and 1 others. 2024. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892
2024 arXiv
-
[18]
Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng
Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng. 2025. https://arxiv....
2025 arXiv
-
[19]
Imanol Schlag, Kazuki Irie, and J \"u rgen Schmidhuber. 2021. Linear transformers are secretly fast weight programmers. In International conference on machine learning, pages 9355--9366. PMLR
2021
-
[20]
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. 2024. https://api.semanticscholar.org/CorpusID:271098045 Flashattention-3: Fast and accurate attention with asynchrony and low-precision . ArXiv, abs/2407.08608
2024 arXiv
-
[21]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30
2017
-
[22]
Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo. 2024. https://doi.org/10.18653/v1/2024.acl-long.352 LL a MA pro: Progressive LL a MA with block expansion . In Proceedings of the 62nd Annual Meeting of the Association for Computationa...
2024 doi
-
[23]
Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. 2024. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.06484
2024 arXiv
-
[24]
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. 2025. https://arxiv.org/abs/2502.11089 Native sparse attention: Hardware-aligned ...
2025 arXiv
-
[25]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[26]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.