Pith. sign in

REVIEW 3 major objections 5 minor 26 references

RWKV-X: A Linear Complexity Hybrid Language Model

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read RWKV-X: a hybrid RWKV that retrieves passkeys at 64K with linear training cost.

desk verdict Real retrieval numbers, but the training-complexity claim collapses under its own Eq. 4. read the letter →

arxiv 2504.21463 v2 pith:B7A2URNV submitted 2025-04-30 cs.CL

classification cs.CL
keywords RWKV-XhybridlanguagemodellinearcomplexitysparseattentionKVcachemanagementlong-contextmodelingpasskeyretrievalcontinualpretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RWKV-X is a hybrid language model that grafts a sparse-attention block onto the RWKV-7 linear-recurrent backbone, aiming to fix the long-range recall failures of linear models without paying the quadratic cost of full attention. The paper's central claim is that this hybrid trains in linear time in sequence length and decodes in constant time per token with a fixed-size cache, while a 3.6B version continually pretrained on 64K-token sequences reaches near-perfect accuracy on the 64K passkey retrieval benchmark. If true, long-context language modeling would no longer face the memory and time wall that full-attention hybrids inherit. The short-context scores stay close to the underlying RWKV-7 model, so the added long-range mechanism is not bought by sacrificing general language ability.

What carries the argument

The load-bearing mechanism is the Top-$k$ Chunk Sparse Attention block paired with a top-$m$ KV-cache manager. In the block, each query scores every chunk by $s_i = q \cdot (1/B)\sum_{j=1}^B k_j^{(i)}$, selects the $k$ highest-scoring chunks, and runs softmax attention only over those chunks. The cache manager splits the cache into the recent observation window and the earlier past, computes an importance score for each past key as the column sum of softmax attention from observation-window queries, keeps the top-$m$ past keys and values, and concatenates them with the window. This fixed-size cache is what converts decoding from growing memory into constant memory and, the paper argues, keeps per-token cost independent of context length.

What would settle it

Run decoding on a fixed model with context lengths 64K, 128K, 256K, 512K, and 1M, and profile the cache-management step. If per-token latency or memory grows with context length, or if the importance-score computation in Equation (7) visits every past key at each step, the O(1) decoding claim is falsified.

Watch

Extended reading notes

Core claim

The paper establishes that a fixed RWKV-7 checkpoint can be widened with periodically inserted sparse-attention blocks through block expansion, then aligned on short text and continually pretrained on 64K sequences, producing a model that retrieves a hidden passkey almost perfectly at 64K context. The sparse attention divides the sequence into chunks, scores each chunk by the inner product of the query with its mean-pooled keys, attends only to the top-$k$ chunks, and compresses the past KV cache to a constant budget by keeping the top-$m$ past keys by cumulative observation-window attention. In the reported S-NIAH evaluation, the 3.6B RWKV-X holds near-perfect accuracy through 8K context and 95.6--99.8 accuracy on the harder UUID and number-in-haystack tasks at 8K, where RWKV-7 baselines collapse. The paper therefore claims that the long-context weakness of linear RNNs is fixable by a sparse-attention component that remains linear in training and constant-memory in decoding.

Load-bearing premise

The constant-time and constant-memory decoding claim rests on the assumption that the top-$m$ cache selection can be updated incrementally in constant time per generated token; the paper gives the importance-score formula as a sum over the whole past cache, and no incremental update rule is shown, so if that update is missing the decoding claim would fail.

Editorial extensions

If this is right

  • Long-context training of a hybrid RWKV-style model stays linear in sequence length, so 64K--1M contexts can be handled without the quadratic memory blowup of full-attention hybrids.
  • Because the sparse blocks are inserted into an existing RWKV-7 checkpoint and aligned before long-context pretraining, the recipe is a candidate drop-in upgrade path for deployed RWKV models.
  • Near-perfect 64K passkey retrieval after only 1B tokens of continual pretraining indicates long-range recall can be regained cheaply relative to training from scratch.
  • Decoding latency and memory remain flat out to 1M tokens when the cache is fixed at 64K entries, provided the importance-scoring update can be made constant-time.
  • The fixed-size cache also makes the model a plausible backbone for generation over very long contexts, since released memory does not grow with the prompt.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the heuristic chunk scorer is never stress-tested at chunk boundaries; placing the passkey near or across a chunk edge would show whether top-$k$ selection can miss retrievable information.
  • Beyond the paper: the same observation-window importance scoring should transfer to other sparse-attention hybrids with growing KV caches, converting them to constant-memory decoders if the top-$m$ update can be made incremental.
  • Beyond the paper: the authors' own limitation note says sparse-attention decoding is currently slower than vanilla RWKV in wall-clock time; the constant-time claim is asymptotic, and fused kernels are the concrete test of whether it becomes visible in practice.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. RWKV-X is a hybrid language model that interleaves frozen/expanded RWKV-7 blocks with newly added Top-k Chunk Sparse Attention blocks, together with a KV-cache compression scheme inspired by SnapKV. The paper claims O(N) training complexity, O(1) per-token decoding complexity and memory, and demonstrates near-perfect S-NIAH passkey retrieval after continual pretraining on 64K-token sequences, while retaining competitive short-context accuracy. The authors release checkpoints and code.

Significance. If the complexity claims were established, RWKV-X would be a practically valuable hybrid: it builds directly on RWKV-7 checkpoints, adds long-range retrieval capability, and maintains a bounded cache during decoding. The paper reports genuine S-NIAH measurements (Table 2) and informative ablations (LongCE loss, attention-layer ratio, model size, positional encoding), and it ships code and checkpoints. However, the analysis of both training and decoding complexity is incomplete, which leaves the two headline claims ('linear-time training' and 'constant-time decoding') unsupported as written. With the gaps fixed, the work could be a solid systems/empirical contribution.

major comments (3)
  1. [Sec. 3.2, Eq. (4); Sec. 3.2.2] The claimed training complexity O(kBN)=O(N) omits the cost of computing the chunk relevance scores in Eq. (4). For each query token q, the model computes a dot product with the mean-pooled key of every one of the n=N/B chunks, which costs n operations per query and O(N^2/B) total per layer. Since B is described as a small constant, this term is quadratic in N. The paper does not describe any hierarchical, approximate, or shared chunk-selection scheme that would reduce this cost; Appendix D addresses only KV-cache compression, not training-time chunk selection. Consequently, the abstract's 'linear-time complexity during training' is unsupported by the method as written.
  2. [Sec. 3.2.1, Eq. (7); Sec. 3.2.2] The constant-time decoding claim rests on the ability to compute the importance scores C in Eq. (7) over a bounded set. As written, Kpast is the 'earlier cached states' split from the past cache; if that cache grows with the generated sequence, the sum over Kpast costs O(N_past) per decoding step, making per-token decoding O(N) rather than O(1). The paper does not state that Kpast has already been truncated to a constant size before Eq. (7) is evaluated, nor does it provide an incremental update of the importance scores (the 'Update cluster' element in Figure 7 is not described in the text). Without one of these, the headline claim of constant-time, constant-memory decoding up to 1M tokens is not established.
  3. [Sec. 3.2.1 and Appendix D] The cache management procedure is underspecified in a way that affects both correctness and memory. The text says that 'top-m keys and values are selected based on C' from Kpast, and then the cache is reconstructed by concatenating the selected entries with the observation window. If Kpast already has size m-Lobs (for a total budget m), selecting 'top-m' is ill-defined, and concatenating with Kobs would produce more than m entries. The paper should state explicitly the invariant on the cache size (e.g., select top-(m-Lobs) from Kpast and then append the observation window) and specify the sizes of Kpast and Kobs in terms of m and Lobs.
minor comments (5)
  1. [Sec. 3.3.1 vs Appendix A, Table 7] The alignment-stage context length is inconsistent: Section 3.3.1 says 'short texts with a context length of 1024', while Appendix A lists 4,096 tokens for the 3.6B model's alignment phase. Please reconcile the two descriptions.
  2. [Table 2] The headline claim of near-perfect accuracy on the 64K passkey retrieval benchmark is supported only by Figure 1(c); Table 2 reports S-NIAH results only up to 8K. Adding a 64K column (or a separate table) with the numerical values would make the central empirical claim directly verifiable.
  3. [Sec. 2.2] The description of Mamba as 'integrates retrieval-based and generative components' is inaccurate; Mamba is a selective state-space model, not a retrieval-augmented architecture.
  4. [Sec. 4.4, Figure 4] The decoding-latency comparison between RWKV-X-3.6B and RWKV-7-2.9B confounds architecture differences with parameter count and cache configuration; a footnote or sentence explaining that RWKV-7 is a recurrent model with no KV cache would clarify the comparison.
  5. [Sec. 4.5.2, Figure 5] The ablation on the percentage of attention layers does not state the training data or the number of training tokens used for the 126M models; please provide this information for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical claims are benchmark-measured; the training-complexity gap is an omitted-cost error, not a circular derivation.

full rationale

The paper's derivation chain is not circular. RWKV-X is an explicit combination of RWKV-7 blocks, top-k chunk sparse attention, block expansion, and LongCE loss, with each component attributed to prior work and none defined in terms of the target results. The S-NIAH and short-context numbers are external benchmark measurements, and no fitted constant is relabeled as a prediction. The self-citations to RWKV-7 (Peng et al., 2025) involve overlapping authors, but the citation is used as a base-model reference, not as evidence for the new long-context claims, so it is not load-bearing. The main technical defect is a complexity-analysis error: Eq. (4) computes a relevance score for all n = N/B chunks per query before selection, which costs O(N^2/B) and is omitted from the stated O(kBN) training complexity. This makes the linear-training claim unsupported as written, but it is an omitted-cost mistake and not an equivalence between inputs and outputs by construction. Therefore the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the RWKV-7 recurrence taken from prior work, on benchmark validity of S-NIAH, and on two unverified modeling assumptions: that mean-pooled chunk keys rank chunks well enough for sparse-needle retrieval, and that the top-m cache can be updated in constant time per decoding step. The key method constants (B, k, m, Lobs) are never disclosed, so they function as unreported free parameters.

free parameters (5)
  • chunk size B
    Hand-chosen hyperparameter of Top-k Chunk Sparse Attention (Sec. 3.2). The value used in the 3.6B model is never reported, so the central result is not reproducible from the text.
  • number of selected chunks k
    Hand-chosen hyperparameter for top-k chunk selection (Eq. 5). Not reported in text or appendix.
  • KV cache memory budget m = fixed 64K cache (Sec. 4.4)
    Predefined memory budget for cache compression (Sec. 3.2.1). No ablation on m's effect on retrieval accuracy is provided.
  • observation window size Lobs
    Used in Eq. (7) and in the decoding complexity O(kB + Lobs); value not reported.
  • sparse-attention layer ratio = 25% of layers
    Chosen from the Fig. 5 ablation as the validation-loss minimizer; the headline 3.6B model uses this ratio. A hyperparameter tuned on validation data and then baked into the reported model.
assumptions (4)
  • domain assumption Eq. (2)-(3) from RWKV-7 correctly describe the state evolution and transition matrix of the base model.
    The paper builds RWKV-X on RWKV-7 (Peng et al., 2025) without re-deriving or verifying these recurrences; any error in the base model propagates.
  • domain assumption RULER S-NIAH scores are a valid measure of long-context retrieval capability.
    The headline long-context evidence is S-NIAH alone (Sec. 4.2, Table 2). The RULER S-NIAH-1 default variant is known to be format-exploitable, and the paper does not use variable needles or multi-needle variants.
  • domain assumption Mean-pooled chunk keys (Eq. 4) provide a sufficient relevance signal for top-k chunk selection in long-context retrieval.
    The selection mechanism is trained end-to-end, but no analysis shows that a sparse needle survives mean pooling well enough to be retrieved; the near-perfect results suggest the model learns around this, yet the assumption is structural.
  • ad hoc to paper The top-m cache importance scores (Eq. 7) can be updated in constant time per decoding step without scanning the full past cache.
    This unstated assumption is required for the O(1) decoding complexity claim in Sec. 3.2.2; the paper does not describe an incremental cluster-update algorithm that would make Eq. (7) constant-time.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RWKV-X: A Linear Complexity Hybrid Language Model." pith.science (2026). https://pith.science/paper/B7A2URNV

@misc{pith2026250421463,
  author       = {Pith},
  title        = {Pith review of: RWKV-X: A Linear Complexity Hybrid Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B7A2URNV}},
  note         = {Machine review of arXiv:2504.21463}
}
read the original abstract

In this paper, we introduce RWKV-X, a novel hybrid architecture that combines the efficiency of RWKV for short-range modeling with a sparse attention mechanism designed to capture long-range context. Unlike previous hybrid approaches that rely on full attention layers and retain quadratic complexity, RWKV-X achieves linear-time complexity in training and constant-time complexity in inference decoding. We demonstrate that RWKV-X, when continually pretrained on 64K-token sequences, achieves near-perfect accuracy on the 64K passkey retrieval benchmark. It consistently outperforms prior RWKV-7 models on long-context benchmarks, while maintaining strong performance on short-context tasks. These results highlight RWKV-X as a scalable and efficient backbone for general-purpose language modeling, capable of decoding sequences up to 1 million tokens with stable speed and memory usage. To facilitate further research and analysis, we have made the checkpoints and the associated code publicly accessible at: https://github.com/howard-hou/RWKV-X.

Figures

Figures reproduced from arXiv: 2504.21463 by the authors.

Figure 1
Figure 1. Passkey retrieval performance of RWKV-X models on documents up to 64K tokens. Results are shown for: (a) RWKV-7 pretrained with a 4K context length; (b) RWKV-7 after continual pretraining with a 128K context length; and (c) RWKV-X trained with continual pretraining on a 64K context length. petitive performance compared to Transformers un￾der similar model sizes and training budgets, while significantly reducing infe… view at source ↗
Figure 2
Figure 2. The architecture of RWKV-X, a hybrid model that combines RWKV-7 blocks with Sparse Attention [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Prefill latency comparison between RWKV-X [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Validation loss vs. percentage of attention [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Training efficiency comparison between RWKV-X and RWKV-7 [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Illustration of KV cache management for Top- [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 2 canonical work pages

  1. [1]

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. 2023. https://arxiv.org/abs/2312.04927 Zoology: Measuring and improving recall in efficient language models . Preprint, arXiv:2312.04927

  2. [2]

    Yingfa Chen, Xinrong Zhang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun. 2024. Stuffed mamba: State collapse and state capacity of rnn-based long-context modeling. arXiv preprint arXiv:2410.07145

  3. [3]

    Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. 2025. https://arxiv.org/abs/2410.23771 What is wrong with perplexity for long-context language modeling? Preprint, arXiv:2410.23771

  4. [4]

    Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2025 a . https://arxiv.org/abs/2410.02660 How to train long-context language models (effectively) . Preprint, arXiv:2410.02660

  5. [5]

    Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi, Junjie Lai, Hayden Kwok-Hay So, Ting Cao, Fan Yang, and Mao Yang. 2025 b . https://arxiv.org/abs/2410.13276 Seerattention: Learning intrinsic sparse attention in your llms . Preprint, arXiv:2410.13276

  6. [6]

    Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. 2024. https://arxiv.org/abs/2405.16712 Zamba: A compact 7b ssm hybrid model . Preprint, arXiv:2405.16712

  7. [7]

    Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752

  8. [8]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. https://arxiv.org/abs/2404.06654 Ruler: What's the real context size of your long-context language models? Preprint, arXiv:2404.06654

Show all 26 references
  1. [9]

    Jean Kaddour. 2023. https://arxiv.org/abs/2304.08442 The minipile challenge for data-efficient language models . Preprint, arXiv:2304.08442

  2. [10]

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156--5165. PMLR

  3. [11]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. https://arxiv.org/abs/2404.14469 Snapkv: Llm knows what you are looking for before generation . Preprint, arXiv:2404.14469

  4. [12]

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Haim Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avshalom Manevich, Nir Ratner, Noam Rozen,...

  5. [13]

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, and 6 others. 2025. https://arxiv.org/abs/2502.131...

  6. [14]

    MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, and 71 others. 2025. https://arxiv.org/abs/2501.0831...

  7. [15]

    Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. 2023. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pages 26670--26698. PMLR

  8. [16]

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, and 1 others. 2023. Rwkv: Reinventing rnns for the transformer era. In Findings of the Association for Computational Linguistics: EMN...

  9. [17]

    Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, and 1 others. 2024. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892

  10. [18]

    Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng

    Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng. 2025. https://arxiv....

  11. [19]

    Imanol Schlag, Kazuki Irie, and J \"u rgen Schmidhuber. 2021. Linear transformers are secretly fast weight programmers. In International conference on machine learning, pages 9355--9366. PMLR

  12. [20]

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. 2024. https://api.semanticscholar.org/CorpusID:271098045 Flashattention-3: Fast and accurate attention with asynchrony and low-precision . ArXiv, abs/2407.08608

  13. [21]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

  14. [22]

    Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo. 2024. https://doi.org/10.18653/v1/2024.acl-long.352 LL a MA pro: Progressive LL a MA with block expansion . In Proceedings of the 62nd Annual Meeting of the Association for Computationa...

  15. [23]

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. 2024. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.06484

  16. [24]

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. 2025. https://arxiv.org/abs/2502.11089 Native sparse attention: Hardware-aligned ...

  17. [25]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  18. [26]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.