Pith. sign in

REVIEW 2 major objections 7 minor 35 references

ModRWKV: Transformer Multimodality in Linear Time

T0 review · 2 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ModRWKV: a linear-time recurrent backbone for multimodal LLMs, tested on vision, audio, and time series.

desk verdict Credible vision/audio results for RWKV7 as a multimodal backbone, but the time-series 'zero-shot' evaluation is compromised by train/eval overlap. read the letter →

arxiv 2505.14505 v1 pith:4TLK2PF5 submitted 2025-05-20 cs.CL cs.AI

Jiale Kang , Ziyin Yue , Qingyu Yin , Jiang Rui , Weile Li , Zening Lu , Zhouran Ji This is my paper
classification cs.CLcs.AI
keywords multimodalLLMrecurrentneuralnetworkRWKV7linear-timeinferencevision-languagemodelspeechrecognitiontime-seriesforecastingsequencecompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a modern recurrent neural network, RWKV7, can serve as the backbone of a multimodal large language model handling image, audio, and time-series inputs, matching or beating Transformer-based models of similar size at a fraction of inference cost. To show this, the authors build ModRWKV, a decoupled framework where each modality has its own encoder and a single lightweight MLP adapter projects features into the RWKV7 embedding space. On seven vision-language benchmarks, ModRWKV outperforms a comparable state-space model (VL-Mamba) and approaches larger LLaVA-style models; on speech recognition it reaches a 2.43% word error rate on LibriSpeech clean; on time-series forecasting it reports competitive zero-shot MSE. The paper also identifies configuration choices—encoder selection, adapter width, and 1D convolution compression—that balance accuracy and cost. If the claim holds, linear recurrent models become a practical low-cost route to multimodal AI.

What carries the argument

The load-bearing element is the RWKV7 layer, a recurrent unit whose hidden state $s_t$ evolves as $s_t = G_t s_{t-1} + a_t k_t v_t^T$, with a vector-valued gating matrix $G_t = (I - a_t k_t k_t^T)\mathrm{diag}(e^{-e^{w_t}})$ and an in-context learning rate $a_t$ that controls how new key-value information is written into the state. This gives the backbone rich, input-dependent memory at constant inference cost. Around it, ModRWKV places a decoupled encoder set (SigLIP2/CLIP for vision, WavLM/Whisper for audio, WaveNet/Timer for time series), a single two-layer MLP adapter for dimension alignment, and a 1D convolution layer that compresses long token sequences before they enter the RWKV blocks.

What would settle it

Evaluate ModRWKV on a time-series benchmark that is provably disjoint from any training data (e.g., a newly collected dataset) and compare its MSE to the zero-shot baselines; if the gap to TimeFM, Timer, and TTM disappears or reverses, the claimed generalization is overstated. Alternatively, run the same vision and audio recipes with a Transformer backbone of equal parameter count and training budget to test the parity claim directly.

Watch

Extended reading notes

Core claim

The central discovery is that RWKV7's recurrent state, which updates with an input-dependent decay and in-context learning rate, can absorb and reason over non-textual features when they are projected through a minimal adapter. Across vision, audio, and time-series benchmarks, ModRWKV achieves accuracy comparable to or better than existing Transformer-based multimodal models of similar parameter count, while keeping constant memory use during inference and enabling sequence compression through 1D convolutions. The authors further find that the choice of pretrained text weights matters more than encoder size: a post-trained 'g1' RWKV7 checkpoint improves vision-language accuracy by up to 28% on ScienceQA, and a smaller SigLIP2 encoder outperforms a much larger CLIP encoder.

Load-bearing premise

The zero-shot time-series results assume that training on GIFT-Eval does not overlap with the ECL, ETT, WTH, and Traffic test sets; the paper reports no explicit disjoint split for those benchmarks.

Editorial extensions

If this is right

  • Multimodal LLMs can be built with constant-memory backbones, enabling long-context and on-device deployment at lower cost than attention-based models.
  • The plug-and-play encoder design means new modalities can be added by swapping the front-end encoder and retraining only a small adapter.
  • Sequence compression via 1D convolution offers a tunable accuracy/latency trade-off, shown to cut token count by 50% with minimal score loss.
  • Pretraining the language backbone on reasoning-oriented text data can improve multimodal understanding more than increasing encoder size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The vision results suggest RNN backbones may close the gap with Transformers further if scaled to 7B+ parameters, which the paper does not test.
  • The audio results use a QA-style prompt, implying the same RWKV7 backbone can handle speech recognition and instruction following without task-specific heads.
  • The time-series results should be read cautiously until the train/test split issue is resolved, because GIFT-Eval is itself a benchmark containing those public datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper proposes ModRWKV, a multimodal framework built on the RWKV7 linear-time RNN backbone, with lightweight modality-specific encoders (CLIP/SigLIP2 for vision, WavLM/Whisper for audio, WaveNet/Timer for time series) and MLP adapters. It reports competitive vision-language results on standard benchmarks, ASR results on LibriSpeech and Aishell-1, and zero-shot time-series forecasting results on ECL, ETTh, ETTm, WTH, and Traffic. The central claim is that modern RNN architectures are a viable alternative to Transformers for multimodal large language models, based on these three modality pillars.

Significance. If the results hold, the paper makes a meaningful contribution by demonstrating that a linear recurrent backbone can perform cross-modal fusion at lower inference cost than quadratic-complexity Transformers. Strengths include the use of widely used public benchmarks for vision and audio, comparison against external baselines such as LLaVA and VL-Mamba, systematic ablations of encoders, sequence compression, adapter scaling, and pretrained weights, and the release of code. However, the time-series evidence is seriously weakened by a likely train/evaluation overlap, and the absence of error bars or significance tests makes it hard to judge the reliability of the reported improvements. The vision and audio results are reasonably controlled, but the time-series pillar needs a clean held-out evaluation before the broad central claim is fully supported.

major comments (2)
  1. [Section 4.2, Tables 4 and 5] The time-series 'zero-shot' results are compromised by a likely train/evaluation overlap. The model is fine-tuned on GIFT-Eval (and GIFT-Eval + UTSD), as described in Section 4.1, but GIFT-Eval is a benchmark assembled from existing public time-series datasets, which include ECL, ETTh1/ETTh2, ETTm1/ETTm2, WTH, and Traffic. These are exactly the datasets used for evaluation in Tables 4 and 5, and the paper does not specify any disjoint split between the training data and the evaluation series. Consequently, the reported MSE values likely reflect in-distribution forecasting accuracy rather than zero-shot generalization, and the claim in the same section that the training strategy 'enables the model to maintain stable predictions even under distribution shifts' is not supported by the presented experiments. Because the time-series results are one of the three pillars supporting the central claim, this issue is load-bearing. I request a clean held-out evaluation, either by training on datasets entirely disjoint from the evaluation series or by specifying and honoring a temporal/entity-based split within GIFT-Eval, and by relabeling the results accordingly.
  2. [Sections 4.2 and 4.3] The main performance claims are reported as point estimates with no error bars, seed variance, or statistical significance tests. This is particularly problematic for the time-series comparisons (e.g., Table 4, ETTh1: MODRWKV 0.648 vs. TTM 0.368; Table 5, 4× vs. 8× adapter scaling differences of roughly 0.1) and for the vision ablations (Table 8, e.g., SQA I improvement of 28% at 0.4B), where small differences are used to support architectural conclusions. Please report at least three seeds with mean and standard deviation, or a significance test, for the headline comparisons in the main tables.
minor comments (7)
  1. [Section 4.1] In the audio training paragraph, 'All experiments used 44×090 GPUs' appears to be a typo; please clarify the GPU configuration (e.g., 4×A100 90GB or 4×H100).
  2. [Table 2] In the LLaVA-1.5-13B row, '86.267.7' is missing a separator and should read '86.2 67.7'.
  3. [Section 4.2, Table 3] The audio evaluation reports WER/CER values without any comparison to existing ASR systems or baselines, so the claimed 'effectiveness' is not contextualized; please add reference numbers or at least a well-known baseline for LibriSpeech and Aishell-1.
  4. [Section 3.3, Equation (5)] The stride notation in the Conv1D formula is confusing: the index 's·t+j' suggests a stride equal to the kernel size in some positions, and the output length formula uses a floor but does not state how padding p is chosen to obtain the reported token counts (e.g., 577, 288, 192, 144 in Table 7); please clarify the relationship between the formula and the actual sequence lengths.
  5. [Section 4.3] The formula 'p= 1− Pn i=m n i 2n' appears without any context, definition of variables, or connection to the surrounding text on adapter scaling; it appears to be a leftover fragment and should be removed or properly introduced and derived.
  6. [Abstract and Introduction] The paper claims to be 'the first RNN-based linear model' extending to the cross-modal domain, but it compares against VL-Mamba, which is a state-space-model-based multimodal architecture; this novelty claim is overstated and should be softened to avoid confusion.
  7. [Throughout] The dataset name is inconsistently spelled as both 'GIFT-Eval' and 'Gift-Evel' (e.g., Table 5 and the paragraph following Table 5); please unify the spelling.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ModRWKV's central claim is an empirical systems result, not a derivation that reduces to its inputs.

full rationale

The paper's central claim is an empirical systems claim: a linear-recurrent RWKV7 backbone with lightweight adapters can perform multimodal understanding competitively on vision, audio, and time-series benchmarks. There is no derivation chain in which an output quantity is defined in terms of the quantity it is supposed to predict. The vision results in Table 2 are measured on external public benchmarks (VQAv2, GQA, TextVQA, ScienceQA, POPE, MMBench, MMMU) against external baselines such as LLaVA and VL-Mamba, and the training data (LLaVA-595K/665K) is disjoint from those evaluation sets. The audio results in Table 3 use held-out LibriSpeech and Aishell-1 test sets. The use of RWKV7 pretrained weights is an engineering input, not a self-citation that supplies the conclusion; RWKV7 is prior work being evaluated, not a theorem invoked to force the paper's outcome. The time-series 'zero-shot' tables may raise a genuine benchmark-validity concern: the paper trains on GIFT-Eval and then evaluates on public datasets that GIFT-Eval compiles, without stating a disjoint split. However, that is a potential training/evaluation overlap or labeling issue, not a case where the reported prediction is identical to the training input by construction. No equation or definition in the paper forces the reported MSE values, and the paper does not claim a first-principles derivation of its time-series numbers. Therefore no circular step is established, and the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on standard ML pipeline assumptions and no new physical entities are introduced. Adapter scaling and compression hyperparameters are free choices tuned on validation data.

free parameters (3)
  • Adapter scaling factor = 4x
    Chosen by validation across 2x, 4x, 8x; central to time-series results (Table 5).
  • Conv1D kernel/stride = (3,2), (4,3), (5,4); default 577 tokens
    Sequence compression configuration selected for efficiency/performance balance (Table 7).
  • GIFT-Eval training fraction = 100% (vs 25%)
    Data fraction chosen after observing improved zero-shot MSE; this is model selection.
assumptions (3)
  • domain assumption RWKV7 state update equations and pretrained weights are valid and can be fine-tuned for multimodal inputs
    The entire framework relies on RWKV7 (Peng et al., 2025) as backbone; Sections 2 and 4.1.
  • domain assumption Frozen pretrained modality encoders (CLIP, SigLIP2, WavLM, Whisper, WaveNet, Timer) preserve information needed for downstream fusion
    Section 3.1, encoders are frozen during Phase II; if this fails, cross-modal reasoning is impossible.
  • domain assumption Benchmark evaluation protocols for prior models (LLaVA, VL-Mamba, etc.) are comparable without re-running those models under identical conditions
    Table 2 compares against published numbers without controlling for training details of baselines.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ModRWKV: Transformer Multimodality in Linear Time." pith.science (2026). https://pith.science/paper/4TLK2PF5

@misc{pith2026250514505,
  author       = {Pith},
  title        = {Pith review of: ModRWKV: Transformer Multimodality in Linear Time},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4TLK2PF5}},
  note         = {Machine review of arXiv:2505.14505}
}
read the original abstract

Currently, most multimodal studies are based on large language models (LLMs) with quadratic-complexity Transformer architectures. While linear models like RNNs enjoy low inference costs, their application has been largely limited to the text-only modality. This work explores the capabilities of modern RNN architectures in multimodal contexts. We propose ModRWKV-a decoupled multimodal framework built upon the RWKV7 architecture as its LLM backbone-which achieves multi-source information fusion through dynamically adaptable heterogeneous modality encoders. We designed the multimodal modules in ModRWKV with an extremely lightweight architecture and, through extensive experiments, identified a configuration that achieves an optimal balance between performance and computational efficiency. ModRWKV leverages the pretrained weights of the RWKV7 LLM for initialization, which significantly accelerates multimodal training. Comparative experiments with different pretrained checkpoints further demonstrate that such initialization plays a crucial role in enhancing the model's ability to understand multimodal signals. Supported by extensive experiments, we conclude that modern RNN architectures present a viable alternative to Transformers in the domain of multimodal large language models (MLLMs). Furthermore, we identify the optimal configuration of the ModRWKV architecture through systematic exploration.

Figures

Figures reproduced from arXiv: 2505.14505 by the authors.

Figure 1
Figure 1. ModRWKV network architecture. Multi-modality data streams undergo initial processing via an encoder, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance and efficiency of MODRWKV. Left. The scaling curve of tokens with the performance score. Right. The inference time of MODRWKV with the number of tokens. dimension [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 5 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong, and Doyen Sahoo. 2024. Gift-eval: A benchmark for general time series forecasting model evaluation. arxiv preprint arxiv:2410.10393

    arXiv 2024

  3. [3]

    Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017. http://arxiv.org/abs/1709.05522 Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline

    arXiv 2017

  4. [4]

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022. https://doi.org/10.1109/jstsp.2022.3188113 Wavlm: Large-scale self-supervised pre-training for full stack speech proces...

    arXiv 2022

  5. [5]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. http://arxiv.org/abs/2205.14135 Flashattention: Fast and memory-efficient exact attention with io-awareness

    arXiv 2022

  6. [6]

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. http://arxiv.org/abs/2410.00037 Moshi: a speech-text foundation model for real-time dialogue

    arXiv 2024

  7. [7]

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. 2025. http://arxiv.org/abs/2409.06666 Llama-omni: Seamless speech interaction with large language models

    arXiv 2025

  8. [8]

    Yash Goyal, Tejas Khot, Douglas Summers - Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in V isual Q uestion A nswering

    work page 2017

Show all 35 references
  1. [9]

    Albert Gu and Tri Dao. 2024. http://arxiv.org/abs/2312.00752 Mamba: Linear-time sequence modeling with selective state spaces

    2024 arXiv

  2. [10]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. 2019. https://api.semanticscholar.org/CorpusID:152282269 Gqa: A new dataset for real-world visual reasoning and compositional question answering . pages 6693--6702

    2019

  3. [11]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji rong Wen. 2023. https://api.semanticscholar.org/CorpusID:258740697 Evaluating object hallucination in large vision-language models

    2023

  4. [12]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . http://arxiv.org/abs/2310.03744 Improved baselines with visual instruction tuning

    2024 arXiv

  5. [13]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. http://arxiv.org/abs/2304.08485 Visual instruction tuning

    2023 arXiv

  6. [14]

    Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. 2024 b . Timer: Generative pre-trained transformers are large time series models. arXiv preprint arXiv:2402.02368

    2024 arXiv

  7. [15]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024 c . http://arxiv.org/abs/2307.06281 Mmbench: Is your multi-modal model an all-around player?

    2024 arXiv

  8. [16]

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and A. Kalyan. 2022. https://api.semanticscholar.org/CorpusID:252383606 Learn to explain: Multimodal reasoning via thought chains for science question answering . ArXiv, abs...

    2022 arXiv

  9. [17]

    Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, and Xie Chen. 2024. http://arxiv.org/abs/2402.08846 An embarrassingly simple approach for llm with strong asr capacity

    2024 arXiv

  10. [18]

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. https://doi.org/10.1109/ICASSP.2015.7178964 Librispeech: An asr corpus based on public domain audio books . In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page...

    2015

  11. [19]

    Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng

    Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng. 2025. http://arxiv.o...

    2025 arXiv

  12. [20]

    Yanyuan Qiao, Zheng Yu, Longteng Guo, Sihan Chen, Zijia Zhao, Mingzhen Sun, Qi Wu, and Jing Liu. 2024. http://arxiv.org/abs/2403.13600 Vl-mamba: Exploring state space models for multimodal learning

    2024 arXiv

  13. [21]

    Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo, Aoying Zhou, Christian S Jensen, Zhenli Sheng, et al. 2024. Tfb: Towards comprehensive and fair benchmarking of time series forecasting methods. arXiv preprint arXiv:2403.20150

    2024 arXiv

  14. [22]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. http://arxiv.org/abs/2103.00020 Learning transferable visual models from natural lang...

    2021 arXiv

  15. [23]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. http://arxiv.org/abs/2212.04356 Robust speech recognition via large-scale weak supervision

    2022 arXiv

  16. [24]

    Stephan Rasp, Peter D Dueben, Sebastian Scher, Jonathan A Weyn, Soukayna Mouatadid, and Nils Thuerey. 2020. Weatherbench: a benchmark data set for data-driven weather forecasting. Journal of Advances in Modeling Earth Systems, 12(11):e2020MS002203

    2020

  17. [25]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. https://api.semanticscholar.org/CorpusID:85553602 Towards vqa models that can read . pages 8309--8318

    2019

  18. [26]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. http://arxiv.org/abs/2302.13971 Llama: ...

    2023 arXiv

  19. [27]

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. 2025. http://arxiv.org/abs/2502.14786 Sigl...

    2025 arXiv

  20. [28]

    Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu, et al. 2016. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 12

    2016 arXiv

  21. [29]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. http://arxiv.org/abs/1706.03762 Attention is all you need

    2023 arXiv

  22. [30]

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. 2024 a . http://arxiv.org/abs/2312.06635 Gated linear attention transformers with hardware-efficient training

    2024 arXiv

  23. [32]

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. 2025. http://arxiv.org/abs/2406.06484 Parallelizing linear transformers with the delta rule over sequence length

    2025 arXiv

  24. [33]

    Qingyu Yin, Xuzheng He, Xiang Zhuang, Yu Zhao, Jianhua Yao, Xiaoyu Shen, and Qiang Zhang. 2024. Stablemask: Refining causal masking in decoder-only transformer. arXiv preprint arXiv:2402.04779

    2024 arXiv

  25. [34]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. htt...

    2024 arXiv

  26. [35]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  27. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.