Pith. sign in

REVIEW 17 cited by

RWKV-7 "Goose" with Expressive Dynamic State Evolution

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.14456 v2 pith:D4UEVNNC submitted 2025-03-18 cs.CL cs.AIcs.LG

Bo Peng , Ruichong Zhang , Daniel Goldstein , Eric Alcaide , Xingjian Du , Haowen Hou , Jiaju Lin , Jiaxing Liu
show 10 more authors
Janna Lu William Merrill Guangyu Song Kaifeng Tan Saiteja Utpala Nathan Wilce Johan S. Wind Tianyi Wu Daniel Wuttke Christian Zhou-Zheng
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords rwkv-7billionlanguagemodelsconstantdatasetgoosehttps
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

We present RWKV-7 "Goose", a new sequence modeling architecture with constant memory usage and constant inference time per token. Despite being trained on dramatically fewer tokens than other top models, our 2.9 billion parameter language model achieves a new 3B SoTA on multilingual tasks and matches the current 3B SoTA on English language downstream performance. RWKV-7 introduces a newly generalized formulation of the delta rule with vector-valued gating and in-context learning rates, as well as a relaxed value replacement rule. We show that RWKV-7 can perform state tracking and recognize all regular languages, while retaining parallelizability of training. This exceeds the capabilities of Transformers under standard complexity conjectures, which are limited to $\mathsf{TC}^0$. To demonstrate RWKV-7's language modeling capability, we also present an extended open source 3.1 trillion token multilingual corpus, and train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on this dataset. To foster openness, reproduction, and adoption, we release our models and dataset component listing at https://huggingface.co/RWKV, and our training and inference code at https://github.com/RWKV/RWKV-LM all under the Apache 2.0 License.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A First-Principles Theory of Slow Thinking and Active Perception

    cs.AI 2026-07 conditional novelty 7.5 of 10

    Active lifting of data distributions via latent-sequence sampling and max-rate uncertainty reduction formally derives slow-thinking LLMs and places them on representation and sampler hierarchies that can be climbed.

  2. A Compositional Theory of Causally Masked Transformers

    cs.FL 2026-07 accept novelty 7.0 of 10

    NoPE finite-precision causal transformers realize definite, R-trivial, locally R-trivial, or star-free languages according to whether attention is width-one window, sharp soft, cascaded, or ordinary floating-point soft.

  3. Advances in Neural Controlled Differential Equations

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Linear NCDEs replace non-linear vector fields with linear ones, enabling parallel-in-time training via associative scans while retaining maximal theoretical expressivity and achieving state-of-the-art time series perf...

  4. Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoning

    cs.LG 2026-03 conditional novelty 7.0 of 10

    A symplectic LQR layer inserted as an adapter into pretrained LLMs yields large gains on MATH-500, AMC and AIME by solving a latent optimal-control problem at inference time.

  5. Learning State-Tracking from Code Using Linear RNNs

    cs.LG 2026-02 unverdicted novelty 7.0 of 10

    Linear RNNs with negative eigenvalues track permutation state from REPL traces under next-token prediction, but probabilistic state-tracking with partial reveals forces exponential belief decay without nonlinear renor...

  6. Selective Rotary Position Embedding

    cs.CL 2025-11 unverdicted novelty 7.0 of 10

    Selective RoPE replaces RoPE's fixed rotation angles with input-dependent, learnable angles and improves recall-focused tasks in gated linear and softmax transformers.

  7. AudioRWKV: Efficient and Stable Bidirectional RWKV for Audio Pattern Recognition

    cs.SD 2025-09 conditional novelty 6.0 of 10

    AudioRWKV, an RWKV7-based audio backbone with 2D convolution and bidirectional WKV, beats AST and AuM baselines on five benchmarks at linear complexity.

  8. AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.

  9. RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding

    cs.CL 2025-05 conditional novelty 6.0 of 10

    RAD selects redundant attention layers via self-speculative decoding throughput, replaces them with SSM blocks, and distills them into hybrid models that beat the original on GSM8K and CRUX.

  10. When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models

    cs.CL 2026-03 conditional novelty 5.0 of 10

    Perplexity-based scoring systematically understates teacher–student gaps in distilled hybrid LLMs and can reverse design-choice rankings; generation-based evaluation is the more reliable protocol.

  11. LADY: Linear Attention for Autonomous Driving Efficiency without Transformers

    cs.AI 2025-12 conditional novelty 5.0 of 10

    LADY shows that an end-to-end driving model using only linear attention can match transformer-based planners on NAVSIM/Bench2Drive while fusing arbitrary-length historical sensor frames at constant per-frame cost.

  12. ENA: Efficient N-dimensional Attention

    cs.LG 2025-08 conditional novelty 5.0 of 10

    ENA combines linear recurrence with hardware-friendly sliding tile attention to model images and videos efficiently, claiming Transformer-level accuracy at roughly 70% attention sparsity.

  13. ATLAS: Learning to Optimally Memorize the Context at Test Time

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A new recurrent memory module (Atlas) that optimizes its memory over a window of past tokens and uses Muon-style second-order updates, reporting gains over Transformers and existing linear RNNs on long-context tasks.

  14. DualComp: End-to-End Learning of a Unified Dual-Modality Lossless Compressor

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A unified RWKV-based model compresses both images and text losslessly with a shared vocabulary, modality-specific contextual layers, and mixture-of-experts routing, matching or improving on much larger SOTA compressor...

  15. Memory for Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    The paper organizes LLM memory architectures along three axes—implicit vs explicit representation, offline vs online update, and short vs long-term persistence—plus finer-grained update rules.

  16. Understanding Transformer from the Perspective of Associative Memory

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Frames the Transformer as associative memory, derives retrieval SNR for linear, softmax, ReLU, and SoLU kernels, and proposes DeltaFormer, a softmax-plus-delta-rule variant claimed to exceed TC0 expressivity.

  17. ModRWKV: Transformer Multimodality in Linear Time

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A linear RNN backbone (RWKV7) with lightweight adapters can handle vision, speech, and time-series inputs, with competitive vision and speech results but unreliable time-series evaluation.

Pith tools