REVIEW 5 cited by
Were RNNs All We Needed?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The introduction of Transformers in 2017 reshaped the landscape of deep learning. Originally proposed for sequence modelling, Transformers have since achieved widespread success across various domains. However, the scalability limitations of Transformers - particularly with respect to sequence length - have sparked renewed interest in novel recurrent models that are parallelizable during training, offer comparable performance, and scale more effectively. In this work, we revisit sequence modelling from a historical perspective, focusing on Recurrent Neural Networks (RNNs), which dominated the field for two decades before the rise of Transformers. Specifically, we examine LSTMs (1997) and GRUs (2014). We demonstrate that by simplifying these models, we can derive minimal versions (minLSTMs and minGRUs) that (1) use fewer parameters than their traditional counterparts, (2) are fully parallelizable during training, and (3) achieve surprisingly competitive performance on a range of tasks, rivalling recent models including Transformers.
Forward citations
Cited by 5 Pith papers
-
Pretraining Recurrent Networks without Recurrence
SMT trains nonlinear RNNs by imitating one-step memory-transition labels generated by a Transformer, replacing BPTT's unrolled credit assignment with time-parallel supervised learning.
-
EEvAct: Early Event-Based Action Recognition with High-Rate Two-Stream Spiking Neural Networks
A high-rate two-stream spiking network with a lightweight gated fusion unit achieves 94.9% on THU EACT-50 and enables early prediction within 100 ms.
-
AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.
-
GraphMinNet: Learning Dependencies in Graphs with Light Complexity Minimal Architecture
A gated recurrent graph architecture that applies minGRU-style gating to node embeddings with Laplacian positional encoding, reporting state-of-the-art on several benchmarks and claiming linear-time long-range depende...
-
Temporal horizons in forecasting: a performance-learnability trade-off
Longer training horizons improve forecast quality but worsen learnability, with loss-landscape roughness growing exponentially for chaotic dynamics and linearly for limit cycles.
Discussion (0). Continue with ORCID to comment.