REVIEW 1 cited by
RecurrentGemma: Moving Past Transformers for Efficient Open Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce RecurrentGemma, a family of open language models which uses Google's novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide two sizes of models, containing 2B and 9B parameters, and provide pre-trained and instruction tuned variants for both. Our models achieve comparable performance to similarly-sized Gemma baselines despite being trained on fewer tokens.
Forward citations
Cited by 1 Pith paper
-
Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory
Decoupling the forget gate from the input gate in a discrete state-space recurrence lets one fixed-size state both preserve old bindings over long horizons and overwrite stale ones.
Discussion (0). Continue with ORCID to comment.