REVIEW 4 cited by
Addressing Some Limitations of Transformers with Feedback Memory
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Transformers have been successfully applied to sequential, auto-regressive tasks despite being feedforward networks. Unlike recurrent neural networks, Transformers use attention to capture temporal relations while processing input tokens in parallel. While this parallelization makes them computationally efficient, it restricts the model from fully exploiting the sequential nature of the input. The representation at a given layer can only access representations from lower layers, rather than the higher level representations already available. In this work, we propose the Feedback Transformer architecture that exposes all previous representations to all future representations, meaning the lowest representation of the current timestep is formed from the highest-level abstract representation of the past. We demonstrate on a variety of benchmarks in language modeling, machine translation, and reinforcement learning that the increased representation capacity can create small, shallow models with much stronger performance than comparable Transformers.
Forward citations
Cited by 4 Pith papers
-
Full-bandwidth transformer
Fusing the previous top-layer hidden state into the next input token through a gated linear unit improves transformer pretraining data efficiency and decoding accuracy at 1B scale.
-
Pretraining Recurrent Networks without Recurrence
SMT trains nonlinear RNNs by imitating one-step memory-transition labels generated by a Transformer, replacing BPTT's unrolled credit assignment with time-parallel supervised learning.
-
Maglev: Sliding Recurrent Memory
Using a full-context prefiller to write memory targets, Maglev trains a sliding-window recurrent transformer that keeps bounded memory at inference and improves language-modeling benchmarks.
-
ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversations
ContextQFormer adds a queue of [CLS] image and text features to a Q-Former style module and reports a 4.2% available-rate gain over LoRA-only on the new TMDialog benchmark.
Discussion (0). Continue with ORCID to comment.