REVIEW 8 cited by
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost which have both financial and/or environmental impact. The goal of this paper is to present scaling insights from pretraining and finetuning Transformers. While Kaplan et al. presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss. Therefore, it is still unclear if these set of findings transfer to downstream task within the context of the pretrain-finetune paradigm. The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient. To this end, we present improved scaling protocols whereby our redesigned models achieve similar downstream fine-tuning quality while having 50\% fewer parameters and training 40\% faster compared to the widely adopted T5-base model. We publicly release over 100 pretrained checkpoints of different T5 configurations to facilitate future research and analysis.
Forward citations
Cited by 8 Pith papers
-
On the Fitness Landscape in the $NK$ Model
For the NK fitness landscape with K/N tending to alpha, exact limits for free energy and maximum fitness are identified, together with the geometry of near-fittest peaks.
-
Can Interpretation Predict Behavior on Unseen Data?
Presence of hierarchical attention heads on in-distribution data predicts hierarchical out-of-distribution generalization across 270 small transformers, independent of causal support.
-
Progressive Scaling Visual Object Tracking
A progressive scaling training strategy with small-teacher distillation and masked-input alignment improves tracking accuracy and powers a new 12-dataset benchmark.
-
An Efficient Private GPT Never Autoregressively Decodes
Applying speculative decoding to secure two-party GPT inference, with a latency-insensitive multi-token forward and optimized private sampling, gives 2.1x to 6.0x speedups without changing the output distribution.
-
MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections
Dynamic multiway dense connections across Transformer layers let a 2.8B model match a 6.9B model's pretraining quality and few-shot performance.
-
Random Initialization Can't Catch Up: The Advantage of Language Model Transfer for Time Series Forecasting
Pretrained T5 language weights give a persistent validation-loss advantage over random initialization for low-data time series forecasting, and the advantage does not vanish within the training budget.
-
Text-to-LoRA: Instant Transformer Adaption
A hypernetwork can generate task-specific LoRA adapters from a text description, and when trained with supervised fine-tuning it zero-shot outperforms a multi-task LoRA baseline on ten benchmarks.
-
Large-Scale AI and Foundation Models for Neuroscience: A Comprehensive Review
This paper is a survey: it organizes existing foundation-model work in neuroscience into five application domains and lists public datasets, without presenting new experiments.
Discussion (0). Continue with ORCID to comment.