REVIEW 4 cited by
Transferring Inductive Biases through Knowledge Distillation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are a limiting factor, or where training data is not perfectly representative of the conditions at test time. However, defining, designing and efficiently adapting inductive biases is not necessarily straightforward. In this paper, we explore the power of knowledge distillation for transferring the effect of inductive biases from one model to another. We consider families of models with different inductive biases, LSTMs vs. Transformers and CNNs vs. MLPs, in the context of tasks and scenarios where having the right inductive biases is critical. We study the effect of inductive biases on the solutions the models converge to and investigate how and to what extent the effect of inductive biases is transferred through knowledge distillation, in terms of not only performance but also different aspects of converged solutions.
Forward citations
Cited by 4 Pith papers
-
Can Interpretation Predict Behavior on Unseen Data?
Presence of hierarchical attention heads on in-distribution data predicts hierarchical out-of-distribution generalization across 270 small transformers, independent of causal support.
-
Propositional Logic for Probing Generalization in Neural Networks
Standard neural architectures generalize to unseen variable and operator combinations, but systematically fail when negation is applied to an operator that was hidden during training.
-
Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
Different procedural pretraining tasks create complementary, transferable structures in a transformer's attention and MLP weights, and structures from different tasks can be combined into one initialization.
-
Procedural Pretraining: Warming Up Language Models with Abstract Data
A short warm-up on procedural data (brackets, sorting, sets) makes language models more accurate and more data-efficient on language, code, and informal math.
Discussion (0). Sign in to comment.