REVIEW 13 cited by
Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training. Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019). However, surprisingly, the simple baseline of just pre-training and fine-tuning compact models has been overlooked. In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work. Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation. The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements. Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data. One surprising observation is that they have a compound effect even when sequentially applied on the same data. To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.
Forward citations
Cited by 13 Pith papers
-
Transporting Task Vectors across Different Architectures without Training
Theseus transports task vectors across models of different widths by functionally matching activations via orthogonal Procrustes alignment, yielding a closed-form solution that improves over baselines without training.
-
Basis Transformers for Multi-Task Tabular Regression
Basis transformers beat fine-tuned LLMs on 34 multi-task tabular regression datasets while using five times fewer parameters and no data preprocessing.
-
Learn from A Rationalist: Distilling Intermediate Interpretable Rationales
Distilling a teacher rationale-extraction model's feature selections and predictions into smaller students improves student accuracy by up to ~14 points on CIFAR-10 while keeping the same rationale sparsity.
-
Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation
A one-step belief propagation refinement with a repulsive Potts prior increases attention entropy and improves downstream accuracy of small Transformers, with GTD as a diagnostic for multi-hop attention.
-
GenRecal: Generation after Recalibration from Large to Small Vision-Language Models
A learnable Recalibrator bridges different tokenizers so that small VLMs can distill knowledge from any large VLM, improving their benchmark scores.
-
Clustering and Median Aggregation Improve Differentially Private Inference
Clustering seed texts and privately aggregating median token logits improves representativeness and reduces reported privacy cost for DP synthetic text generation.
-
Team DACTYL at PAN 2026: Bayesian Data Mixing and Empirical X-risk Minimization for AI-text Detection
A Bayesian-uncertainty text filter, partial-AUROC training, and MCGrad calibration produce the second-ranked AI-text detector (0.974 mean score) on the PAN 2026 leaderboard.
-
The Impact of Background Speech on Interruption Detection in Collaborative Groups
Interruption detection in small-group dialogue remains accurate in simulated multi-group background noise when acoustic and textual features are combined, while an overlap-based heuristic collapses.
-
ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generation
ReqBrain, a LoRA-fine-tuned Zephyr-7b-beta model, produces software requirements that human evaluators could not reliably tell apart from human-authored ones, with automatic metrics favoring it over untuned ChatGPT-4o.
-
X-ARES: A Comprehensive Framework for Assessing Audio Encoder Performance
X-ARES evaluates 13 audio encoders on 22 speech, sound, and music tasks using linear probing and nearest-neighbor classifiers, revealing strong domain-dependent performance differences.
-
Compressed Models are NOT Trust-equivalent to Their Large Counterparts
Compressed BERT models share at most 67% of their top decision features with BERT-base and show different calibration profiles even at similar accuracy, so accuracy parity does not ensure trust-equivalence.
-
Semantic Convergence: Investigating Shared Representations Across Scaled LLMs
Gemma-2-2B and Gemma-2-9B align most strongly on SAE-derived features in middle layers, with preliminary evidence for shared multi-token concept subspaces.
-
QFFN-BERT: An Empirical Study of Depth, Performance, and Data Efficiency in Hybrid Quantum-Classical Transformers
Replacing the feed-forward networks of a tiny BERT with four-qubit quantum circuits yielded 81.19% versus 79.59% on SST-2, but the result rests on a single seed and a CLS-only circuit application.
Discussion (0). Sign in to comment.