Pith. sign in

REVIEW 22 cited by

TinyBERT: Distilling BERT for Natural Language Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1909.10351 v5 pith:NCY5SNFA submitted 2019-09-23 cs.CL cs.AIcs.LG

Xiaoqi Jiao , Yichun Yin , Lifeng Shang , Xin Jiang , Xiao Chen , Linlin Li , Fang Wang , Qun Liu This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords tinybertbertdistillationlanguageinferenceknowledgelayersteacher
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is difficult to efficiently execute them on resource-restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large teacher BERT can be effectively transferred to a small student Tiny-BERT. Then, we introduce a new two-stage learning framework for TinyBERT, which performs Transformer distillation at both the pretraining and task-specific learning stages. This framework ensures that TinyBERT can capture he general-domain as well as the task-specific knowledge in BERT. TinyBERT with 4 layers is empirically effective and achieves more than 96.8% the performance of its teacher BERTBASE on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT with 4 layers is also significantly better than 4-layer state-of-the-art baselines on BERT distillation, with only about 28% parameters and about 31% inference time of them. Moreover, TinyBERT with 6 layers performs on-par with its teacher BERTBASE.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 137 citations worldwide. Full citation record

  1. DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce

    cs.LG 2026-02 conditional novelty 6.0 of 10

    DynamiQ compresses gradients for multi-hop all-reduce with per-block bit-width allocation, block reordering, and fused recompression, matching BF16 accuracy at ~5 bits per coordinate and improving time-to-accuracy by ...

  2. Multimodal Medical Image Binding via Shared Text Embeddings

    eess.IV 2025-06 conditional novelty 6.0 of 10

    Five modality-specific CLIP-like medical models are aligned through a shared, distilled text embedding space, enabling zero-shot cross-modal retrieval and improved few-shot classification without paired image data.

  3. Optimizing Knowledge Distillation in Transformers: Enabling Multi-Head Attention without Alignment Barriers

    cs.CV 2025-02 reject novelty 6.0 of 10

    Squeezing-Heads Distillation mixes several teacher attention maps into one per-sample weighted map, enabling knowledge distillation between transformers with different head counts without extra parameters.

  4. Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile

    cs.CV 2025-02 conditional novelty 6.0 of 10

    A three-stage pipeline combining sparse 'tile' attention with multi-step consistency distillation makes Open-Sora-Plan video generation up to 7.8x faster while keeping the aggregate VBench final score within 1%.

  5. FHECore: Rethinking GPU Microarchitecture for Fully Homomorphic Encryption

    cs.AR 2026-02 reject novelty 5.0 of 10

    A GPU-integrated 16×8 systolic modulo-multiply-accumulate unit with Barrett reduction is claimed to speed CKKS NTT and base conversion by roughly 1.6–2.1x at 2.4% area cost.

  6. SindBERT, the Sailor: Charting the Seas of Turkish NLP

    cs.CL 2025-10 conditional novelty 5.0 of 10

    SindBERT releases Turkish RoBERTa base/large models trained on 312GB of text; they match existing models, with the large variant best on two of four tasks and little scaling gain.

  7. Expandable Residual Approximation for Knowledge Distillation

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A new knowledge distillation method decomposes the teacher-student feature gap into multiple residual steps and reports improved accuracy on ImageNet and COCO.

  8. GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replay

    cs.CL 2025-08 conditional novelty 5.0 of 10

    A fixed 1,000-sample replay set from pretraining text, trained with a threshold-based margin loss on last-layer hidden states, reduces forgetting across 15 sequential finetuning tasks in Llama-3.1-8B.

  9. Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Structural pruning with finetuning plus hidden-state distillation recovers most performance in multimodal LLMs, with 5% of training data sufficient at moderate compression levels.

  10. AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes

    cs.AI 2025-06 reject novelty 5.0 of 10

    AgentDistill distills agent capabilities without any training by having a teacher generate reusable MCP tool boxes that small-model students invoke at inference time.

  11. Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FedProj combines client-side gradient projection onto a global-knowledge loss with server-side ensemble distillation and outperforms existing federated learning methods on non-IID image and NLP benchmarks.

  12. A Survey of LLM $\times$ DATA

    cs.DB 2025-05 conditional novelty 5.0 of 10

    A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.

  13. NoteBar: An AI-Assisted Note-Taking System for Personal Knowledge Management

    cs.CL 2025-09 reject novelty 4.0 of 10

    NoteBar introduces a persona-conditioned synthetic note dataset and shows DeBERTa-v3 reaches 0.78 accuracy and 0.76 F1 on multi-label note classification, while claiming user-facing benefits it does not actually measure.

  14. Enhanced Arabic Text Retrieval with Attentive Relevance Scoring

    cs.CL 2025-07 conditional novelty 4.0 of 10

    An Arabic dense retriever using a trainable attentive scoring module instead of dot-product similarity reports improved top-k passage retrieval on ArabicaQA.

  15. Resource-Efficient Automatic Software Vulnerability Assessment via Knowledge Distillation and Particle Swarm Optimization

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A PSO-guided knowledge distillation framework compresses a CodeBERT vulnerability assessor to 0.6% of its original size while retaining 89.3% of its accuracy.

  16. MobileRAG: A Fast, Memory-Efficient, and Energy-Efficient Method for On-Device RAG

    cs.DB 2025-07 conditional novelty 4.0 of 10

    A fully on-device RAG pipeline using a partitioned, partially disk-loaded graph index and selective sentence-window reduction claims 1.72-8.89x faster vector search and up to 40.2% lower power than baselines, with com...

  17. Toward Edge General Intelligence with Multiple-Large Language Model (Multi-LLM): Architecture, Trust, and Orchestration

    cs.NI 2025-07 conditional novelty 4.0 of 10

    A survey of multi-LLM systems in edge computing, covering architectures, enabling technologies, trust mechanisms, applications, and open datasets for edge general intelligence.

  18. SEMFED: Semantic-Aware Resource-Efficient Federated Learning for Heterogeneous NLP Tasks

    cs.CL 2025-05 reject novelty 4.0 of 10

    A framework combining semantic-aware client selection, heterogeneous model sizes, and feature compression reports 98.5% accuracy and 80.5% less communication on a synthetic 10-client NLP task.

  19. LLM Modules: Knowledge Transfer from a Large to a Small Model using Enhanced Cross-Attention

    cs.CL 2025-02 reject novelty 4.0 of 10

    A frozen Qwen2-1.5B teacher passes its hidden states through gated cross-attention into GPT-Neo-125M, which after 15 epochs generates more coherent arithmetic responses than the base small model.

  20. AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Models

    cs.CV 2025-08 reject novelty 3.0 of 10

    AMMKD claims large gains from adaptively weighted two-teacher CLIP distillation, but its equations are internally inconsistent, its baselines are unverifiable, and its tests do not match its stated retrieval goal.

  21. Collaborative Distillation Strategies for Parameter-Efficient Language Model Deployment

    cs.CL 2025-07 reject novelty 2.0 of 10

    The paper claims that combining weighted output fusion, entropy-driven teacher weighting, and feature alignment produces a better distilled language model.

  22. Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation

    cs.CL 2025-05 unverdicted novelty 2.0 of 10

    A survey of small language models that organizes known methods into taxonomies but adds no new models, data, or validated benchmarks.

Pith tools