Pith. sign in

REVIEW 2 cited by

DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.01900 v4 pith:QHLBWBFS submitted 2021-10-05 cs.CL eess.AS

Heng-Jui Chang , Shu-wen Yang , Hung-yi Lee This is my paper · ORCID
classification cs.CLeess.AS
keywords speechdistilhuberthubertlearningpre-trainingbertdatahidden-unit
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Self-supervised speech representation learning methods like wav2vec 2.0 and Hidden-unit BERT (HuBERT) leverage unlabeled speech data for pre-training and offer good representations for numerous speech processing tasks. Despite the success of these methods, they require large memory and high pre-training costs, making them inaccessible for researchers in academia and small companies. Therefore, this paper introduces DistilHuBERT, a novel multi-task learning framework to distill hidden representations from a HuBERT model directly. This method reduces HuBERT's size by 75% and 73% faster while retaining most performance in ten different tasks. Moreover, DistilHuBERT required little training time and data, opening the possibilities of pre-training personal and on-device SSL models for speech.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VeS: Teaching Pixels to Listen Without Supervision

    cs.CV 2025-07 conditional novelty 5.0 of 10

    On a multilingual Indian speech dataset, dense token-level audio-visual matching beats global pooling by 59% relative Recall@1 and yields sharper zero-shot localization heatmaps.

  2. Lillama: Large Language Models Compression via Low-Rank Feature Distillation

    cs.LG 2024-12 conditional novelty 5.0 of 10

    Lillama compresses LLMs by SVD-initialized low-rank layers trained with a local Teacher plus Student activation distillation loss, achieving 20-40% parameter reduction with only 13 million calibration tokens.

Pith tools