Don't Throw Away Data: Better Sequence Knowledge Distillation

Colin Cherry; Eleftheria Briakou; Hamid Dadkhahi; Jun Wang; Rishabh Agarwal; Trevor Cohn

Don't Throw Away Data: Better Sequence Knowledge Distillation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.10456 v1 pith:K3MVBMTV submitted 2024-07-15 cs.CL

Jun Wang , Eleftheria Briakou , Hamid Dadkhahi , Rishabh Agarwal , Colin Cherry , Trevor Cohn This is my paper · ORCID

classification cs.CL

keywords distillationknowledgesequencedataenglishoutputsstudentteacher

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

A critical component in knowledge distillation is the means of coupling the teacher and student. The predominant sequence knowledge distillation method involves supervised learning of the student against teacher-decoded outputs, and is exemplified by the current state of the art, which incorporates minimum Bayes risk (MBR) decoding. In this paper we seek to integrate MBR more tightly in distillation training, specifically by using several high scoring MBR translations, rather than a single selected sequence, thus capturing a rich diversity of teacher outputs. Our experiments on English to German and English to Japanese translation show consistent improvements over strong baseline methods for both tasks and with varying model sizes. Additionally, we conduct a detailed analysis focusing on data efficiency and capacity curse aspects to elucidate MBR-n and explore its further potential.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages
cs.CL 2025-07 conditional novelty 5.0 of 10

Generating several candidate translations per source sentence for knowledge distillation yields better small multilingual translators than standard single-hypothesis distillation, especially in low-resource settings.

Pith tools