Knowledge distillation for semi-supervised domain adaptation

Akshay Pai; Christian Igel; Jorge Cardoso; Lauge S{\o}rensen; Mads Nielsen; Marc Modat; Mauricio Orbes-Arteaga; Sebastien Ourselin

REVIEW 5 major objections 5 minor 14 references

Knowledge distillation for semi-supervised domain adaptation

T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Knowledge distillation with soft teacher labels can perform semi-supervised domain adaptation of segmentation networks, and on white-matter hyperintensity MRI it beats adversarial domain adaptation without dataset-specific tuning.

desk verdict A plausible but thinly novel distillation-based adaptation method whose core significance claim is contradicted by its own table, and whose adversarial comparator is modified and untuned. read the letter →

arxiv 1908.07355 v1 pith:4A4W3MQ4 submitted 2019-08-16 cs.LG eess.IVstat.ML

Mauricio Orbes-Arteaga , Jorge Cardoso , Lauge S{o}rensen , Christian Igel , Sebastien Ourselin , Marc Modat , Mads Nielsen , Akshay Pai This is my paper · ORCID

classification cs.LGeess.IVstat.ML

keywords knowledgedistillationsemi-superviseddomainadaptationadversarialwhitematterhyperintensitiesMRIsegmentationteacher-studentsoftlabelsshift

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that knowledge distillation is a general, hyperparameter-light way to adapt a deep segmentation network to a new data domain when no annotated target data is available. Instead of training a domain discriminator as adversarial methods do, the authors train a teacher on the source domain and then train a student on soft labels the teacher produces for both source and target scans. On cross-scanner white-matter hyperintensity segmentation, the student consistently outperforms both the source-only baseline and adversarial domain adaptation in dice score, except in one scanner-pair direction. The appeal is that the method needs only a temperature parameter, not a carefully chosen discriminator architecture.

What carries the argument

The machinery is teacher–student knowledge distillation with a temperature-controlled softmax. A teacher $f_s$ is trained on source pairs $(x_s,y_s)$ by cross-entropy; it then emits soft probability maps $\sigma(f_s(x_i)/T)$ for every scan in the union of source and target data, and a student $f_t$ is trained to match those softened maps via the distillation loss (Eq. 3). The temperature $T>1$ controls how much inter-class similarity information the student receives, making it the only extra hyperparameter compared with ordinary supervised training.

What would settle it

Take the Utrecht-to-Singapore setup, compute the teacher's dice on the Singapore target set, then train the KD student and compare per-subject student dice to per-subject teacher dice: if the student's dice never exceeds the teacher's on any subject where the teacher is confident but wrong, that would confirm the student cannot correct teacher errors and directly test the weakest assumption.

Watch

Extended reading notes

Core claim

The central claim is that a teacher network trained on source data with manual labels can be reused to train a student network on the union of source and unlabeled target data, using the teacher's softened probability maps as targets, and that this yields a target-domain model with higher WMH dice than adversarial domain adaptation. The student's objective (Eq. 3) replaces hard one-hot labels with temperature-softened teacher predictions, and the same student architecture as the teacher is trained on the mixture. The paper reports significantly higher dice scores for KD over ADA in most cross-clinic directions, with the exception of adapting from Utrecht to Singapore; it also shows KD works in an on-the-fly setting where adaptation is performed per test subject. The authors infer that the soft labels let the student learn label similarities, producing cleaner lesion boundaries, particularly for small deep-white-matter lesions.

Load-bearing premise

The load-bearing assumption is that the teacher's soft labels on target-domain scans are a trustworthy training signal—since no target annotations exist, the student is never corrected when the teacher is wrong, and any teacher error on the target domain can be baked into the student.

Editorial extensions

If this is right

For a new scanner with no annotations, a segmentation model can be adapted by running a pre-trained teacher over a handful of unlabeled scans and retraining a student on the teacher's soft labels.
Because there is no discriminator to design, the method transfers across datasets without dataset-specific architecture search; only temperature needs setting.
The on-the-fly experiments show adaptation can be applied per test subject, so a clinical pipeline could adapt to each incoming scan rather than to a batch.
KD's dice gains over ADA are concentrated in small lesions in deep white matter, where soft labels preserve boundary and similarity information that hard labels discard.
The one failure direction (Utrecht to Singapore) suggests improvement is bounded by teacher quality; the paper shows that using an ADA-improved teacher lifts KD from 0.65 to 0.69 dice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If teacher quality is the bottleneck, a natural extension is to make the teacher itself domain-robust—for example, by ensembling teachers or by first applying a cheap unsupervised alignment—before distilling; the paper's ADA-teacher experiment already hints at this direction.
The temperature $T$ could be treated as a per-domain or per-class parameter; since soft labels encode inter-class similarity, adjusting $T$ might trade off boundary precision against false positives on small lesions.
The same soft-label strategy could be tested on other multi-site medical segmentation tasks, such as prostate or brain-structure segmentation, where scanner variation is the main domain shift; one would predict gains wherever the teacher's confusions are structured rather than random.
Because the student never sees target ground truth, the method cannot verify its own target performance; a practical deployment would need a monitoring signal, such as the entropy of student predictions, to flag when target shift exceeds what the teacher can encode.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Knowledge distillation for semi-supervised domain adaptation is a simple, clean idea applied to cross-scanner WMH segmentation: train a teacher on labeled source data, then train a student on soft teacher outputs over the union of source and target unlabeled scans. As the authors themselves note, this is very close to omni-supervised learning, so the novelty is thin. The main positive is the evaluation is on a public benchmark with held-out ground truth, and the method requires only a temperature parameter, which is genuinely more convenient than tuning an adversarial discriminator. So the paper does useful work in showing that a distillation-based adaptation can be competitive.

The problem is the reporting. The abstract says KD achieves "significantly higher" dice than ADA, but Table 2 shows significance is not universal, and the text in §4.3 contradicts the table on the Utrecht-to-Singapore direction: the text says ADA was significantly better, but the table gives KD the higher mean in that cell and in the reverse direction. This is a load-bearing error because the whole selling point is the significance claim. The reader's stress-test is correct, and I couldn't find any way to reconcile the numbers.

The ADA comparator is also set up to make KD look better. The authors modify Kamnitsas et al. by attaching the discriminator to the final layer, and they don't report tuning for ADA. The paper motivates KD by ADA's need for careful hyperparameter selection, but then doesn't give ADA that care. That's a fairness problem. Additionally, the method's reliance on teacher soft labels as an unvalidated training signal is a real concern, though it doesn't make the evaluation circular because held-out ground truth is independent.

Who is this for? A reader working on semi-supervised DA in medical imaging might find the method easy to implement and worth a baseline. But the statistical contradiction means no one should rely on the central claim without repairs.

Recommendation: This deserves a serious referee rather than a desk reject, because the method is plausible and the flaws are fixable. I would send it out with a clear request to correct the text/table conflict, report exact p-values for every comparison, and re-run the ADA baseline under a fair tuning budget. If those are fixed, it's a reasonable MICCAI-grade contribution; as is, it fails its own claim.

Referee Report

5 major / 5 minor

Summary. The paper proposes knowledge distillation (KD) as a semi-supervised domain adaptation method for white matter hyperintensity (WMH) segmentation in MRI. A teacher DNN is trained on labeled source-domain scans; its soft predictions on the union of source and unlabeled target scans are then used as supervision for a student network. The method is compared with a lower-bound baseline, an upper-bound baseline, and adversarial domain adaptation (ADA) on cross-clinic WMH challenge data, including a scenario where adaptation is performed subject-wise on the fly. The abstract claims that KD achieves significantly higher dice scores than both the baseline and ADA.

Significance. If the empirical claims were fully supported, the paper would offer a simple, hyperparameter-light alternative to adversarial domain adaptation, which is practically attractive for medical imaging where target-domain annotations are unavailable. The use of a public multi-scanner dataset, the cross-clinic evaluation protocol, and the on-the-fly adaptation scenario are genuine strengths, and the held-out ground-truth evaluation is not circular. However, the current manuscript contains internal contradictions between the text and result tables, the statistical support for the abstract's headline claim is incomplete, and the ADA comparison is not presented as a fair tuned baseline. These issues currently prevent the claimed significance from being established.

major comments (5)

[§4.3, Tables 2 and 3] The prose and the tables contradict each other for the Utrecht-to-Singapore direction. The text states: 'For domain adaptation from Utrecht clinic to Singapore clinic, ADA was significantly better than KD.' However, Table 2 (Test row 'Singapore', Training column 'Utrech') lists KD 0.6924 (0.2103) versus ADA 0.6859 (0.2036), i.e., KD has the higher mean in that cell; Table 3 gives KD 0.6945 versus ADA 0.6680 in the same direction, again favoring KD. As printed, the table and prose cannot both be correct, and the abstract's blanket claim that the method 'achieves significantly higher WMH dice scores' than ADA is not supported by the reported numbers.
[Abstract and §4.3] The abstract's claim of significantly higher dice than the baseline DNN is not backed by any statistical test. §4.3 explicitly states that 'the statistical comparison are made only between ADA and KD,' and no p-values or significance tests are reported for KD versus L-bound. Since the comparison with the baseline is a central part of the abstract's claim, the authors need either to add those statistical comparisons or to soften the claim to a descriptive improvement.
[§4.2, ADA baseline] The ADA comparator is modified by attaching the discriminator to the final layer 'to be consistent with KD,' in contrast to the setup in Kamnitsas et al. [9], and no hyperparameter search for ADA is reported. Because the paper motivates KD by ADA's need for 'careful dataset-specific selection of hyperparameters,' comparing against an untuned, modified ADA is not a fair test of that motivation. The authors should tune ADA, report its hyperparameters, or compare with the original published setup.
[§3.1 and §4.2] No ablation isolates the effect of soft distillation labels from plain self-training. In Eq. (3), the student is trained on the teacher's predictions on unlabeled target data; a natural baseline is self-training with hard pseudo-labels (the teacher's argmax) on the same union of source and target data. Without this baseline, the reported gains cannot be attributed to the distillation temperature or to soft labels specifically, which is the paper's central contribution.
[§3.1 and §5] Because the student's only target-domain supervision is the teacher's own output, the method cannot correct systematic teacher errors on the target domain, and the paper never validates teacher reliability. The Discussion's experiment replacing the teacher with an ADA-adapted network and observing an increase from 0.65 to 0.69 confirms that the student's performance is constrained by teacher quality. The claim that the student 'learns a better mapping' than the teacher needs additional evidence or a mechanism.

minor comments (5)

[Throughout] There are multiple typographical and formatting errors: 'Sceince' in the affiliations, 'Utrech' for Utrecht, 'Bold fond' for 'Bold font,' and Table 3 entries such as '0.6285 ( 0.097 0.7465(0.0855)' are missing parentheses and alignment.
[§4.3 and Table 4] The text refers to 'Figure 4,' but no figure with that number appears in the manuscript; Table 4 is a caption without an image. Either include the figure or remove the reference.
[Tables 2 and 3] The p-value ranges in the captions ('0.0002 < p < 0.02') are too coarse. Report per-comparison p-values or confidence intervals, and clarify what 'bold' indicates for each statistically significant winner.
[§4.2] The experimental setup should specify how the two folds are split, including how the 10 target subjects used for adaptation and the 10 used for testing are chosen, and whether the same test subjects are used across all methods so that the paired t-tests are properly paired.
[Eq. (3)] The notation 'σ(T^{-1} f_s(x_i))' is ambiguous; writing 'σ(f_s(x_i)/T)' would make clear that the temperature is applied to the logits before the softmax.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central KD-vs-ADA claim is evaluated against held-out ground-truth WMH annotations, so it is not encoded in the method's inputs.

full rationale

The derivation chain is self-contained. Equation (1) trains the teacher on source labels; Equation (3) trains the student to match teacher soft labels on the union of source and target data. This is a training objective, not a prediction generated from the evaluation labels. All dice scores in Tables 2 and 3 are computed on held-out target subjects with manual annotations (Sections 4.2 and 4.3), so the abstract's claim that KD beats the baseline and ADA is an external measurement rather than a consequence of construction. The paper's reliance on teacher soft labels for unlabeled target data is a genuine limitation, and the authors acknowledge it in Section 5: 'This may have led to an inferior teacher performance that the student network relies on.' But that is a robustness or error-propagation concern, not equation-level circularity: the student is not defined in terms of the held-out labels, and no fitted parameter is renamed as a prediction (the temperature is fixed at T=2). There are no load-bearing self-citations: references [2], [9], and [11] are independent prior works, and no uniqueness theorem is imported from the authors. The reported inconsistency between Table 2 and the prose in Section 4.3 regarding the Utrecht-Singapore direction is a statistical reporting issue outside the circularity definition. Overall, no step reduces to its own input.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim relies on only a few hand-chosen experimental parameters. The temperature and the ADA baseline configuration are the most consequential; the number of adaptation scans is also a choice. No new explanatory entities are introduced.

free parameters (3)

distillation temperature T = 2
Controls the softness of the teacher's probability outputs in the distillation loss (Eq. 3). The paper states it is chosen based on source-domain performance, but no selection procedure is reported; it remains a hand-chosen hyperparameter.
ADA discriminator architecture = 4 conv layers (8,16,32,64 filters) + 3 FC layers (64,128,2)
The adversarial baseline uses this hand-picked discriminator, placed on the final layer of the segmentation network rather than the configuration in Kamnitsas et al. This choice affects the comparison and is not tuned.
number of unlabeled target scans used for adaptation = 10
Exactly 10 of the 20 target subjects are used for adaptation in the main experiments; the central claim depends on this split.

assumptions (3)

domain assumption Source and target domains share the same label space and the WMH segmentation task is transferable across scanners.
The method assumes that soft labels produced by a source-trained teacher are relevant for training on target scans; this is the premise of the cross-clinic experiments in §4.
ad hoc to paper The student network can learn a better mapping from the teacher's soft labels on unlabeled target data.
Stated in §3.1 as an expectation, with no proof or ablation. This is the key unvalidated premise of the adaptation step.
standard math Standard optimization and loss-function assumptions (softmax, cross-entropy, Adam) hold.
The experiments rely on standard deep learning training; no new mathematical claims are made.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Knowledge distillation for semi-supervised domain adaptation." pith.science (2026). https://pith.science/paper/4A4W3MQ4

@misc{pith2026190807355,
  author       = {Pith},
  title        = {Pith review of: Knowledge distillation for semi-supervised domain adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4A4W3MQ4}},
  note         = {Machine review of arXiv:1908.07355}
}

read the original abstract

In the absence of sufficient data variation (e.g., scanner and protocol variability) in annotated data, deep neural networks (DNNs) tend to overfit during training. As a result, their performance is significantly lower on data from unseen sources compared to the performance on data from the same source as the training data. Semi-supervised domain adaptation methods can alleviate this problem by tuning networks to new target domains without the need for annotated data from these domains. Adversarial domain adaptation (ADA) methods are a popular choice that aim to train networks in such a way that the features generated are domain agnostic. However, these methods require careful dataset-specific selection of hyperparameters such as the complexity of the discriminator in order to achieve a reasonable performance. We propose to use knowledge distillation (KD) -- an efficient way of transferring knowledge between different DNNs -- for semi-supervised domain adaption of DNNs. It does not require dataset-specific hyperparameter tuning, making it generally applicable. The proposed method is compared to ADA for segmentation of white matter hyperintensities (WMH) in magnetic resonance imaging (MRI) scans generated by scanners that are not a part of the training set. Compared with both the baseline DNN (trained on source domain only and without any adaption to target domain) and with using ADA for semi-supervised domain adaptation, the proposed method achieves significantly higher WMH dice scores.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 7 canonical work pages

[9]

In: International conference on information processing in medical imaging, Springer (2017) 597--609

Kamnitsas, K., Baumgartner, C., Ledig, C., Newcombe, V., Simpson, J., Kane, A., Menon, D., Nori, A., Criminisi, A., Rueckert, D., et al.: Unsupervised domain adaptation in brain lesion segmentation with adversarial networks. In: International conference on information processing in medical imaging, Springer (2017) 597--609

work page 2017
[1]

In: Advances in Neural Information Processing Systems

Oliver, A., Odena, A., Raffel, C.A., Cubuk, E.D., Goodfellow, I.: Realistic evaluation of deep semi-supervised learning algorithms. In: Advances in Neural Information Processing Systems. (2018) 3239--3250

work page 2018
[2]

arXiv preprint arXiv:1503.02531 (2015)

Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

arXiv 2015
[3]

arXiv preprint arXiv:1511.03643 (2015)

Lopez-Paz, D., Bottou, L., Sch \"o lkopf, B., Vapnik, V.: Unifying distillation and privileged information. arXiv preprint arXiv:1511.03643 (2015)

arXiv 2015
[4]

arXiv preprint arXiv:1301.3224 (2013)

Hoffman, J., Rodner, E., Donahue, J., Darrell, T., Saenko, K.: Efficient learning of domain-invariant image representations. arXiv preprint arXiv:1301.3224 (2013)

arXiv 2013
[5]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer (2018) 476--484

Karani, N., Chaitanya, K., Baumgartner, C., Konukoglu, E.: A lifelong learning approach to brain mr segmentation across scanners and protocols. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer (2018) 476--484

work page 2018
[6]

In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2017) 7167--7176

work page 2017
[7]

In: European Conference on Computer Vision, Springer (2016) 443--450

Sun, B., Saenko, K.: Deep coral: Correlation alignment for deep domain adaptation. In: European Conference on Computer Vision, Springer (2016) 443--450

work page 2016

Show all 14 references

[8]

arXiv preprint arXiv:1711.03213 (2017)

Hoffman, J., Tzeng, E., Park, T., Zhu, J.Y., Isola, P., Saenko, K., Efros, A.A., Darrell, T.: Cycada: Cycle-consistent adversarial domain adaptation. arXiv preprint arXiv:1711.03213 (2017)

2017 arXiv
[10]

In: Proceedings of the IEEE conference on computer vision and pattern recognition

Gupta, S., Hoffman, J., Malik, J.: Cross modal distillation for supervision transfer. In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2016) 2827--2836

2016
[11]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer (2018) 572--580

Huang, R., Noble, J.A., Namburete, A.I.: Omni-supervised learning: Scaling up to large unlabelled medical datasets. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer (2018) 572--580

2018
[12]

, " * write output.state after.block = add.period write

ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...
[13]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
[14]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION stupid.colon after.authors 'output.state := FUNCTION insert.comma output.state before.all = 'skip between...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [9]

In: International conference on information processing in medical imaging, Springer (2017) 597--609

Kamnitsas, K., Baumgartner, C., Ledig, C., Newcombe, V., Simpson, J., Kane, A., Menon, D., Nori, A., Criminisi, A., Rueckert, D., et al.: Unsupervised domain adaptation in brain lesion segmentation with adversarial networks. In: International conference on information processing in medical imaging, Springer (2017) 597--609

work page 2017

[2] [1]

In: Advances in Neural Information Processing Systems

Oliver, A., Odena, A., Raffel, C.A., Cubuk, E.D., Goodfellow, I.: Realistic evaluation of deep semi-supervised learning algorithms. In: Advances in Neural Information Processing Systems. (2018) 3239--3250

work page 2018

[3] [2]

arXiv preprint arXiv:1503.02531 (2015)

Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

arXiv 2015

[4] [3]

arXiv preprint arXiv:1511.03643 (2015)

Lopez-Paz, D., Bottou, L., Sch \"o lkopf, B., Vapnik, V.: Unifying distillation and privileged information. arXiv preprint arXiv:1511.03643 (2015)

arXiv 2015

[5] [4]

arXiv preprint arXiv:1301.3224 (2013)

Hoffman, J., Rodner, E., Donahue, J., Darrell, T., Saenko, K.: Efficient learning of domain-invariant image representations. arXiv preprint arXiv:1301.3224 (2013)

arXiv 2013

[6] [5]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer (2018) 476--484

Karani, N., Chaitanya, K., Baumgartner, C., Konukoglu, E.: A lifelong learning approach to brain mr segmentation across scanners and protocols. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer (2018) 476--484

work page 2018

[7] [6]

In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2017) 7167--7176

work page 2017

[8] [7]

In: European Conference on Computer Vision, Springer (2016) 443--450

Sun, B., Saenko, K.: Deep coral: Correlation alignment for deep domain adaptation. In: European Conference on Computer Vision, Springer (2016) 443--450

work page 2016