An In-Depth Analysis of Adversarial Discriminative Domain Adaptation for Digit Classification

Edmund Young; Eugene Choi; Julian Rodriguez

REVIEW 3 major objections 4 minor 16 references

An In-Depth Analysis of Adversarial Discriminative Domain Adaptation for Digit Classification

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that ADDA improves target-domain accuracy in five of the six MNIST/USPS/SVHN domain shifts, with significant gains in four and minimal in-domain loss for the simpler domains.

desk verdict A transparent replication-extension of ADDA that fills in the missing digit-shift experiments, but the headline 'significant' claims are undercut by single runs and no error bars. read the letter →

arxiv 2412.19391 v2 pith:EZFFBHKW submitted 2024-12-27 cs.CV cs.AIcs.LG

Eugene Choi , Julian Rodriguez , Edmund Young This is my paper · ORCID

classification cs.CVcs.AIcs.LG

keywords domainadaptationadversariallearningADDAdigitclassificationMNISTUSPSSVHNt-SNE

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test whether Adversarial Discriminative Domain Adaptation (ADDA) helps a digit classifier generalize across all six source-target pairings of MNIST, USPS, and SVHN, not just the three pairings the original ADDA paper reported. It implements ADDA from scratch and measures both out-of-domain target accuracy and post-adaptation in-domain accuracy. The central claim is that ADDA improves target accuracy in five of the six shifts, with substantial improvements in four, while leaving in-domain accuracy almost unchanged for MNIST and USPS but sharply lower when SVHN is the source. If correct, the result maps where a standard adversarial adaptation recipe helps and where it fails, and points to a fixed training budget as a likely cause of the failures.

What carries the argument

The machinery is ADDA's asymmetric mapping: a source encoder and classifier are pretrained on labeled source data and frozen, then a separate target encoder with unshared weights is trained against a domain discriminator under the standard GAN loss so that the target encoder learns to map target images into the already discriminative source feature space. At test time, target data passes through the target encoder and the frozen source classifier. The paper instantiates this with a modified LeNet encoder, a width-500 classifier, and a three-layer discriminator with hidden width 500, and evaluates it with fixed budgets of 80 source-pretraining epochs and 150 adversarial-adaptation epochs for every shift.

What would settle it

Train USPS to SVHN with adversarial adaptation extended well beyond 150 epochs, for example 300, 600, and 1200 epochs, using the paper's architectures, and record the target accuracy. If it rises above the 0.2567 baseline and keeps climbing, the reported failure is an under-training artifact of the fixed budget; if it stays at or below the baseline with longer training, the failure is a genuine limitation of ADDA for that shift.

Watch

Extended reading notes

Core claim

The central discovery is that one adversarial adaptation recipe, applied with the same training budget to every shift, improves out-of-domain generalization for five of the six source-target combinations among MNIST, USPS, and SVHN, and the paper characterizes four of those improvements as significant. The largest improvement is MNIST to USPS, rising from 0.4305 to 0.6886 target accuracy, while the only decline is USPS to SVHN, falling from 0.2567 to 0.2542. The paper also measures post-adaptation in-domain accuracy: MNIST and USPS lose at most 0.0105 in source accuracy, whereas SVHN loses as much as 0.5025, a pattern the paper attributes to the larger gap between SVHN's feature distribution and those of MNIST and USPS. Confusion matrices and t-SNE plots show that when adaptation works, the target encoder forms distinct digit clusters, and when it fails, no such structure appears.

Load-bearing premise

The load-bearing assumption is that a single fixed training budget of 80 source epochs and 150 adversarial epochs is fair for every one of the six domain shifts, even though the paper itself notes that SVHN-target shifts may need more training time.

Editorial extensions

If this is right

ADDA's practical gains are concentrated in shifts between visually similar digit domains: the two MNIST-USPS directions improve most, while shifts into SVHN gain little or nothing.
The in-domain measurements imply that adversarial adaptation can substantially degrade source accuracy for a more complex dataset such as SVHN, so users should check post-adaptation source accuracy before deployment.
The fixed 80-epoch and 150-epoch budgets mean the reported SVHN-target accuracies are tied to a specific compute allocation; per-shift tuning could change the comparison.
The confusion-matrix pattern, with 7, 0, and 4 misclassified as 1, implies that remaining errors concentrate in digit pairs that share visual structure rather than being random.
The t-SNE results imply that successful adaptation is associated with visible digit clusters in the target encoder output, giving a quick diagnostic for whether ADDA is working.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that its single failure case, USPS to SVHN, is confounded with the fixed training budget; a reader should not treat that failure as a proven property of ADDA.
The success pattern suggests a testable rule: adversarial discriminative adaptation helps when source and target feature spaces are close enough for one frozen classifier to remain useful, which would predict that extra training helps SVHN-target shifts less than MNIST-related shifts.
The large in-domain drop for SVHN hints that ADDA's mapping may collapse complex real-world digit features into a shared space that is poorly separated; applying gradient-based visual explanations to the adapted encoder could test where the classifier actually looks.
Combining ADDA with source-side data augmentation, such as the domain randomization the paper mentions in future work, could be a direct way to reduce the SVHN-as-source in-domain degradation.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things up front. First, this paper is exactly what it says it is: a from-scratch ADDA implementation, run on all six digit domain shifts, with code released. That is a legitimate contribution because the original ADDA paper only reported three of six shifts and never reported post-adaptation source accuracy. Second, the central claim that ADDA 'significantly improves' in four shifts is not supported by the evidence as presented. Every number in Table 2 is a single run at a single checkpoint. No seeds, no error bars, no significance tests. The deciding margins are tiny—SVHN→USPS improves by +0.009 and USPS→SVHN drops by -0.0025—so the five-of-six qualitative result could easily flip under reseeding.

What the paper does well: it is honest. The authors explicitly flag the fixed 80/150 epoch budget as a confound, they note that USPS→SVHN may need more training time, and they provide confusion matrices and t-SNE plots that let readers see the failure modes. The in-domain degradation analysis is sensible: MNIST and USPS keep most of their source accuracy, while SVHN drops sharply, which is consistent with SVHN being a harder, visually richer domain. The related work is standard and the method description matches Tzeng et al. closely, which is what you want in a replication study.

The soft spots are real but not fatal. The fixed epoch budget is a genuine confound for any SVHN-target comparison, and the authors admit it. The 'significant' language in the abstract, Section 5.1, and the conclusion should be softened to 'large improvements' or 'consistent improvements' given the missing variance estimates. They also never compare their baseline numbers to the original ADDA paper, so I cannot independently verify that their implementation reproduces the original setup, though the numbers look plausible.

The paper does not overreach. It does not propose new theory, and it does not claim to change practice. It documents a measurement gap in a well-known method. For someone who works on domain adaptation or wants a clean example of why error bars matter in deep learning replication, this is a useful read. The code availability is a plus, even without a commit hash.

I would send this to peer review. It deserves referee time because it is a concrete, reproducible measurement of ADDA across all digit shifts, and the limitations are openly stated. The requested revision is straightforward: add multiple seeds, report standard deviations, and do a longer-schedule check for USPS→SVHN. With those, the paper would be solid for a workshop or a small empirical venue.

Referee Report

3 major / 4 minor

Summary. The paper implements Adversarial Discriminative Domain Adaptation (ADDA) from scratch and evaluates it on all six directed domain shifts among MNIST, USPS, and SVHN, extending the original ADDA paper's three reported shifts. The authors report source-pretrained baseline accuracy, ADDA target accuracy, and post-ADDA source (in-domain) accuracy for each shift, together with confusion matrices and t-SNE visualizations. Their central claim, stated in §5.1 and the Conclusion, is that ADDA improves out-of-domain accuracy in five of six shifts, with significant improvements in four, and that in-domain accuracy is minimally degraded for MNIST and USPS but drops substantially for SVHN as source. The paper also candidly acknowledges that a single fixed training budget was used for all shifts due to time limitations.

Significance. If the quantitative claims are robust, the paper is a useful reproducibility study: it independently confirms ADDA's behavior on three of the original shifts, adds results on the three previously unreported shifts, and contributes a systematic in-domain degradation analysis that the original paper does not provide. The public code, the all-six-shift coverage, and the explicit discussion of the fixed-epoch-budget limitation are strengths. However, the central empirical claim is currently supported only by single-run, single-checkpoint accuracies, and the deciding margins are so small that the five-of-six summary could easily invert under reseeding. The significance of the contribution therefore hinges on whether the accuracy results are shown to be stable and not artifacts of the fixed training schedule.

major comments (3)

[§5.1 / Tab. 2 / Conclusion] The central claim that ADDA improves out-of-domain accuracy in five of six shifts, with significant improvements in four, is not supported by the reported evidence because each entry in Tab. 2 is a single run at a single epoch with no error bars, no reruns across random seeds, and no significance tests. The two entries that decide the qualitative claim are within plausible seed-level noise: SVHN→USPS improves by only +0.0090 and USPS→SVHN drops by only -0.0025. Please report mean and standard deviation over at least 5 seeds (or bootstrap confidence intervals from test predictions) and a paired significance test, and state explicitly whether the five-of-six pattern survives reseeding.
[§4.3 / §6 / Tab. 2] The fixed training budget of 80 source epochs and 150 adversarial adaptation epochs, chosen 'due to time limitations' (§4.3), is a confound for the main conclusion. The authors themselves note in §4.3 and §6 that the optimal epoch count likely differs per shift and that SVHN-target shifts may need longer training. Since the only negative result (USPS→SVHN, -0.0025) and the smallest positive result (SVHN→USPS, +0.0090) both involve SVHN as target, it is currently not possible to attribute these outcomes to ADDA's properties rather than to under-training. Please report learning curves or run longer training for the SVHN-target shifts to show that the accuracies have plateaued by the end of the budget.
[§5.1, ADDA-source paragraph] The explanation that 'drastic domain shifts result in lower in-domain accuracy' is extrapolated from six shifts without an independent measure of domain shift, so the causal wording is stronger than the evidence supports. If the in-domain degradation analysis is a central contribution, please add a quantitative domain-similarity proxy (for example, baseline target accuracy or a feature-space distance) to justify the ordering as 'drastic,' or soften the claim to a correlation rather than a causal mechanism.

minor comments (4)

[§4.3] There is a spacing typo: 'with200 images' should read 'with 200 images.'
[§5.2.2] The text refers to the t-SNE plot of SVHN→MNIST evaluated on MNIST as 'Fig. 7 lower panel,' but that plot is the upper panel of Fig. 7; please correct the pointer.
[Abstract / §5.1] The abstract uses 'significant' without statistical support, while §5.1 uses the same word informally; please align the wording with the evidence after the requested variance analysis, and reserve 'significant' for results that survive a stated significance criterion.
[Abstract] The abstract says 'Code is here' with no visible URL in the paper body; please include a proper reference to the code repository in the abstract or footnote.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical replication study with direct measurements; no circular steps identified.

full rationale

The paper is an empirical replication study rather than a derivation. It implements the ADDA method exactly as described in the original ADDA paper by Tzeng et al. [3], which is external prior work and not authored by the present authors. The authors then train source and target encoders on fixed digit datasets and report measured accuracies in Table 2. The central claim that "ADDA improves out-of-domain generalization ability" in five of six domain shifts is a direct reading of those measurements, not the result of any fitted parameter being renamed a prediction, any self-referential uniqueness theorem, any ansatz smuggled in via self-citation, or any definitional identity linking inputs to outputs. The paper explicitly treats the original ADDA formulation as a method to be replicated, not as a conclusion to be derived. Concerns about single-run results, missing error bars, and the fixed 80/150-epoch training budget are statistical and internal-validity issues, not circularity. No load-bearing self-citations appear; all references to prior work are to the external ADDA method, standard datasets, and common deep-learning techniques. Under the hard rule that circularity must be exhibited by quoting a specific reduction, none can be exhibited here. The paper is self-contained as an empirical measurement study, so the circularity score is 0.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two classes of choices: fixed hyperparameters (epochs, learning rates, batch size, discriminator width) and domain assumptions (grayscale SVHN preserves labels, the source classifier transfers once features align). No new entities or fitted parameters are introduced; the only parameters are standard training hyperparameters, several of which were held constant across all six shifts and are acknowledged as a limitation.

free parameters (7)

Source pretraining epochs = 80
Fixed for all source datasets due to compute limits (Section 4.3); may be too few for SVHN and affects every reported number.
Adversarial adaptation epochs = 150
Fixed for all six shifts (Section 4.3); authors suggest USPS-to-SVHN may need more training, so the sole failure could be a training-time artifact.
Pretraining learning rate = 1e-3
Adam learning rate used while training the source encoder (Section 4.3).
Adaptation learning rate = 1e-4
Adam learning rate used for discriminator and target encoder (Section 4.3).
Batch size = 200
Mini-batch size used in both training stages (Section 4.3).
Discriminator hidden width = 500
Chosen after a 100-node version performed poorly (Section 4.1); the final value matches the original ADDA architecture.
Random seed = not reported
No seeds are listed, so run-to-run variance cannot be assessed and exact reproduction is not guaranteed.

assumptions (4)

domain assumption Adam-based mini-batch training, run for the stated number of epochs, yields a usable, representative model for each component.
Section 4.3; this is an unproved practical assumption about optimization.
domain assumption A classifier trained only on source labels remains valid for target data once the target encoder maps into the source feature space.
Core ADDA assumption used in the testing phase, Section 3.
domain assumption Grayscaling SVHN and resizing to 28 by 28 preserves digit-class information.
Stated in Section 4.2; if false, SVHN results are systematically depressed.
domain assumption t-SNE visualizations reflect the meaningful structure of the learned encoder features.
Qualitative claims in Section 5.2.2 interpret cluster quality as encoder quality.

how reviews work

0 comments

Cite this review

Pith. "Pith review of An In-Depth Analysis of Adversarial Discriminative Domain Adaptation for Digit Classification." pith.science (2026). https://pith.science/paper/EZFFBHKW

@misc{pith2026241219391,
  author       = {Pith},
  title        = {Pith review of: An In-Depth Analysis of Adversarial Discriminative Domain Adaptation for Digit Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EZFFBHKW}},
  note         = {Machine review of arXiv:2412.19391}
}

read the original abstract

Domain adaptation is an active area of research driven by the growing demand for robust machine learning models that perform well on real-world data. Adversarial learning for deep neural networks (DNNs) has emerged as a promising approach to improving generalization ability, particularly for image classification. In this paper, we implement a specific adversarial learning technique known as Adversarial Discriminative Domain Adaptation (ADDA) and replicate digit classification experiments from the original ADDA paper. We extend their findings by examining a broader range of domain shifts and provide a detailed analysis of in-domain classification accuracy post-ADDA. Our results demonstrate that ADDA significantly improves accuracy across certain domain shifts with minimal impact on in-domain performance. Furthermore, we provide qualitative analysis and propose potential explanations for ADDA's limitations in less successful domain shifts. Code is at https://github.com/eugenechoi2004/COS429_FINAL .

Figures

Figures reproduced from arXiv: 2412.19391 by the authors.

**Figure 3.** SVHN → MNIST domain shift confusion matrices are often misclassified as 1. 4 and 9 are also close to each other in both plots, as well as 3 and 5. The SVHN → USPS domain shift evaluated on USPS ( [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 5.** USPS → SVHN domain shift confusion matrices In addition, further qualitative analysis such as GradCAM [16] would be very insightful in providing visual explanations for poor model performance (particularly domain shifts evaluated on SVHN as the target). These visual explanations would be useful in understanding the current limitations of ADDA and potentially outline improvements to the training process. ADDA can a… view at source ↗

Figures from the paper (2 more)

**Figure 7.** Figure 7: SVHN → MNIST t-SNE Plots mance degradation compared to pre-ADDA in-domain accuracies. We find that there are minimal performance drops for MNIST and USPS, while SVHN suffers large performance drops. Furthermore, we provide detailed qualitative analysis through confus…

**Figure 8.** Figure 8: SVHN → USPS t-SNE Plots References [1] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks, 2015. 1 [2] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 15 canonical work pages

[1]

Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks, 2015. 1

work page 2015
[2]

Deep domain confusion: Maximizing for domain invariance, 2014

Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance, 2014. 1

work page 2014
[3]

Adversarial discriminative domain adaptation, 2017

Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation, 2017. 1, 2

work page 2017
[4]

Lecun, L

Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(11):2278–2324, 1998. 1, 2

work page 1998
[5]

J.J. Hull. A database for handwritten text recognition re- search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(5):550–554, 1994. 1, 2

work page 1994
[6]

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bis- sacco, Bo Wu, and Andrew Y . Ng. Reading digits in natural (a) USPS → SVHN evaluated on SVHN (b) USPS → SVHN evaluated on USPS Figure 9. USPS → SVHN t-SNE Plots images with unsupervised feature learning. In NIPS Work- shop on Deep Learning and Unsupervised Feature Learning 2011, 2011. 1, 2

work page 2011
[7]

Kouw and Marco Loog

Wouter M. Kouw and Marco Loog. An introduction to do- main adaptation and transfer learning, 2019. 1

work page 2019
[8]

Hospedales

Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M. Hospedales. A simple feature augmentation for domain generalization. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8866–8875,

work page 2021

Show all 16 references

[9]

Domain-adversarial train- ing of neural networks, 2016

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pas- cal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial train- ing of neural networks, 2016. 1

2016
[10]

Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. 1 7

2014
[11]

Adversarial autoencoders,

Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders,
[12]

Domain randomization for transferring deep neural networks from simulation to the real world, 2017

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Woj- ciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world, 2017. 1, 5

2017
[13]

Caffe: Convolutional architecture for fast feature embedding, 2014

Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding, 2014. 2

2014
[14]

Kingma and Jimmy Ba

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. 2

2017
[15]

Visualizing data using t-sne

Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(86):2579–2605, 2008. 3, 4

2008
[16]

Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Ba- tra

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Ba- tra. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Com- puter Vision, 128(2):336–359, October 2019. 5 8

2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks, 2015. 1

work page 2015

[2] [2]

Deep domain confusion: Maximizing for domain invariance, 2014

Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance, 2014. 1

work page 2014

[3] [3]

Adversarial discriminative domain adaptation, 2017

Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation, 2017. 1, 2

work page 2017

[4] [4]

Lecun, L

Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(11):2278–2324, 1998. 1, 2

work page 1998

[5] [5]

J.J. Hull. A database for handwritten text recognition re- search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(5):550–554, 1994. 1, 2

work page 1994

[6] [6]

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bis- sacco, Bo Wu, and Andrew Y . Ng. Reading digits in natural (a) USPS → SVHN evaluated on SVHN (b) USPS → SVHN evaluated on USPS Figure 9. USPS → SVHN t-SNE Plots images with unsupervised feature learning. In NIPS Work- shop on Deep Learning and Unsupervised Feature Learning 2011, 2011. 1, 2

work page 2011

[7] [7]

Kouw and Marco Loog

Wouter M. Kouw and Marco Loog. An introduction to do- main adaptation and transfer learning, 2019. 1

work page 2019

[8] [8]

Hospedales

Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M. Hospedales. A simple feature augmentation for domain generalization. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8866–8875,

work page 2021