{"id":"8164a2c0-be2b-46f2-8997-756788ffee4b","arxiv_id":"2412.19391","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"In a replication of the ADDA method, target-domain accuracy improved in five of six digit domain shifts, with the largest gains for MNIST-USPS pairs and a small loss for USPS-to-SVHN.","lead":"This paper re-implements the ADDA adversarial domain adaptation method from scratch and tests it on all six digit-classification domain shifts between MNIST, USPS, and SVHN. It reports that ADDA improves target-domain accuracy in five of six shifts, while degrading in-domain accuracy most severely when SVHN is the source.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The five-of-six claim rests on single-run final-epoch accuracies with no error bars; the deciding margins (+0.009, -0.0025) are within plausible seed noise, so 'significant' is unsupported.","rationale":"The paper's contribution is an empirical claim about which domain shifts ADDA helps, so the validity of the headline depends on the reproducibility of the accuracy deltas in Tab. 2. The smallest deltas are precisely the ones that separate 'five of six' from 'four of six,' and a single-run comparison gives no way to know whether those deltas are signal or noise. The paper's own limitation statements in §4.3 and §6 acknowledge that the fixed training budget is a confound, especially for SVHN-target shifts, which reinforces the concern but does not replace the need for seed variance. I do not see an internal inconsistency or a fatal flaw: the qualitative pattern—large gains for MNIST/USPS pairs and trouble with SVHN as target—is plausible and consistent with the original ADDA direction. However, the strength of the wording ('significant improvements in four domain shifts') exceeds what the evidence supports. Since the reader's verdict is already CONDITIONAL and my concern is a concrete condition that would strengthen the paper, the appropriate recommendation remains CONDITIONAL (UNCHANGED). A multi-seed, multi-checkpoint study would settle whether the concern lands; if it lands, the paper would need to soften the 'significant' language and re-examine the USPS→SVHN result under a longer schedule. My agreement with the reader is partial: they emphasize the fixed epoch budget, while I emphasize the missing error bars and significance testing, with the fixed budget as a closely related confound.","tokens_in":6138,"tokens_out":6833,"duration_ms":65330,"concrete_test":"For each of the six domain shifts, run the provided code (or a minimal re-implementation) with at least 5 random seeds, recording target-domain and source-domain accuracy every 25 epochs of adversarial adaptation up to at least 300 epochs. Compute mean ± std and a paired permutation test comparing baseline accuracy to ADDA accuracy at both the final epoch and the best epoch on a fixed target validation split. If SVHN→USPS's +0.0090 or USPS→SVHN's -0.0025 does not remain a consistent directional effect across seeds, or if USPS→SVHN becomes positive with extended training, the five-of-six claim and its 'significant' wording should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §5.1 and the Conclusion is that ADDA improves target accuracy in five of six domain shifts and does so 'significantly' in four. But every entry in Tab. 2 is a single run at a single checkpoint: no random seeds, no error bars, and no significance tests are reported. The cases that decide whether the claim is 'five of six' are near zero: SVHN→USPS improves by +0.0090 and USPS→SVHN drops by -0.0025. For deep digit classifiers, such small differences are easily within one standard deviation of seed-to-seed variation, so the qualitative 'five of six' result could invert under reseeding. The problem is compounded by §4.3's fixed 80/150-epoch budget, chosen 'due to time limitations,' which the authors themselves say may hurt SVHN-target shifts and that USPS→SVHN may need more training time. Thus the only negative result and the smallest positive result are confounded by training schedule as well as by missing variance information. Without variance estimates, the word 'significant' is not justified, and the attribution of the USPS→SVHN failure to dataset difficulty rather than to under-training is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper implements Adversarial Discriminative Domain Adaptation (ADDA) from scratch and evaluates it on all six directed domain shifts among MNIST, USPS, and SVHN, extending the original ADDA paper's three reported shifts. The authors report source-pretrained baseline accuracy, ADDA target accuracy, and post-ADDA source (in-domain) accuracy for each shift, together with confusion matrices and t-SNE visualizations. Their central claim, stated in §5.1 and the Conclusion, is that ADDA improves out-of-domain accuracy in five of six shifts, with significant improvements in four, and that in-domain accuracy is minimally degraded for MNIST and USPS but drops substantially for SVHN as source. The paper also candidly acknowledges that a single fixed training budget was used for all shifts due to time limitations.","tokens_in":6334,"tokens_out":3237,"duration_ms":28247,"significance":"If the quantitative claims are robust, the paper is a useful reproducibility study: it independently confirms ADDA's behavior on three of the original shifts, adds results on the three previously unreported shifts, and contributes a systematic in-domain degradation analysis that the original paper does not provide. The public code, the all-six-shift coverage, and the explicit discussion of the fixed-epoch-budget limitation are strengths. However, the central empirical claim is currently supported only by single-run, single-checkpoint accuracies, and the deciding margins are so small that the five-of-six summary could easily invert under reseeding. The significance of the contribution therefore hinges on whether the accuracy results are shown to be stable and not artifacts of the fixed training schedule.","major_comments":[{"comment":"The central claim that ADDA improves out-of-domain accuracy in five of six shifts, with significant improvements in four, is not supported by the reported evidence because each entry in Tab. 2 is a single run at a single epoch with no error bars, no reruns across random seeds, and no significance tests. The two entries that decide the qualitative claim are within plausible seed-level noise: SVHN→USPS improves by only +0.0090 and USPS→SVHN drops by only -0.0025. Please report mean and standard deviation over at least 5 seeds (or bootstrap confidence intervals from test predictions) and a paired significance test, and state explicitly whether the five-of-six pattern survives reseeding.","section":"§5.1 / Tab. 2 / Conclusion"},{"comment":"The fixed training budget of 80 source epochs and 150 adversarial adaptation epochs, chosen 'due to time limitations' (§4.3), is a confound for the main conclusion. The authors themselves note in §4.3 and §6 that the optimal epoch count likely differs per shift and that SVHN-target shifts may need longer training. Since the only negative result (USPS→SVHN, -0.0025) and the smallest positive result (SVHN→USPS, +0.0090) both involve SVHN as target, it is currently not possible to attribute these outcomes to ADDA's properties rather than to under-training. Please report learning curves or run longer training for the SVHN-target shifts to show that the accuracies have plateaued by the end of the budget.","section":"§4.3 / §6 / Tab. 2"},{"comment":"The explanation that 'drastic domain shifts result in lower in-domain accuracy' is extrapolated from six shifts without an independent measure of domain shift, so the causal wording is stronger than the evidence supports. If the in-domain degradation analysis is a central contribution, please add a quantitative domain-similarity proxy (for example, baseline target accuracy or a feature-space distance) to justify the ordering as 'drastic,' or soften the claim to a correlation rather than a causal mechanism.","section":"§5.1, ADDA-source paragraph"}],"minor_comments":[{"comment":"There is a spacing typo: 'with200 images' should read 'with 200 images.'","section":"§4.3"},{"comment":"The text refers to the t-SNE plot of SVHN→MNIST evaluated on MNIST as 'Fig. 7 lower panel,' but that plot is the upper panel of Fig. 7; please correct the pointer.","section":"§5.2.2"},{"comment":"The abstract uses 'significant' without statistical support, while §5.1 uses the same word informally; please align the wording with the evidence after the requested variance analysis, and reserve 'significant' for results that survive a stated significance criterion.","section":"Abstract / §5.1"},{"comment":"The abstract says 'Code is here' with no visible URL in the paper body; please include a proper reference to the code repository in the abstract or footnote.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"This reads like an undergraduate course project: it is transparent about limitations and provides code, but the central claim is overstated relative to the evidence. A revision that adds multiple seeds, confidence intervals, and a proper treatment of the epoch-budget confound could make it a solid reproducibility study for a workshop or a domain-adaptation venue. The current version is not yet there; however, the flaws are fixable within the manuscript's scope, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things up front. First, this paper is exactly what it says it is: a from-scratch ADDA implementation, run on all six digit domain shifts, with code released. That is a legitimate contribution because the original ADDA paper only reported three of six shifts and never reported post-adaptation source accuracy. Second, the central claim that ADDA 'significantly improves' in four shifts is not supported by the evidence as presented. Every number in Table 2 is a single run at a single checkpoint. No seeds, no error bars, no significance tests. The deciding margins are tiny—SVHN→USPS improves by +0.009 and USPS→SVHN drops by -0.0025—so the five-of-six qualitative result could easily flip under reseeding.\n\nWhat the paper does well: it is honest. The authors explicitly flag the fixed 80/150 epoch budget as a confound, they note that USPS→SVHN may need more training time, and they provide confusion matrices and t-SNE plots that let readers see the failure modes. The in-domain degradation analysis is sensible: MNIST and USPS keep most of their source accuracy, while SVHN drops sharply, which is consistent with SVHN being a harder, visually richer domain. The related work is standard and the method description matches Tzeng et al. closely, which is what you want in a replication study.\n\nThe soft spots are real but not fatal. The fixed epoch budget is a genuine confound for any SVHN-target comparison, and the authors admit it. The 'significant' language in the abstract, Section 5.1, and the conclusion should be softened to 'large improvements' or 'consistent improvements' given the missing variance estimates. They also never compare their baseline numbers to the original ADDA paper, so I cannot independently verify that their implementation reproduces the original setup, though the numbers look plausible.\n\nThe paper does not overreach. It does not propose new theory, and it does not claim to change practice. It documents a measurement gap in a well-known method. For someone who works on domain adaptation or wants a clean example of why error bars matter in deep learning replication, this is a useful read. The code availability is a plus, even without a commit hash.\n\nI would send this to peer review. It deserves referee time because it is a concrete, reproducible measurement of ADDA across all digit shifts, and the limitations are openly stated. The requested revision is straightforward: add multiple seeds, report standard deviations, and do a longer-schedule check for USPS→SVHN. With those, the paper would be solid for a workshop or a small empirical venue.","headline":"A transparent replication-extension of ADDA that fills in the missing digit-shift experiments, but the headline 'significant' claims are undercut by single runs and no error bars.","tokens_in":6882,"tokens_out":1614,"would_cite":false,"duration_ms":15627,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that ADDA improves target-domain accuracy in five of the six MNIST/USPS/SVHN domain shifts, with significant gains in four and minimal in-domain loss for the simpler domains.","keywords":["domain adaptation","adversarial learning","ADDA","digit classification","MNIST","USPS","SVHN","t-SNE"],"falsifier":"Train USPS to SVHN with adversarial adaptation extended well beyond 150 epochs, for example 300, 600, and 1200 epochs, using the paper's architectures, and record the target accuracy. If it rises above the 0.2567 baseline and keeps climbing, the reported failure is an under-training artifact of the fixed budget; if it stays at or below the baseline with longer training, the failure is a genuine limitation of ADDA for that shift.","tokens_in":5880,"feed_emoji":"🔢","tokens_out":5303,"duration_ms":45373,"temperature":0.7,"pith_summary":"The paper sets out to test whether Adversarial Discriminative Domain Adaptation (ADDA) helps a digit classifier generalize across all six source-target pairings of MNIST, USPS, and SVHN, not just the three pairings the original ADDA paper reported. It implements ADDA from scratch and measures both out-of-domain target accuracy and post-adaptation in-domain accuracy. The central claim is that ADDA improves target accuracy in five of the six shifts, with substantial improvements in four, while leaving in-domain accuracy almost unchanged for MNIST and USPS but sharply lower when SVHN is the source. If correct, the result maps where a standard adversarial adaptation recipe helps and where it fails, and points to a fixed training budget as a likely cause of the failures.","feed_headline":"ADDA beats the baseline in five of six digit shifts","feed_subtitle":"Re-implemented from scratch, ADDA lifts cross-domain accuracy most when MNIST is involved and fails only USPS to SVHN.","key_machinery":"The machinery is ADDA's asymmetric mapping: a source encoder and classifier are pretrained on labeled source data and frozen, then a separate target encoder with unshared weights is trained against a domain discriminator under the standard GAN loss so that the target encoder learns to map target images into the already discriminative source feature space. At test time, target data passes through the target encoder and the frozen source classifier. The paper instantiates this with a modified LeNet encoder, a width-500 classifier, and a three-layer discriminator with hidden width 500, and evaluates it with fixed budgets of 80 source-pretraining epochs and 150 adversarial-adaptation epochs for every shift.","core_discovery":"The central discovery is that one adversarial adaptation recipe, applied with the same training budget to every shift, improves out-of-domain generalization for five of the six source-target combinations among MNIST, USPS, and SVHN, and the paper characterizes four of those improvements as significant. The largest improvement is MNIST to USPS, rising from 0.4305 to 0.6886 target accuracy, while the only decline is USPS to SVHN, falling from 0.2567 to 0.2542. The paper also measures post-adaptation in-domain accuracy: MNIST and USPS lose at most 0.0105 in source accuracy, whereas SVHN loses as much as 0.5025, a pattern the paper attributes to the larger gap between SVHN's feature distribution and those of MNIST and USPS. Confusion matrices and t-SNE plots show that when adaptation works, the target encoder forms distinct digit clusters, and when it fails, no such structure appears.","pith_inferences":["The paper leaves implicit that its single failure case, USPS to SVHN, is confounded with the fixed training budget; a reader should not treat that failure as a proven property of ADDA.","The success pattern suggests a testable rule: adversarial discriminative adaptation helps when source and target feature spaces are close enough for one frozen classifier to remain useful, which would predict that extra training helps SVHN-target shifts less than MNIST-related shifts.","The large in-domain drop for SVHN hints that ADDA's mapping may collapse complex real-world digit features into a shared space that is poorly separated; applying gradient-based visual explanations to the adapted encoder could test where the classifier actually looks.","Combining ADDA with source-side data augmentation, such as the domain randomization the paper mentions in future work, could be a direct way to reduce the SVHN-as-source in-domain degradation."],"forward_implications":["ADDA's practical gains are concentrated in shifts between visually similar digit domains: the two MNIST-USPS directions improve most, while shifts into SVHN gain little or nothing.","The in-domain measurements imply that adversarial adaptation can substantially degrade source accuracy for a more complex dataset such as SVHN, so users should check post-adaptation source accuracy before deployment.","The fixed 80-epoch and 150-epoch budgets mean the reported SVHN-target accuracies are tied to a specific compute allocation; per-shift tuning could change the comparison.","The confusion-matrix pattern, with 7, 0, and 4 misclassified as 1, implies that remaining errors concentrate in digit pairs that share visual structure rather than being random.","The t-SNE results imply that successful adaptation is associated with visible digit clusters in the target encoder output, giving a quick diagnostic for whether ADDA is working."],"supporting_citations":[{"why":"Supplies the ADDA method, the network architectures, and the digit-classification setup that the paper replicates and extends.","marker":"[3]"},{"why":"Supplies the MNIST dataset and the modified LeNet encoder architecture used for the source and target encoders.","marker":"[4]"},{"why":"Supplies the USPS dataset used in the domain-shift experiments.","marker":"[5]"},{"why":"Supplies the SVHN dataset with its real-world digit images, the main source of difficulty in the shifts.","marker":"[6]"},{"why":"Supplies the Adam optimizer used for both source pretraining and adversarial adaptation training.","marker":"[14]"},{"why":"Supplies the t-SNE visualization method used to assess whether the target encoder forms discriminative digit clusters.","marker":"[15]"}],"fun_headline_variants":["ADDA beats baseline in five of six digit shifts, fails USPS→SVHN","Digit adaptation: ADDA improves five, loses one","USPS to SVHN stumps ADDA; MNIST to USPS shines","ADDA scores 5/6 on digit domain transfers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a single fixed training budget of 80 source epochs and 150 adversarial epochs is fair for every one of the six domain shifts, even though the paper itself notes that SVHN-target shifts may need more training time.","fun_headline_variants_meta":{"raw":{"variants":["ADDA beats baseline in five of six digit shifts, fails USPS→SVHN","Digit adaptation: ADDA improves five, loses one","USPS to SVHN stumps ADDA; MNIST to USPS shines","ADDA scores 5/6 on digit domain transfers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000391,"raw_usage":{"total_tokens":2038,"prompt_tokens":905,"completion_tokens":1133,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1056}},"tokens_in":521,"tokens_out":1133,"duration_ms":8916,"temperature":1.0,"reasoning_tokens":1056,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:37:52.473458+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train USPS to SVHN with adversarial adaptation extended well beyond 150 epochs, for example 300, 600, and 1200 epochs, using the paper's architectures, and record the target accuracy. If it rises above the 0.2567 baseline and keeps climbing, the reported failure is an under-training artifact of the fixed budget; if it stays at or below the baseline with longer training, the failure is a genuine limitation of ADDA for that shift.","supporting_citations":[{"cited_title":"Adversarial discriminative domain adaptation, 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the ADDA method, the network architectures, and the digit-classification setup that the paper replicates and extends."},{"cited_title":"Lecun, L","cited_arxiv_id":null,"evidence_quote":"Supplies the MNIST dataset and the modified LeNet encoder architecture used for the source and target encoders."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the USPS dataset used in the domain-shift experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SVHN dataset with its real-world digit images, the main source of difficulty in the shifts."},{"cited_title":"Kingma and Jimmy Ba","cited_arxiv_id":null,"evidence_quote":"Supplies the Adam optimizer used for both source pretraining and adversarial adaptation training."},{"cited_title":"Visualizing data using t-sne","cited_arxiv_id":null,"evidence_quote":"Supplies the t-SNE visualization method used to assess whether the target encoder forms discriminative digit clusters."}],"review_version":1}