{"id":"a1d5560c-a073-46e5-92f4-8e0b76491154","arxiv_id":"2608.09630","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A comparative study showing that conditional domain adaptation combined with uncertainty-based multitask balancing improves gamma-ray event reconstruction under extreme label shift in CTAO LST simulations.","lead":"This paper combines domain adaptation with multitask learning to help gamma-ray telescopes trained on simulations work on noisy real observations where gamma rays are extremely rare compared to background protons. The authors find that an importance-weighted variant of an adversarial method, paired with automatic loss balancing, gives the best reconstruction results on simulated telescope data, and they release the code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CTAO ranking that selects CDANN+UW is based on single runs and on target-test-set selection; the reported edge over CDANN+GN is within plausible seed noise.","rationale":"The reader's weakest assumption identifies the external validity of the Poisson-only target degradation as the main risk. That is a real limitation, and the authors acknowledge it. However, the more load-bearing problem is internal: the CTAO comparison that produces the headline ranking has no error bars and the winner is chosen by looking at the target test set. Even if the simulation-based target were perfectly representative of real observations, the reported differences between CDANN+UW and its closest competitor are small enough that they could be seed noise, so the central claim would still lack statistical support. The reader's other observations, such as the unreproduced DeepJDOT baseline, reinforce the need for caution but do not change the verdict. The proposed concrete test would settle the internal question directly: re-run with seeds and a separate validation split. If the ranking survives that test, the conclusion becomes substantially stronger; if not, the paper should be revised to conditional language about the selected method. This keeps the reader's CONDITIONAL verdict rather than moving it to ACCEPT or REJECT, because the concern is concrete but potentially fixable by additional experiments.","tokens_in":19252,"tokens_out":5930,"duration_ms":57995,"concrete_test":"Re-run each CTAO configuration in Tables 7-9 with ten random seeds, matching the digit-protocol Section 4.2, and split the target set once into a model-selection subset and a final-test subset. Choose alpha and the winning method using only the selection subset, then evaluate the chosen model on the untouched final-test subset, reporting mean and standard deviation over seeds for E_mu, E_sigma, theta_sigma, and AUC. If CDANN+UW is not significantly better than the runner-up on the held-out subset, for example by a paired Wilcoxon test with p<0.05, then the conclusion that CDANN+UW is the most promising tool should be relaxed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim selects CDANN+UW as the most promising configuration from the CTAO comparison. That selection is not statistically grounded. Unlike the digit experiments, which average ten seeds per configuration (Section 4.2), the CTAO ablation Tables 7-9 report single numbers per configuration with no error bars and no run counts. The differences that drive the choice are small: in Table 7, CDANN+UW versus CDANN+GN gives E_mu 0.08 vs 0.09, E_sigma 0.31 vs 0.33, theta_sigma 0.26 vs 0.27, and AUC 0.82 vs 0.82. These differences are within the typical seed-to-seed variability of deep multitask networks, so the ranking could invert under resampling. In addition, the alpha sweep for GN (Section 5.4.3) and the final choice of the winning method are made using metrics computed on the target test set, the same set used to report the headline numbers. This is selection-on-test and biases the comparison in favour of the chosen method. The external-validity caveat about Poisson-only degradation is acknowledged, but even within the paper's controlled setup the support for the headline ranking is missing. If the comparison were repeated with seeds and a proper target validation split, CDANN+UW might still win, but the paper as written does not demonstrate it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to combine unsupervised domain adaptation (UDA) with multitask balancing (MTB), introducing conditional versions of DANN, DeepJDOT, and DeepCORAL that restrict the adaptation objective to the majority source class in order to handle extreme label shift. The methods are first validated on digit benchmarks (MNIST, USPS, SVHN, MNISTM) with ten-seed averages, and then applied to a controlled CTAO LST-1 simulation setup in which the target domain is built by adding Poisson noise (δλ = 0.46) to simulated images and imposing a gamma/proton ratio of 10^-4. The central claim, stated in the conclusion, is that CDANN paired with Uncertainty Weighting (UW) is the most promising configuration for real gamma-ray source detection and analysis. The paper also introduces a Gradient Layer (GL) scheduler for domain-adaptation gradients and publishes code and results on Zenodo.","tokens_in":19554,"tokens_out":9413,"duration_ms":81056,"significance":"If the central claim holds, the paper provides a practical recipe for applying UDA to gamma-ray astronomy under extreme label shift, a regime that is rarely studied and is highly relevant for CTAO. The controlled simulation framework with ground truth labels is well chosen, and the comparison covers three UDA families (adversarial, optimal-transport, and correlation alignment) with multiple MTB strategies. The digit experiments use ten seeds, and the paper explicitly reports convergence issues and hyperparameter sensitivities, which is honest. The open-source release of code and results is a concrete strength. The significance is currently limited by the lack of statistical grounding in the CTAO experiments and by the use of the target test set for model selection, both of which weaken the support for the headline ranking.","major_comments":[{"comment":"The CTAO ablation results are reported as single numbers without error bars or run counts, whereas the digit experiments are averaged over ten seeds (Section 4.2). The differences that motivate the selection of CDANN+UW over CDANN+GN (e.g., Table 7: E_mu 0.08 vs 0.09, E_sigma 0.31 vs 0.33, theta_sigma 0.26 vs 0.27, AUC 0.82 vs 0.82) are within the typical seed-to-seed variability of deep multitask networks. The authors should repeat each CTAO configuration with multiple seeds and report means with standard deviations or confidence intervals. Without this, the ranking that drives the paper's central claim is not statistically supported.","section":"Section 5.4.4, Tables 7–9"},{"comment":"The GradNorm hyperparameter alpha and the final winning method are selected using metrics computed on the target test set, which is the same set used to report the headline numbers. This is selection-on-test: the comparison is biased in favor of the chosen configuration, and the reported edge over alternatives is optimistic. To support the conclusion, the authors should either hold out a target validation set for all hyperparameter decisions and report results on a separate target test set, or explicitly state that the reported figures are the result of oracle (test-set) selection and should be interpreted with that caveat.","section":"Section 5.4.3 and Tables 7–9"},{"comment":"The DeepJDOT baseline does not reproduce the performance reported in the original DeepJDOT paper [25]; on SVHN→MNIST, the authors report accuracies of 0.65–0.75 depending on the MTB strategy, and state that they 'did not manage to retrieve the results of [25] in this particular scenario.' Since DeepJDOT is one of the three UDA baselines used in the CTAO comparison (Table 9 and Section 5.4.5), a baseline that is not validated against the published method makes the comparative conclusions potentially unfair. The authors should either tune DeepJDOT to reproduce the published digit results, or provide a detailed analysis of why their implementation is the correct one for a fair comparison in the CTAO setting.","section":"Section 4.3, Table 2"},{"comment":"The conclusion states that the comparative study was conducted on simulations with 'multiple NSB and label shift,' but the actual UDA method comparison (Tables 7–9) uses a single target configuration: δλ = 0.46 and a gamma/proton ratio of 10^-4. Table 5 varies δλ only for the non-adapted baseline model, and Table 6 varies the ratio only for DANN without the conditional or MTB variants. The ranking that selects CDANN+UW is therefore established at only one operating point. To support the conclusion, the authors should either present UDA results across the δλ range of Table 5 and across the ratio range of Table 6, or soften the claim to reflect the single-point comparison.","section":"Section 6 (Conclusion) vs Sections 5.4.1–5.4.5"}],"minor_comments":[{"comment":"The definition of the gradient layer derivative is ambiguous: dG/dx = (−1)R f(i) × I should be written more clearly, e.g., dG/dx = −R·f(i)·I for DANN and dG/dx = +f(i)·I for DeepJDOT/DeepCORAL, or with the exponent written as (−1)^R · f(i). The current notation makes it hard to see which sign is used in each case.","section":"Eq. (12)–(13)"},{"comment":"The 'performance gap' values in Table 4 are not tied to Table 2. For example, the +21.8 gap for DeepJDOT with UW is much larger than any difference visible in Table 2; the reader cannot tell whether Table 2 already includes the Gradient Layer or whether the gap is computed relative to a different base configuration. Please clarify the relationship between the two tables.","section":"Table 4"},{"comment":"The claim that 'there is no work trying to associate UDA and MTB using GN or UW' is a strong novelty statement that is not substantiated by a systematic literature search. Given the popularity of both techniques, it would be safer to soften the claim or provide a more thorough review of prior attempts at automatic loss balancing in UDA.","section":"Section 2.4"},{"comment":"The sentence 'all our methods have a better performance when paired with GN' is too broad: in Table 7, DANN with UW has AUC 0.83 versus 0.82 with GN, and after conditioning CDANN+UW outperforms CDANN+GN on E_mu and theta_sigma. The text later acknowledges the change, but the earlier sentence should be qualified to refer to the non-conditional setting and to the specific metrics.","section":"Section 5.4.3, Tables 7–9"},{"comment":"Minor grammatical issue: 'DeepCORAL and DeepJDOT fails at converging' should be 'fail to converge'.","section":"Section 5.4.3"},{"comment":"The phrase 'most promising tool for the detection and analysis of real gamma-ray sources' in the conclusion goes beyond the evidence, which is based on a controlled simulation target. The authors do add the caveat that real data will be studied later, but the wording of the central claim could be softened to 'most promising in the controlled simulation framework tested' to avoid overstating the external validity.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope for the journal and addresses a timely problem. The central idea—conditioning UDA on the majority source class under extreme label shift—is sensible and the controlled simulation setup is appropriate. The main weakness is statistical: the CTAO comparison that selects CDANN+UW lacks error bars and uses test-set selection, so the headline ranking is not established. These issues are fixable with additional runs and a proper validation/test split, and the digit experiments demonstrate that the authors already have a multi-seed protocol in place. I see no grounds for rejection, but the revision needs to provide the missing statistical support and address the DeepJDOT baseline issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful engineering paper with real reproducibility credit, but its headline claim—CDANN+UW is the best configuration for CTAO LST analysis—rests on single CTAO runs and on selecting hyperparameters and the winning method on the target test set. The digit study is properly seeded; the CTAO tables are not.\n\nWhat's new: conditional versions of DANN, DeepJDOT, and DeepCORAL that restrict alignment to source protons under extreme label shift, plus a comparison of GradNorm and uncertainty weighting in that setting. That is a sensible, practical extension rather than a deep theoretical leap. The paper is honest about scope: it uses Poisson-degraded simulations as a controlled proxy and explicitly acknowledges that real observations may contain other discrepancies. Code and results on Zenodo are real and valuable. The digit benchmarks give ten-seed averages and show that multitask balancing can replace manual weighting. The label-shift sensitivity table is also informative.\n\nSoft spots: the CTAO comparison that drives the conclusion has no error bars or run counts, and the differences between CDANN+UW and CDANN+GN are small enough to invert under seed noise. The alpha sweep and the final choice of method are made using metrics on the target test set, the same set used to report the headline numbers. That is selection-on-test, and it biases the ranking in favor of the chosen method. The DeepJDOT baseline also does not reproduce its published SVHN→MNIST performance; the authors mention hyperparameter sensitivity but do not quantify it. The conditioning variants themselves are essentially importance weighting from prior work applied to existing UDA losses, which the authors acknowledge, so the novelty is mostly empirical and comparative.\n\nThis is not a broken paper. It is a solid engineering study with an overstated conclusion. It deserves serious peer review, but a revision should add seeds and error bars for the CTAO experiments, use a proper target validation split for selection, and ideally rerun the comparison before claiming CDANN+UW is the most promising tool. If the ranking survives resampling, the paper will be genuinely useful for CTAO and for anyone doing UDA under severe label shift.","headline":"Useful, code-sharing engineering study of conditional UDA under extreme label shift, but the CTAO conclusion that CDANN+UW wins is not statistically supported by single-run, target-test-selected numbers.","tokens_in":20063,"tokens_out":1578,"would_cite":true,"duration_ms":17010,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that conditional domain-adversarial training combined with uncertainty weighting is the most promising unsupervised domain adaptation plus multitask configuration for CTAO LST-1 gamma-ray image analysis under extreme…","keywords":["unsupervised domain adaptation","multitask learning","label shift","importance weighting","conditional domain adaptation","gamma-ray astronomy","Cherenkov Telescope Array","night sky background"],"falsifier":"Apply the same benchmark to real LST-1 observations of the Crab Nebula, using the Monte-Carlo simulations as labeled source and the real data as unlabeled target, then compare CDANN+UW against the non-conditional and GradNorm variants with the paper's IRF metrics; if CDANN+UW does not improve AUC, energy bias, or angular resolution over the alternatives, the simulation-based ranking is falsified. A cheaper check is to compare the distribution of cleaned image moments between real data and Poisson-degraded simulations; a mismatch beyond the 0.46 noise level would break the controlled-target premise that supports the ranking.","tokens_in":19046,"feed_emoji":"🔭","tokens_out":7394,"duration_ms":62732,"temperature":0.7,"pith_summary":"The paper proposes combining unsupervised domain adaptation (UDA) with automatic multitask balancing so that a deep network trained on labeled gamma-ray simulations can be adapted to unlabeled real telescope images despite an extreme class imbalance. Its central claim, stated in the conclusion, is that the most promising configuration for the CTAO first Large-Sized Telescope is conditional domain-adversarial training (CDANN) paired with uncertainty weighting (UW). The paper builds a controlled target dataset by adding Poisson noise at rate 0.46 to simulated images and fixing a gamma/proton ratio of $10^{-4}$, then compares DANN, DeepCORAL and DeepJDOT, each with and without conditioning and with manual, uncertainty or GradNorm balancing. On digit benchmarks it shows automatic balancing matches or beats manual weight tuning, and on the LST simulations the conditional variants recover performance lost to label shift.","feed_headline":"Conditional domain adaptation wins the CTAO shift test","feed_subtitle":"Noise-degraded simulations and a 10^-4 gamma/proton ratio favor CDANN paired with uncertainty weighting.","key_machinery":"The central object is the conditional domain-adaptation loss, a modification of three UDA objectives (DANN, DeepJDOT, DeepCORAL) that restricts the domain-alignment term to source protons paired with unlabeled target samples, so the extreme gamma/proton label shift does not poison the adaptation. A second mechanism is the gradient layer (GL), a scheduler that weights the domain-adaptation gradient during backpropagation and was found helpful for DeepJDOT and DeepCORAL but harmful for DANN. The third is automatic multitask balancing: uncertainty weighting (UW), which sets loss weights from task-dependent homoscedastic uncertainty, and GradNorm (GN), which sets them from gradient norms and inverse training rates.","core_discovery":"The paper's core claim is that the dominant discrepancy between simulations and telescope observations can be handled by a conditional UDA loss that aligns only source protons with the unlabeled target batch, using the known gamma/proton ratio as an importance weight. For DANN, conditioning means the domain-classifier loss is masked so that source gammas do not participate in the alignment; for DeepJDOT and DeepCORAL the analogous step restricts the transport cost and the covariance statistics to source protons. The paper reports that CDANN combined with uncertainty weighting yields the best energy bias and resolution on the controlled target, and that the conditional variants of DeepCORAL and DeepJDOT improve energy reconstruction only when paired with GradNorm. It also reports that DeepJDOT, while theoretically attractive, is unstable and impractical at CTAO scale because of quadratic cost and large initial gradients. The authors explicitly restrict the claim to controlled simulations and say results on real telescope data will be presented in a subsequent work.","pith_inferences":["The paper does not test whether aligning only source protons is enough when real gamma events have different morphology than simulated ones; a gamma-specific residual shift would remain uncorrected if that difference is large.","Because the AUC stays nearly constant across target ratios from 10^-1 to 10^-5 in the paper's Table 6, the method may tolerate fairly large errors in the estimated gamma/proton ratio, which is a testable property for real observations.","A natural extension, which the paper does not run, is to add a second degradation such as a modified point-spread function or atmospheric absorption to see whether CDANN+UW remains the best configuration when the shift is not purely Poisson.","The gradient layer only scales gradient amplitude and does not resolve direction conflicts; an editorial next step is to combine conditional UDA with gradient-surgery balancing, which the paper mentions as future work."],"forward_implications":["CDANN+UW, if correct, becomes the recommended configuration for adapting CTAO LST-1 models trained on simulations to real unlabeled observations under night-sky background and extreme label shift.","Conditional UDA makes the adaptation stable even when the target gamma/proton ratio is as low as 10^-4, so the method does not require knowing the exact source flux in advance.","Automatic balancing with UW or GN removes the grid search over task weights and, on digit benchmarks, matches or outperforms manual weighting.","DeepJDOT's quadratic optimal-transport cost and training instability make it less suited than CDANN for large-scale astrophysical datasets.","The ranking is currently validated only on Poisson-degraded simulations at delta-lambda=0.46 and ratio 10^-4, so real-data validation is the explicit next step."],"supporting_citations":[{"why":"Supplies the gamma-PhysNet full-event reconstruction architecture whose multitask branches this paper extends with UDA.","marker":"[6]"},{"why":"Defines DANN and the gradient reversal layer that CDANN conditions on source protons.","marker":"[9]"},{"why":"Provides the importance-weighting theory for conditional and label shift that motivates the conditional UDA variants.","marker":"[12]"},{"why":"Defines the DeepCORAL correlation-alignment loss that CDeepCORAL computes from source protons.","marker":"[23]"},{"why":"Defines DeepJDOT's optimal-transport loss and the two-step procedure used by CDeepJDOT.","marker":"[25]"},{"why":"Introduces uncertainty weighting, the multitask balancing method paired with CDANN in the recommended configuration.","marker":"[8]"},{"why":"Introduces GradNorm, the gradient-norm balancing baseline used in the GN comparisons.","marker":"[40]"},{"why":"Establishes the Poisson-noise-degraded target setup and the earlier UDA results without label shift that this paper extends.","marker":"[51]"},{"why":"Documents the observed gamma/proton ratio and dataset context that set the 10^-4 label-shift level in the controlled target.","marker":"[11]"}],"fun_headline_variants":["Proton-only alignment wins CTAO label-shift test","CDANN with uncertainty weighting beats CTAO shift","Extreme label shift: conditional UDA for CTAO wins","DeepJDOT unstable; CDANN best for CTAO images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's ranking of methods rests on the assumption that adding Poisson noise at rate 0.46 and fixing the gamma/proton ratio at $10^{-4}$ reproduces the main gap between simulated and real telescope images; if real observations differ in other ways, the winning method could change.","fun_headline_variants_meta":{"raw":{"variants":["Proton-only alignment wins CTAO label-shift test","CDANN with uncertainty weighting beats CTAO shift","Extreme label shift: conditional UDA for CTAO wins","DeepJDOT unstable; CDANN best for CTAO images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1326,"prompt_tokens":897,"completion_tokens":429,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":358}},"tokens_in":513,"tokens_out":429,"duration_ms":4243,"temperature":1.0,"reasoning_tokens":358,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:46:09.527177+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the same benchmark to real LST-1 observations of the Crab Nebula, using the Monte-Carlo simulations as labeled source and the real data as unlabeled target, then compare CDANN+UW against the non-conditional and GradNorm variants with the paper's IRF metrics; if CDANN+UW does not improve AUC, energy bias, or angular resolution over the alternatives, the simulation-based ranking is falsified. A cheaper check is to compare the distribution of cleaned image moments between real data and Poisson-degraded simulations; a mismatch beyond the 0.46 noise level would break the controlled-target premise that supports the ranking.","supporting_citations":[{"cited_title":"Adversarial Unsupervised Domain Adaptation with Conditional and Label Shift: Infer, Align and Iterate","cited_arxiv_id":"2107.13469","evidence_quote":"Provides the importance-weighting theory for conditional and label shift that motivates the conditional UDA variants."},{"cited_title":"Dell’aiera, T","cited_arxiv_id":null,"evidence_quote":"Establishes the Poisson-noise-degraded target setup and the earlier UDA results without label shift that this paper extends."}],"review_version":1}